Holiday let and second home owners have complained that activity at a yard where fishermen store boats and equipment is ruining their peace and quiet.