You have done nothing wrong. If you were peering in the windows with your face up against the glass, then I'd side with the tenants - but currently they just sound like idiots.
I have faced the same complaint from tenants claiming that I'm harassing them by always loitering around their...