Nice call
I agree with everyone. Firstly tell the agent you will not deal direct with the landlord (she/he is just pasign the buck of dealing with a painful LL), if you wanted to deal directly with a LL you would have found a house that was sefl managed, you did not sign up for this. I woudl tell the owner next time they rock up at the property that they can not have access as you were not informed via the correct chanels. What ever you do, do not allow them access again, ever, Not without the correct permission. This will teach them that they will not get away with the curretn behaviour.
I agreee abut the padlock on the gate as suggested. That is a great idea, you are within your right and it will make it difficult for anyone to gain access when you are not home. It is a smart move.
Grab a copy of the tenants rights booklet, and excercise them.
BTW I once had the same situation whereby I knew the owner of a property I was renting and she would call and ask if she could pop over. I said no after the 3rd time and told her that when I am at home I am relaxing and do not want to be disturbed without prior arrangement. She took it very well and was always respectful from that time onwards.
...........Until I decided to move out and she wanted to self manage and when she conducted the first open for inspection she went into all of my cupboards and asked me to wipe the shelves in my bathroom and kitchen, and asked if I could remove a towel which I had folded nicely over my washing machine to protect the top of it because it didnt look "good"
I wrote a very direct email telling her she was out of line and that she could no longer gain access without me being present. She was very apologetic and said she had lost her brain due to recently having a baby. I accepted her apology, luckily I nipped everything in the bud super early though.
Good luck with it all,