I think it really comes down to who engaged the fence contractor to do the works. If it was yourself or the agent then the contractor should only take instructions from yourself or your agent.
Given that you haven't paid, I would be expressing the above to the contractor and require them to get...