Sure.
Reread your Agency Agreement. It defines the terms of your agreement with your PM.. If its like most Agency Agreements they can pretty much do whatever they want, or dont want, and not be in breach of it.
If the argument escalates their watertight Agency Agreement will trump your...