You don't owe the broker anything, they did work for you in getting the loans in the first place and were paid a commission then, and they get paid a commission now to service the loan as a first port of call instead of the bank.
If they're not providing you service then there's nothing to...