All mortgage brokers are not equal and it sounds like poor advice from the broker you saw.
I find it interesting after receiving advice from a number of brokers on here that said it would be possible to do what you want that you didn't use one of them?