That is pretty bad by your branch lender to not think of this as a potential issue upfront.
I would be pushing to do as investment and state that it was a misunderstanding and your long term intention is to live in it but short term you will rent it out. Perhaps arrange your own rental...