If you owner build, without the right permits, and then tried to refinance, the bank may find out, as suddenly having a house on what they thought was a vacant block might make the assessor ask questions. If you went to a second bank, it might be ok, but why go to the trouble of owner building...