Don't even think of it.
For a start the extra to pay off your mortgage will be a non-deductible debt, so you will be no better off. You will still have to pay the interest and won't get a tax deduction as you do with the interest on the IP loan.
Even worse is the fact that you will contaminate the deductible debt with the private use, which will be an accounting nightmare for as long as you hold the IP.
Marg