Don't even think of it.
For a start the extra to pay off your mortgage will be a non-deductible debt, so you will be no better off. You will still have to pay the interest and won't get a tax deduction as you do with the interest on the IP loan.
Even worse is the fact that you will...