If servicing is strong - and LVR is less than 80% then refinancing that small loan over shouldn't have much of an impact other than getting you a lower rate.
Just keep it variable and if you need to refi to another lender later - the costs won't be too high.
If you paid LMI on the...