Hiya
Was the $96k set up as a separate loan? If so - should be fine. If not - a bit of a mess......ask your accountant if you can apportion it.
With the $80k loan - make sure it's set up as a separate loan! That way - there shouldn't be any issues with deductions given the purpose.
p.s...
