The issue of him borrowing to repay family may not satisfy tax tests for a loan refinance used to acquire the property. This needs to be explored in detail to ascertain the original nature and use of the funds to acquire and if a trail is conclusive.
Don't assume that just because family...