IP offset account tax implications

[tdh78au]

Hi folks,
long time reader first time poster.
I've just returned from a visit to my accountant ... and he's totally freaked me out.
This is why:
I have an investment property with an interest-only investment loan @ 80% LVR. There is an offset account attached to the IP loan. Am I able to put all my income (salary & rent) into the offset account without having any tax implications if I take it back out for personal expenses (holiday, travel, day to day bills, etc)? I paid off my PPOR loan couple of months back & hence have no bad debt (with clear uncumbered PPOR title to my name).
My accountant seems to think that I am not at liberty to use the IP loan offset account for personal expenses (please note that the offset account is actually a separate deposit account to my IP loan account).
Please help!
any links to ato site and/or rulings re: same would be really appreciated as well.