Strictly speaking I think what you're planning is okay. If the offset account is never used for anything other investment purposes it should be fine.
In practice though I'd discourage it. It's way too easy to muck something up and compromise the detectability of the loan. If you're...