That's why people have offset accounts. :D
You can do that, the offset account is really just a savings account which reduces the amount of interest you pay on your mortgage. You're not reducing the debt with it, so you can put money in/out of it as you please. This however is not the case...