The bank wants to limit its "exposure" on your property to a certain limit. That's why banks don't lend you 100% of the property value in the first place.
So, there is a maximum percentage for which you'll be able to get finance. This is the LVR - Loan to Valuation ratio.
To gain access to your equity you will usually take out a LOC (Line of Credit), and the total debt against your property will be subject to the LVR. Most people keep their LVR at 80% or less, partly because most banks require Lenders Mortgage Insurance (LMI) to be paid if you exceed 80% LVR. In other words, you must pay an insurance premium to protect the bank if you want them to increase their risk by increasing the LVR (if you can get them to agree to it in the first place).
At 80% LVR, the most you can borrow on a $100K property is $80K. Since you've already got an $80K mortgage, you can't borrow anymore.
If you went to 90% LVR, that permits $90K total borrowings against a $100K property, and since you've only got an $80K mortgage, they might lend you $10K more.
But, either way, the amount in your circumstance is probably not enough to something worthwhile with. You certainly can't get access to your whole 20% deposit.