If IP tax benefits were to be suddenly withdrawn, and therefore IP investors started selling up and leaving the market, housing prices would most likely drop and a lot more people would be able to afford their own homes. I don't think many people are that financially illiterate that they can't buy a house - it's just that the prices are beyond affordability.
Obviously you are young and don't remember the time this actually happened for a small amount of time. It caused so much upheaval that they quickly re-introduced it at a higher level than before.
For what it's worth, it is not investors that drive the market, it is home owners who get emotional about their "perfect home" and will buy at any price (so long as they can get the finance). A savvy investor will stay unemotional and not buy unless a property stays within their criterior.
There are many, many investors out there that choose houses with little to no tax benefits, prefering to have something that pays for itself. An older home has little in the way of depreciation too.
If what you say is true, then there are many low income suburbs that would be considerably cheaper than they are now. Lets take Western Sydney, for example. A place I know well. Most of the houses are around 40-50 years old, at a guess, meaning no depreciation unless they have been upgraded. The yields are much higher than closer to the city and you can easily afford a home here if you are on an average income. Yet, I know many people/families that rent by choice rather than own. My ex-neighbours, for example,
earn a 6 figure income, yet rent (and spend all disposable income) a small ex-commission home.
Anyway, during the last boom most of the investors had already left the market (at least the ones that knew what they were doing) while prices kept moving steadily upwards. This price was fuelled by people wanting to buy their home, because they thought that prices would continue the upwards trend endlessly.
After prices pulled back, the investors returned. Especially when the interest rates were low. Heck, you could get a home with an 8% yeild and a loan of 4.99%. Prices where CHEAPER, yet the owner occupiers were hardly anywhere to be seen.
AGAIN, please note, it was actually cheaper to buy than to rent.
There are many places that offer better yeilds than Western Sydney. Just tune into some of Nathan's posts and you will again see many examples of homes that are cheaper to buy than to rent, yet people rent them by choice.