At its heart your argument is, if conditions change then their could be more supply then required.
I agree completely.
Look at pictures of the great depression. People on the streets, people in shanty towns etc.
If we had a depression then house prices would almost certainly be in a tail spin...