Yes last time Richmond (for example) was a rough place to live. But at that time, places like Toorak were not. Can we safely say that Richmond has outperformed Toorak? Hell no. People still pay more for Toorak houses than they do Richmond houses.
And no - rich people don't live in Frankston...