We have always owned a holiday house, it is basically a second home to me.
My folks got the place years ago, and have spent very little money on it since. With the cost of a holiday anywhere for 5 people over the years (30 of them) this place has paid for itself many times over. Plus it has meant that we are able to (and regularly do) spend weekends away - either as a family or just on our own.
Over the years we have set the place up so that if you want to go away you can literally pack some clothes and leave - everything else is there and ready. Ill often go there for just one night to finish off some work in peace and quiet. Or drive up there for the day to go for a dive.
Its now owned by us 'kids'. Mum and dad can spend the rest of their cash - but this one stays with us
In reality it has no monetary value to us - we will never sell it so dont look at it in that way.
In the meantime we have all still travelled quite extensively throughout the world. I wouldn't trade the house for a trip overseas (I dont like flying at the best of times).
I think if you were looking to buy a holiday house it comes to how quickly you can get to it (ie anything more that 3-4hours is too much hassle for a weekend). Which will determine how often you can use it. Once you own it you can set it up to cost you very little, and if you use it regularly then the annual cost/usage is pretty minimal (obviously subject to 1,000 assumptions).
If you can afford it then I would say go for it! Use it often. Why else would we spend our lives investing/working if it wasn't to purchase something we can enjoy?
Blacky