Stick to whatever you can afford. If you can't afford a house, buy a townhouse. There is no point over-extending yourself, resulting in both financial and mental stress to you and your partner.
You will have time to buy more property later. Just get moving.
BR