This sounds like a go to me, so you should consider it now when your family is young and you don't need a big house
If by swapping properties your cash flow improves considerably then IMO it's a no brainer.
However, nicely renovated PPOR's can lose their presentation when they get rented...