I am not sure he is wrong. As has been said before, you haven't reached settlement yet so if you pull out now I think you would definitely forfeit your deposit. And as Aaron_C has said, in property law you can only enforce specific performance - i.e. force the seller to complete the contract as agreed. You would then be entitled to sue for damages but you can't just pull out and expect to get you deposit back.
we are giving the default notice and hope for the tenants to still being in the house for the 14 days after that so the contract will then be done and dusted.