I have seen this happen a few times. One case went to th esupreme court of Vic over a few inches. The units had to be pulled down and the builder went bankrupt.
So I would say none of this is your fault and you really can do what ever you please.
Lyfes a Beach !