And I can empathize with you, having trouble with the neighbor is a shitty feeling!
And it's similar. We are also building into a gap in the buildings, but this one is 2500m2. The neighbors take liberties with everything. They knowingly build over OUR property with part of their parking space, drive over our property with their construction vehicles, bicycles, use our property as storage for their construction work, and so on and so forth. We don't care at all, let them do it, we have enough space around the house, we haven't even approached their border in the slightest. But at the same time, every single, and I mean every single, move on our property is watched with eagle eyes. Recently, our gardener sprayed the weeds with an organic agent, and then there was writing 40m over OUR property about what he was doing.
Presumably, our mistake was that we should have shown our neighbors "their" borders much earlier. The mistake of our neighbors is that to avoid having neighbors and their construction site, they should have bought our property. Actually, quite simple!