I would also first slope off at the property boundaries and wait. Spending 10-20k€ on L-stones that will later be buried in the ground is definitely not worth it for me.
You don't have to argue with the neighbor right away, but if he just throws out his opinion like that, you can definitely tell him that it's pretty rude of him.