Communication beforehand is simply crucial here.
We are building into the extended garden of the parents-in-law, but both houses are about 70 meters apart, with outbuildings in between, so you can't even see each other at first. In summer, that might happen more easily; the used part of the parents-in-law’s garden including the terrace/grill is practically our "neighbor," they spend a lot of time there, but an existing embankment separates the two properties. This natural privacy screen was/is important to us. And not only to us, but also to them... that is the decisive factor. We said beforehand that we want to be able to have our peace, the other "side" feels the same, even though we get along really well.
Discuss everything beforehand, then decide if you want it. I wouldn’t have built door to door, but as described above, it is absolutely right for us. You have to make the decision for yourself anyway, feedback aside ;)