Hard to say. But the fact that he then moves his terrace away from you and around the corner is worth a lot - that really gives much more privacy. We also live in a semi-detached house but have a wrap-around terrace and like to sit "around the corner" from the neighbors, as otherwise you really feel like you're sitting side by side, even if there is still a privacy screen wall in between. You hear everything and are also heard. We like our neighbors very much, but sometimes you just want to sit alone and in peace or have a truly private conversation. If the sun is not blocked too much, I would definitely consider that.