We would certainly calculate that once. Maybe the plot is also divisible and then we would actually do it 50:50 with Viebrockhaus. But first, let's see...
Well, if anything, you should act before the next buyer finds their misfortune. In case of doubt, explain the story so far to the city council or municipal council and then buy according to the standard land value (possibly discounted) as a divided plot without building obligation. Then the story would have a happy ending :-)
If it is financially feasible, it could be a good investment. It increases the living value for you and your other end house neighbor, increases the value for a possible future buyer, and possibly you can even make 1-2 wall openings for additional windows in ten or twenty years. But it lowers the maintenance value here in the forum. And, after all, you have to pay it off...