Hello,
I would talk to the municipality again, especially §31 Baugesetzbuch works in your favor because although your case contradicts the regulations, it does not deviate so much that people or the place itself would suffer.
With the neighbors' consent, that could work, at least it would be an argument for the municipality that they cannot easily dismiss.
Best regards Dirk Grafe