Influence of outbuildings on boundary construction

The building authority now wants to check whether the neighbor is using the building approved as a stable as living space. If he does, we have to maintain 5 m, without windows on this side. Otherwise 6 meters if with 5 windows

No, then the building authority has to prohibit that. Legal violations by the neighbor cannot justify restrictions for you.

The building authority also now wants to check whether our building is too large. That other neighbors have built just as large or larger in recent months is dismissed with "Yes, maybe that was not even approved"

Building law is not a wish concert; the building authority has nothing it wants. It is an authority and must apply the applicable law. If neighbors have built illegally, a demolition order must be issued if necessary. If, on the other hand, they have been approved, the authority must adhere to the principle of discretion (unless these approvals were unlawfully granted). "The last one gets bitten by the dogs" is not permissible. But I had assumed you had been living there for a long time ...

Now... If others are allowed to build however they want without a permit - then maybe it's better if we build without one too

Get a good lawyer and sue for your building permit. As a layperson, I wouldn't bother dealing with the building authority alone anymore, hearing it like this.