Cost of photovoltaics with storage

or the vultures of bankruptcy roam the country, collect down payments and file for insolvency

My concerns exactly encompass the points you mentioned :(

On the one hand, I like to trust and have so far had a pretty good impression of the friendly and reputable small business "around the corner," but when it comes to almost five-figure amounts, I have to think twice.

On the other hand, if you think about it "benevolently," you can somehow understand the small local provider not wanting to set up a five-figure system and then possibly have to chase his money without being able to remove the system from the roof?
I guess I need to sleep on it.

What other recommendable options would there be to safely reach the goal in the interest of both parties?

Direct debit?
Bank guarantee?
….?

Payment for the material upon delivery to the property. Payment of the remaining amount after commissioning

A different question: Is it actually common to pay in advance for the material when commissioning the local installation company? In my case, it amounts to 65% of the total sum and unfortunately does not inspire me...

It is actually uncommon in photovoltaics. First invoice after material delivery. Second invoice after installation. Third invoice after final commissioning.

We had agreed at that time, because it was not clear how the feed-in tariff would look in the future, that we have a special right of withdrawal should this be discontinued by the time of implementation.

What is the current situation regarding the feed-in tariff? We are currently also planning a photovoltaic system... he said something like yesterday: "Commissioning around January 2022."

Since the gigawatt cap has been lifted, the remuneration continues. It just decreases month by month. The decisive factor for the individual amount of remuneration is the date of commissioning. The amount at this time is then guaranteed for the entire duration, even if it continues to decrease afterward.
The plan should be to generate as much self-consumption as possible due to the comparatively low remuneration by now. The remuneration is now just a nice-to-have. You can somewhat trick the commissioning date with the solar installer so that it is as early as possible and you get as many cents as possible for the duration of the remuneration.