Without knowing the exact facts or your actual financial scope in more detail now, but with the given information, I consider it more reasonable to carry out the project. All options seem modest, but what could happen to you in the worst case? You run out of money along the way. However, the shell and part of the building services should be doable. In other words, a dry warm place with a functioning bathroom and an improvised kitchen. You don't really need much more to survive at first. And you can manage like that for a while. Inflation initially works in your favor when paying off the loan. Everything else will show itself. A lot will still happen until you actually know where you stand in the end.
Try to possibly counteract the uncertainties with the materials by ordering very early and store heating, tiles and bathroom ceramics, bricks, photovoltaics, kitchen somewhere in between (rental garage, relatives, ...). Extraordinary times require extraordinary measures. And moving into a turnkey house is currently not an option under the given circumstances. Better to live in your own half-finished house than in an old shabby rental apartment. I hope you can still do one thing or another yourself because, in the long run, of course, there is quite a bit to do. But even inexperienced people can now do a lot more themselves (in the representative area – stay away from gas/water/electricity if you have no idea), thanks to YouTube and such great forums like this one, than ten or twenty years ago.
The problem is that if you stop now, you will lose your funding and the first 50k will be gone. You won’t recover that quickly if you break away from KfW55. Besides, the interest rate is running away from you; you can’t even keep up at the moment.
It may be that the big bubble bursts, but that initially means nothing more than the market has to completely reorganize itself afterwards. Raw material and energy prices will still be there. Labor costs will still rise anyway. In other words, in my assessment, building cheaper won't be possible anytime soon.
The good thing about your starting position: wages will sooner or later follow inflation. And in a few years your installments will be relatively cheaper. So sooner or later the garden will also be possible.
I was born in '82. In my childhood, it was more or less common that neighbors in new housing developments had piles of earth, purchased cranes, and mini excavators standing in their front yards for years.
In this sense, I wish you all the best.