Opinions do differ somewhat. In my opinion, one aspect is always somewhat overlooked: for the money, you do get something in return (the house with the plot) – and given how building land and real estate prices have developed over the last 10 years, at least the value of the land will not decrease.
By building a house (and paying off the loan), you are basically creating added value.
Of course, the finances have to add up overall. And I know how much many people like to plan everything. I am neither in favor of naive approaches nor of detailed financial planning for the next 50 years. Based on the situation described by the OP, I believe that financing around €400,000 is definitely doable.
(Off-Topic: I have been following the development of building land prices in the Palatinate since 2004. If good infrastructure is available – shopping facilities, doctors, schools, etc. – then prices have increased extremely. It is developing roughly like 10 years earlier in Switzerland – and even today there is no sign of a price decrease for building land there. By this I mean: if I can buy building land and realize a house construction today, then waiting makes no sense!).