Facts:
Property value approx. €100,000
Apartment value today minus outstanding debts approx. €100,000
Income €3,150 + child benefit = approx. €3,350
Desired house 150m² on developed land without a basement = approx. €400,000.
So only €300,000 needs to be financed, with a house value of €500,000.
If you opt for a 15-year fixed interest rate, 4% repayment, that results in an interest rate of 0.71% and a payment of €1,172.50 with an outstanding balance of €110,000 after 15 years. The first years might be a bit challenging, but as soon as you earn some income again, it is completely manageable. If the parents can gift land, they can surely also help out with a euro or two in particularly difficult times.
Do it? Yes.
P.S. It is possible that your husband is not actually afraid of the financing, but of the idea of living next to his parents-in-law. Justified? Probably.