If family planning changes, then the office, guest room, gaming room, hobby room simply becomes a children's room.
Through personal effort and the money allocated for it, the OP could manage the costs. He says it will be around 350,000 euros.
@ OP
With your income, financing should be feasible without becoming a slave to your house. Maybe some luxuries won't be possible anymore – the world won't end because of that. Salary increases will also come.
Please make sure to consider all costs marked as [Bauseits]. There are also threads here, for example, why house construction keeps getting more expensive, which I would look at again.