Hello Camille,
You have more or less described your life situation. The quite correct initial responses from the other participants deal almost exclusively with the topic of financing. That is of course also the first question you have to ask yourself. What can I afford and what do I want to afford? Thus, one has to reconcile the ideas and wishes with what one can afford financially. However, there are many possibilities here as well. You don't have to have your house paid off by retirement age. My wife and I are moving into our new house in Hamburg in two months. We are both in our late 30s, earn well, but due to the high overall costs in Hamburg, we will not have the entire financing paid off by retirement age. But we have consciously accepted that. By retirement, our daughter will probably be out of the house. Then we will be sitting (at best, if things go well for us!) alone in a house. Maybe it will be too big for us then and we will sell and move into a smaller apartment. Who knows.... In any case, we are looking forward to hopefully many nice years in the house, but we also know that we must not be stuck to it. If interest rates rise sharply after the financing expires or if something else happens (job loss, illness, divorce, etc.), then that is how it is and you have to deal with it. If you are only negative, then you are never allowed to start anything that could end badly.
If you should decide to build, it is like caring for and raising a child. There are a few basic rules that are reasonable and good. You should definitely follow these. But much is also due to personal, local, and time-related circumstances! Many people could probably share their positive and negative experiences with you here. Whether that would also apply to your case depends on your circumstances. Therefore, take basic rules as a foundation and be prepared to go your own (often long and exhausting!) way.
Best regards
Andy