There is a massive lack of equity here. You would certainly get a loan and also find someone to start building the house. However, the probability is high that the costs will get out of hand during construction and you will not be able to bear the double burden from delays. The strain of building a house, especially with 2 small children, should not be underestimated. You write that the children come first; with the credit limit and income, that would certainly not be the case for the next 2 years.
If it absolutely has to be and the idea has already taken hold in your mind, then at least change the plans. Local construction company, standard gable roof house with 100-110sqm of living space without any extras, pave a small driveway and have grass planted everywhere for now. That is affordable and you can then stay there with small children without being forced to slog every day for months after moving in.