Some can't afford it anymore, some can only build smaller.
Then example calculation with smaller volume (money and house).
Earlier:
600k€ with 1.2% interest = 600€/month interest
100 people: 100 * 600€ = 60,000€/month
Today:
300k€ with 2.4% interest (fictional as example) = 600€/month interest
Or to extend your statement, and some can still afford this interest (I would be interested in how many still borrow 600k€ at 3.5% interest)
Then the sum mixes:
A few:
600k€ with 3.5% = 1750€/month interest
A few more (build smaller):
300k€ with 3.5% interest = 875€/month interest
Out of 100 people:
10 * 1750€ = 17,500€/month
50 * 875€ = 43,750€/month
40 (can no longer build)
= 61,250€/month
Interest per month remains the same