But apparently my considerations are all nonsense,
They are absolutely understandable according to classical school physics understanding, only that in practice thousands of cracked stones, which you know nothing about, but which are nevertheless cracked, make the resulting concerns seem less dramatic. If you
and the load is thus distributed over other points, which could lead to crack formations in other places
"think consistently further," for example, higher frequencies of cracks would occur in houses where (at least larger) window openings in the upper and ground floors are not aligned. But they do not. Such a house is tougher than you think. And even if no one catches them doing it, construction workers improvise very regularly. More precisely: so regularly that over time "improvising" becomes :-)