Unfortunately, you are mixing quite a few things.
I have calculated the heat gains from controlled residential ventilation for you.
Here’s an estimate for windows:
The heat gain from unshaded windows can quickly be around 200W/m² of window area.
With a nice lift-and-slide door measuring 5x2m, that alone means 2000W from just this one window, which is 100 times as much as from controlled residential ventilation with heat recovery. ;-)
In principle, this exactly matches your feeling.
Insulation neither lets heat in nor out, controlled residential ventilation also neither lets heat in nor out.
Windows, however, do let heat in well, if they are not shaded. So here by radiation and not by convection.
But from your statements, I assume you don’t have external blinds (Raffstores)?
We have roller shutters; we just hadn’t expected such a high heat buildup (as you just calculated) from the pure solar radiation through the windows in the first spring in the new house. We naively assumed that since the windows are heat-insulated, the house would neither let heat in nor out. We eventually understood how important shading is, but those first nice days in May caused a lot of heat inside the house.
As you already said, the house neither lets heat in nor out. So we had to actively cool and since then have paid better attention to shading. However, we still notice that the house continues to warm up in summer; we cool regularly. But we also don’t sit in a bunker; closing all the roller shutters makes the house very dark and dull. So we take a reasonable middle ground between darkening and cooling.
If I were to build again and had the money, I would probably invest in smart home and automatically controlled shading. This was not at all clear to me when building the first time, how important it is.
Would you keep the controlled residential ventilation running normally in summer? We don’t have a summer bypass, so it runs with heat recovery. If it’s 32°C outside, 24°C inside (due to the air conditioning), and I then run the controlled residential ventilation, it will get warmer inside again, right? The air cooled by the AC is transported outside, and the 32°C from outside comes inside? Then I have to expend energy again to cool it down. Or am I thinking about this wrong?