So cucumber slices and paper bags don't bother me. It stops with hazardous waste or liquid residues from construction chemicals.
The paper bag might partially contribute to a better U-value, but I'm not really keen on that :-D
Jokes aside.
Politely but firmly address your discovery with the contractor and ask him to support you in keeping the bricks in their factory condition, meaning no further fillers applied on site.
I've stopped counting how many buckets of trash I've disposed of myself by now. Just get used to it, that's that. Many companies hardly dare to reprimand their employees for fear they might take it personally and leave – at least that's my impression.
The plasterers have cleaned their hoses on previously uncontaminated ground, thrown leftovers of the 2K coating next to the light well, poured the remaining primer in front of the front door, wiped silicone, etc. on the bricks and of course much more. At some point, you just clean it up yourself. I can't hear the phrase "this is a construction site, not an operating room" anymore. If someone can't distinguish a construction site from a landfill, there's no helping them anyway.