1:1 with the Queen
Yes, I think thats correct.
Example: If a lens is at 100mm and f20, then the entry pupil is still 5mm. The wavelength of light is in the nanometer range and I dont think that much diffraction can occur. Even at f40 the entry pupil is 2.5mm.
There is another point. Macro lenses probably have more precise and thinner aperture blades, while other lenses, that are not made for small apertures might have thicker blades that reflect more light so they could degrade image quality when small aperture is used.
Thats what I personally think about it.