A very curious effect

See below for several photos of bricks taken through a circular hole in a board. I was stationary and did not adjust the focal range -- stayed at 125 mm. The only thing I changed was the aperture. As you can see, I began at 2.8 and stopped down a full stop until getting to f/16.

Here's what's not surprising: the focal plane grew wider as I stopped down. No mystery there. The borders of the hole is increasingly blurry as I opened the aperture up.

What IS surprising: it appears that the diameter of the circle shrinks as I stop down the lens. Why does this happen when I don't move closer or zoom in?

Zooming compresses things that are at different distances, but this effect does not appear to explain this particular phenomena.

Explanations are welcome!

SOS