Is the 70-200mm f4 NON IS better than the IS version for fast action shooting?

I'm wanting to use a 70-200mm f4 lens for fast action street dance close focus photography.

I'm wanting to capture facial expressions and also badges/decorations  that performers often wear.

I'll be using 1/1000 sec shutter speed on a 5D III to freeze action.

I've settled on the f4 version of Canon's 70-200mm L lens, the f2.8 being too expensive and heavy for my purposes.

What I'd like to know which version will give me the best results for my intended use?

At 1/1000 of a sec I won't need IS and I'm paying a lot for it. I'm thinking that a set of IS elements in a focus train will tend to slow down focus speed - if there is more glass to move it takes longer to start moving, also longer to stop when in focus - even if the IS mechanism is switched off. Am I thinking right here? Are the differences so small as to be ignorable?

Curious about the differences in optical performance between the two versions used close up, typically I'll be using at 2 or 3 meters at f4 to 6.3 depending on lighting, as I believe the IS version is a bit sharper, but would I notice it under these conditions.

Your comments and experiences would be appreciated.