I've not seen the tests but you're paying for speed and the ability to be reasonably sharp at the fast apertures. This doesn't mean the lens will be any better (or even as good) stopped down than a slower, less expensive alternative. Grinding and polishing large elements is a lot more difficult to hold tolerance than for smaller elements.
I'm wondering whether there is a "sweet spot" with current design/manufacturing technology for lens speed if all we want is the best possible stopped down performance across the frame, e.g. at f/5.6 or f/8 on full frame.
Let's say the target MSRP for a new 17-40 zoom is $1200. Can Canon make a zoom that performs significantly better at f/5.6 within this budget when it is a constant f/4 compared to f/2.8? One would think so, but there is really not much evidence that this is true. And if it indeed is true, it's a shame that those slower lenses are not made. A lot of landscape/architecture photographers would rather get a slower WA zoom with better stopped down corners than the faster lens for the same price.