Looking for more sharpness

The little 40mm is a good way to test the waters of primes to see if you like them before getting an L lens.  It's also new enough that used prices should be very close to new, and though my 35L experience is limited, the differences I've noticed have been minimal throughout the range where they share aperture settings (the 40mm really struggles at f/1.4-f/2.5 ;).

Since you're stopping them down, you may also want to try renting one of your desired primes/zooms for a weekend or something similar to see if you like them and find them worth the price.  $50 or $100 is a pretty cheap price to pay before shelling out $1500-$2500 on a lens that's purpose is to be sharper than an already sharp lens.