Three or four primes?

I am thinking of switching marques and purchasing a Nikon D90 or just possibly a D300s.

With this I am thinking of getting 3-4 prime lenses which is what I have at the moment. These would be lenses in the range of 20-24mm, 35mm, 50mm and 85mm. I know that Nikon have a rep for producing excellent zooms but I enjoy using primes (not least because they are discrete on the street) and am easy about giving up the lack of flexibility or convenience that one might enjoy with a zoom.

My question is whether these Nikon primes - I know there is more than one at most focal lengths - are optically good enough to make going zoomless a worthwhile option, not least since either way is expensive. Has anyone found lack of VR a hassle when using primes? I don't imagine I would but one never knows. I'm happy with zooms for wide or longer - 10-20mm-ish or 10mm and over - but the vast majority of my pics are in the 24-70mm or so range.