Traded my D90 to an EP-3 and now starting to ask why I did it.

A suggestion:  Maybe you should sell the EP-3 and look at an EPL5.  It uses a MUCH better sensor than the EP-3, has a flip screen on the back, and is very inconspicuous.  Image quality with the EP-3 sensor is NOT going to be as good as the D90, whereas IQ with the EPL5 will be better than the D90's.

And, yes, you should look into getting one of the primes.  The 14mm f2.5 Panasonic can be had on Ebay for under $200...if that is too wide a focal length for your purposes, the 20mm is about $300  used, or $350 new....

I had a D90 and bought an EPL1 (which uses the same sensor as your EP3) as my second camera two years ago.  After using the EPL1 for a while, I switched to the GH2, which is still M43 format but used a better sensor.  After that, I felt no need to keep the D90, but if there had not been a GH2 level sensor, I would have not switched completely from the Nikon gear at that point.  You are in a similar situation although you did get rid of your Nikon stuff already...However, the solution is to just get an M43 camera with an advanced sensor.  In small body size, the EPL5 has the excellent 16MP sensor that is in the OMD, and will outperform your D90.  You could also look at the Panasonic GX1, which has a larger display (although it does not flip) and more direct controls on the body than the EPL5, and which also has a more advanced 16MP sensor than that in your EP3.

Lots of options, and you do NOT have to go back to a DSLR to get the same output quality as your D90 had.  However, yes, lenses are not cheap....but fast primes for anything (except for the Nikon 50mm f1.8) are not inexpensive, so I am not sure that part of the argument makes a whole lot of sense.  (Plus, if you are feeling adventurous, and want to try manual lens operation with zone focusing for street shooting, you could get any vintage lens you want, and use that with a cheap adapter on your M43....easy way to spend very little money on good glass, and have some fun at the same time   )

Good luck!

-J