Yes, I believe the upgrade to the E-P3 is just fine...
Look, the first mindset we have to periodically alter is the one that seemingly embraces the view that the only camera upgrade should be to the latest and greatest model camera. Because the latest and greatest may not always net you the results (both IQ-wise and emotionally) that you were hoping to achieve.
Over the past decade I've acquired a large collection of cameras that I've kept due mostly to their higher IQ standards and my emotional attachments to them (see profile). The E-P3 is one of those cameras and it is staying in my collection for a variety of reasons. I have it mated to the Panasonic 20 MM f1.7 and those two were made for each other, IMO. In fact, with these two together, the output looks like it came from a much higher MP camera and the detail will surprise you.
In addition, the E-P3 is a gorgeous looking camera and performs quite well. Does it have class leading high ISO capabilities (well, for Micro 43, that is)? No, it doesn't, and I find that I don't like to go over 2000 - 2500 or so in ISO with the E-P3.
But it does a great job of capturing superb images. And if you can find one at a great price, don't hesitate to act. Here are a few IQ samples - most taken with the Panny 20 MM f1.7. It's one of those cameras that makes you want to keep it - call it emotional, instinctive, or whatever - it just happens that way. All of this is my opinion, of course.

ISO 800, RAW, AWB, Aperture Priority Mode - using the Olympus FL-50 flash in bounce mode (and the Gary Fong Lightsphere dome unit).

ISO 200, RAW, AWB, Aperture Priority Mode.

ISO 200, RAW, AWB, Aperture Priority Mode.

ISO 200, RAW, AWB, Aperture Priority Mode.
-- hide signature --
Semper Fidelis...
Ben Herrmann
US Marine Corps (Retired)
North Carolina, USA