Call me crazy - not sure about the X-E1 and if I should keep it!

I had earlier posted about the merits of the OVF on the XP1 vs the EVF on the X-E1, and although I'm used to an optical viewfinder, an extraordinary deal on the X-E1 tipped me, I decided to get the X-E1 with the kit 18-55 f2.8-f4. Keep in mind that prior to this purchase, I had the chance to use a friend's X-E1 with the same lens in a real shooting scenario: I shot about 200 shots for about 45 minutes. So, it wasn't a whole day photo outing, but certainly much more than just a few minutes handling in a store, and I felt I handled the camera sufficiently to assess it. I thought it was a fine camera, EVF and all.

In addition, I had read one review which said that the EVF on the X-E1 was jittery when you panned, but by focusing on a spot first and then moving, the EVF image cleared up substantially. When I tried this with my friend's camera, this certainly seemed to be the case, and I was shooting basketball players warming up - not an easy scene to shoot! The camera also felt quite comfortable in hand. So, I had no concerns about getting the X-E1.

... but when my purchase arrived the other day, I found that the EVF seemed far more jittery when I panned than I remember seeing on my friend's X-E1. I thought perhaps that I'm just so used to an optical viewfinder (Nikon D3s), that I've forgotten the limits to an EVF - except when I picked up my 2 year old Panasonic GH2, I discovered that its EVF when panning - although not as high resolution as the Fuji - looked far, far less broken up. The Fuji also seemed smaller in hand and less comfortable to hold than I remembered, although maybe this just takes some getting used to.

I really, really want to like the X-E1 - the images are fantastic and the 18-55 lens puts all other short zoom lens to shame - but the jittery EVF is more bothersome to me than I would have thought.

Is it just me? Is this a real limitation? Is there a setting I can adjust? If I decided on the XP1 and its OVF, would it "solve" my EVF issue, or would there be another issue I'd then have to deal with?