Fujifilm X-E1 or Olympus OM-D E-M5?

I also have both, the X-E1 and E-M5.

Regarding image quality, I think the Fuji has better JPGs ooc, as they are cleaner and have less artefacts due to heavier sharpening and denoising (or simply worse algorithms for those two parameters) on the Olympus, (also completely tuned down). Not to say Olys JPGs are bad, they are a lot better than from most other cameras, but the Fuji's are just very close to perfect. Looking at the RAWs (via Silkypix, as LR/ACR still doesn't support the X-Transsensors in an acceptable manner), it's an incredibly close race. The Fuji resolves slightly more very fine details (no AA-filter on the Fuji vs very weak AA-filter on the Oly) and produces somewhat less luminance noise (a lot less chroma noise). But both differences are miniscule and I only might guess the right camera when watching direct comparisons in 100% view. I did comparisons up to ISO 6400 and simply don't see any high ISO advantages worth mentioning for the Fuji in the RAWs, they are incredibly close to each other with the Fuji only very slightly ahead. In situations where Oly's IBIS can be used, it pulls ahead without any doubt at all, regarding available light shots. By the way, due to the heavy use of Sony's (great) sensors in most recent cameras, image quality in general doesn't look that different at all anymore between cameras from different companies (at least the RAWs). Fuji or Olympus or Nikon or Pentax, etc. doesn't matter that much...

Concerning lenses, well Fuji still has very few, whereas m43 has plenty. Fuji's system is "a lot" younger, so no real surprise here. But if you already find the lenses you need (or at least in the near future of the roadmap), who cares for the rest of the lineup. If you don't find what you need, who cares for the cameras attached to those lenses you don't need... Both have excellent and not so excellent glass, don't see much difference regarding picture quality.

Features and handling. The E-M5 can be costumized in a trillion ways. The Fuji a lot less, you can't even remove AE-L from the half pressed shutter on the X-E1 (why the hell?!). The Olympus has a myriad of features, quite a lot very useful ones. The Fuji only has a few basic ones. The Olympus has blazingly quick and completely silent AF and the stabilized EVF view is godsend for MF, especially regarding tele lenses. AF on the Fuji is very usable (not silent) and MF finally pretty decent, even with native lenses. The Olympus feels and behaves like a very fast and responsive DSLR (and in many ways even better), the Fuji a lot more like a rangefinder.

This last sentence above is exactly what really sets the two apart, if you ask me. The E-M5 is an incredibly potent jack of all trades in a small package, whereas the X-E1 very elegantly reduces your approach to the basics of photography called aperture, shutter speed, exposure compensation and focus (ok, and ISO), in an equally small package. It completely depends what kind of camera you are looking for in order to chose the right one for you. It' not about the lenses, the image quality or whatever, as those differences are not really worth mentioning. It's only about the type of camera you prefer. If you can't decide, like myself, you'll end up with both, using the one better suited for the given day and the other one on the next...