Fujifilm X-E1 or Olympus OM-D E-M5?

Yes until I tried the Oly.

Don't get me wrong, in no objective way did I hate the Oly, I just didn't like the kit zoom IQ and the ergonomics. Having slightly largish hands the control set and buttons on the Oly are cramped and hard to distinguish by feel, and it is if anything too customisable. However if you add the grip it becomes Frankencamera - the "extra" dial and shutter just killed it for me - I cant pick the camera up without pressing half the controls.

This is a very personal thing. People argue about it but it's a waste of time honestly (some people even buy both) but now I have the X-E1 (also own Xpro1) I have to say it just works for me. It makes sense, even to the point that the quirks make sense - it is in essence such a simple camera that you know when you f**ed up and when the camera did (moreso than the Xpro1), and in the latter case what to do about it. Workarounds are simple and mostly obvious (and I don't have to remember what I programmed the 5 programmable buttons to do, so I can do all of it without looking).

And the first time I saw the bokeh on the 60mm I fell in love - I don't care HOW slow it focuses.

In essence two cameras after the same market with with diametrically opposite approaches. I would love some of the Oly features on my Xe1 (IBIS, long exposure previews, flash control), but then I would like the Oly to look like and work like my Xe1 too. I find the Xe1s deficiencies easier to live with. I guess I am just coming from the same place as Fuji's designers.

I also think Fuji are developing fast. From nowhere to the Xe1 in two years with 4 excellent lenses and more on the way is amazing. Fuji have claimed that phase detect is not far off (Xpro2 perhaps?). Oly on the other hand seem to have reached their zenith with the EM5. The sensor is mighty good but how much further can one push an MFT Bayer sensor in terms of DR and resolution. APSC Xtrans is in its first iteration and already beats the Oly for IQ (not by much at low ISO perhaps, granted, but like I said it's only the start). Future development of Xtrans has real possibilities.

I think as a long term investment Fuji is very compelling. The quality of the lenses (build as well as optics) is amazing for the price. Much as I like some of the Oly primes they are also expensive on the whole and I am much less convinced by the zooms, except the wide angle possibly. They are slow and just not that amazing, whereas the 18-55 lens on the Xe1 seems to be amazing - for a zoom.

However I will qualify this and say "my other camera is a D600". I have an option for studio and event photography with all the flash, high speed continuous AF and other features you could dream of. I am not using my Xe1 or Xpro1 as all-rounders, just street and travel cameras.