Should I sell my E-M5 for a Fuji X-E1? Help needed!

You make mention that you mostly frame with swiveling LCD, the Fuji's LCD is fixed.  It doesn't articulate.  That's a big difference.  I know myself I have come to like and use the swiveling screen.

Like you, I have seen that the Fuji is better at the higher ISO's.  But I don't anticipate that their EVF will be any better Olympus's or Sony's.  So in low light, you're back to using the LCD either way.  Granted the IBIS system won't compensate for motion blur of slower shutter speeds but we can agree that it's an awesome feature and very handy in most imaging situations and usable with ANY lens now.

The temptation to improve one's gear is always going to be there, especially in this age of new and improved every 9 months or so.  But you appear to have some nice glass and adding the 12mm f2 might be the best way to go (dollar-wise).

Don't forget that the lenses for the Fuji camera will be larger making the whole system noticably larger, that could turn your discreet street shooting system into one that may not be as discreet or easy on carry for long periods of time.