I have a D700, X100, E-PL3, and LX3. The X100 has the best Auto White Balance and jpeg rendering of any camera I've ever used, including the above cameras, several digital Elphs, and the Canon 1DSMarkIII. The X100 is the only one of my cameras that I shoot jpeg.
Sometimes I turn on the X100 and compose a frame, even if there's no shot to take, just to see how the Fuji will render color in a particular light. It's that good.
The operational quirks aren't bad as many report, especially once you get the camera set up the way you like. But, if you shoot people, especially kids and candids, focussing can be much more troublesome than m4/3. In my opinion, face detection is one of the greatest advancements in photography since we went digital. It allows you to lock focus on a person and change composition to get the best photo. You're not stuck with focus and recompose, or repositioning a focus sensor, like you are with the X100 and other cameras that don't have face detection.
I like my X100 very much and it makes the most film-like images of any of my cameras, but I am not leaving m4/3. I sold my GF-1 and bought an Olympus E-PL3. m4/3 is great for travel and other outdoors photography and the 7-14mm is an incredible lens. I don't expect to ever take my D700 and a backpack full of lenses on a family vacation again. Well, maybe...