NEX 6? Any downsides?

If you put the standard 18-55 Sony zoom on the NEX then it isn't very compact.

Whether it's compact enough for you or not only you can decide. Most people figure if it doesn't fit in a pocket then you're carrying it on a strap, and once you go to that there's not much practical difference in size.

If you are OK with a camera with a large protruding lens, then also consider Nikon 1 cameras and Olympus PEN cameras. Thom Hogan has a good website on mirrorless cameras that you can link to from bythom.com. He reviews them and ranks them in various ways. Unlike most people here, he has actually used most (all?) of them.