There's a point at which desirable attributes become "good enough" and more isn't necessarily better.
Are you happy with your lenses? If so, a newer Fuji camera would likely address the focusing speed issues of your X-Pro1, and the non-OVF versions are more compact. (Also, it'd be worth making sure you have the latest firmware on both your camera and lenses.)
The Leica CL would represent a good step up in image quality, with both the 18/2.8 and the 35/1.4. However, the 35/1.4 is relatively large, and cropping the 18/2.8 down to 50mm equivalent would probably not be satisfying. (It might be! We all have our personal tastes. If so, the CL & 18/2.8 might combine to make a perfect travel kit for you.) If having a fast aperture isn't important, the 18-56 zoom is smaller than the 35mm prime (!), and the zoom's image quality would likely be entirely satisfactory. Also, if the size of the 35 doesn't bother you -- the lens would mostly be in your bag, after all -- the improvement in quality might be welcome.
Enter the Q. The lens appears fantastic and the sensor does it justice: clearly, the Q offers the best image quality at 28mm of this bunch. The Q is also about the same size as your X-Pro1 with XF 35/1.4. So it wouldn't be getting larger, but it wouldn't be getting smaller, either. I haven't closely examined 50mm-like crops from it, so I can't comment on that.
But, if they're all more than good enough in terms of image quality, the question might reduce to one of size and ergonomics. The CL & 18/2.8 would be very small but is operated by two thumb dials, which is okay but not great for me. The Fuji's aperture ring and finger-operated shutter speed dial are quite nice. The Q also has those features, and adds a very functional manual focus ring and a viewfinder that (by all accounts I've seen) is a pleasant step up.
Personally, I like interchangeable lenses; as good as the Q is, it isn't for me. The CL would be far more appealing to me if I could get one in hand and try adjusting the left dial with my right index finger, reaching over the top plate. If that's comfortable, I'd have some preference for the superior Leica lenses. There's a reason, though, that I currently shoot more with my Fuji than my Leica: the Fuji is good enough where it needs to be, and offers me other benefits that I don't get from my Leica. But I do miss my M lenses, even if I don't need them.