At the risk of being humiliated for my lack of camera knowledge, has anyone compared these two cameras side by side? The E5 features a 40MP sensor, a Film Sim dial, and is marginally smaller. But inferior on these features: LCD Res, Drive FPS, less than 1/2 battery life, tilt vs fully articulated screen, no built-in flash, and costs $400 more. Maybe the E5 is more user-friendly? I love the look of it, I admit. The size difference is primarily due to the larger battery in the S20, but it also gives you a better grip. BTW, I have the X-S20, kit lens, and 70-300 plus Fuji 2X teleconvertor (900mm equiv).
I have an X-S20 + X-E4 combo and I'll probably replace the 4 with the 5, so I'll bite with my very personal take on this.
I bought the X-S20 because the X-E4 is severely lacking in the video department. It stopped recording unpredictably before the official limit of 30 minutes, ruining what I was working on and forcing me to start over. What’s also frustrating is that the tripod mount is a few millimeters away from the SD card compartment, so you have to take your camera off the tripod or arm you’re using before you can access the SD card. Now that I’ve got the X-S20, I’ve come to appreciate the grip for semi-pro photography work when I don’t need to watch the weight I’m carrying. But for family holidays, it's too big a camera to just grab and have fun with.
Enter the X-E5. I can use it as a full second body next to my X-S20: it has no recording limit and has IBIS for my photography work. So it's a much more comparable camera to the X-S20 in that respect. And the tripod mount is now further away from the SD card compartment!
But as a travel camera, it's much better that the X-S20. Aside from obviously being smaller and lighter, you can also use the 'digital teleconverter'. 1.4x at 20mp and 2x at 10mp. If you're not going to print larger than A3 size, those 10mp are sufficient. But it turns a prime into a very useful triple prime! The 18mm is then also a 25 and 36mm (28mm, 35-ish, 50-ish equiv). Or the new 23mm becomes a handy 35 + 46 (35, 50, 70mm equiv).
The X-E5 also seems to have more physical controls, though I should check that. For my work, the C1-C4 on the X-S20 is more practical than the film sim dial on the X-E5: C1 video, C2 Acros photos, C3 chrome etc. With the X-E5 I could use the 3 custom sim recipes but I'd have to configure my photo/video settings by hand.
Those are my thoughts in a nutshell. I obviously haven't used an X-E5 yet, but comparing the specs vs my experience with the X-E5, the latter keeps the X-E physical appearance, which remains the biggest difference with the X-S20.