Have you considered the X-E3?
That with the 27 2.8 is smaller and lighter. Plus you can change lenses

The 27 2.8 will also be much sharper wide open the the 23 on the X100 series.
You will loose the OVF and pull up ISO adjuster. But you gain the ability to change lenses, a decent touchscreen, and a smaller more pocket-able form factor!
Hope this is helpful!
John
And what do you think about trying an X-A5?
It is even smaller than a T20 but offer a lot of her functionality: in fact even something that other fuji cameras does not have like the possibility to produce stunning 4K 30fps timelapse video directly in camera (included intervalometer).
I give a try to this camera because I have found one complete new open box kit with the fujinon XC 15-45 zoom lens with OIS for less than 350 euros on amazon warehouse!
I'm still discovering the full potential of this tiny camera but what a big surprise! It is so fast on startup, the 24megapixel bayer APSC sensor retain so much details , you have all the famous fuji film simulation (except eterna and acros) and the possibility to do raw conversion in camera.
A lot of parameters like ISO selection, AF/MF selection, focus magnification aid, film selection, exposure compensation, white balance, can be changed via physical dials or virtual buttons on the touch screen. Battery life is incredible in the mirrorless scenario (estimated 450shots per charge).
It is so good that Fuji engineers leave so much of the creative power through customisation into such a tiny and economic digital camera.
Another big advantage is the possibility to change lenses!
I have mounted a tiny voigtlander 21 mm lens using a fuji-x to leica m39 adaper and this duo is so small.

Fuji X-A5 with a Voigtlander 21mm f4.5 (fuji x to leica m39 adapter).