This is a challenging question. I've shot with all the cameras mentioned though, so I'll just weigh in with my perspective in case it's helpful.
Strictly between the cameras in the OP, I would choose the PEN-F. And I would do it for ergonomic reasons because I never got along with the X-T10/X-T20 style bodies. The feel and charm of the X-Pro and X-T1/2 is lost and it feels like a cheap X-T2 but not in a good way.
The PEN-F feels like a sculpture steeped in quality manufacturing. Not weather sealed, but also incredibly dense and re-assuringly constructed.
I reviewed the PEN-F not that long ago and was really surprised by the results:
I loved it.
IQ-wise, it's a strange situation. The Fujifilm will get you better dynamic range and there are lenses in the ecosystem that out-do the equivalent Olympus ones for sheer optical quality and character. I just did a shoot-out between the Fuji 35mm ƒ/2 and ƒ/1.4 and the Olympus 25mm ƒ/1.2 PRO...interesting results.
Anyway: you'll probably get more critically sharp shots out of the PEN-F. There's a whole section about it in the review, but it comes down to the contribution that IBIS makes.
I'll also mention that the topic of colours is more nuanced than people make it out to be. I love the Fujifilm film simulations, but you can get there with the PEN-F's JPG tools as well, it just takes more work. The much-maligned creative dial on the front is hiding the Colour Creator mode, which lets you effectively design your own film sim with
way more control than you get elsewhere.
Once you design your own film sim, you can use it just by turning the dial to it. But that initial setup is less convenient than the Fuji way where most of the work is done for you.
Anyway, I won't re-hash the entire review but in the end my impression is that both cameras are
fantastic and any differences in IQ won't become meaningful unless you regularly shoot in extremely difficult lighting situations. I have 13x19 prints from both in my home and they're indistinguishable.
The PEN-F will hit its maximum IQ potential more easily and more consistently thanks to IBIS, but the X-T20 has higher image quality potential if your technique (shooting and processing) is up to snuff. And if you like the ergonomics, which I don't.
If you were to replace the X-T20 with an X-T2 or X-Pro 2 in your question then I'd give the nod to those, but answering the question as asked: go with the PEN-F.