You're right. I moved to M4:3 full time in 2014 after a career spent in Canon film (I've been a fill time photojournalist since 1984) and digital cameras. I started with the Panasonic GF1 and their pancake lenses in 2010 as a supplement to my 5D series bodies and L lenses.
I liked the GF1 a lot and used it for work when I could but the autofocus was just okay and the high ISO was pretty ragged (I didn't take it above 800). It was a great proof of concept camera though and I liked it a lot.
In 2013 I got an Olympus E-P5 and started switching to the Oly primes. The E-P5 was a huge improvement. The single shot AF was excellent, high ISO was better and the IBIS was a game changer. I started using the E-P5 as a main camera and put the 5D Mark III bodies away, only dragging them out when I needed something the 5D M3 offered the E-P5 did not - continous autofocus or long lens (the longest I had then was the 45mm f1.8) or available darkness.
In 2014 I gave up completely on "full frame." (Although really, to me, full frame refers more to photo technique than sensor size. Full frame means you don't crop, but I digress.) I got a couple of E-M5 Mark II bodies, the f2.8 Oly pro zooms and 75mm f1.8 prime. This combo did everything I need and delivered great results doing it. The IBIS meant I could use lower ISOs than I could with the 5D MkIII, and the Oly zooms covered me out to 420mm (with the excellent 1.4X teleconverter).
Now I'm using the E-M1 Mark II bodies and I love them. The AF, even continuous, is excellent. The IBIS still leads the industry. The lenses are outstanding. The only place it lags is high ISO but it's better than it was on the 5D Mk2 (IMO) and with some noise reduction it's very usable.
There's been a lot of noise in the FF mirrorless market this month, but not one of those cameras interests me. The mirrorless lenses are even bigger, and heavier, than the legacy lenses for dSLRs and the prices are out of reach. I have no regrets about moving into the M4:3 universe.