What would you do?

The only compelling reason not to switch, in my humble opinion, depends greatly on how much you use telephoto lenses. If you are a birder or like to shoot with long lenses, FF is probably not the best choice, especially moving from 4/3 to FF.

If you don't do a lot of long lens work, then go for it. High quality lenses are expensive regardless of the system and Canon has a number of high quality lenses just like Olympus does. Luckily we don't need the highest quality lenses on any system to turn out decent photos.