Finding the XE-1 EVF awful

After a week with the XE-1, I am finding the EVF almost impossible to work with. It takes just way too long to adjust to different light conditions. Not sure how anyone can deal with this. It literally takes almost a second or so if I point the camera towards a bright sunlit area after it has been looking towards a shaded one.

I tried an XPRO-1 last summer (returned it for various reasons) but mainly used the OVF. I don't remember anything like this.

Is anyone else having this problem or do I maybe have a defective camera?