Shutter speed Olympus 45 1.8 vs Panasonic 20 1.7 on EM5

Not at all surprising - with such differing FOV, even the same subject would meter differently - and that is assuming the lighting is constant.

Try the same test on a grey card to get any meaningful comparison. It has to fill the frame on the 20, be the same distance for both lenses.