XZ-1 Limitations

I am trying to decide between the S100 and the XZ-1. It has come down to size versus image quality. I have seen some S100 sample pictures that have the type of quality that I would like, whereas I have consistently seen many XZ-1 samples that are simply outstanding. I have searched the web for hours and the XZ-1 is the clear IQ winner for my tastes. It consistently impresses me.

So I will need to make a decision whether that image quality is worth the size trade-off (and the need to shoot RAW to get the most out of time for best results). I don't care about video or GPS.

Where I'm still not clear, however, is under what shooting circumstances the higher ISO difficulty that the XZ-1 has would be a problem. Due to the great lens, it appears that the majority of shots in a variety of light conditions are ISO 100 and 200 with the odd 400 or 800. So when is the poor high ISO performance relative to the S95/100 ever an issue?