whumber
Senior Member
The claim in the E-M1X review that the G9 IBIS is superior to both the E-M1ii and E-M1X was something I just couldn't make sense of based on my experience owning all three of those cameras. I've been trying to get details of how DPReview is conducting their test, but outside of the extremely vague description they include in some of the reviews they seem to have no interest in responding to any of my requests for more details. I decided to try actually measuring the difference between the two cameras to see if my overall impressions matched up with an attempt at objective measurements.
To take some measurements, I'm using a standard ISO-12233:2000 test chart, or at least the central portion of it, to do some slant edge MTF calculations. I have the central portion of the chart printed at 13"x19" which would correspond to a full test chart around 33" wide. I'm using a variable LED light stand so that I can maintain a constant aperture throughout all the tests.
All shots are taken while standing, unsupported, 6.5 ft from the test chart with the camera at the same level as the center of the chart. Best focus is attained by using each cameras "punch in" feature and focusing until a best focus is attained and then a minimum of 50 shots are taken in burst mode. Electronic shutter is used in all cases for best sharpness except on the G9 for the 2s exposures as electronic shutter is not available, in that case the mechanical shutter with electronic first curtain is used. All images are taken with the Olympus 12-40mm f/2.8 lens at 12mm | f/5.6| ISO 100; I chose f/5.6 to attempt to minimize the effect of forward and back sway while not going too far into diffraction territory.
To prepare for processing the results I also took a series of tripod mounted images in order to establish control values as I expected to see significant differences in the MTF calculations between the two cameras just due to factors such as the AA filter and any hidden processing that might occur from the Adobe RAW converter. Fortunately, the two ended up showing almost identical MTF values, although all the MTF values are normalized by the maximum MTF values measured from the tripod control images. For the MTF calculations I used an older version of Reikan Focal's "Focus Consistency" tool where it allows you to load files manually (although the EXIF has to be hacked for my version to make it think that you're using an older Canon or Nikon body). If I was going to be doing this on a larger scale I'd put something together in matlab but this was easy for something quick and dirty since the control tests showed credible results.
Tests were done at 1/10s, 1/5s, 1/2.5s, 1s, and 2s. I only performed the tests with IS enabled as I'm only concerned about the relative performance of the E-M1X and G9 rather than the absolute degree of stabilization relative to an unstabilized system. The results are summarized in the chart below. One important note, the green line represents the MTF value above which I would consider very sharp while the red line is the MTF value threshold I would consider the image to be soft but useable; anything below the red line is blurred. These values are something I had to set arbitrarily but I believe I was fairly aggressive in setting the limits. To give an idea of what is considered sharp, soft, and blurred I'm including images that correlate to each of the thresholds.
Sharp
Soft
Blurred
The main takeaway is that, just like I suspected, the IBIS in the E-M1X is superior to the G9. In addition, DPReview's reporting showed that at 0.2s they saw approximately 20% sharp images, 30% soft images, and 50% blurry. In my testing, and this is with just the IBIS rather than IBIS + OIS of the 12-100, I see 0 out of 54 images fall below the sharp threshold. At 0.2s we also see the E-M1X and G9 performing very similarly with a slight advantage to the E-M1X on average. Note for the histograms below, I had to artificially add a count in the lowest and highest bin for each plot simply for reasons of laziness when creating the plots. Also the green, yellow, and red parts of the plots represent sharp, soft, and blurred zones respectively with the red zone truncated to a single histogram bin.
Similarly at 0.5s they report 0 sharp images, 20% soft, and then the rest blurred. I see 60 above the sharp threshold, 2 soft, and 0 blurred images at 0.4s. Why 0.4s instead of 0.5s? Because I did the test at 0.4s and didn't feel like going back and reprocessing everything, =). At 0.4s we also see the E-M1X really start to pull ahead of the G9. The E-M1X maintains almost a perfect sharp rate but the G9 is down to 64% sharp.
At 1s, the E-M1X is finally starting to show some weakness as it gets its first truly blurred images but it still maintains a 71% sharp rate. The G9 on the other hand really starts to struggle at this point. It still manages to get sharp images but the sharp rate is down to 27% and the blurred rate is 57%.
Finally at 2s the G9 seems to have given up. I didn't get a single image out of 50 that classified as sharp or even soft, all were well into the blurred range. The E-M1X on the other hand is definitely struggling at 2s but is still managing a 33% sharp rate. Interestingly, there's a bimodal behavior in this range; it seems like the images either come out sharp or completely blurred with very little in between.
Now before the first person comes out and starts screaming that they can get 2s exposures on their G9 while dancing the jitterbug I'm not saying it's not possible, just that with IBIS only at 12mm and a 6.5ft focus distance it's much easier to do with the E-M1X. After about a year with the G9 I think I've got a pretty good feel for it but if someone knows a secret setting that will greatly improve the IBIS performance I'd certainly be interested.
This testing methodology is certainly not perfect, but I honestly don't understand what DPReview is doing with their test to get the results they publish. My best guess is that they're doing something goofy like standing right next to the test chart when doing the 24mm equivalent measurements which will change translational camera movements from a secondary driver of blur to a primary driver. I've asked them repeatedly for details about their test procedure but I'm always left hanging. Maybe they'll open up their methodology in the future, but in the meanwhile I believe there's some fundamental methodology flaw in their testing and I would recommend taking anything they report with a huge grain of salt.
EDIT: DPReview is doing something silly with most of pictures so you'll have to click on "View Original Size" below the picture to actually see anything.
To take some measurements, I'm using a standard ISO-12233:2000 test chart, or at least the central portion of it, to do some slant edge MTF calculations. I have the central portion of the chart printed at 13"x19" which would correspond to a full test chart around 33" wide. I'm using a variable LED light stand so that I can maintain a constant aperture throughout all the tests.
All shots are taken while standing, unsupported, 6.5 ft from the test chart with the camera at the same level as the center of the chart. Best focus is attained by using each cameras "punch in" feature and focusing until a best focus is attained and then a minimum of 50 shots are taken in burst mode. Electronic shutter is used in all cases for best sharpness except on the G9 for the 2s exposures as electronic shutter is not available, in that case the mechanical shutter with electronic first curtain is used. All images are taken with the Olympus 12-40mm f/2.8 lens at 12mm | f/5.6| ISO 100; I chose f/5.6 to attempt to minimize the effect of forward and back sway while not going too far into diffraction territory.
To prepare for processing the results I also took a series of tripod mounted images in order to establish control values as I expected to see significant differences in the MTF calculations between the two cameras just due to factors such as the AA filter and any hidden processing that might occur from the Adobe RAW converter. Fortunately, the two ended up showing almost identical MTF values, although all the MTF values are normalized by the maximum MTF values measured from the tripod control images. For the MTF calculations I used an older version of Reikan Focal's "Focus Consistency" tool where it allows you to load files manually (although the EXIF has to be hacked for my version to make it think that you're using an older Canon or Nikon body). If I was going to be doing this on a larger scale I'd put something together in matlab but this was easy for something quick and dirty since the control tests showed credible results.
Tests were done at 1/10s, 1/5s, 1/2.5s, 1s, and 2s. I only performed the tests with IS enabled as I'm only concerned about the relative performance of the E-M1X and G9 rather than the absolute degree of stabilization relative to an unstabilized system. The results are summarized in the chart below. One important note, the green line represents the MTF value above which I would consider very sharp while the red line is the MTF value threshold I would consider the image to be soft but useable; anything below the red line is blurred. These values are something I had to set arbitrarily but I believe I was fairly aggressive in setting the limits. To give an idea of what is considered sharp, soft, and blurred I'm including images that correlate to each of the thresholds.
Sharp
Soft
Blurred
The main takeaway is that, just like I suspected, the IBIS in the E-M1X is superior to the G9. In addition, DPReview's reporting showed that at 0.2s they saw approximately 20% sharp images, 30% soft images, and 50% blurry. In my testing, and this is with just the IBIS rather than IBIS + OIS of the 12-100, I see 0 out of 54 images fall below the sharp threshold. At 0.2s we also see the E-M1X and G9 performing very similarly with a slight advantage to the E-M1X on average. Note for the histograms below, I had to artificially add a count in the lowest and highest bin for each plot simply for reasons of laziness when creating the plots. Also the green, yellow, and red parts of the plots represent sharp, soft, and blurred zones respectively with the red zone truncated to a single histogram bin.
Similarly at 0.5s they report 0 sharp images, 20% soft, and then the rest blurred. I see 60 above the sharp threshold, 2 soft, and 0 blurred images at 0.4s. Why 0.4s instead of 0.5s? Because I did the test at 0.4s and didn't feel like going back and reprocessing everything, =). At 0.4s we also see the E-M1X really start to pull ahead of the G9. The E-M1X maintains almost a perfect sharp rate but the G9 is down to 64% sharp.
At 1s, the E-M1X is finally starting to show some weakness as it gets its first truly blurred images but it still maintains a 71% sharp rate. The G9 on the other hand really starts to struggle at this point. It still manages to get sharp images but the sharp rate is down to 27% and the blurred rate is 57%.
Finally at 2s the G9 seems to have given up. I didn't get a single image out of 50 that classified as sharp or even soft, all were well into the blurred range. The E-M1X on the other hand is definitely struggling at 2s but is still managing a 33% sharp rate. Interestingly, there's a bimodal behavior in this range; it seems like the images either come out sharp or completely blurred with very little in between.
Now before the first person comes out and starts screaming that they can get 2s exposures on their G9 while dancing the jitterbug I'm not saying it's not possible, just that with IBIS only at 12mm and a 6.5ft focus distance it's much easier to do with the E-M1X. After about a year with the G9 I think I've got a pretty good feel for it but if someone knows a secret setting that will greatly improve the IBIS performance I'd certainly be interested.
This testing methodology is certainly not perfect, but I honestly don't understand what DPReview is doing with their test to get the results they publish. My best guess is that they're doing something goofy like standing right next to the test chart when doing the 24mm equivalent measurements which will change translational camera movements from a secondary driver of blur to a primary driver. I've asked them repeatedly for details about their test procedure but I'm always left hanging. Maybe they'll open up their methodology in the future, but in the meanwhile I believe there's some fundamental methodology flaw in their testing and I would recommend taking anything they report with a huge grain of salt.
EDIT: DPReview is doing something silly with most of pictures so you'll have to click on "View Original Size" below the picture to actually see anything.
Last edited:
