DxOMark's measured ISOs vs. manufacturer ISOs

Started Mar 13, 2013 | Discussions thread
GordonBGood Veteran Member • Posts: 6,308
Re: DxOMark's measured ISOs vs. manufacturer ISOs
4

cptobvious wrote:

GordonBGood wrote:

cptobvious wrote:

Yes, the ISO standard is silly, that's why comparisons based on a single testing methodology is needed.

Yes, the as the ISO standard only applies to output formats and isn't all that exact even at that, DxOMark had to come up with their own sensitivity "standard: for the purposes of comparing linear raw sensor outputs, and there is nothing wrong with their "standard" for its intended purpose.

However, your concern about the comparison between the raw sensitivity and the camera ISO isn't really a concern as DxOMark shows that the actual Olympus OM-D E-M5 sensor performs normally for the size of the sensor as to performance so that after making whatever post processing adjustments are necessary for the actual raw sensitivity, the image quality is in line with a rating based on sensor size, while the Fuji X100 does have a slight amount more noise in the bright tones than other APS-C sensors; the overall DxOMark sensor scores for these cameras are mostly down graded because the sensors do not use as low a real sensitivity as some other camera sensors do and that the digital data acquisition circuits put a cap on Dynamic Range (DR) at about the camera measured ISO 400 sensitivity for the Olympus.

In short, for real equivalent use, although the Olympus E-M5 does have about a stop less high ISO sensitivity usability due to its smaller sensor, it's real limit is more due a limited DR as compared to the best sensors for low ISO sensitivity use (but not everyone needs or uses this) and the Fuji X100 has this same limit with in a additional slightly less efficiency in photon gathering efficiency per unit area than the best sensors; other than the difference in maximum DR, there is only about a half a stop in difference in high ISO use between these cameras and the best sensors, which won't be a major factor in choice of cameras - other factors such as user interface and usability or lens selection will weigh much more.

Regards, GordonBGood

Your points are well taken. But my concern with the E-M5's ISO calculation methodology is that it has misled reviewers to claim that it has the same noise performance as larger sensors that obviously perform better but had their ISOs calculated more conservatively. Some examples:

http://www.imaging-resource.com/PRODS/omd-em5/omd-em5RAW.HTM (scroll down for D7000 comparison)

http://www.stevehuffphoto.com/2012/05/31/crazy-comparison-the-olympus-om-d-e-m5-vs-nikon-d800-for-high-iso/ (claiming the E-M5 holds its own in ISO performance against the D800)

http://www.cameralabs.com/reviews/Olympus_OM-D_E-M5/Olympus_E-M5_vs_Sony_NEX-7_vs_Nikon_D3200_noise_RAW.shtml

http://dslr-check.at.webry.info/201204/article_10.html (Japanese site)

http://www.pcmag.com/article2/0,2817,2403444,00.asp (claims better noise performance than the NEX-7)

Even DPReview's studio shot comparison tool uses different aperture settings:

http://www.dpreview.com/reviews/studio-compare#baseDir=%2Freviews_data&cameraDataSubdir=boxshot&indexFileName=boxshotindex.xml&presetsFileName=boxshotpresets.xml&showDescriptions=false&headerTitle=Studio%20scene&headerSubTitle=Standard%20studio%20scene%20comparison&masterCamera=oly_em5&masterSample=p1010013.acr&slotsCount=4&slot0Camera=oly_em5&slot0Sample=p1010013.acr&slot0DisableCameraSelection=true&slot0DisableSampleSelection=true&slot0LinkWithMaster=true&slot1Camera=nikon_d7000&slot1Sample=dsc1_1891.acr&slot2Camera=sony_nex5n&slot2Sample=dsc00790.acr&slot3Camera=pentax_k5&slot3Sample=imgp3870.acr&x=-0.18003992015968062&y=-0.8257205072370949&extraCameraCount=0

I have seen posted on reviews and on forums numerous times that the E-M5's sensor is as good or even better than the Sony Exmor APS-C equivalents at noise performance. The links I just pulled up are from just one search on Google in 10 minutes. To me this is just dishonest on the part of Olympus, and is inevitably going to lead to a 'race to the bottom' in future generation of cameras by manufacturers playing looser with their reported ISOs.

The DxOMark results for the Olympus OM-D E-M5 show that the raw sensor response is about a half stop worse than the best APS-C sensors for equivalent raw sensor sensitivity for typical none wide Dynamic Range (DR) use, which I believe to be correct.

Comparing images on the basis of JPEG output with all the variables of various ways of processing including differing sharpening and Noise Reduction (NR) methods and amounts, many of which cannot be entirely disabled, is always a very "iffy" and subjective comparison.  I know that is why you have dug up all these reviews that compare conversions from raw sensor data, but even those can be skewed by the raw converter used as some such as Adobe Camera Raw (ACR) as used by DPR has custom processing applied by camera model so that the results can be affected. But many of these reviews don't really prove your point, as follows (in reverse order):

  1. DPR has admitted that the studio light levels aren't always consistent across different cameras nor have attempt been made in the past to keep consistent aperture and shutter speed settings, unlike as Image-Resource sample studio images do.  Even at that, the E-M5 results as per the posted comparison link is not better than the Nikon D7000 nor the Pentax K-5 at ISO 3200 and is likely about a half stop worse just as DxOMark would indicate.  The E-M5 does look better than the Sony NEX-5N but that may be due to the reasons as stated as the Sony is less exposed by about 2/3's of a stop.
  2. The unsupported-by-images statement by PCMag that the E-M5 has less noise than the Sony NEX-7 is true on a per pixel basis given that the NEX-7 has 50% more of them.
  3. The Japanese website/blog showing that the E-M5 has close to the same noise as that of the Sony NEX-7 is probably correct according to DxOMark as it shows the objects are about the same size, meaning that the NEX-7 image has likely been down sampled by some unknown algorithm of the SilkyPix raw converter program used.
  4. CameraLabs use ACR for their comparison, with the usual provisos:  it is known to use different processing depending on camera model and the 24 Megapixel images have been sized to match the 16 Megapixel output of the E-M5.  The results don't really prove anything other than that the cameras are quite close in image quality for a given viewing size.
  5. The Steve Huff article proves nothing other than that the per photosite noise of the E-M5 has a similar level to that of the Niikon D800 when each is set to the same camera ISO sensitivity, which comes as no surprise given that DxOMark also predicts this for the two.
  6. The best and most consistent comparison is that of the Imaging-Resource (IR) link but again there are no surprises revealed in that DxOMark results predict that the E-M5 has better image quality that any of the cameras compared:  The Olympus E-P3, the Panasonic G3, and the Samsung NX200.  Looking at the exposure data for the sample images we see that the E-M5 is exposed by about 20% more than the Samsung NX200, but that the raw histogram still shows the raw sensor data for the E-M5 to be still less exposed than that of the NX200.

Now, other than for the IR comparison (which also uses the open-source dcraw converter as a verifiable camera-model-neutral application), it is difficult to dig out the exposures used for the other reviews to verify your clams that ISO settings are distorted excluding the DPR comparisons where they state themselves that there is every indication to believe that metered ISO sensitivity is standard as stated by the camera for output images within a small number of stops; there are no indications that there is any "foolery" done by Olympus other than to underexpose the raw captures for  more "raw highlight headroom" with metering set to be slightly compensate this by slight over exposure.  Raw-Digger examination of DPR and IR raw example files confirm this:  the E-M5 raw captures have more "highlight headroom" than comparable cameras even though they meter to expose slightly more.

However, every raw comparison you have cited has adjusted exposure in raw conversion and thus effective ISO sensitivity in order to compensate for the dark raw histogram else the IR comparison would show the EM-5 as much darker than the result from the Samsung NX200, for example.  Thus, the reviews use of the raw conversion programs (or in many cases automatic compensation applied by those programs) has done pretty much what the scaling and shifting of the DxOMark results due to their determination of real raw sensor sensitivity does.  In other words, raw conversion whether by camera or by raw converter application pretty much cancels out the effect of underexposure of the raw captures.

The conclusion remains:  DxOMark results stand that the Olympus OM-D E-M5 has a pretty good sensor but not quite as good as the best of the new APS-C sensors (or 35 mm. sensors) when compared on a per image size viewed basis.  None of the reviews to which you've linked refute this and only take a slightly different approach at their comparisons which don't make the conclusion as clear.  An alternate summary could be that you are wasting everyone's time in constantly harping that the E-M5 isn't as good a camera as many believe it to be based on this; it likely doesn't have quite the image quality of the best competing APS-C sensors, but that has little or nothing to do with Olympus's use of the sensor or their choice of metering sensitivity.

Regards, GordonBGood

Post (hide subjects) Posted by
tko
tko
Keyboard shortcuts:
FForum PPrevious NNext WNext unread UUpvote SSubscribe RReply QQuote BBookmark MMy threads
Color scheme? Blue / Yellow