DxOMark's measured ISOs vs. manufacturer ISOs

Started Mar 13, 2013 | Discussions
cptobvious Contributing Member • Posts: 815
DxOMark's measured ISOs vs. manufacturer ISOs
1

It's been stated numerous times in this forum that certain manufacturers overstate their ISOs in order to make their cameras seem to have better noise performance than they actually do.  Out of curiosity, I took a sample (non-scientific) of 22 tested cameras from 8 manufacturers on DxOMark, divided their measured ISO by the manufacturer ISO, and took the average of the results across the tested ISO range.

The number reflects what percentage the measured ISO is to the manufacturer ISO.  In the one case (GH-1) where the percentage exceeds 100%, the measured ISO was higher than the manufacturer ISO.  In all other cases it was lower.  The two worst offenders were the Olympus OM-D E-M5 and the Fuji X100.

Canon:

7D: 78%

5D Mark III: 85% (if removing ISO 50 as an outlier, it would be 78%)

6D: 83% (if removing ISO 50 as an outlier, it would be 76%)

Fuji:

X100: 50% (if removing ISO 3200 and 6400 as outliers, it would be 63%

X10: 99%

(no data available on X-Trans sensor cameras)

Nikon:

D7000: 83%

D800: 78% (if removing ISO 50 as an outlier, it would be 70%)

D600: 85% (if removing ISO 50 as an outlier, it would be 77%)

Olympus:

E-P1: 71%

E-P3: 64%

OM-D E-M5: 49%

Panasonic:

GH-1: 138%

GH-2: 98%

GH-3: 64% (if removing ISO 12800 and 25600 as outliers, it would be 74%)

Pentax:

K-x: 89%

K-7: 85%

K-5: 91%

Samsung:

NX100: 77%

NX200: 85%

Sony:

NEX-5N: 69%

NEX-6: 75%

NEX-7: 81%

Lenni Vilen Regular Member • Posts: 139
Re: DxOMark's measured ISOs vs. manufacturer ISOs
1

DxOMark does not measure ISO, but something I like to call DxOISO

ISO is a output format standard and DxOMark measues the raw data.

Additionally the ISO standard is quite silly and it allows the manufacturers to set the ISOs in pretty much any way they like, thus they don't overstate or understate their ISOs but just state their ISOs differently as the standard allows. Ie. there are no "offenders" as you stated.

OP cptobvious Contributing Member • Posts: 815
Re: DxOMark's measured ISOs vs. manufacturer ISOs
2

Yes, the ISO standard is silly, that's why comparisons based on a single testing methodology is needed.

GordonBGood Veteran Member • Posts: 6,308
Re: DxOMark's measured ISOs vs. manufacturer ISOs
1

cptobvious wrote:

Yes, the ISO standard is silly, that's why comparisons based on a single testing methodology is needed.

Yes, the as the ISO standard only applies to output formats and isn't all that exact even at that, DxOMark had to come up with their own sensitivity "standard: for the purposes of comparing linear raw sensor outputs, and there is nothing wrong with their "standard" for its intended purpose.

However, your concern about the comparison between the raw sensitivity and the camera ISO isn't really a concern as DxOMark shows that the actual Olympus OM-D E-M5 sensor performs normally for the size of the sensor as to performance so that after making whatever post processing adjustments are necessary for the actual raw sensitivity, the image quality is in line with a rating based on sensor size, while the Fuji X100 does have a slight amount more noise in the bright tones than other APS-C sensors; the overall DxOMark sensor scores for these cameras are mostly down graded because the sensors do not use as low a real sensitivity as some other camera sensors do and that the digital data acquisition circuits put a cap on Dynamic Range (DR) at about the camera measured ISO 400 sensitivity for the Olympus.

In short, for real equivalent use, although the Olympus E-M5 does have about a stop less high ISO sensitivity usability due to its smaller sensor, it's real limit is more due a limited DR as compared to the best sensors for low ISO sensitivity use (but not everyone needs or uses this) and the Fuji X100 has this same limit with in a additional slightly less efficiency in photon gathering efficiency per unit area than the best sensors; other than the difference in maximum DR, there is only about a half a stop in difference in high ISO use between these cameras and the best sensors, which won't be a major factor in choice of cameras - other factors such as user interface and usability or lens selection will weigh much more.

Regards, GordonBGood

Andrew Westlake Senior Member • Posts: 2,928
Re: DxOMark's measured ISOs vs. manufacturer ISOs
7

cptobvious wrote:

It's been stated numerous times in this forum that certain manufacturers overstate their ISOs in order to make their cameras seem to have better noise performance than they actually do.

It may have been stated numerous times, but it's been equally wrong each time, too.

-- hide signature --

Andy Westlake
dpreview.com

 Andrew Westlake's gear list:Andrew Westlake's gear list
Fujifilm FinePix F30 Zoom Olympus Stylus Verve S Olympus XZ-2 iHS Canon EOS 5D Panasonic Lumix DMC-GF1 +11 more
Barrie Davis
Barrie Davis Forum Pro • Posts: 21,460
Re: DxOMark's measured ISOs vs. manufacturer ISOs

Andy Westlake wrote:

cptobvious wrote:

It's been stated numerous times in this forum that certain manufacturers overstate their ISOs in order to make their cameras seem to have better noise performance than they actually do.

It may have been stated numerous times, but it's been equally wrong each time, too.

Which just about puts the tin lid on THAT, then.

Next? 

-- hide signature --

Regards,
Baz
"Ahh... But the thing is, these guys were no ORDINARY time travellers!"

 Barrie Davis's gear list:Barrie Davis's gear list
Konica Minolta DiMAGE A2
John Sheehy Forum Pro • Posts: 21,725
Re: DxOMark's measured ISOs vs. manufacturer ISOs
2

cptobvious wrote:

It's been stated numerous times in this forum that certain manufacturers overstate their ISOs in order to make their cameras seem to have better noise performance than they actually do. Out of curiosity, I took a sample (non-scientific) of 22 tested cameras from 8 manufacturers on DxOMark, divided their measured ISO by the manufacturer ISO, and took the average of the results across the tested ISO range.

The number reflects what percentage the measured ISO is to the manufacturer ISO. In the one case (GH-1) where the percentage exceeds 100%, the measured ISO was higher than the manufacturer ISO. In all other cases it was lower. The two worst offenders were the Olympus OM-D E-M5 and the Fuji X100.

DxO's ISO ratings have nothing whatsoever to do with the issues of manufacturers mis-stating ISOs.  DxO is concerned with rating the ISOs to meet a standard amount of RAW headroom; the issue with manufacturers has to do with metering, as cameras will vary in the real exposure the metering gives when set to an ISO.  The two are completely independent of each other, other than the fact that there is a practical limit to how little headroom you could have above metered middle gray.

OP cptobvious Contributing Member • Posts: 815
Re: DxOMark's measured ISOs vs. manufacturer ISOs
1

GordonBGood wrote:

cptobvious wrote:

Yes, the ISO standard is silly, that's why comparisons based on a single testing methodology is needed.

Yes, the as the ISO standard only applies to output formats and isn't all that exact even at that, DxOMark had to come up with their own sensitivity "standard: for the purposes of comparing linear raw sensor outputs, and there is nothing wrong with their "standard" for its intended purpose.

However, your concern about the comparison between the raw sensitivity and the camera ISO isn't really a concern as DxOMark shows that the actual Olympus OM-D E-M5 sensor performs normally for the size of the sensor as to performance so that after making whatever post processing adjustments are necessary for the actual raw sensitivity, the image quality is in line with a rating based on sensor size, while the Fuji X100 does have a slight amount more noise in the bright tones than other APS-C sensors; the overall DxOMark sensor scores for these cameras are mostly down graded because the sensors do not use as low a real sensitivity as some other camera sensors do and that the digital data acquisition circuits put a cap on Dynamic Range (DR) at about the camera measured ISO 400 sensitivity for the Olympus.

In short, for real equivalent use, although the Olympus E-M5 does have about a stop less high ISO sensitivity usability due to its smaller sensor, it's real limit is more due a limited DR as compared to the best sensors for low ISO sensitivity use (but not everyone needs or uses this) and the Fuji X100 has this same limit with in a additional slightly less efficiency in photon gathering efficiency per unit area than the best sensors; other than the difference in maximum DR, there is only about a half a stop in difference in high ISO use between these cameras and the best sensors, which won't be a major factor in choice of cameras - other factors such as user interface and usability or lens selection will weigh much more.

Regards, GordonBGood

Your points are well taken. But my concern with the E-M5's ISO calculation methodology is that it has misled reviewers to claim that it has the same noise performance as larger sensors that obviously perform better but had their ISOs calculated more conservatively. Some examples:

http://www.imaging-resource.com/PRODS/omd-em5/omd-em5RAW.HTM (scroll down for D7000 comparison)

http://www.stevehuffphoto.com/2012/05/31/crazy-comparison-the-olympus-om-d-e-m5-vs-nikon-d800-for-high-iso/ (claiming the E-M5 holds its own in ISO performance against the D800)

http://www.cameralabs.com/reviews/Olympus_OM-D_E-M5/Olympus_E-M5_vs_Sony_NEX-7_vs_Nikon_D3200_noise_RAW.shtml

http://dslr-check.at.webry.info/201204/article_10.html (Japanese site)

http://www.pcmag.com/article2/0,2817,2403444,00.asp (claims better noise performance than the NEX-7)

Even DPReview's studio shot comparison tool uses different aperture settings:

http://www.dpreview.com/reviews/studio-compare#baseDir=%2Freviews_data&cameraDataSubdir=boxshot&indexFileName=boxshotindex.xml&presetsFileName=boxshotpresets.xml&showDescriptions=false&headerTitle=Studio%20scene&headerSubTitle=Standard%20studio%20scene%20comparison&masterCamera=oly_em5&masterSample=p1010013.acr&slotsCount=4&slot0Camera=oly_em5&slot0Sample=p1010013.acr&slot0DisableCameraSelection=true&slot0DisableSampleSelection=true&slot0LinkWithMaster=true&slot1Camera=nikon_d7000&slot1Sample=dsc1_1891.acr&slot2Camera=sony_nex5n&slot2Sample=dsc00790.acr&slot3Camera=pentax_k5&slot3Sample=imgp3870.acr&x=-0.18003992015968062&y=-0.8257205072370949&extraCameraCount=0

I have seen posted on reviews and on forums numerous times that the E-M5's sensor is as good or even better than the Sony Exmor APS-C equivalents at noise performance. The links I just pulled up are from just one search on Google in 10 minutes. To me this is just dishonest on the part of Olympus, and is inevitably going to lead to a 'race to the bottom' in future generation of cameras by manufacturers playing looser with their reported ISOs.

Martin.au
Martin.au Forum Pro • Posts: 13,293
Re: DxOMark's measured ISOs vs. manufacturer ISOs
1

You do realize they're measuring two different things, don't you?

Did you read the rest of the thread, in particular Andy Westlake's comment?

 Martin.au's gear list:Martin.au's gear list
Olympus OM-D E-M5 Olympus E-M1 II Olympus M.Zuiko Digital ED 9-18mm F4.0-5.6 Panasonic Lumix G Fisheye 8mm F3.5 Olympus M.Zuiko Digital ED 12-50mm 1:3.5-6.3 EZ +7 more
edhannon
edhannon Senior Member • Posts: 1,761
Re: DxOMark's measured ISOs vs. manufacturer ISOs

Gordon, not sure I see the usefullness of ISO rating, either for comparison between cameras or use in a camera I use.

IMHO better measures are the number of stops saturation is above middle grey meter reaading and the number of stops a SNR of 0 is below meter reading.

That gives the useable dynamic range and whether the camera under/over exposes.

For day to day use those numbers allow you to adjusrt exposure for difficult subject brightness ranges.

One complication is that the saturation number varies based on quality of light (sunny, cloudy, shade, tungsten) and type of lens (new Pentax, M42, other manufactures).

Comments?

Ed Hannon

http://www.pbase.com/edhannon

 edhannon's gear list:edhannon's gear list
Pentax K10D Pentax K-3 Pentax smc DA 16-45mm F4 ED AL Pentax smc FA 77mm 1.8 Limited +1 more
Jack Hogan Veteran Member • Posts: 6,988
Re: DxOMark's measured ISOs vs. manufacturer ISOs
4

Andy Westlake wrote:

cptobvious wrote:

It's been stated numerous times in this forum that certain manufacturers overstate their ISOs in order to make their cameras seem to have better noise performance than they actually do.

True

It may have been stated numerous times, but it's been equally wrong each time, too.

Bertrand Russel would disapprove How wouldyou even know?

The OP provides evidence that the EM5 is an outlier - strangely enough massively in its favor, resulting in some interesting false-positive publicity, per his later post. This is one of my favorites, where the EM5 obviously performs as good as physical limits allow for its format.  Does a neutrino travel faster than the speed of light?  The Italians think so

While we are at it, who said 'You can fool all the people all of the time if the advertising is right and the budget is big enough'?

Right.

Jeff Charles Veteran Member • Posts: 7,514
Re: DxOMark's measured ISOs vs. manufacturer ISOs
1

Jack Hogan wrote:

Andy Westlake wrote:

cptobvious wrote:

It's been stated numerous times in this forum that certain manufacturers overstate their ISOs in order to make their cameras seem to have better noise performance than they actually do.

True

It may have been stated numerous times, but it's been equally wrong each time, too.

Bertrand Russel would disapprove How wouldyou even know?

The OP provides evidence that the EM5 is an outlier - strangely enough massively in its favor, resulting in some interesting false-positive publicity, per his later post. This is one of my favorites, where the EM5 obviously performs as good as physical limits allow for its format. Does a neutrino travel faster than the speed of light? The Italians think so

While we are at it, who said 'You can fool all the people all of the time if the advertising is right and the budget is big enough'?

Right.

The problem is the ISO testing procedure used by DPR and others. They equalize image brightnesses by adjusting actual exposures. They should be doing it by adjusting gain, while keeping exposures the same. Adjusting actual exposures gives an advantage to cameras whose nominal ISOs overstate their actual ISOs.

-- hide signature --

Jeff
"Please proceed..."

 Jeff Charles's gear list:Jeff Charles's gear list
Sony RX100 III Fujifilm X100T Olympus E-M1 Fujifilm X-E2 Olympus M.Zuiko Digital ED 40-150mm F4-5.6 R +5 more
timedrun Contributing Member • Posts: 683
Re: DxOMark's measured ISOs vs. manufacturer ISOs
6

shut up! The omd is a miracle camera that, as well as turning physics on its head, is also really cute and looks like a 1970's slr.

I wont hear a bad word said about any aspect of that camera (until the next model comes out, and after ive ebayed mine). Hey, did i mention that it also looks almost exactly like a 1970's slr?

-- hide signature --

A little green rosetta......a tiny green rosetta........

 timedrun's gear list:timedrun's gear list
Samsung NX20 Samsung NX 30mm F2 Pancake Samsung NX 50-200mm F4-5.6 OIS Samsung NX 16mm F2.4 Pancake Samsung NX 45mm F1.8
tkbslc Forum Pro • Posts: 13,170
Why should we trust DxO?
1

If essentially EVERY camera is 20-30% off according to DxO, then perhaps DxO are the ones that are off?

John Sheehy Forum Pro • Posts: 21,725
Re: Why should we trust DxO?
3

tkbslc wrote:

If essentially EVERY camera is 20-30% off according to DxO, then perhaps DxO are the ones that are off?

What DxO is measuring is not the metering of the camera.  If they did, there would be 3 data points for each ISO; camera-stated ISO, DxO-ISO, and camera-metered ISO.  The latter is not reported by DxOMark.

DxO's "measured ISO" is nothing more than this simple idea: the exposure index of the RAW tonal level in the green channel, some fixed number of stops below RAW saturation.  Why else do you think that "expanded" low ISOs are often the same "measured ISO" as the lowest normal base ISO?  This is a relatively unique metric which has no relationship to what we are usually talking about when we question whether or not a particular camera is mis-stating the ISO that it is metering for.

frank-in-toronto Senior Member • Posts: 1,034
Re: Why should we trust DxO?

and the very obvious reason manufacturers do this is to snare sales.  high-iso sells.  even if it's fake.

gwlaw99 Senior Member • Posts: 1,113
Re: DxOMark's measured ISOs vs. manufacturer ISOs
1

What is baffling about this topic in general is that no one has developed a standardized way to test sensors so that consumers can make apples to apples comparisons.

Detail Man
Detail Man Forum Pro • Posts: 17,044
Re: DxOMark's measured ISOs vs. manufacturer ISOs
2
CAcreeks
CAcreeks Forum Pro • Posts: 13,535
should we standardize on 80%
1

Thank you for doing this tabulation!

I say there should be a new ISO standard saying that digital ISO (DISO) should be 80% of the DxO-ISO to (film) ISO ratio.

After all, ISO makes money by selling standards, so what could be better than another standard? Then we could go back to calling it  "ASA" again.

Captain Obvious wrote:

It's been stated numerous times in this forum that certain manufacturers overstate their ISOs in order to make their cameras seem to have better noise performance than they actually do. Out of curiosity, I took a sample (non-scientific) of 22 tested cameras from 8 manufacturers on DxOMark, divided their measured ISO by the manufacturer ISO, and took the average of the results across the tested ISO range.

The number reflects what percentage the measured ISO is to the manufacturer ISO. In the one case (GH-1) where the percentage exceeds 100%, the measured ISO was higher than the manufacturer ISO. In all other cases it was lower. The two worst offenders were the Olympus OM-D E-M5 and the Fuji X100.

Canon:

7D: 78%

5D Mark III: 85% (if removing ISO 50 as an outlier, it would be 78%)

6D: 83% (if removing ISO 50 as an outlier, it would be 76%)

Fuji:

X100: 50% (if removing ISO 3200 and 6400 as outliers, it would be 63%

X10: 99%

(no data available on X-Trans sensor cameras)

Nikon:

D7000: 83%

D800: 78% (if removing ISO 50 as an outlier, it would be 70%)

D600: 85% (if removing ISO 50 as an outlier, it would be 77%)

Olympus:

E-P1: 71%

E-P3: 64%

OM-D E-M5: 49%

Panasonic:

GH-1: 138%

GH-2: 98%

GH-3: 64% (if removing ISO 12800 and 25600 as outliers, it would be 74%)

Pentax:

K-x: 89%

K-7: 85%

K-5: 91%

Samsung:

NX100: 77%

NX200: 85%

Sony:

NEX-5N: 69%

NEX-6: 75%

NEX-7: 81%

ultimitsu
ultimitsu Veteran Member • Posts: 6,650
Re: DxOMark's measured ISOs vs. manufacturer ISOs
3

Andy Westlake wrote:

cptobvious wrote:

It's been stated numerous times in this forum that certain manufacturers overstate their ISOs in order to make their cameras seem to have better noise performance than they actually do.

It may have been stated numerous times, but it's been equally wrong each time, too.

Then why does DPR continue to measure iso performance the wrong way? I am not being sarcastic, I am seriously curious.

Keyboard shortcuts:
FForum MMy threads