DxOMark's measured ISOs vs. manufacturer ISOs

Started Mar 13, 2013 | Discussions
Jack Hogan Veteran Member • Posts: 6,982
Re: I wish I could favorite this 1,000 times!

texinwien wrote:

Andy Westlake wrote:

cptobvious wrote:

It's been stated numerous times in this forum that certain manufacturers overstate their ISOs in order to make their cameras seem to have better noise performance than they actually do.

It may have been stated numerous times, but it's been equally wrong each time, too.

I'm seriously laughing over here.

True, concise and hilarious. Touché!

I love sarcasm

Martin.au
Martin.au Forum Pro • Posts: 13,248
Re: DxOMark's measured ISOs vs. manufacturer ISOs

Jack Hogan wrote:

Mjankor wrote:

Jack Hogan wrote:

So our hapless EM5 owner, having evaluated his scene, sets ISO in both cameras up so that their sensors will both saturate at the same Hsat = 0.0525 lx-s, which just so happens to correspond to an in-camera setting of ISO 3200 for the EM5 and ISO 1600 for the G3 (or Ssat of 1489 and 1481 respectively). And this is the response he gets:

Noise performance when in-camera ISO is chosen so that both *sensors/ADCs* will provide the same mean outputl when presented with the same Exposure

Whoa, un-wow, right? The G3 is not so bad after all

Can you please include some hypothetical shutter, iso, etc numbers for these examples.

For example, the first two at ISO 3200 would be:

OMD - 1/100, ISO 3200, f5

G3 1/100, ISO 3200, f5.

And for the second

OMD - 1/100, ISO3200, f5

G3 - 1/50, ISO 1600, f5

In my simplified example above both cameras use the same lens and both sensors are set up* to clip the same scene highlights at the same Exposure (Hsat). Since DSC sensor performance below saturation is linear, that means that the mean Raw value produced by either sensor/ADC as a fraction of full scale will be about the same for any scene Exposure below this value - resulting in an apples to apples, objective comparison of the foundation of IQ: 'for a given amount of light (Exposure) from the scene, camera A produces a SNR of x and camera B a SNR of y'.

Exposure is fully defined by shutter speed and f/number only. So to make sensor Exposure the same one can use the same shutter speed and f/number in both cameras, or any other combination that keeps the Exposure Value constant. In the apples to apples image above with the same lens, the settings were

EM5 f/6.3 1/800s
G3 f/6.3 1/640s

The two shutter speeds should be identical but they are in fact 0.3 stops off because of differences in metering - see the PS of my previous message.

*In-camera ISO settings do not matter for this comparison and are simply a consequence of having set up both cameras' sensors/ADCs to respond similarly to scene light, so that they will provide the same mean output level for a given input. If you are curious, that meant 3200 for the EM5 vs 1600 for the G3.

Cheers,
Jack

So, just to clarify, you think that for the same scene: OM-D f6.3, 1/800, ISO 3200 will give a similar result as G3, f6.3, 1/640, ISO 1600. If so, I refer you to my experiment above.

 Martin.au's gear list:Martin.au's gear list
Olympus OM-D E-M5 Olympus E-M1 II Olympus M.Zuiko Digital ED 60mm 1:2.8 Macro Olympus M.Zuiko Digital ED 12-50mm 1:3.5-6.3 EZ Panasonic Lumix G Fisheye 8mm F3.5 +7 more
Jack Hogan Veteran Member • Posts: 6,982
Re: DxOMark's measured ISOs vs. manufacturer ISOs
2

Andy Westlake wrote:

Jack Hogan wrote:

The OP provides evidence that the EM5 is an outlier

The OP is using DxO's 'measured ISO' to reinterpret tests that are calibrated to a different ISO definition (ISO 12232:2006 Standard Output Sensitivity). This is entirely erroneous.

Bertrand Russle is turning in his grave.  Are you sure it's SOS in every single one of the cases he mentions?

It seems to me that any of the recommended parameters in that there standard could have been used, depending on camera and manufacturer: Saturation Speed, Noise Speed, SOS or REI.

No matter, he's got a point

Jack Hogan Veteran Member • Posts: 6,982
Re: DxOMark's measured ISOs vs. manufacturer ISOs

Mjankor wrote:

Noise performance when in-camera ISO is chosen so that both *sensors/ADCs* will provide the same mean outputl when presented with the same Exposure

So, just to clarify, you think that for the same scene: OM-D f6.3, 1/800, ISO 3200 will give a similar result as G3, f6.3, 1/640, ISO 1600. If so, I refer you to my experiment above.

Not quite. If DxO's Ssat measurements are correct (no reason to doubt that they are not), the mean Raw value produced by a Middle-Gray tone illuminated by the same source at the same exposure should be about the same, within measurement error, when settings are:

EM5 f/6.3, 1/800 ISO 3200
G3 f/6.3 1/800* ISO 1600

Of course this says nothing about relative IQ. It simply allows you to compare SNR objectively and possibly begin to draw some conclusions off an image like the one in this post.

Jack
*You don't really thing the G3 is better than the EM5, do you? I'll take a look at your experiment.

Randell Tober
Randell Tober Regular Member • Posts: 248
Re: DxOMark's measured ISOs vs. manufacturer ISOs

I want to thank all of you at DPR for your great work over the years. I’ve followed you off and on as a hobbyist for many. Maybe a new article or link that shows just what goes into all of the testing/comparisons at DPR would be beneficial. I’ve seen so much discussion and banter throughout many of the forums. Once again “Thanks” for all of the great articles/reviews and thought.

 Randell Tober's gear list:Randell Tober's gear list
Canon PowerShot SD4000 IS Nikon D7100 Nikon AF-S DX Nikkor 35mm F1.8G Nikon AF Nikkor 24-85mm f/2.8-4D IF Nikon AF-S Nikkor 70-300mm f/4.5-5.6G VR +2 more
texinwien Veteran Member • Posts: 3,326
No Sarcasm Here

Jack Hogan wrote:

texinwien wrote:

Andy Westlake wrote:

cptobvious wrote:

It's been stated numerous times in this forum that certain manufacturers overstate their ISOs in order to make their cameras seem to have better noise performance than they actually do.

It may have been stated numerous times, but it's been equally wrong each time, too.

I'm seriously laughing over here.

True, concise and hilarious. Touché!

I love sarcasm

No sarcasm - Mr. Westlake is absolutely correct and has made his point with style.

tex

 texinwien's gear list:texinwien's gear list
Panasonic Lumix DMC-GM5 Olympus E-M5 II Olympus 12-40mm F2.8 OnePlus One Canon EOS 300D +20 more
Jack Hogan Veteran Member • Posts: 6,982
Re: DxOMark's measured ISOs vs. manufacturer ISOs
4

Mjankor wrote:

As you can see, both cameras meter about the same and record photos of similar exposure. If anything the E520 is a touch underexposed compared to the OM-D.

You are mixing several variables that are quite different and need to be kept apart: Metering/Recording, Input Exposure/ Output Brightness, Raw/rendered data.  Plus you are forgetting an important one, in fact the only one that this thread is really about: IQ.

Metering: In theory for a given scene and ISO speed (S,going back to film days, therefore NOT the latest crappy definitions, but the old one) your camera's spot meter should provide the same parameters for ss and f/n (Exposure) as a hand held reflected light meter - for instance Sunny 16.  The meter's variable is called the Reflected Light Meter constant K, which for CaNikon is 0.125.  In theory all meters that have the same k will provide the same Exposure for a set S.  This determines the input signal and has nothing to do with output brightness.

Recording: In theory a camera with a spot meter as above will meter off of and record a tone half way between perceived maximum diffuse white and black (called Middle Gray) in Raw values around 12.5% of full Scale, or 2.5 stops below saturation.  Is this good? Can't say, it depends on the scene.  Most modern cameras actually record Middle Gray 3-4 stops below full scale.  This determines what gets recorded in the Raw data and it still has nothing to do with output brightness.

Rendered Image: your in-camera/in-computer rendering engine takes the raw data and applies a number of very severe linear and non linear transformations to the data in order to produce an image on the output medium of choice of pleasing brightness/contrast/saturation etc.  This is what you see on your monitor: it has a lot to do with output brightness but it has very little to do with input Exposure.

IQ, for this thread SNR:  SNR is really what ties together input Exposure and Brighness (ok, more than just brightness, bet let's simplify).

As you know, you may have accidentally severely underexposed a capture, but still produce a pleasingly bright output on your monitor by changing rendering parameters in PP - of course it may also be a tad noisier than desired...

Problem is, some DSC manufacturers use output image Brightness (instead of recording as a function of raw full scale) to help determine input Exposure by using it to determine S which affects metering, but as I hope you now see, the two are not really related - and IQ may suffer badly as a result.  In fact, this horrible behavior is sanctioned in one of the least well thought out ISO non-standards around, the aforementioned 12232:2006.  What is one to do?  DxO to the rescue.

Conclusion:

The OM-D definitely did not require a shutter speed twice as slow as the E520 to take a similarly exposed image.

So you should now see that this means the reflected meter constants are similar.

Ergo the hypothesis that DXO ISO is comparable to the camera ISO is wrong.

So you should see that we can make any image look as bright/contrasty as another (in-camera or in-cpmputer), but that the real question for this thread is 'which one is cleaner'?

Cheers,
Jack

Jack Hogan Veteran Member • Posts: 6,982
Re: No Sarcasm
2

texinwien wrote:

Jack Hogan wrote:

texinwien wrote:

Andy Westlake wrote:

cptobvious wrote:

It's been stated numerous times in this forum that certain manufacturers overstate their ISOs in order to make their cameras seem to have better noise performance than they actually do.

It may have been stated numerous times, but it's been equally wrong each time, too.

I'm seriously laughing over here.

True, concise and hilarious. Touché!

I love sarcasm

No sarcasm - Mr. Westlake is absolutely correct and has made his point with style.

It may appear stylish to you, to others it may look curt and unsubstantiated - which does not mean they do not appreciate DPR or its community.

Jack

Jeff Charles Veteran Member • Posts: 7,514
Re: DxOMark's measured ISOs vs. manufacturer ISOs

Gordon - Thanks for your incredibly informative posts, as usual.

GordonBGood wrote:

  1. DPR has admitted that the studio light levels aren't always consistent across different cameras nor have attempt been made in the past to keep consistent aperture and shutter speed settings,unlike as Image-Resource sample studio images do. 

DPR not maintaining consistent lighting or exposure does call into question the validity of their ISO tests.

I do not know if IR has changed testing procedures, but in the past, they have varied light levels and/or exposure to equalize output brightness between cameras. I raised the issue in an IR forum thread a few years ago. My post is number 3, and Dave Etchell's response is number 4.

-- hide signature --

Jeff
"Please proceed..."

 Jeff Charles's gear list:Jeff Charles's gear list
Sony RX100 III Fujifilm X100T Fujifilm X-E2 Olympus E-M1 Fujifilm XF 18-55mm F2.8-4 R LM OIS +5 more
texinwien Veteran Member • Posts: 3,326
Re: No Sarcasm

Jack Hogan wrote:

texinwien wrote:

Jack Hogan wrote:

texinwien wrote:

Andy Westlake wrote:

cptobvious wrote:

It's been stated numerous times in this forum that certain manufacturers overstate their ISOs in order to make their cameras seem to have better noise performance than they actually do.

It may have been stated numerous times, but it's been equally wrong each time, too.

I'm seriously laughing over here.

True, concise and hilarious. Touché!

I love sarcasm

No sarcasm - Mr. Westlake is absolutely correct and has made his point with style.

It may appear stylish to you, to others it may look curt and unsubstantiated - which does not mean they do not appreciate DPR or its community.

Jack

I do understand that, Jack. I think the difficult thing for the DPReview guys is that this topic has been hashed and rehashed, ad nauseum (and I mean that literally ). The DPReview guys have answered the misconceptions in great detail - some misconceptions I've seen you posting in this thread for instance.

They have made their cases to skeptical digital imaging experts (people who are engineers in the field), generally, after much toil and trouble, to those experts' satisfaction, as far as I've seen.

But every few months, the same non-conspiracy boils to the top again, this time with new actors who are acting in good faith and making arguments that seem logical and common sensical, but which are either based on some degree of ignorance regarding the topic, some degree of ignorance about DPReview's testing regimes, or a combination of those things.

The problem is that they've been here before, but the new crops of people who come in and make these arguments every few months don't bother to go back and familiarize themselves with all of the details before coming up with their brilliant arguments that they're simply 100% certain are correct.

So I understand the DPReview team's annoyance at having to deal with this same topic, over and over again, with people who haven't done the hard work to learn the details about the topic and/or the details about how DPReview runs its tests.

tex

 texinwien's gear list:texinwien's gear list
Panasonic Lumix DMC-GM5 Olympus E-M5 II Olympus 12-40mm F2.8 OnePlus One Canon EOS 300D +20 more
Jack Hogan Veteran Member • Posts: 6,982
Re: No Sarcasm
2

texinwien wrote:I do understand that, Jack. I think the difficult thing for the DPReview guys is that this topic has been hashed and rehashed, ad nauseum (and I mean that literally ). The DPReview guys have answered the misconceptions in great detail - some misconceptions I've seen you posting in this thread for instance.

They have made their cases to skeptical digital imaging experts (people who are engineers in the field), generally, after much toil and trouble, to those experts' satisfaction, as far as I've seen.

But every few months, the same non-conspiracy boils to the top again, this time with new actors who are acting in good faith and making arguments that seem logical and common sensical, but which are either based on some degree of ignorance regarding the topic, some degree of ignorance about DPReview's testing regimes, or a combination of those things.

The problem is that they've been here before, but the new crops of people who come in and make these arguments every few months don't bother to go back and familiarize themselves with all of the details before coming up with their brilliant arguments that they're simply 100% certain are correct.

So I understand the DPReview team's annoyance at having to deal with this same topic, over and over again, with people who haven't done the hard work to learn the details about the topic and/or the details about how DPReview runs its tests.

tex

Fair enough, tex.  Now me being someone who likes to learn, would you mind setting me straight on the misconceptions I may have helped to perpetrate?

Jack

tko Forum Pro • Posts: 12,770
why not?
1

After reading this entire thread and seeing all the math and reasoning people throw up to obfuscate this simple topic, it seems very simple. Test the cameras at the same or equivalent aperture and shutter speed.

A calculated ISO based on sensor data is simply useless in the real world. Mathematical games with no meaning.

Why can't you have the the same light and the same exposure? Not that hard to measure light, and not that hard to adjust. Diffusers, snoots, and other devices. Perhaps even, gasp, a flash. Shutter speed can be measured, and so can len's properties. Actually, you could measure the light intensity behind the lens in a lens test jig, and adjust all lenses for the same value, then test at that setting on camera.

So, definitely doable to adjust the light and keep a level playing field.

Bottom line is that high ISO is used to control shutter speed is low light environments. A camera that gives half the shutter speed under the exact same circumstances is only half as good (for this area.) Manipulated the numbers all you want, but that's the take away.

ultimitsu wrote:

Just so we do not waste time beat around the bush. In iso 6400 test images, Pana G3 and OMD used the same lens (oly 50mm prime), both shot at F6.3, G3 1/2500, OMD 1/1600. that is a very significant 2/3 stop more light for OMD.

I know DPR insist that they cannot ensure lighting is the same for every shot. but I would guess the difference is in any case minimal. and the variation of exposure in DPR's images coincide with Imaging Resource too.

texinwien Veteran Member • Posts: 3,326
Re: No Sarcasm

Jack Hogan wrote:

texinwien wrote:I do understand that, Jack. I think the difficult thing for the DPReview guys is that this topic has been hashed and rehashed, ad nauseum (and I mean that literally ). The DPReview guys have answered the misconceptions in great detail - some misconceptions I've seen you posting in this thread for instance.

They have made their cases to skeptical digital imaging experts (people who are engineers in the field), generally, after much toil and trouble, to those experts' satisfaction, as far as I've seen.

But every few months, the same non-conspiracy boils to the top again, this time with new actors who are acting in good faith and making arguments that seem logical and common sensical, but which are either based on some degree of ignorance regarding the topic, some degree of ignorance about DPReview's testing regimes, or a combination of those things.

The problem is that they've been here before, but the new crops of people who come in and make these arguments every few months don't bother to go back and familiarize themselves with all of the details before coming up with their brilliant arguments that they're simply 100% certain are correct.

So I understand the DPReview team's annoyance at having to deal with this same topic, over and over again, with people who haven't done the hard work to learn the details about the topic and/or the details about how DPReview runs its tests.

tex

Fair enough, tex. Now me being someone who likes to learn, would you mind setting me straight on the misconceptions I may have helped to perpetrate?

Jack

Jack, here are four posts I have written in the last couple of days on the subject, each of which quotes and links to a number of outside sources  - previous statements by Andy Westlake and Richard Butler, articles at DPReview, Articles at DxOMark, an article at Imatest, the Wiki article on Film Speed. Basically, a number of the important sources that one should really read if he wants to get a good understanding of the topics - both the question of ISO, the question of DPReview testing regimes, and the question of how ISO Settings (DxO Manufacturer ISO) relate to ISO Sensitivity (DxO Measured ISO).

http://forums.dpreview.com/forums/post/51036625

http://forums.dpreview.com/forums/post/51037198

http://forums.dpreview.com/forums/post/51042816

http://forums.dpreview.com/forums/post/51042893

It's all there, I believe. You may want to skip my pontifications and go straight to the sources I link to and quote.

Good luck!

tex

 texinwien's gear list:texinwien's gear list
Panasonic Lumix DMC-GM5 Olympus E-M5 II Olympus 12-40mm F2.8 OnePlus One Canon EOS 300D +20 more
Jack Hogan Veteran Member • Posts: 6,982
Re: why not?
1

tko wrote:

So, definitely doable to adjust the light and keep a level playing field.

OK.  How would you work ISO into that equation if you wanted to compare apples to apples?

barkingghost Regular Member • Posts: 190
This surprises you?

Many manufacturers do this on many of their products. Have you tried measure the contrast ratio of a flat panel and compared it to what the maker advertised? How about a projector? If you buy into the marketing, then that's your fault.

I'd much rather see what others have found, than believe the blind what a marketing nut pushed onto me.

robert1955 Veteran Member • Posts: 4,220
Re: No Sarcasm
1

texinwien wrote:


They have made their cases to skeptical digital imaging experts (people who are engineers in the field), generally, after much toil and trouble, to those experts' satisfaction, as far as I've seen.

But every few months, the same non-conspiracy boils to the top again, this time with new actors who are acting in good faith and making arguments that seem logical and common sensical, but which are either based on some degree of ignorance regarding the topic, some degree of ignorance about DPReview's testing regimes, or a combination of those things.

The problem is that they've been here before, but the new crops of people who come in and make these arguments every few months don't bother to go back and familiarize themselves with all of the details before coming up with their brilliant arguments that they're simply 100% certain are correct.

tex

Sadly, that sums up much of the web. Well put.

 robert1955's gear list:robert1955's gear list
Fujifilm X-E3 Fujifilm XF 16mm F1.4 R WR Fujifilm 50mm F2 R WR Samyang 12mm F2 NCS CS XF 90mm
texinwien Veteran Member • Posts: 3,326
Re: DxOMark's measured ISOs vs. manufacturer ISOs

Jack Hogan wrote:

Andy Westlake wrote:

cptobvious wrote:

It's been stated numerous times in this forum that certain manufacturers overstate their ISOs in order to make their cameras seem to have better noise performance than they actually do.

True

It may have been stated numerous times, but it's been equally wrong each time, too.

Bertrand Russel would disapprove How wouldyou even know?

The OP provides evidence that the EM5 is an outlier - strangely enough massively in its favor, resulting in some interesting false-positive publicity, per his later post. This is one of my favorites, where the EM5 obviously performs as good as physical limits allow for its format. Does a neutrino travel faster than the speed of light? The Italians think so

While we are at it, who said 'You can fool all the people all of the time if the advertising is right and the budget is big enough'?

Right.

Jack - you're totally, 100% off base here. You seem like a smart guy, so I really hope you'll go through the reference material with care.

It is a complex subject, but I am certain you are intelligent enough to understand it.

I would caution you, however - keep an open mind and don't go into this journey convinced that you're right, because it will take you a lot longer to reach the correct conclusion, which is that you are 100% wrong.

I know, because I made the same journey, and did it without keeping an open mind. My journey took longer than it should have because I was convinced I'd figured it all out. I hadn't.

And neither have you - yet...

tex

 texinwien's gear list:texinwien's gear list
Panasonic Lumix DMC-GM5 Olympus E-M5 II Olympus 12-40mm F2.8 OnePlus One Canon EOS 300D +20 more
olliess Senior Member • Posts: 1,349
Re: DxOMark's measured ISOs vs. manufacturer ISOs

texinwien wrote:

Jack - you're totally, 100% off base here. You seem like a smart guy, so I really hope you'll go through the reference material with care.

[...]

I know, because I made the same journey, and did it without keeping an open mind. My journey took longer than it should have because I was convinced I'd figured it all out. I hadn't.

And neither have you - yet...

tex

So are you saying this to the Raw as one of the Converted?

Jack Hogan Veteran Member • Posts: 6,982
Re: DxOMark's measured ISOs vs. manufacturer ISOs
3

texinwien wrote:

Jack - you're totally, 100% off base here.

...My journey took longer than it should have because I was convinced I'd figured it all out. I hadn't.

And neither have you - yet...

I'll read the material and leave notes in your threads. But there are two issues here (and you are introducing a third):

1) The fact that the ISO definition of S is too messy and lax and therefore it makes it difficult to compare apples to apples
2) The possibility that this laxity could be used to someone's advantage with naive customers - the subject of this thread and what you are responding to in this post
3) That DPR's methods are flawed (which brought you in Andy's defence)

1) I think we all agree with
2) I think we'll never know, one way or the other - if I were Olympus I sure wouldn't be telling anyone at DPR - hence it's pretty dumb to come down hard one way or the other
3) I never considered it a target in any of my posts until you jumped in to defend it - now I'll have to form an opinion on it

As far as I am concerned manufacturers can literally call in-camera ISO whatever they want while producing images of whatever brightness they want OOC - the standard allows them to to it. So when someone deviates from accepted norm, which incidentally results in them looking better than the competition, they must be prepared for some added scrutiny. Which may be misguided, but fun anyway. We'll never know.  But to compare apples to apples it behooves reviewers and readers alike not to use such ISO nonsense: thank god for DxO.

So are you 100% sure that this thread is 100% off base?

OP cptobvious Contributing Member • Posts: 810
Re: DxOMark's measured ISOs vs. manufacturer ISOs
2

It's hilarious how much deflecting in going on this thread. People can argue until they are blue in the face how Olympus (or Fuji) has a right to calculate their ISOs as they do because the ISO standard permits it. That is really not the point of the thread. The point of this thread as I started it was to point out that their measurements are outside the norm, and conveniently these measurements give a false impression of the E-M5's equal noise performance to Exmor APS-C/APS sensors when all exposure variables are not controlled.

In February 2011, DxOMark wrote their article explaining why measured ISOs are lower than manufacturer ISOs, because manufacturers base their ISO measurements on JPEG output and DxOMark bases their ISO measurements on RAW files: http://www.dxomark.com/index.php/Publications/DxOMark-Insights/Pushed-ISO-Let-s-make-it-clear/RAW-ISO-measures-are-inferior-to-manufacturer-ISOs-is-this-a-problem

However, this article was written before the X100 and E-M5 were tested. When this article was written, was there ever an camera released where the discrepancy between the two measurements was so significant? I have yet to find one. Maybe texinwien can tell us.

Keyboard shortcuts:
FForum MMy threads