DxO score and measurements WT... ???

I'm having serious problems understanding DxO ratings and measurements, even though I read all about how they test and score etc.

Let's take the D7000 and the new D5200. We'll also see the Pentax K5-IIs.

Now DxO tells us the D5200 sensor is the king of APSC sensors, and when you look at the overall score, you can see that. The new sensor scores 84 while the old one scores 80. Pentax is at 82.



Looking at the ratings for sports, portrait and landscape you can see the new sensor scores better for low light and color depth, and equal for DR.

But looking at the measurements for signal-to-noise, we see this:



The new sensor is significantly worse at some ISO's and just worse at some. The Pentax K5-IIs is rated much better at high ISO's, and yet it scores lower for low-light.

Let's look at Tonal Range.



Yep, the new sensor is worse here too than the old one, especially at low ISOs, and the Pentax K5 IIs is again significantly better.



While Nikon sensors scores equal in DR, there is a significant difference. One is better at low ISOs one is better at high ISOs. So what the overall score tell us ?

And here we have another problem. How can the D5200 score the same for DR compared with the D7000 when we all saw banding and strong green color shift when pushing shadows ? From my understanding DxO test the best DR that can be achieved by pushing and pulling in PP, I'm sure no sensor has 14 EV DR out of the box as you open the raw file.

The K5-IIs again looks the best, at lower ISOs, while equal at higher ISOs with the new sensors.

And then we have color sensitivity, where the new sensor again scores better.



Well, at low ISO the old sensor seems to be better again, while as ISO increase, the new sensors is better, but only very very slightly better. Looking at the K5-IIs, it's far superior at anything above ISO 1600.

Any comments on how to interpret the scores and measurements above ?

P.S. Since most people don't print I selected the screen option in DxO applet.

A few pointers



You did nearly confuse me. All DXO mark scores are based on an 12 by 8 print. The Screen scores you used were added after consumer pressure but really mean nothing. DXO mark scores rightly normalise for output dimensions



Secondly the DXO graphs IMHO are great. But the scores are junk. Just a way of over simplifying.



Thirdly looking at the graphs for prints these sensors are so close that you couldn't tell them apart in the real world. That's another DXO mark weakness. They use way too much precision in the their measurements. I don't believe that they can really get a reproducible iso value of 4 significant figures

The SNR graph you captured is for the native pixel SNR, which doesn't account for the resolution difference between the sensors. The print tab normalizes for sensor surface area, which then shows an SNR delta within DxO's margin of error.

As for DR, DxO's measurements are quantitative only and don't account for qualitative issues like banding or color shifts. When those are accounted for the D7100 has a lower usable base ISO DR, depending on what your workflow is and how well it resolves the banding and color shift issues.

Very good explanations, thanks. So the print tab is not actually a print, but just a dowscale of all the cameras to 8MP ? And the screen tab is practically individual pixel quality ? It makes sense.

Still don't understand how they come up with the overall low-light, portrait, landscape scores, I'll take a look again.

d3xmeister said:

d3xmeister wrote:

Very good explanations, thanks. So the print tab is not actually a print, but just a dowscale of all the cameras to 8MP ? And the screen tab is practically individual pixel quality ? It makes sense.

Still don't understand how they come up with the overall low-light

Click to expand...

I think low light is the iso for a particular noise level (in a print)

d3xmeister said:

, portrait,, landscape scores, I'll take a look again.

Click to expand...

I think these are judged only at base iso

d3xmeister said:

d3xmeister wrote:

Very good explanations, thanks. So the print tab is not actually a print, but just a dowscale of all the cameras to 8MP ? And the screen tab is practically individual pixel quality ? It makes sense.

Click to expand...

Print refers to how they would compare if you print all images at the same final size (at 8MP). Screen means that's what you get, relatively, at 100% viewing for all cameras.

d3xmeister said:

Still don't understand how they come up with the overall low-light, portrait, landscape scores, I'll take a look again.

Click to expand...

Read their documents, they explain how they do them. The scores come from cutpoints. For example, high ISO is the ISO cutpoint for certain SNR and DR thresholds. If a camera rates at 1,200 and another at 2,400, it means the latter will show similar SNR/DR at 2,400ISO to the first at 1,200ISO, basically.

http://www.dxomark.com/index.php/About/What-is-DxOMark2

The pattern noise we see in shadows is something that their measurements won't detect; SNR measures FWC, basically, an individual photosite spec (then made global).

--
Renato.
http://www.flickr.com/photos/rhlpedrosa/
OnExposure member
http://www.onexposure.net/
Good shooting and good luck
(after Ed Murrow)

According to Emil Martinec, no:

Bottom line: High exposure zones and/or high ISO, where photon noise and pre-amplification read noise dominate the noise, are rather insensitive to what ISO is chosen once a choice of exposure is selected and care is taken not to clip highlights. Underexposing by a stop, and doubling the raw values in post-processing (that is, applying exposure compensation), yields the same image quality as 'proper' exposure under these conditions. On the other hand, in lower exposure zones at low ISO, where post-amplification read noise becomes important, the read noise goes down by a bit less than a factor of two (in electrons) when the ISO doubles. In this situation, underexposing by a stop and doubling the raw values in post-processing, yields more noise than proper exposure, particularly in shadows.

I also recall Iliah Borg indicating that every DSLR has an optimum ISO for the cleanest shadows, which would be higher than base ISO.

rhlpetrus said:

http://www.dxomark.com/index.php/About/What-is-DxOMark2

The pattern noise we see in shadows is something that their measurements won't detect; SNR measures FWC, basically, an individual photosite spec (then made global).

Click to expand...

Not sure what FWC stands for. But by definition you can't measure the SNR of a single photosite

john Clinch said:

john Clinch wrote:

john Clinch said:

http://www.dxomark.com/index.php/About/What-is-DxOMark2

The pattern noise we see in shadows is something that their measurements won't detect; SNR measures FWC, basically, an individual photosite spec (then made global).

Click to expand...

Not sure what FWC stands for. But by definition you can't measure the SNR of a single photosite

Click to expand...

Full well capacity.

JimPearce said:

JimPearce wrote:

According to Emil Martinec, no:

Bottom line: High exposure zones and/or high ISO, where photon noise and pre-amplification read noise dominate the noise, are rather insensitive to what ISO is chosen once a choice of exposure is selected and care is taken not to clip highlights. Underexposing by a stop, and doubling the raw values in post-processing (that is, applying exposure compensation), yields the same image quality as 'proper' exposure under these conditions. On the other hand, in lower exposure zones at low ISO, where post-amplification read noise becomes important, the read noise goes down by a bit less than a factor of two (in electrons) when the ISO doubles. In this situation, underexposing by a stop and doubling the raw values in post-processing, yields more noise than proper exposure, particularly in shadows.

I also recall Iliah Borg indicating that every DSLR has an optimum ISO for the cleanest shadows, which would be higher than base ISO.

Click to expand...

I thought that base ISO was defined as the value for which DR is the highest possible, or was it SNR?

"Sports & action photography: Low-Light ISO. An SNR value of 30dB means excellent image quality. Thus low-light ISO is the highest ISO setting for a camera that allows it to achieve an SNR of 30dB while keeping a good dynamic range of 9 EVs and a color depth of 18bits."

You will see that the values for sports are equal to iso level where the line passes the 30 db.



lock

E.M. is not quibbling with the definition of base ISO here, nor is he disagreeing with the notion that very large jumps - say 5 stops or more - are best shot at base ISO. What he is saying is that for moderate "effective" ISOs there will be less noise in the shadows if you shoot at the "correct" ISO.

JimPearce said:

I also recall Iliah Borg indicating that every DSLR has an optimum ISO for the cleanest shadows, which would be higher than base ISO.

Click to expand...

Some cameras have the cleanest shadows at base ISO, some not. And for some cameras there is no point to go higher than base ISO. One needs to test each of his pairs, camera + converter, because different converters behave differently, and that determines the final outcome.

Could it also depend on the amount of the push, as E.M. indicates Iliah?

Iliah Borg said:

Iliah Borg wrote:

Iliah Borg said:

I also recall Iliah Borg indicating that every DSLR has an optimum ISO for the cleanest shadows, which would be higher than base ISO.

Click to expand...

Some cameras have the cleanest shadows at base ISO, some not. And for some cameras there is no point to go higher than base ISO. One needs to test each of his pairs, camera + converter, because different converters behave differently, and that determines the final outcome.

--
http://www.libraw.org/
http://www.rawdigger.com/

Click to expand...

This is interesting from an academic perspective, but, IMO, less so from the practical one. Every time I press the shutter release I make a choice between the depth of field, shutter speed, and ISO. I don't use a calculator and don't go by DxO numbers, I base it exclusively on my personal experience with a particular camera in a particular environment, and dial in ballpark numbers which should get me an acceptable result. Do I compromise more on the noise, or risk motion blur, or get more or less DOF than would be desirable, -- these are often momentary judgements which could be on or could be off, occasionally by some margin. Luckily, digital gives you an instant feedback, so you can try to make finer adjustments on the spot. But the bottom line -- I wouldn't split hairs over DxO scores, elementary lapses of judgement would more often than not matter a lot more than slight differences in DxO scores. If one shoots under highly controlled and repeatable conditions and carefully sets up every shot -- this might be different, -- but then the sensor and the camera as a whole are not likely to be pushed to their limits in such an environment.

But most of us don't really make decisions on every shot, nor do we have that luxury in many cases. We acquire effective habits. As a wildlife shooter, generally shooting at moderate ISOs from 200 to 800, I think I can see that certain ISOs offer the most pleasing results. The D300, for instance seems to really do well at ISO 320; the D7100 at ISO 250, at least with Capture NX2. A perceptive photographer will build habits around this, possibly without even explicitly noting it.

I suspect that even DXO engineers would say that you are over-analyzing this. Somewhere on their web site they state that sensors giver or take five points are photographically equal for most purposes.

That said, my in-depth experiences with a half dozen cameras (i always shoot raw, so DXO's values are most relevant to me) largely comports with the strengths and weaknesses that DxO measures.

But, really, aside from resolution differences, i think that you'll find these sensors comparable . . . and i print large.

-- gary ray

Semi-professional in early 1970s; just a putzer since then. interests: historical sites, virginia, motorcycle racing. A nikon user more by habit than choice; still, nikon seems to work well for me.

d3xmeister said:

d3xmeister wrote:

Very good explanations, thanks. So the print tab is not actually a print, but just a dowscale of all the cameras to 8MP ? And the screen tab is practically individual pixel quality ? It makes sense.

Still don't understand how they come up with the overall low-light, portrait, landscape scores, I'll take a look again.

Click to expand...

They have an arbitrary cut off. Even though the pentax scores better at highiso, what they are saying is that below x level the cameras are both so bad neither is usable. I think the level of the cutoff are fine and tend to agree. So just draw a horizontal line on that graph around 1100.

You have taught yourself the real usefullness of dxo. Theyhave the graph. People hate on them, but you are welcome to draw your own cutoff line. So if you shot above 1600 the pentax looks better. But again both images are potentially very noisy at that point, so they ignore the portion past 1100.

For me dx 800-1600 and fx 3200 match what I see out of my cameras and whatis the highest I prefer to shoot.

But hey people love to say their 6d looks better than the d600 at 25600. Me ill take the camera that performs better in the ranges I actually use. I absolutely am annoyed how much people get a kick crippling their high performance cameras wrt noise and dynamic range down to the level of a quite mediocre camera.

Also print is the way to go imo. the screen doesn't tell you much because while there is more noise in d800 than a d600, viewed at the same size or down sampled the noise is a wash as dxo shows, matching my experience.

yray said:

Every time I press the shutter release I make a choice between the depth of field, shutter speed, and ISO.

Click to expand...

Many do the same. Does not mean there are no better ways.

> Luckily, digital gives you an instant feedback

No it does not. And the price of the distorted and misleading "feedback" it provides is taking the eye from the viewfinder.

--
http://www.libraw.org/
http://www.rawdigger.com/