Which DXOmark Metrics Do We Believe?

[TheGrammarFairy]

I get the sense that members of this forum have strong opinions pro and con about by the measurements published by DXOMark.

But I was wondering if is there a consensus on which of their measurements are well regarded and useful.

Here are some of the most prominent: SNR 18%, Dynamic Range, Tonal Range, Color Sensitivity, Full SNR, Color Response, Full CS, Portrait (Color Depth), Landscale (Dynamic Range), and Sports (Low-Light ISO).

Thanks

 

[bclaff]

TheGrammarFairy wrote:
Sports (Low-Light ISO).

This one is a total joke.

It will easily rate a camera with lots of ugly read noise as superior to one with the same sensor size and low, random read noise, with just slightly less QE. It weighs far too heavily for high-ISOs with photon noise, and photon noise is usually the least of your problems at high ISOs, in the most problematic, darker tones.

One problem with the DxOMark Sport Score is that

The DxOMark Sports Score is an ISO value determined by the minimum of three criteria:
1) A Color Score of 18bits
2) A DR of 9EV
3) An SNR value of 30dB

As a combination of three things it's rather opaque.

It is weird enough that it comes down to just one of three, but my point remains that all of these are heavily correlated to photon noise, determined by sensor size and QE. None of the three have anything to do with problem noise in the darker tones of high ISOs.

FWIW, I think #2 A DR of 9EV does

Yes, photon noise increases by 41% for each halving of exposure, but (pre-gain) read noise increases by 100% (except at the cross-over point of dual-conversion-gain sensors).

Except for dual conversion gain, as you point out, pre-gain read noise is unaffected by the ISO setting so I don't understand where that 100% comes from.

It comes from the fact that I didn't say "input referred read noise" or "absolute read noise", but used the more common concept of noise, which is relative to signal, or the inverse of SNR.

I still don't understand how read noise is affected by halving the exposure.

 

[Erik Kaffehr]

TheGrammarFairy wrote:
Sports (Low-Light ISO).

This one is a total joke.

It will easily rate a camera with lots of ugly read noise as superior to one with the same sensor size and low, random read noise, with just slightly less QE. It weighs far too heavily for high-ISOs with photon noise, and photon noise is usually the least of your problems at high ISOs, in the most problematic, darker tones.

One problem with the DxOMark Sport Score is that

The DxOMark Sports Score is an ISO value determined by the minimum of three criteria:
1) A Color Score of 18bits
2) A DR of 9EV
3) An SNR value of 30dB

As a combination of three things it's rather opaque.

It is weird enough that it comes down to just one of three, but my point remains that all of these are heavily correlated to photon noise, determined by sensor size and QE. None of the three have anything to do with problem noise in the darker tones of high ISOs.

DR is ratio of used well capacity to dark noise expressed in EV. With dark noise I mean noise you have with lens cap on lens.

Yes, photon noise increases by 41% for each halving of exposure, but (pre-gain) read noise increases by 100% (except at the cross-over point of dual-conversion-gain sensors).

Except for dual conversion gain, as you point out, pre-gain read noise is unaffected by the ISO setting so I don't understand where that 100% comes from.

It comes from the fact that I didn't say "input referred read noise" or "absolute read noise", but used the more common concept of noise, which is relative to signal, or the inverse of SNR.

Best regards

Erik

 

[57even]

... Landscape (Dynamic Range), and Sports (Low-Light ISO).

...

I don't know quite how they measure DR. I think the read noise estimates at P2P are more interesting but I don't know that DR or PDR are all that useful on their own. Really need to look at the whole photon transfer curve to see what's really going on, but read noise is pretty low these days and largely irrelevant. Other problems are more likely to mess up deep shadows.

...

FWIW, DxOMark Landscape Dynamic Range is entirely based on normalizing read noise as if the sensor were 8MP.

PhotonsToPhotos Photographic Dynamic Range (PDR) does consult the Photon Transfer Curve (PTC).
Here's an illustration :



The black dot marks where PDR for this camera is located on the PTC.
Note that the PTC axes are logarithmic and this point is at a Signal to Noise Ratio (SNR) of about 2 at the pixel level. PDR is 9.81

DxOMark didn't test this camera but they would do the following. Read noise is 2.602.
They would decrease that by sqrt(24MP / 8PM) = sqrt(3) so 2.602 / sqrt(3) = 1.190 would be the normalized read noise and DxOMark Landscape Dynamic Range would be 14-1.190 = 12.81

I am aware how it is calculated, but if you use P2P read noise numbers and calculate EDR for SNR = 1 on the PTC, the 'screen' values for DxO always seem a bit low. In fact, it never seems to exceed the bit-depth.

You allude to other factors, among them is pattern noise which PhotonsToPhotos presents in the Sensor Heatmaps; for example :


Must be viewed riginal size"

The colors are not particularly important, they serve to make patterns appear more prominent. The random patterns shown above indicate well controlled pattern noise.

This is part of the problem, along with flare and other issues, but the real problem is that no single number tells me much about what noise level I will see at a given level in the image because it doesn't account for gamma and tone curves.

I can't really think of a simple way to account for this, other than finding a default tone curve for some common raw converter and applying the transformation to the PTC.

--
"A designer knows he has achieved perfection not when there is nothing left to add, but when there is nothing left to take away." Antoine de Saint-Exupery

 

[John Sheehy]

One problem with the DxOMark Sport Score is that

The DxOMark Sports Score is an ISO value determined by the minimum of three criteria:
1) A Color Score of 18bits
2) A DR of 9EV
3) An SNR value of 30dB

As a combination of three things it's rather opaque.

It is weird enough that it comes down to just one of three, but my point remains that all of these are heavily correlated to photon noise, determined by sensor size and QE. None of the three have anything to do with problem noise in the darker tones of high ISOs.

DR is ratio of used well capacity to dark noise expressed in EV. With dark noise I mean noise you have with lens cap on lens.

I'm not sure why you wrote that, but a DR of 9EV can happen with quite a bit of read noise, so it is therefore a mainly photon noise criterion.

 

[bclaff]

... Landscape (Dynamic Range), and Sports (Low-Light ISO).

...

I don't know quite how they measure DR. I think the read noise estimates at P2P are more interesting but I don't know that DR or PDR are all that useful on their own. Really need to look at the whole photon transfer curve to see what's really going on, but read noise is pretty low these days and largely irrelevant. Other problems are more likely to mess up deep shadows.

...

FWIW, DxOMark Landscape Dynamic Range is entirely based on normalizing read noise as if the sensor were 8MP.

PhotonsToPhotos Photographic Dynamic Range (PDR) does consult the Photon Transfer Curve (PTC).
Here's an illustration :



The black dot marks where PDR for this camera is located on the PTC.
Note that the PTC axes are logarithmic and this point is at a Signal to Noise Ratio (SNR) of about 2 at the pixel level. PDR is 9.81

DxOMark didn't test this camera but they would do the following. Read noise is 2.602.
They would decrease that by sqrt(24MP / 8PM) = sqrt(3) so 2.602 / sqrt(3) = 1.190 would be the normalized read noise and DxOMark Landscape Dynamic Range would be 14-1.190 = 12.81

I am aware how it is calculated, but if you use P2P read noise numbers and calculate EDR for SNR = 1 on the PTC, the 'screen' values for DxO always seem a bit low. In fact, it never seems to exceed the bit-depth.

I really can't follow what you're trying to say.

You can't calculate EDR using SNR=1, EDR is SNR=0
DxOMark doesn't publish read noise values but if you curve fit their SNR data you can compute read noise. Read noise for the D7200, for example, is below 1; so that exceed bit-depth.
Again ,using the D7200 as an example the DxOMark curve fit value at ISO 100 is .9029DN which is just a tad higher than the PhotonsToPhotos value of .9010DN
(I would say they match).

--
Bill ( Your trusted source for independent sensor data at PhotonsToPhotos )

 

[bclaff]

One problem with the DxOMark Sport Score is that

The DxOMark Sports Score is an ISO value determined by the minimum of three criteria:
1) A Color Score of 18bits
2) A DR of 9EV
3) An SNR value of 30dB

As a combination of three things it's rather opaque.

It is weird enough that it comes down to just one of three, but my point remains that all of these are heavily correlated to photon noise, determined by sensor size and QE. None of the three have anything to do with problem noise in the darker tones of high ISOs.

DR is ratio of used well capacity to dark noise expressed in EV. With dark noise I mean noise you have with lens cap on lens.

I'm not sure why you wrote that, but a DR of 9EV can happen with quite a bit of read noise, so it is therefore a mainly photon noise criterion.

I'm sure you know DR is a ratio and that DR isn't a function of just the top or the bottom.
Despite photon noise dominating the numerator it's read noise in the denominator that is the controlling factor.

 

[57even]

... Landscape (Dynamic Range), and Sports (Low-Light ISO).

...

I don't know quite how they measure DR. I think the read noise estimates at P2P are more interesting but I don't know that DR or PDR are all that useful on their own. Really need to look at the whole photon transfer curve to see what's really going on, but read noise is pretty low these days and largely irrelevant. Other problems are more likely to mess up deep shadows.

...

FWIW, DxOMark Landscape Dynamic Range is entirely based on normalizing read noise as if the sensor were 8MP.

PhotonsToPhotos Photographic Dynamic Range (PDR) does consult the Photon Transfer Curve (PTC).
Here's an illustration :



The black dot marks where PDR for this camera is located on the PTC.
Note that the PTC axes are logarithmic and this point is at a Signal to Noise Ratio (SNR) of about 2 at the pixel level. PDR is 9.81

DxOMark didn't test this camera but they would do the following. Read noise is 2.602.
They would decrease that by sqrt(24MP / 8PM) = sqrt(3) so 2.602 / sqrt(3) = 1.190 would be the normalized read noise and DxOMark Landscape Dynamic Range would be 14-1.190 = 12.81

I am aware how it is calculated, but if you use P2P read noise numbers and calculate EDR for SNR = 1 on the PTC, the 'screen' values for DxO always seem a bit low. In fact, it never seems to exceed the bit-depth.

I really can't follow what you're trying to say.

You can't calculate EDR using SNR=1, EDR is SNR=0

20Log SNR = 0. Ratio is 1

DxOMark doesn't publish read noise values but if you curve fit their SNR data you can compute read noise. Read noise for the D7200, for example, is below 1; so that exceed bit-depth.
Again ,using the D7200 as an example the DxOMark curve fit value at ISO 100 is .9029DN which is just a tad higher than the PhotonsToPhotos value of .9010DN
(I would say they match).

OK, seems fair, but I would expect at least some MFT sensors to have an EDR >12 at base ISO (screen) but very few do.

--
"A designer knows he has achieved perfection not when there is nothing left to add, but when there is nothing left to take away." Antoine de Saint-Exupery

 

[57even]

TheGrammarFairy wrote:
Sports (Low-Light ISO).

This one is a total joke.

It will easily rate a camera with lots of ugly read noise as superior to one with the same sensor size and low, random read noise, with just slightly less QE. It weighs far too heavily for high-ISOs with photon noise, and photon noise is usually the least of your problems at high ISOs, in the most problematic, darker tones.

One problem with the DxOMark Sport Score is that

The DxOMark Sports Score is an ISO value determined by the minimum of three criteria:
1) A Color Score of 18bits
2) A DR of 9EV
3) An SNR value of 30dB

As a combination of three things it's rather opaque.

It is weird enough that it comes down to just one of three, but my point remains that all of these are heavily correlated to photon noise, determined by sensor size and QE. None of the three have anything to do with problem noise in the darker tones of high ISOs.

FWIW, I think #2 A DR of 9EV does

Yes, photon noise increases by 41% for each halving of exposure, but (pre-gain) read noise increases by 100% (except at the cross-over point of dual-conversion-gain sensors).

Except for dual conversion gain, as you point out, pre-gain read noise is unaffected by the ISO setting so I don't understand where that 100% comes from.

It comes from the fact that I didn't say "input referred read noise" or "absolute read noise", but used the more common concept of noise, which is relative to signal, or the inverse of SNR.

I still don't understand how read noise is affected by halving the exposure.

It depends if we are talking about absolute noise in the captured signal, or perceived noise in an image at some relative brightness level (arguably more relevant to a photographer).

 

[bclaff]

TheGrammarFairy wrote:
Sports (Low-Light ISO).

This one is a total joke.

It will easily rate a camera with lots of ugly read noise as superior to one with the same sensor size and low, random read noise, with just slightly less QE. It weighs far too heavily for high-ISOs with photon noise, and photon noise is usually the least of your problems at high ISOs, in the most problematic, darker tones.

One problem with the DxOMark Sport Score is that

The DxOMark Sports Score is an ISO value determined by the minimum of three criteria:
1) A Color Score of 18bits
2) A DR of 9EV
3) An SNR value of 30dB

As a combination of three things it's rather opaque.

It is weird enough that it comes down to just one of three, but my point remains that all of these are heavily correlated to photon noise, determined by sensor size and QE. None of the three have anything to do with problem noise in the darker tones of high ISOs.

FWIW, I think #2 A DR of 9EV does

Yes, photon noise increases by 41% for each halving of exposure, but (pre-gain) read noise increases by 100% (except at the cross-over point of dual-conversion-gain sensors).

Except for dual conversion gain, as you point out, pre-gain read noise is unaffected by the ISO setting so I don't understand where that 100% comes from.

It comes from the fact that I didn't say "input referred read noise" or "absolute read noise", but used the more common concept of noise, which is relative to signal, or the inverse of SNR.

I still don't understand how read noise is affected by halving the exposure.

It depends if we are talking about absolute noise in the captured signal, or perceived noise in an image at some relative brightness level (arguably more relevant to a photographer).

Here's the problem: there's no such thing as read noise at some relative brightness; read noise is always without any signal.

 

[57even]

TheGrammarFairy wrote:
Sports (Low-Light ISO).

This one is a total joke.

It will easily rate a camera with lots of ugly read noise as superior to one with the same sensor size and low, random read noise, with just slightly less QE. It weighs far too heavily for high-ISOs with photon noise, and photon noise is usually the least of your problems at high ISOs, in the most problematic, darker tones.

One problem with the DxOMark Sport Score is that

The DxOMark Sports Score is an ISO value determined by the minimum of three criteria:
1) A Color Score of 18bits
2) A DR of 9EV
3) An SNR value of 30dB

As a combination of three things it's rather opaque.

It is weird enough that it comes down to just one of three, but my point remains that all of these are heavily correlated to photon noise, determined by sensor size and QE. None of the three have anything to do with problem noise in the darker tones of high ISOs.

FWIW, I think #2 A DR of 9EV does

Yes, photon noise increases by 41% for each halving of exposure, but (pre-gain) read noise increases by 100% (except at the cross-over point of dual-conversion-gain sensors).

Except for dual conversion gain, as you point out, pre-gain read noise is unaffected by the ISO setting so I don't understand where that 100% comes from.

It comes from the fact that I didn't say "input referred read noise" or "absolute read noise", but used the more common concept of noise, which is relative to signal, or the inverse of SNR.

I still don't understand how read noise is affected by halving the exposure.

It depends if we are talking about absolute noise in the captured signal, or perceived noise in an image at some relative brightness level (arguably more relevant to a photographer).

Here's the problem: there's no such thing as read noise at some relative brightness; read noise is always without any signal.

Read noise is always present in the image. It doesn't just disappear when there is a signal, it just makes a relatively smaller contribution to the total noise as the signal increases because other noise sources dominate.

But, from an image perspective, the contribution from read noise is still part of the overall noise profile, and any patterns in that noise are potentially discernible if the other noise sources are relatively low.

At high ISO, the relative contribution is considerably greater since it isn't falling as the root of the signal, so it ultimately limits maximum ISO just as it can ultimately limit useful shadow recovery in high contrast images (but not always).

 

[John Sheehy]

I still don't understand how read noise is affected by halving the exposure.

It depends if we are talking about absolute noise in the captured signal, or perceived noise in an image at some relative brightness level (arguably more relevant to a photographer).

... and the reasonably assumed default unless otherwise qualified. It is just as important to recognize context as it is to recognize numbers and equations.

 

[57even]

I still don't understand how read noise is affected by halving the exposure.

It depends if we are talking about absolute noise in the captured signal, or perceived noise in an image at some relative brightness level (arguably more relevant to a photographer).

... and the reasonably assumed default unless otherwise qualified. It is just as important to recognize context as it is to recognize numbers and equations.

Yes. This point often gets lost in technical discussions on photography. If you can't tell me how different it will look, you haven't told me anything.

 

[John Sheehy]

DR is ratio of used well capacity to dark noise expressed in EV. With dark noise I mean noise you have with lens cap on lens.

I'm not sure why you wrote that, but a DR of 9EV can happen with quite a bit of read noise, so it is therefore a mainly photon noise criterion.

I'm sure you know DR is a ratio and that DR isn't a function of just the top or the bottom.

I am sure, too, and have been for about 2 decades.

Despite photon noise dominating the numerator it's read noise in the denominator that is the controlling factor.

I asked Erik why he wrote that because it seemed to be addressing what I said, but I didn't think that it did. I did not disagree with what he wrote; I questioned the context of him replying it to me.

There is no photon noise in the numerator; just a RAW highlight clipping point. There is a tad of non-linearity at the clipping point from noise dithering, but so far, it is relatively insignificant, but may become significant in future high-density, 1-bit to 3-bit sensors.

 

[John Sheehy]

It depends if we are talking about absolute noise in the captured signal, or perceived noise in an image at some relative brightness level (arguably more relevant to a photographer).

Here's the problem: there's no such thing as read noise at some relative brightness; read noise is always without any signal.

So, it is additive rather than modulatory (like PRNU noise, or square-root-modulatory, like photon noise). That does not mean that it doesn't have an intensity relative to signal, when signal is present.

Relative to signal, halving exposure doubles pre-gain read noise (with the same conversion gain assumed), increases photon noise 1.414x, and does not affect PRNU noise. Changes in post-gain read noise are a doubling of that noise relative to signal, if the same analog gain is used, or it remains the same relative to signal if you double the gain with the halving of the signal.

 

[John Sheehy]

The way I see it, the data on your site is better. DxO-mark shows what is essentially the same data in different ways, which can be quite illustrative. But I feel that SNR, Tonal Range, DR are pretty much the same data, just viewed with different glasses.

SNR18 and DR are almost mutually exclusive. DxO DR is affected by pixel blackframe noise and the square root of the number of pixels, and not at all by photon noise. SNR18 chart trends basically map sensor size times QE (which directly affects photon noise), with a slight pull down at the low ISO end by PRNU noise and post-gain read noise, and the high ISO end by pre-gain read noise, but only a slight pull downward. Read noise, especially read noise with significant spatial correlation (and any modulo-2 patterns in the monochrome RAW, common in readout arrays, blow up into color offsets after demosaicing), can be highly visible at high ISOs, and in deep shadows at low ISOs, even when the SNR18 correlates closely with photon noise.

 

[TheGrammarFairy]

Excellent, thanks. But could you please elaborate on the two points below?

What is it telling us?

“Also, it doesn't tell you what is says it does.”

Why academic? (Assuming you mean no practical relevance.)

SMI is always interesting but somewhat academic

SNR 18% is fairly unambiguous, as is ISO calibration. Tonal range and colour sensitivity are just different integrals of the SNR/signal curve and mostly boring/predictable. Also, it doesn't tell you what is says it does.

SMI is always interesting but somewhat academic

 

[Iliah Borg]

I trust they are making an effort, and are often getting in the ballpark.

Can't speak for "we".

--
http://www.libraw.org/

 

[57even]

Excellent, thanks. But could you please elaborate on the two points below?
What is it telling us?

“Also, it doesn't tell you what is says it does.”

Why academic? (Assuming you mean no practical relevance.)

SMI is always interesting but somewhat academic

Cameras with lowish SMIs are often praised for 'great colour'. So what use is it?

SNR 18% is fairly unambiguous, as is ISO calibration. Tonal range and colour sensitivity are just different integrals of the SNR/signal curve and mostly boring/predictable. Also, it doesn't tell you what is says it does.

Tonal range is not a range. It's just the maximum number of tonal steps above noise. Same for colour sensitivity.

In the end, an 8-bit display has 256 tonal steps and about 16.8 million colour steps. If the camera has more than that, as most do at base ISO, the primary consequence is posterisation. If it has less, then the main consequence is visibility of low contrast detail close to the noise level.

A relationship to 'tonality' or 'gradation' is implied, but not true unless you use 10-bit media. With 8-bit media, noise dithering is necessary to avoid posterisation and reduce aliasing.

Both metrics are almost entirely predictable from SNR. Less noise = more distinguishable tones and colours. It's just that most of them can't be reproduced on a screen anyway.

 

[Christof21]

We all prefer your numbers

Not me, I prefer dxomark.

Photonstophtos can not be used to compare cameras because it does not measure ISO.

Which is a major problem. It gives for instance an unfair advantage to Fuji, I have seen many time fujirumors posting graph from this site to reveal the "superiority" of Fuji sensors thanks to this "trusted" source.

Dxomark is more rigourous, this is a fact.

 

[Christof21]

I get the sense that members of this forum have strong opinions pro and con about by the measurements published by DXOMark.

But I was wondering if is there a consensus on which of their measurements are well regarded and useful.

Here are some of the most prominent: SNR 18%, Dynamic Range, Tonal Range, Color Sensitivity, Full SNR, Color Response, Full CS, Portrait (Color Depth), Landscale (Dynamic Range), and Sports (Low-Light ISO).

Thanks

I think we should believe all measured data. To some end, it is the same data just wrangled different ways.

Measurement is a science in and of itself (Metrology). Theorists often make very bad metrologists because assumptions that are fine to make on paper are often not fine to make with real hardware. Or some "practical" accommodations have to be made to adapt a theory to a real system.

Since DxO made all their own tools, the onus is on them to show that their tools (measurement devices) are precise, accurate, and the calibration is maintained.

They've never done that and have been often wrong in flabbergasting ways, which is either a usage error or a problem with the tool. I will never trust any of their data until they release open validation and calibration information that satisfies my wont for rigor.

DxOMark documented their stuff pretty well early on but for some reason a lot of that has gone missing in recent years and can only be located using something like the Wayback Machine :-(

That said, I regurgitate most of the DxOMark data in the second section at PhotonsToPhotos

Admirable work indeed!

My impression over the years is that there are occasional data collection glitches and occasional processing errors; but the most puzzling results arise from their choice of algorithms, for example the "normalizing" of read noise as Landscape Dynamic Range.

I am not really with you here.

I use their data as a "sanity check" but naturally feel better about the data I collect myself. And, naturally, I think Photographic Dynamic Range (PDR) is better thought out than DxOMark Landscape DR.

In any case, this is becoming moot as DxOMark is testing fewer and fewer cameras.

The only think I miss is "Measured ISO" which is useful in computing Quantum Efficiency (QE) (actually effective QE but we rarely make that distinction).

I guess that I see your point.

The way I see it, the data on your site is better. DxO-mark shows what is essentially the same data in different ways, which can be quite illustrative. But I feel that SNR, Tonal Range, DR are pretty much the same data, just viewed with different glasses.

Having ISO calibrated is a good thing.

It is more than a nice to have, it is a must have imho..

How can you plot SNR graphs from diferent cameras with ISO not calibrated ?

That is why I can not draw any rigourous conclusion with bclaff plots, especially when comparing cameras.

Dxomark for the measures is quite informative and dpreview comparison tool (in raw) are my main references.
Ths is my opinion, absolutely nothing against bclaff, I would certainly change my mind if ISO was measured the same way with all cameras.