Photographic Dynamic Range (PDR) is a More Useful Measure than the DxOMark Landscape Score

Started Apr 17, 2015 | Discussions
OP bclaff Veteran Member • Posts: 8,737
Re: Engineering DR still a needed data point

Horshack wrote:

IMO engineering DR is still very useful because it tells the reader what the deepest shadows will look like irrespective of the SNR further up the tonal scale (what PDR provides). For example, here is a D5000 vs D3s shadow push comparison I did many years ago - although the D3s outperforms the D5000 significantly in terms of QE and shot noise, the D5000 still produces better deep shadows at base ISO, even after discounting the D3s's banding:

Full link: http://www.dpreview.com/forums/post/36104230

I think comparisons of "identical" images in this fashion is very useful.

But let's not forget that a tone curve has been applied, dynamic range has been compressed in some areas and expanding in others.
There is no easy way to relate what is seen in these images back to the linear raw data.

In particular, I'm quite sure you're not seeing anything relevant in any areas that are entirely read noise.

Regarding the D3S and D5000, DxOMark shows a 0.55 stop advantage to the D5000 at ISO 100
So I'm unsure what point you were making with those images.

Regards,

-- hide signature --
Detail Man
Detail Man Forum Pro • Posts: 17,078
Re: DxO Print/Landscape DR vs Photographic Dynamic Range

bclaff wrote:

Jack Hogan wrote:

So IMHO they both have their place: DxO tells you clearly which is the better sensor, which deepest shadows you can push more; PDR tells you how much of a difference it may make in (its view of) typical practice.

Great overview (which I snipped out).

Semantics perhaps ...
I disagree that DxO "tell you clearly which is the better sensor".
Perhaps I'd agree that it tells you which has the better pixel architecture.

I mean we have to say "better sensor" for what purpose.

Horshack wrote:

IMO engineering DR is still very useful because it tells the reader what the deepest shadows will look like irrespective of the SNR further up the tonal scale (what PDR provides).

If it's to take photographs I think DxO is less relevant than PDR.

As you might yourself ask, "relative to what particular concerns surrounding photograhic images ?".

OP bclaff Veteran Member • Posts: 8,737
Re: DxO Print/Landscape DR vs Photographic Dynamic Range

Detail Man wrote:

As you might yourself ask, "relative to what particular concerns surrounding photograhic images ?".

"Devil's advocate" eh?

Remember, DxOMark calls this a Use Case and it is named the Landscape Score.

Is anyone really arguing that read noise in the absence of photon noise is relevant to landscape photography?

Regards,

-- hide signature --
Detail Man
Detail Man Forum Pro • Posts: 17,078
Re: DxO Print/Landscape DR vs Photographic Dynamic Range

bclaff wrote:

Detail Man wrote:

As you might yourself ask, "relative to what particular concerns surrounding photograhic images ?".

"Devil's advocate" eh?

Remember, DxOMark calls this a Use Case and it is named the Landscape Score.

Is anyone really arguing that read noise in the absence of photon noise is relevant to landscape photography?

It is hard for me to conclude that invariably, in all cases, such would not ever materially matter.

Detail Man
Detail Man Forum Pro • Posts: 17,078
Re: Photographic Dynamic Range (PDR) is a More Useful Measure than the DxOMark Landscape Score
2

bclaff wrote:

Horshack wrote:

Detail Man wrote:

bclaff wrote:

... The PDR criteria operates in the PTC region well above SNR=1 (at least until we get 384Mp sensors!).

In this region the subtle interplay of read noise, other noise, small amounts of photon noise, slope due to Full Well Capacity (FWC), etc. is captured.

It seems that (especially thses days, with higher performance image-sensors), Readout/Dark noise component are effectively "numerically swamped" by Photon Shot Noise. Rather "subtle", indeed.

But there's a distinct qualitative difference of read noise vs shot noise. John Sheehy once posted a simulation of an ISO 1,000,000 image with zero read noise and it looked more natural than a much lower ISO image with read noise.

In Detail Man's comment I think "swamped" is too strong a term.

My use of "swamped" refers to the existence of a relatively low numerical output sensitivity-factor existing as a function of Read/Dark noise components.

In this area of the PTC we are still very much in the read noise dominated region.

At (linear) SNR = 20 ?

If an image or an area of an image is entirely read noise then it has no detail and is essentially black.

How so where it comes to periodic noise components ? Random components are "invisible" ?

To have a photograph you must have photons and therefore photon noise.

To generate RAW image-files (with whatever internal system "warts" exist) is "painting with light" (or, more specifically, "painting with some amount of luminance"), aesthetics aside ...

Characterizing the photographic use of a sensor in a region of the PTC that is devoid of photon noise is, as I have stated more subtly above, brain-dead.

The phrase "brain dead" seems a bit dissmissive (of phenomena that can potentially "matter").

Here's a hypothetical.
Two sensors that are the same in every way except that one has a higher FWC.
(You can even imagine them both with zero read noise, DSNU, etc.)
Photographically these sensors are not identical.
You will get more useful dynamic range out of the one with the higher FWC.

The (minimum low-level analyzed) Photon Shot Noise components (relative to Read/Dark noise components) will scale in magnitude with the square-root of increases in "Full Well Capacity".

.

Prior to DxO Labs releasing test results for the Olympus E-M5, I did some comparisons analyzing very deep shadow regions of reliable (sensor-level Exposure matched, nearly identical scene) RAW test shots from Panasonic DMC-GH2 and E-M5 at their base ISO settings. The GH2 clips blacks, the E-M5 does not. Differences in the shapes of the noise distributions could be seen at <= SNR=12 dB - but appear to have had minor numerical impact on the interpolated DxOMark SNR data from which the DxOMark DR spec is derived.

The Sensorgen.info (DxOMark data derived) base-ISO Read Noises listed are comparable:

http://www.sensorgen.info/PanasonicLumix-DMC-GH2.html

http://www.sensorgen.info/OlympusOM-D-E-M5.html

Estimates were made visually. In all groups of 2 images below, the order is GH2 followed by E-M5. Differences seen (as a function of SNRs 12 dB and 18 dB) were not large. As it turnd out, the E-M5 DR surpasses the GH2's DR almost entirely as a result of it's (nearly doubled) FWC compared to GH2.

.

.

SNR ~ 12 dB (where Read/Dark Noise appears to dominate over Photon Shot Noise). The Standard Deviation of the E-M5 is approximately 0.5 EV higher than that of the GH2:

The location of the Mean Value is shown as a vertical gray line in the histograms below:

.

.

(3 pixel-width) arithmetically averaged view; corresponding histograms allowing easier assessment:

.

.

SNR ~ 18 dB (where Photon Shot Noise appears to dominate over Read/Dark Noise). The Standard Deviation of the E-M5 is only then approximately 0.5 EV lower than the GH2:

The location of the Mean Value is shown as a vertical gray line in the histograms below:

.

.

(3 pixel-width) arithmetically averaged view; corresponding histograms allowing easier assessment:

.

It seems that with differing FWCs in such a case as above (the E-M5 image-sensor FWC is around twice that of the GH2 image-sensor) - and where the (Sensorgen derived from DxOMark data) per-photosite Read Noise is very close to being numerically identical (and the pixel-pitches of the individual image-sensors are similar in value) - the "efficacy" of simply having a higher FWC is (remains) less influential - at low (enough) illumination levels.

People attempting to "push" a dim image in processing "fly by their minds' eyes". If shadows deep enough to matter within those persons' in-process viewing exist, Read/Dark noise will matter ?

DM

Horshack Veteran Member • Posts: 7,995
Re: Clarification - Read Noise and Engineering Dynamic Range
3

bclaff wrote:

However, normalized read noise is not photographically relevant; and the DxOMark Landscape Use Case score is supposed to be a photographic use case.

Read noise is not relevant to whose photography specifically? The problem with randomly selecting an SNR cutoff for DR and calling it relevant is that it's only relevant to some and not others. And even for the group it is generally relevant for it may not be relevant for all of their photographic scenarios.

It's the same problem with using any composite metric to convey an overall IQ score, which is why I think both PDR and DxO's Landscape number aren't very useful as a one-size-fits-all metric. Since no composite score can convey IQ I think the best solution is to provide the raw data points, explain how those data points apply to specific shooting situations and then let individuals decide which sensor is better for their shooting scenarios.

Btw, DxO themselves sort of agree with you and think the SNR cutoff should be higher than 1:1. They believe it should be 10:1 [20db], source and quotes:

"The problem, Guichard said, is that the technical measurement of dynamic range is based on a signal-to-noise measurement that's not useful in practice. With that formula, "the dynamic range of film is eight stops more than any sensor on the planet," he said. However, "this threshold doesn't make sense. We have to define another threshold more related to a minimal quality acceptance threshold."

"DxO Labs has studied whether film really does have a better dynamic range than digital cameras. When based on measurements with a signal-to-noise ratio of 0 decibels, Kodak's Portra 160NC film (shown with the green line) has a wider range than the Nikon D3X camera (the green arrow-tipped line is longer than the black one). However, DxO argues that it's more practical to use a signal-to-noise ratio of 20dB, at which point the Nikon outdoes the film. (Nikon's newer D800 camera also outdoes the D3X in terms of dynamic range.)"

Horshack Veteran Member • Posts: 7,995
Re: Photographic Dynamic Range (PDR) is a More Useful Measure than the DxOMark Landscape Score
2

Detail Man wrote:

Horshack wrote:

Detail Man wrote:

bclaff wrote:

... The PDR criteria operates in the PTC region well above SNR=1 (at least until we get 384Mp sensors!).

In this region the subtle interplay of read noise, other noise, small amounts of photon noise, slope due to Full Well Capacity (FWC), etc. is captured.

It seems that (especially thses days, with higher performance image-sensors), Readout/Dark noise component are effectively "numerically swamped" by Photon Shot Noise. Rather "subtle", indeed.

But there's a distinct qualitative difference of read noise vs shot noise.

Seems to be a reasonable statement to make [surrounding specific visual observation(s) made].

John Sheehy once posted a simulation of an ISO 1,000,000 image with zero read noise and it looked more natural than a much lower ISO image with read noise.

Tried to find such a graphic (without success). Can you recall any particular "keywords" used in that post that I could "plug into" a (anything but the on-site DPReview, that is) "search engine" ?

Or, might you have a reference-link to John Sheehy's publishing of that graphic to provide ?

DM

I just attempted myself and couldn't find it. Found lots of messages from John about the simulation but not the simulated image itself - I'm nearly positive he posted the image though. I remember reviewing it.

Jack Hogan Veteran Member • Posts: 7,026
Re: Clarification - Read Noise and Engineering Dynamic Range

Horshack wrote:

"DxO Labs has studied whether film really does have a better dynamic range than digital cameras. When based on measurements with a signal-to-noise ratio of 0 decibels, Kodak's Portra 160NC film (shown with the green line) has a wider range than the Nikon D3X camera (the green arrow-tipped line is longer than the black one). However, DxO argues that it's more practical to use a signal-to-noise ratio of 20dB, at which point the Nikon outdoes the film. (Nikon's newer D800 camera also outdoes the D3X in terms of dynamic range.)"

Yes, that was an interesting paper.  Here is its link .

Jack Hogan Veteran Member • Posts: 7,026
Channel Trajectories

bclaff wrote: So we can have a concrete discussion about this matter consdier the following charts:

D7100 PTC dark corner

D7200 PTC dark corner

D610 PTC dark corner

Bill, I get very similar numbers to yours, within a few hundreds of a stop. The D7100 Dxomark Landscape score looks off, it should be much closer to the D7200's, which looks right. Say within 1/3 of a stop.

Here is a question for you. See all those apparently different trajectories the separate color channels appear to be taking: do you think the read noise of the various channels in units of photoelectrons should therefore be different as well? I have recently convinced myself that they should not.

Jack

Detail Man
Detail Man Forum Pro • Posts: 17,078
Re: Photographic Dynamic Range (PDR) is a More Useful Measure than the DxOMark Landscape Score

Horshack wrote:

Detail Man wrote:

Horshack wrote:

Detail Man wrote:

bclaff wrote:

... The PDR criteria operates in the PTC region well above SNR=1 (at least until we get 384Mp sensors!).

In this region the subtle interplay of read noise, other noise, small amounts of photon noise, slope due to Full Well Capacity (FWC), etc. is captured.

It seems that (especially thses days, with higher performance image-sensors), Readout/Dark noise component are effectively "numerically swamped" by Photon Shot Noise. Rather "subtle", indeed.

But there's a distinct qualitative difference of read noise vs shot noise.

Seems to be a reasonable statement to make [surrounding specific visual observation(s) made].

It seems sort of like tentative, aesthetic territory. In the case of (completely) random noise components with a flat spatial frequency spectrum, the difference might seem to be (only) that sensor illumination is (also) increasing (by the square of the Photon Shot Noise increase). In the case of periodic noise components, individual perceived "qualitative" elements seem potentially unique.

John Sheehy once posted a simulation of an ISO 1,000,000 image with zero read noise and it looked more natural than a much lower ISO image with read noise.

Tried to find such a graphic (without success). Can you recall any particular "keywords" used in that post that I could "plug into" a (anything but the on-site DPReview, that is) "search engine" ?

Or, might you have a reference-link to John Sheehy's publishing of that graphic to provide ?

I just attempted myself and couldn't find it. Found lots of messages from John about the simulation but not the simulated image itself - I'm nearly positive he posted the image though. I remember reviewing it.

Perhaps John may be able to let us know about that at some point.

OP bclaff Veteran Member • Posts: 8,737
Re: Clarification - Read Noise and Engineering Dynamic Range

Horshack wrote:

bclaff wrote:

However, normalized read noise is not photographically relevant; and the DxOMark Landscape Use Case score is supposed to be a photographic use case.

Read noise is not relevant to whose photography specifically?

I think I could have been more explicit.
I mean that a measure that is solely read noise and has no photon noise component is not photographically relevant. (It has engineering relevance.)
If we disagree on that, I guess we should just let it go!

The problem with randomly selecting an SNR cutoff for DR and calling it relevant is that it's only relevant to some and not others.

I don't believe the PDR cutoff is random at all.
It is based on the acuity of the human eye and established values for image quality.

Regards,

-- hide signature --
OP bclaff Veteran Member • Posts: 8,737
Re: Photographic Dynamic Range (PDR) is a More Useful Measure than the DxOMark Landscape Score

Just a couple of point ...

Regarding SNR = 20 see the other clarification.

Regarding E-M5 versus GH2 you intermix two concepts and make it incorrectly appear that the result has to do with FWC when it has entirely to do with read noise only.
That's because sensorgen reports read noise in electrons, which came from read noise in DN and gain, and gain came from FWC / (ADC range).
The one stop difference in the Landscape score comes from the one stop difference in engineering dynamic range. Both are 12-bit ADCs and the read noises in DN are about 1.173 and 2.083 respectively.

BTW, slightly OT, I analyze DxOMark data too and have results similar to but not identical to that at sensorgen. The results differe because we take a different approach to the data.
FWIW here's one of my charts .
(Let's not comment on this side topic here and pull this thread OT)

Regards,

-- hide signature --
J A C S
J A C S Forum Pro • Posts: 14,947
Re: Clarification - Read Noise and Engineering Dynamic Range

bclaff wrote:

Horshack wrote:

bclaff wrote:

However, normalized read noise is not photographically relevant; and the DxOMark Landscape Use Case score is supposed to be a photographic use case.

Read noise is not relevant to whose photography specifically?

I think I could have been more explicit.
I mean that a measure that is solely read noise and has no photon noise component is not photographically relevant. (It has engineering relevance.)

Are you sure that photon noise is not taken into account by DXO (I do not know the answer). Their definition is a bit loose, and regardless of the definition, it would be interesting to know how they actually measure it.

Let is say they measure the read noise of a black frame and convert it into stops below saturation, like 20 e-. But at that level, the photon noise can be computed as a square root. Whether they add it or not, I do not know but in any case, you can add it to the read noise (in quadratures). So it is not difficult to include the shot noise as well; and that is a small correction unless the number is too close to 1 (normalized to 8mp) but we are far from it with most sensors, I believe.

Detail Man
Detail Man Forum Pro • Posts: 17,078
Re: Clarification - Read Noise and Engineering Dynamic Range

bclaff wrote:

Horshack wrote:

bclaff wrote:

However, normalized read noise is not photographically relevant; and the DxOMark Landscape Use Case score is supposed to be a photographic use case.

Read noise is not relevant to whose photography specifically?

I think I could have been more explicit.
I mean that a measure that is solely read noise and has no photon noise component is not photographically relevant. (It has engineering relevance.)

At how may EV of (in-processing) "pushing" do the results cease to be "photographically relevant" ?

If we disagree on that, I guess we should just let it go!

The problem with randomly selecting an SNR cutoff for DR and calling it relevant is that it's only relevant to some and not others.

I don't believe the PDR cutoff is random at all.
It is based on the acuity of the human eye and established values for image quality.

.

Iliah Borg wrote:

Above 10 stops flare starts to play significant role in real-life shooting.

If such is indeed the case, how (at least in relevant, so affected cases) might such phenomena influence the ("practical", "real world") significance of both "PDR" as well as DxOMark numbers ?

DM

OP bclaff Veteran Member • Posts: 8,737
Re: Channel Trajectories

Jack Hogan wrote:

bclaff wrote: So we can have a concrete discussion about this matter consdier the following charts:

Bill, I get very similar numbers to yours, within a few hundreds of a stop. The D7100 Dxomark Landscape score looks off, it should be much closer to the D7200's, which looks right. Say within 1/3 of a stop.

Here is a question for you. See all those apparently different trajectories the separate color channels appear to be taking: do you think the read noise of the various channels in units of photoelectrons should therefore be different as well? I have recently convinced myself that they should not.

In the absence of light we certainly don't expect different position in the Color Filter Array (CFA) to behave differently.

In the case of Nikon we know that the red and blue are stretched in the raw data making their noise measure higher and the SNR lower.

Also, it's a stretch, but in a multichannel readout perhaps the channels are intentionally not calibrated identically. That would show too.

Regards,

-- hide signature --
OP bclaff Veteran Member • Posts: 8,737
Re: Clarification - Read Noise and Engineering Dynamic Range

Detail Man wrote:

bclaff wrote:

Horshack wrote:

bclaff wrote:

However, normalized read noise is not photographically relevant; and the DxOMark Landscape Use Case score is supposed to be a photographic use case.

Read noise is not relevant to whose photography specifically?

I think I could have been more explicit.
I mean that a measure that is solely read noise and has no photon noise component is not photographically relevant. (It has engineering relevance.)

At how may EV of (in-processing) "pushing" do the results cease to be "photographically relevant" ?

No amount of pushing will be relevant in an area of the image that is entirely read noise.
This is equivalent to an infinity stop push and even at infinity I suspect that in this case 0 * infinity = 0.

Regards,

-- hide signature --
Detail Man
Detail Man Forum Pro • Posts: 17,078
Re: Clarification - Read Noise and Engineering Dynamic Range

bclaff wrote:

Detail Man wrote:

bclaff wrote:

Horshack wrote:

bclaff wrote:

However, normalized read noise is not photographically relevant; and the DxOMark Landscape Use Case score is supposed to be a photographic use case.

Read noise is not relevant to whose photography specifically?

I think I could have been more explicit.
I mean that a measure that is solely read noise and has no photon noise component is not photographically relevant. (It has engineering relevance.)

At how may EV of (in-processing) "pushing" do the results cease to be "photographically relevant" ?

No amount of pushing will be relevant in an area of the image that is entirely read noise.
This is equivalent to an infinity stop push and even at infinity I suspect that in this case 0 * infinity = 0.

You seem to be proposing that (at least, random) Read/Dark noise is always "inherently invisible" ?

.

In reply to Horshack, Bill Claff wrote:

If we disagree on that, I guess we should just let it go!

Horshack wrote:

The problem with randomly selecting an SNR cutoff for DR and calling it relevant is that it's only relevant to some and not others.

I don't believe the PDR cutoff is random at all.
It is based on the acuity of the human eye and established values for image quality.

.

Iliah Borg wrote:

Above 10 stops flare starts to play significant role in real-life shooting.

If such is indeed the case, how (at least in relevant, so affected cases) might such phenomena influence the ("practical", "real world") significance of both "PDR" as well as DxOMark numbers ?

I do indeed see an argument (if lens-flare places lower bounds on image-sensor illumination levels) that DR metrics utilizing higher than SNR~1 values may (as a result) be more "representative".

DM

Horshack Veteran Member • Posts: 7,995
Re: Clarification - Read Noise and Engineering Dynamic Range
2

bclaff wrote:

Horshack wrote:

bclaff wrote:

However, normalized read noise is not photographically relevant; and the DxOMark Landscape Use Case score is supposed to be a photographic use case.

Read noise is not relevant to whose photography specifically?

I think I could have been more explicit.
I mean that a measure that is solely read noise and has no photon noise component is not photographically relevant. (It has engineering relevance.)
If we disagree on that, I guess we should just let it go!

The problem with randomly selecting an SNR cutoff for DR and calling it relevant is that it's only relevant to some and not others.

I don't believe the PDR cutoff is random at all.
It is based on the acuity of the human eye and established values for image quality.

Regards,

I've got to run out but this is an interesting discussion to continue later. Until then here are two A7s images for thought - at their full native resolution - one at ISO 100 and the other at 409600 - at the same relative exposure. According to DxO the A7s has 5.76DR @ ISO 409,600 (normalized to 8MP) vs your PDR measurement of 0.93. Looking at the ISO 409,600 image do you believe we cannot discern > 1EV of DR in it?

A7s ISO 100 4 seconds

A7s ISO 409,600 1/1000

OP bclaff Veteran Member • Posts: 8,737
Re: Clarification - Read Noise and Engineering Dynamic Range

Horshack wrote:

here are two A7s images for thought - at their full native resolution - one at ISO 100 and the other at 409600 - at the same relative exposure. According to DxO the A7s has 5.76DR @ ISO 409,600 (normalized to 8MP) vs your PDR measurement of 0.93. Looking at the ISO 409,600 image do you believe we cannot discern > 1EV of DR in it?

A7s ISO 100 4 seconds

A7s ISO 409,600 1/1000

I get two 0 byte images from those links

In any case, I would want to see the raw files since because of tone curve considerations I cannot discern raw levels from jpg files.

Regards,

-- hide signature --
OP bclaff Veteran Member • Posts: 8,737
Re: Clarification - Read Noise and Engineering Dynamic Range

Detail Man wrote:

If such is indeed the case, how (at least in relevant, so affected cases) might such phenomena influence the ("practical", "real world") significance of both "PDR" as well as DxOMark numbers ?

Since PDR measurements are made from photographs I suspect that any other influence, such as flare, would be built in to the results.

Regards,

-- hide signature --
Keyboard shortcuts:
FForum MMy threads