Sensor performance, quantum efficiency, sensorgen and DXOmark

Started Feb 5, 2014 | Discussions
bobn2
bobn2 Forum Pro • Posts: 57,436
Re: Sensor performance, quantum efficiency, sensorgen and DXOmark

rrccad wrote:

Chrisk98 wrote:

Nearly 4 years ago I presented here a way to estimate the quantum efficiency of camera sensors by analizing SNR data from DXOmark. Since than many new cameras were tested by DXO and even a website called sensorgen is providing QE data (besides other sensor parameters). This is a nice way to public the true sensor parameters rather than just the high-level SNR data like DXOmark. The only point is that the calculation of QE is based on several assumption which IMO is not adequate mentioned by sensorgen. Therefore, I have summarized the theory behind and provided also some new results with focus on low light performance. The paper can be found here.

The QE results are also shown in the figure below.

Snip. what's the deal with the 70D on this graph?

The deal you'd expect. a modern sensor with about the same QE as Canon's other modern sensors.

-- hide signature --

Bob

OP Chrisk98 New Member • Posts: 21
Re: Sensor performance, quantum efficiency, sensorgen and DXOmark

bobn2 wrote:

Hi Chris,

First, I'd like to thank you for what you did four years ago - the method that you devised there was indeed the inspiration for sensorgen (and yes, I give you a credit). When you made that first couple of posts people started asking if the same data was available for a wider range of cameras, so I decided to do it. And while you will get naysayers saying what you did was pointless, the site does get over a quarter of a million hits a year - so there are at least a few who are interested.

On the second graph I don't think you've normalised for pixel count, so the 'SNR' is SNR per pixel, i.e., different noise bandwidth for each camera, so not directly comparable.

So, once again - thanks.

Hi Bob,

congratuations, you have done a good job with sensorgen. And I didn't expected the huge traffic. But I have one suggestions: do not use the pixel pitch data from DXO, they are often wrong. For example, canon 7D has 4.29µm and not 4.16µm, or Canon 70D has 4.09 and not 4.29µm. Changes like that will have a big change on QE. I recommend this website which is also very useful.

I didn't understand your second comment on the second graph, could you explain me what you exactly mean (normalised for pixel count)?

Thanks

Chris

 Chrisk98's gear list:Chrisk98's gear list
Panasonic LX100
Jason Rickerby Contributing Member • Posts: 712
Re: Sensor performance, quantum efficiency, sensorgen and DXOmark

Chrisk98 wrote:

This figure is very helpful to select the best camera for low light or astrophotography:

Thanks for doing this. It is indeed very helpful and I especially appreciate the exposure time graph. Clearly I'm way better off with the 5D3 instead of the 40D at prime focus This really is a clear illustration of the advancements made in sensor performance.

I must confess, I'm still unclear of the relationship between read noise, well depth and dynamic range. Will camera A, with half the read noise of camera B, at the same ISO and with the same well depth have one more stop of usable dynamic range?

 Jason Rickerby's gear list:Jason Rickerby's gear list
Canon EOS 5D Mark III
Jack Hogan Veteran Member • Posts: 6,556
Re: Sensor performance, quantum efficiency, sensorgen and DXOmark

Jason Rickerby wrote:

I must confess, I'm still unclear of the relationship between read noise, well depth and dynamic range. Will camera A, with half the read noise of camera B, at the same ISO and with the same well depth have one more stop of usable dynamic range?

The generic approximate answer is yes but the precise answer depends on the exact definition of DR required and sensor technology, including QE

bobn2
bobn2 Forum Pro • Posts: 57,436
Re: Sensor performance, quantum efficiency, sensorgen and DXOmark

Chrisk98 wrote:

bobn2 wrote:

Hi Chris,

First, I'd like to thank you for what you did four years ago - the method that you devised there was indeed the inspiration for sensorgen (and yes, I give you a credit). When you made that first couple of posts people started asking if the same data was available for a wider range of cameras, so I decided to do it. And while you will get naysayers saying what you did was pointless, the site does get over a quarter of a million hits a year - so there are at least a few who are interested.

On the second graph I don't think you've normalised for pixel count, so the 'SNR' is SNR per pixel, i.e., different noise bandwidth for each camera, so not directly comparable.

So, once again - thanks.

Hi Bob,

congratuations, you have done a good job with sensorgen. And I didn't expected the huge traffic. But I have one suggestions: do not use the pixel pitch data from DXO, they are often wrong. For example, canon 7D has 4.29µm and not 4.16µm, or Canon 70D has 4.09 and not 4.29µm. Changes like that will have a big change on QE. I recommend this website which is also very useful.

Generally, sensorgen calculates pixel size from sensor size and pixel count.

I didn't understand your second comment on the second graph, could you explain me what you exactly mean (normalised for pixel count)?

You're showing a 'per pixel' SNR, so different pixel count cameras are showing SNR measured over a different bandwidth.

-- hide signature --

Bob

Jason Rickerby Contributing Member • Posts: 712
Re: Sensor performance, quantum efficiency - how is this really measured?

Jack Hogan wrote:

Jason Rickerby wrote:

I must confess, I'm still unclear of the relationship between read noise, well depth and dynamic range. Will camera A, with half the read noise of camera B, at the same ISO and with the same well depth have one more stop of usable dynamic range?

The generic approximate answer is yes but the precise answer depends on the exact definition of DR required and sensor technology, including QE

Reading the OP's paper, there's a lot of of sophisticated math.

Looking at the following Sensorgen pages, http://www.sensorgen.info/CanonEOS_5D_MkIII.html, http://www.sensorgen.info/NikonD800.html I'm still unclear as to where these measurements actually come from.

At ISO 100 with the Canon 5D Mark III and Nikon D800 we have the following:

5D3: Read Noise - 33.1, Saturation - 67531, DR - 11.0

D800: Read Noise - 2.7, Saturation - 44972, DR - 14.0

As A/D converters aren't producing decimal output, how realistic are these Read Noise and DR figures? Isn't it a situation whether you trust the A/D converter low value bits or not? Are these numbers derived from integer data in RAW files? Are decimal figures an estimate due to a known analog gain applied to the sensor readout?

Does the following approximation work?

For the 5D3, 33.1 requires 2^6 and 67531 would require 2^17. That would be a 2^11 difference between saturation and noise. Does this mean sensor gain can be designed to optimize performance of the 14-bit A/D converters?

For the D800, 2.7 requires 2^2 and 44972 would require 2^16. That would be a 2^14 difference between saturation and noise. This is taking 14-bit A/D converters to the limit?

Is the above an approximation of how sensor DR is calculated? Assuming sensor performance is linear, which it isn't, ideally every additional bit equates to a stop of dynamic range? It seems like as ISO goes up, analogue gain is increased and more low value A/D bits are ignored? Is analog gain optimized for A/D performance, ISO accuracy, power consumption/linearity or all of the above? You can't have more stops of camera dynamic range than A/D bits right?

 Jason Rickerby's gear list:Jason Rickerby's gear list
Canon EOS 5D Mark III
rrccad Forum Pro • Posts: 11,108
Re: Sensor performance, quantum efficiency, sensorgen and DXOmark

bobn2 wrote:

rrccad wrote:

Chrisk98 wrote:

Nearly 4 years ago I presented here a way to estimate the quantum efficiency of camera sensors by analizing SNR data from DXOmark. Since than many new cameras were tested by DXO and even a website called sensorgen is providing QE data (besides other sensor parameters). This is a nice way to public the true sensor parameters rather than just the high-level SNR data like DXOmark. The only point is that the calculation of QE is based on several assumption which IMO is not adequate mentioned by sensorgen. Therefore, I have summarized the theory behind and provided also some new results with focus on low light performance. The paper can be found here.

The QE results are also shown in the figure below.

Snip. what's the deal with the 70D on this graph?

The deal you'd expect. a modern sensor with about the same QE as Canon's other modern sensors.

well except that canon's APS-C sensors have never reached a peak of 51% QE - their best is prior best 42%.

 rrccad's gear list:rrccad's gear list
Sigma DP1 Merrill Sigma DP2 Merrill Canon EOS 5D Mark II Canon EOS M Canon EOS 7D Mark II +2 more
bobn2
bobn2 Forum Pro • Posts: 57,436
Re: Sensor performance, quantum efficiency, sensorgen and DXOmark

rrccad wrote:

bobn2 wrote:

rrccad wrote:

Chrisk98 wrote:

Nearly 4 years ago I presented here a way to estimate the quantum efficiency of camera sensors by analizing SNR data from DXOmark. Since than many new cameras were tested by DXO and even a website called sensorgen is providing QE data (besides other sensor parameters). This is a nice way to public the true sensor parameters rather than just the high-level SNR data like DXOmark. The only point is that the calculation of QE is based on several assumption which IMO is not adequate mentioned by sensorgen. Therefore, I have summarized the theory behind and provided also some new results with focus on low light performance. The paper can be found here.

The QE results are also shown in the figure below.

Snip. what's the deal with the 70D on this graph?

The deal you'd expect. a modern sensor with about the same QE as Canon's other modern sensors.

well except that canon's APS-C sensors have never reached a peak of 51% QE - their best is prior best 42%.

Yes, but they hadn't, until this, designed a new APS-C sensor for a very long time.

-- hide signature --

Bob

Jack Hogan Veteran Member • Posts: 6,556
Re: Sensor performance, quantum efficiency - how is this really measured?

Jason Rickerby wrote:

Jack Hogan wrote:

Jason Rickerby wrote:

I must confess, I'm still unclear of the relationship between read noise, well depth and dynamic range. Will camera A, with half the read noise of camera B, at the same ISO and with the same well depth have one more stop of usable dynamic range?

The generic approximate answer is yes but the precise answer depends on the exact definition of DR required and sensor technology, including QE

Reading the OP's paper, there's a lot of of sophisticated math.

Looking at the following Sensorgen pages, http://www.sensorgen.info/CanonEOS_5D_MkIII.html, http://www.sensorgen.info/NikonD800.html I'm still unclear as to where these measurements actually come from.

They are approximated through Photon Transfer and idealized curve fitting.

At ISO 100 with the Canon 5D Mark III and Nikon D800 we have the following:

5D3: Read Noise - 33.1, Saturation - 67531, DR - 11.0

D800: Read Noise - 2.7, Saturation - 44972, DR - 14.0

As A/D converters aren't producing decimal output, how realistic are these Read Noise and DR figures? Isn't it a situation whether you trust the A/D converter low value bits or not? Are these numbers derived from integer data in RAW files? Are decimal figures an estimate due to a known analog gain applied to the sensor readout?

Except that it turns out that bit depth and DR are only loosely related. And photographic signals effectively work in floating point all the way to the raw file Here are a fewintroductory thoughts on why.

Does the following approximation work?

For the 5D3, 33.1 requires 2^6 and 67531 would require 2^17. That would be a 2^11 difference between saturation and noise. Does this mean sensor gain can be designed to optimize performance of the 14-bit A/D converters?

Well, the question is: "if random noise swamps the bottom 6 bits of every value of the raw file as per your example above, are you recording any information in those 6 bits or are you simply wasting time, power and space converting them/storing them"?

So perhaps it would be better to design an ADC so that each raw level sees just enough noise at base ISO to provide desirable dithering - but no more. The design sweetspot today appears to be read noise of around 0.5-1.5ADUs (Raw Levels), with John Sheehy suggesting that 1.3ADUs was optimum a few years ago.

By that yardstick the 5D3's ADC would have one raw level every 25.5e- [33.1/1.3], saturation occurring at level 2648 (67531/25.5), a figure which fits easily into 12 bits. I am not sure why the 5DIII would need any more than that  .

For the D800, 2.7 requires 2^2 and 44972 would require 2^16. That would be a 2^14 difference between saturation and noise. This is taking 14-bit A/D converters to the limit?

Let's see, with the same criterion as above: 1 raw level = 2.1e- [2.7/1.3]. Saturation = 21653 raw levels [44972/2.1], or somewhat more than 14 bits (However we know that the sensorgen.info value for D800 read noise is understated by a good e- or two, so in the end 14 bits are enough)

Is the above an approximation of how sensor DR is calculated? Assuming sensor performance is linear, which it isn't,

Why not? I think for the purposes of this discussion we can assume that raw data is linear with incoming luminance.

ideally every additional bit equates to a stop of dynamic range? It seems like as ISO goes up, analogue gain is increased and more low value A/D bits are ignored?

Rather than ignored, they are swamped by the amplified sensor noise. Here is a graph of what happens if modelled ideally.

Is analog gain optimized for A/D performance, ISO accuracy, power consumption/linearity or all of the above?

You can't have more stops of camera dynamic range than A/D bits right?

Sure, the sky is the limit in the right conditions (1 bit audio DACs come to mind). For photographic cameras one of the key design parameters is the amount of read noise.

Jack

Jason Rickerby Contributing Member • Posts: 712
Thanks for the informative response

Thanks for the helpful response.

Regard sensor linearity, I used to do astrophotography with a CCD and as it was anti-blooming, so one had to ensure to underexpose to do any consistent photometry. I was assuming that CMOS sensors would have similar constraints.

 Jason Rickerby's gear list:Jason Rickerby's gear list
Canon EOS 5D Mark III
Jack Hogan Veteran Member • Posts: 6,556
Peak QE of the 70D

bobn2 wrote:

rrccad wrote:

bobn2 wrote:

rrccad wrote: Snip. what's the deal with the 70D on this graph?

The deal you'd expect. a modern sensor with about the same QE as Canon's other modern sensors.

well except that canon's APS-C sensors have never reached a peak of 51% QE - their best is prior best 42%.

Yes, but they hadn't, until this, designed a new APS-C sensor for a very long time.

Actually that's a good question. By following Chris's method with the values at Sensorgen I get peak QE for the 70D of 45.2% and 43.6% at ISO 800 and 1600 respectively (with average photon energy computed at 550nm and 4.29 micron pitch), in line with Sensorgen's 45% figure.

Chris considers average photon energy at 530nm and uses a pixel pitch of 4.09 micron for the 70D, which at ISO800 result in peak QE of 51.6% with the same calculations.

Imho the better figure is 550nm because the denominator of formula (14) is effectively a naive estimate of the number of photons arriving on the photosite's area for the given saturation exposure. If one assumes uniform spectral distribution of the incoming light, the average energy of the arriving photons used in computing the denominator of forumula (14) should be based on the light's average wavelength. Since the band observed is 400-700nm, the average wavelength is 550nm. The wavelength at which the CFA peaks should be immaterial for this discussion, since for the stated assumptions the only requirement is that the normalized area under the Photopic curve and (weighted) average Green/Blue/Red CFA curves be the same - and this area is independent of peak wavelength.

As for pixel pitch, Raw Digger shows 5568x3708 including the masked pixels and 5496x3670 for a 70D sensor size spec'd at 22.5x15mm. That would yield pitches of 4.045 and 4.09 microns respectively. I wonder whether manufacturers spec the size of the sensor including masked pixels? Using the masked pixel count and 550nm peak QE by Chris' method is 50.9% and 49.1% at ISO800 and 1600 resp.

Jack

PS I found Chris' paper quite interesting and I have several comments/questions on it. When I have time I will start a thread in the Photographic Science and Technology forum about it.

rrccad Forum Pro • Posts: 11,108
Re: Peak QE of the 70D

Jack Hogan wrote:

bobn2 wrote:

rrccad wrote:

bobn2 wrote:

rrccad wrote: Snip. what's the deal with the 70D on this graph?

The deal you'd expect. a modern sensor with about the same QE as Canon's other modern sensors.

well except that canon's APS-C sensors have never reached a peak of 51% QE - their best is prior best 42%.

Yes, but they hadn't, until this, designed a new APS-C sensor for a very long time.

Actually that's a good question. By following Chris's method with the values at Sensorgen I get peak QE for the 70D of 45.2% and 43.6% at ISO 800 and 1600 respectively (with average photon energy computed at 550nm and 4.29 micron pitch), in line with Sensorgen's 45% figure.

Chris considers average photon energy at 530nm and uses a pixel pitch of 4.09 micron for the 70D, which at ISO800 result in peak QE of 51.6% with the same calculations.

Imho the better figure is 550nm because the denominator of formula (14) is effectively a naive estimate of the number of photons arriving on the photosite's area for the given saturation exposure. If one assumes uniform spectral distribution of the incoming light, the average energy of the arriving photons used in computing the denominator of forumula (14) should be based on the light's average wavelength. Since the band observed is 400-700nm, the average wavelength is 550nm. The wavelength at which the CFA peaks should be immaterial for this discussion, since for the stated assumptions the only requirement is that the normalized area under the Photopic curve and (weighted) average Green/Blue/Red CFA curves be the same - and this area is independent of peak wavelength.

As for pixel pitch, Raw Digger shows 5568x3708 including the masked pixels and 5496x3670 for a 70D sensor size spec'd at 22.5x15mm. That would yield pitches of 4.045 and 4.09 microns respectively. I wonder whether manufacturers spec the size of the sensor including masked pixels? Using the masked pixel count and 550nm peak QE by Chris' method is 50.9% and 49.1% at ISO800 and 1600 resp.

Jack

PS I found Chris' paper quite interesting and I have several comments/questions on it. When I have time I will start a thread in the Photographic Science and Technology forum about it.

except with the 70D I would suspect the pixel pitch is actually lower with each pixel being split in half.

which makes canon raising the QE by nearly 10% even more remarkable.

 rrccad's gear list:rrccad's gear list
Sigma DP1 Merrill Sigma DP2 Merrill Canon EOS 5D Mark II Canon EOS M Canon EOS 7D Mark II +2 more
Jack Hogan Veteran Member • Posts: 6,556
Re: Peak QE of the 70D

rrccad wrote:

Jack Hogan wrote: As for pixel pitch, Raw Digger shows 5568x3708 including the masked pixels and 5496x3670 for a 70D sensor size spec'd at 22.5x15mm. That would yield pitches of 4.045 and 4.09 microns respectively. I wonder whether manufacturers spec the size of the sensor including masked pixels? Using the masked pixel count and 550nm peak QE by Chris' method is 50.9% and 49.1% at ISO800 and 1600 resp.

except with the 70D I would suspect the pixel pitch is actually lower with each pixel being split in half.

which makes canon raising the QE by nearly 10% even more remarkable.

Yes, but keep in mind that when we speak about QE here we are not just talking charge collection efficiency and fill factor. The photosite sits under a microlens and various filters (including AA, CFA, IR) the combined effects of which make up the lion's share of what is being termed 'peak QE' in this thread. It's much easier for instance to weaken the CFA in line with current trends then to change/improve fabrication technology. And we know that doing that results in compromises elsewhere...

Jack

rrccad Forum Pro • Posts: 11,108
Re: Peak QE of the 70D

Jack Hogan wrote:

rrccad wrote:

Jack Hogan wrote: As for pixel pitch, Raw Digger shows 5568x3708 including the masked pixels and 5496x3670 for a 70D sensor size spec'd at 22.5x15mm. That would yield pitches of 4.045 and 4.09 microns respectively. I wonder whether manufacturers spec the size of the sensor including masked pixels? Using the masked pixel count and 550nm peak QE by Chris' method is 50.9% and 49.1% at ISO800 and 1600 resp.

except with the 70D I would suspect the pixel pitch is actually lower with each pixel being split in half.

which makes canon raising the QE by nearly 10% even more remarkable.

Yes, but keep in mind that when we speak about QE here we are not just talking charge collection efficiency and fill factor. The photosite sits under a microlens and various filters (including AA, CFA, IR) the combined effects of which make up the lion's share of what is being termed 'peak QE' in this thread. It's much easier for instance to weaken the CFA in line with current trends then to change/improve fabrication technology. And we know that doing that results in compromises elsewhere...

Jack

yes, but canon did that around the 7D / 60D / 18Mp sensor level - weakened the CFA?

canon obviously had to change fab technology somewhat to accommodate 40mp PD's on a APS-C sensor.

 rrccad's gear list:rrccad's gear list
Sigma DP1 Merrill Sigma DP2 Merrill Canon EOS 5D Mark II Canon EOS M Canon EOS 7D Mark II +2 more
Jack Hogan Veteran Member • Posts: 6,556
What Peak is being measured?

Chrisk98 wrote:

Nearly 4 years ago I presented here a way to estimate the quantum efficiency of camera sensors by analizing SNR data from DXOmark. ...The paper can be found here.

I had a chance to re-read your fine paper. There are a couple of passages that I am having difficulty following:

1) What Peak QE is being measured? Not green, right?

I get the feeling that the green channel is implied, since you mention (correctly I think) that that's the channel that most likely saturates first when DxO takes its measurements.

However the photoelectron Signal that the QE estimates are based on is computed from a mix of the three channels: DxO averages the channels' measured SNRs to compute a 'grayscale' Full SNR curve - from which we derive the Signal.

As an example, here is how a normalized photon and photoelectron count could look like under the constant spectral radiance assumption of the paper. The peak strength of the color filters in the example is 0.5, 0.35 and 0.25 for G,B, and R resp. The shapes are the photopic eye response's as assumed in the paper.

Note that constant spectral radiance does not mean constant photons/wavelenght throughout the range, which here is taken to be 400-700nm (the paper says 500-700 but I assume that's a typo).

We can figure out peak QE for this example as discussed in the paper by determining the Signal/lx-s/Area (FWC) in e- as the sum of the photoelectrons under each curve collected by the four photosites of an RGGB quartet, divided by four. One can compute the denominator of equation (14) (/lx-s/Area) at a pivot wavelength of 530nm as the paper suggests. The result is a peak QE of 39.2%, obviously a weighted average of the four peak QEs - not 50%. Am I correct?

2) Assuming that all three channels' CFA curves have the same shape (we know this is a stretch and never true in practice) peak QE as discussed in the paper can be calculated as follows from the peak QE's of the individual channels:

QEpeak = (­sqrt(QEpk_G)/2+sqrt(QEpk_B)/4+sqrt(QEpk_R)/4)^2

Plugging in the peak QE values for the CFA channels of the example in 1), yields a QEpeak of 39.2% as calculated earlier.

Taking a more practical example, Christian Buil's Figure 3 for the 5DII - with QEpk_G=33.8%, QEpk_B=29.3% and QEpk_R=16.8% - the same formula would result in a peak QE of 27.9%.

Buil states that "QE is the conversion rate between the incident photons and photo-electrons generated in the detector."

If one actually integrates the curves, effectively counting the photoelectrons hitting the four RBBG photosites and then applies formula (14) to the quartet one gets a Peak QE of 24.8%, indicating that the shapes of the three color filters are dissimilar, as we know. Your table shows a 5DII of 33%.

So I am still unclear: can you better define what QE the method in your paper is estimating?

Thanks for your input.

Jack

Jack Hogan Veteran Member • Posts: 6,556
Ratio of 'Peak' QE to Effective QE

Chrisk98 wrote:

The ratio of my peak QE and your effective QE should equal to the ratio of 13600 photons/lx-s/micron² you are using, and 304768/78=3907 photons/lx-s/micron² what I am using. From your data (70D) this is indeed the case: peak QE / effective QE = 51/14 = 3.6. 13600/3907=3.5. It is not matching exactly as your FWC values are slightly different to my values.

Assuming (1) constant spectral radiance and (2) Photopic curve shaped Color Filters, the ratio of 'Peak' QE to Effective QE is always equal to the bandwidth under consideration divided by the integral of the Photopic curve, in this case 400nm/106.86nm = 3.74, with lambda zero in your calculation equal to the wavelenth in the middle of the range.

However Effective QE assumes neither (1) nor (2): it assumes instead the spectral radiance of a blackbody lambertian source (an approximation of D50) and it makes no attempt at guessing the shape of the color filter. Therefore ratios will be slightly different from the figure above, varying with the interaction of the differing radiance with the actual average shape of the color filters.

The ratio of 'peak' QE to Effective QE will on the other hand remain constant (at a value most likely other than 3.74) within a pair of specific camera models, say as calculated at different ISOs. Here is the D610 as an example with a narrower bandwidth (380-710nm), that takes into consideration the IR limits of most modern CFAs

Jack Hogan Veteran Member • Posts: 6,556
Re: Ratio of 'Peak' QE to Effective QE

And since much of the data earlier was about the 70D, here are its values:

Keyboard shortcuts:
FForum MMy threads