Unbiased noise test.

Started Feb 28, 2009 | Discussions thread
sidelight Senior Member • Posts: 1,013
Re: article

Rcihardc wrote:

sidelight wrote:

Rcihardc wrote:


The noise in the analog readout of a CMOS image sensor is different
from a continuous reading of a sensor in real time where there is
spikes of noise and interference. It is just a different world, you
cannot really compare directly, please read the Clark articles, there
is no threshold like that...

DxO only measure by looking at raw images, they can never measure
real noise which is only possible by disassembling the camera and
wire-bonding the sense amplifier lines to an Agilent spectrum

Again, I am not following you. I'm not a sensor or electronics guy,
but I find it difficult to understand when you say the sensor noise
cannot be measured using their procedure. I can understand that
there may be different ways to measure noise and under varied
conditions which they may not cover completely. Noise should
manifest itself in signal variation and have a standard deviation.
At low light conditions, I understand it to be mainly read noise. At
higher levels it is a mix of read and photon noise...[snip]

you don't get it DxO is not measuring any signal, they are looking at
a raw files, a raw file is not an analog electrical signal. Also
shot noise power is proportional to sqrt(N) N being the number of
photons detected, signal power is proportional to N, so S/N for shot
noise is prop. to sqrt(N) which is high at good light and low at low
light (that is why you can't have a single photon detector becuase
you would never know when it arrives). So shot noise is important at
low ligh not at high light. Clearly you did not read any of the
Clark's article, please READ the article first and you will
undrestand why DxO is not a good benchmark.

Sorry to get back to you so late - I had not seen your reply earlier. Richard, I'm sorry but you seem to have a lack of understanding of how the tests at DxO, and for that matter, at any other review site, are conducted. The review sites do not take electronic readings off the sensor. Yes, they use the RAW files, which are converted to images digitally of step gray scale step wedges and color patch charts. They then use measurement tools to read the variation in black, gray, and color patches and determine the standard deviations. This is the signal I was referring to.

Here is how Roger Clark does it:


It is similar to dpr and DxOmark. DPR plots StdDev vs ISO. DxOMark presents more results with a greater variety of plots. Below are their procedures (details on each site).

Here is dpr's procedure:

Noise Measurement
By Phil Askey

Standard Test Chart

ISO equivalence on a digital camera is the ability to increase the sensitivity of the sensor. The works by turning up the "volume" (gain) on the sensor's signal amplifiers (remember the sensor is an analogue device). By amplifying the signal you also amplify the noise which becomes more visible at higher ISO's. Many modern cameras also employ noise reduction and / or sharpness reduction at higher sensitivities.

To measure noise levels we take a sequence of images of a GretagMacBeth ColorChecker chart (controlled artificial daylight lighting). The exposure is matched to the ISO (ie. ISO 200, 1/200 sec for consistency of exposure between cameras).

dpreview.com Noise Measurement Tool version 1.4

The biggest change introduced with the latest version of our noise measurement tool is the measurement of standard deviation of the black patch as well as the inclusion of a crop of the patch in the review visualization. The black patch is a fairly good representation of noise typically seen in dark 'shadow' areas of an image, certain cameras which may have a relatively good mid-tone noise profile perform less well in the shadows.

Our current noise measurement tool performs the following functions:

Measures luminance noise of the middle gray patch
Measures luminance noise of the black patch ('shadow noise')
Measures chroma noise of the middle gray patch

Takes crops of the gray, black and detail and provides these for review visualization
Outputs result data for production of luminance and chroma noise graphs

How does DxOMark conduct their test?

To take the measurements we publish on dxomark.com, we perform the following steps:

adjust test targets, lights, and the selected camera
shoot a set of images of the target at different camera settings

process the target images with DxO Analyzer (machine vision algorithms enable automatic processing)
evaluate the data and graphs reported through the DxO Analyzer interface
Read more about DxO Analyzer.

All dxomark.com measurements are performed on cameras that are bought or rented from a reseller’s store. It is DxO Labs’ policy to test and publish results for only publicly-available versions of each camera, rather than for cameras lent by the manufacturer or for pre-series models.

To eliminate setup bias, settings, instrument calibration, light levels, cleanliness (including that of the camera), and all other parameters are checked and rechecked before each camera is tested to ensure that all cameras are tested under exactly the same conditions.

Finally, each measurement is validated by at least two technicians who take their measurements on different days to minimize the risk of manipulation errors."

They then determine noise, signal to noise, dynamic range, etc and plot out these numeric readings to generate plots.

I think you should take some time to read each sites procedures. If you dispute the validity of their procedures and results, please write to them and ask for a response. Please share this response here.

-- hide signature --


Post (hide subjects) Posted by
(unknown member)
(unknown member)
(unknown member)
(unknown member)
(unknown member)
(unknown member)
(unknown member)
(unknown member)
(unknown member)
(unknown member)
(unknown member)
(unknown member)
(unknown member)
(unknown member)
(unknown member)
(unknown member)
Keyboard shortcuts:
FForum PPrevious NNext WNext unread UUpvote SSubscribe RReply QQuote BBookmark MMy threads
Color scheme? Blue / Yellow