downsampling to reduce noise - how much?

[Les Olson]

Averaging pixels 4:1 obviously increases the number of photons
collected per-pixel by a factor of 4, so the shot noise, sqrt,
goes down by a factor of 2, so the SNR goes up by about 3dB or 1
f-stop.

But that's per-pixel noise. It's even more obvious that, since
sensor sizes aren't changing, the average number of photons collected
per-image is constant from generation to generation. So more
megapixels just give the photographer the option of extended
resolution when plenty of light is available.

The data to answer your question are in the DxO database. DxO's formula shows that reducing 15MP to 10MP (eg) gains a maximum of 1.76 dB in SNR. This maximum gain applies when (and only when) the sensor is photon noise limited - when light is plentiful, in other words. As light becomes less plentiful downsampling will help progressively less because the sensor stops being photon-noise limited.

What happens when light is not plentiful can be observed by anyone who cares to by looking at the DxO SNR graphs. Note, however, that you have to look at the graphs under the "Full SNR" tab and use the logarithmic scale graphs, not the linear ones. For example, at 1% reflectance and ISO 200 the LX-3 has an 8-9 dB advantage in SNR over the G10.
--
2 November 1975.

'... Ma come io possiedo la storia,
essa mi possiede; ne sono illuminato:
ma a che serve la luce?'

 

[Kikl]

A downsampling algorithm isn't a "smart" thing, it doesn't recognize
"grain size" or know about "noise", which is kind of the point that
we've been making and that many people have been ignoring.

Noise is part of the information contained in the image file. The sensor recording this file is not part of the information of the image file. In fact, image files may be generated without any sensor; see the gaussian noise images above. Downsampling changes the image file and may therefore effect the noise level. Downsampling is merely a particular lowpass-filter, which reduced the high frequency components (the fine detail) from the image. Any noise contained in the high frequency parts of the image is removed from the image by downsampling. Therefore, downsampling reduces both image noise and image information above the cut-off frequency of the filter.

This is plain and simple and has been demonstrated to you by way of example and through clear and concise reasoning. Nevertheless, you choose to ignore this.

Regards,

kikl

 

[Phil Askey]

http://blog.dpreview.com/editorial/2008/11/downsampling-to.html#grainsize

--
Phil Askey
Editor, dpreview.com

 

[Kikl]

http://blog.dpreview.com/editorial/2008/11/downsampling-to.html#grainsize

--
Phil Askey
Editor, dpreview.com

Those are nice images, they show that downsampling reduces noise above the cutoff frequency of the filter. If no noise above the cutoff frequency is contained in the image, then the filter has no effect on the noise level. For this reason images files, on which noise reduction has already been performed, may not contain any information above the cutoff frequency of the downsampling filter. The downsampling filter has no visible effect on the noise contained in that image. But, in general noise is contained in the whole frequency band of an image.

By increasing the average grain size, the frequency distribution of the noise is shifted to lower frequencies. That is why the picture with smaller grains is more heavily affected by downsampling than the image with larger average grain size. Fine grained noise is preferable, because its frequency distribution is shifted to high frequencies. This high-frequency noise can be reduces more effectively by noise filters. Nevertheless, the noise level on all three images is noticably improved by downsampling.

I wish you would take this into account in future tests. I am criticizing you because I feel that a good review could be made a lot better. I am trying to be constructive and have made several suggestions as have other readers of DPreview. I wish you and your online magazine all the best!

Regards,

kikl

 

[Ernst Dinkla]

Phil,

A good test in my opinion but at the same time I wonder what the result would be in a similar test done on an original high resolution RAW FF sensor image (at say a more moderate 400 ISO) with for example Qimage raw (dcraw) import + anti-aliasing routine for downsampling etc or along another route with Imagemagick. Software mentioned in other threads on the subject. I expect a more uniform and random noise pattern that will not bridge more than one photosite and better handling of the noise in the downsampling routines mentioned.

While I understand your policy to use the average consumer workflow for testing, it gets in conflict with the practice of users that buy the top range of the cameras you test. I also guess that one of your assumptions is that the in camera JPEG creation reflects the manufacturer's ability to create quality in general. That may be the case for Nikon today but has not been the case for some other manufacturers/cameras. I agree that usually there's some correlation between the two aspects. I also see manufacturers that have their act together on both lenses and cameras but some do not. Splitting camera/lenses/software test results could tell a lot more than using the average consumer standard.

From 10 years digital printing practice I learned that downsampling is seen as a cure for image resolution problems while in practice it can be as bad for the image quality as upsampling can be. Scanned files of grainy film can be terrible in downsampling, grain will be rougher in upsampling but that's what you expect. For both there are good tools available.

Like with high jumping one should lay the bar higher after every good jump and with the increasing quality of the cameras I rather see the bar set higher than what DxO does with their overall mark being based on an 8 MP 20x30cm print. A stretch for the compacts but a target too small for the high resolution FFs.

Ernst Dinkla

 

[SteveGJ]

I'd agree that the DXOmark tests need to be done at more than one scaling size (and if that means upscaling the images for some cameras then that's fine as long as the mathematics work). However, the general principle applies that for comparing the output oif cameras it has to be at common sizes. "Per Pixel" might by fine for ultimate capabilities, but it is no way to compare the appearance of output of two different cameras.

On the point about processing workflows, then I'd totally agree that for the most demanding situations then most serious photographers will follow a very different path to that of just using the manufacturer's defaults. I process a lot of photos taken using high ISO under coloured stage lighting. They go through a whole set of processes, including PP noise filtering, unusual tone curves etc. You can never do better than the sensor is capable of in terms of rendering detail, but there's a huge amount that can be done (I'll admit some of it is aesthetic rather than pure technical content).

Fundamentally what I'm interested in is what the limit of image quality is under certain circumstances. I certainly don't expect to get an 11EV DR image at 6400 ISO with enough resolution to print to 24 x 36 inch output. But what I do know is that from my DSLR I can get usable (for some purposes) ISO 3200 shots at 8 x 12 inches and a lot better at 6 x 9 inches.

 

[jbr]

Phil is right. The S/N improvement expected is based on averaging or combining noise from independent samples and clearly with a Bayer sensor the noise is spread over the pixels used in the interpolation and is not independent and Phil refers to this as grain.

So in a 2:1 downsize there are only two independent Green pixels and one each for Blue and Red. From these you need to generate a new R, G and B. The best you could expect to do is to reduce the noise on the Green channel by Root 2 with no noise reduction on Red or Blue. This is a simplistic way to view it but its the way it is.

Joe

 

[ejmartin]

image analysis tool. I filled it with middle gray and added Gaussian
noise. I then resampled the image to 2/3 the image area (the

--

Apart from pattern noise (banding), which is a small component of sensor read noise, the noise in a raw file IS Gaussian random noise, to a rather good approximation.

See Figures 1 and 2 at
http://theory.uchicago.edu/~ejm/pix/20d/tests/noise/

--
emil
--



http://theory.uchicago.edu/~ejm/pix/20d/

 

[ejmartin]

Remember the bayer pattern means that you'll never have noise with a
one pixel grain size. If you do a little experiment you'll see that
(as we'd expect) the larger the grain size the less averaging from
downsampling actually achieves anything.

Clearly you didn't read the entire post. In the latter half of the post, I demosaiced the greyscale noise image, extracted the green channel, and went through the same procedure. The result was that noise was reduced, but by a somewhat lesser amount.

There are serveral salient points here:

1. The demosaic process is a (nonlinear) filter applied to the Bayer data, which does not result in much noise power near the Nyquist frequency. The point at which the noise power spectrum begins to tail off is what is commonly referred to as the "grain size".

2. Downsampling is a filter whose effect is to move the Nyquist frequency to a lower value. If done properly, it leaves the image data below the new Nyquist frequency intact. Photoshop Bicubic (in any of its variants) does not downsample properly with respect to noise.

3. The width of the histogram in a uniform tonality patch is a measure of the noise at Nyquist.

So, the effect on the width of the histogram of downsampling a demosaiced image is to measure the noise power of the original image at a slightly lower spatial frequency (below the original Nyquist by the percentage of the size reduction). That noise power is lower, but not by the percentage of the size reduction, since the linear rise in noise power of truly random noise instead tailed off a bit near Nyquist due to the Bayer interpolation (grain size).

When comparing such a downsampled image to one that is natively that size, the native image has its noise power tailing off at the Nyquist frequency, and so the histogram width is already reduced relative to random noise; the downsampled image noise at the new Nyquist is being compared to a quantity from the native image that has its noise power already depleted at that scale, which is why it doesn't seem that the noise has reduced as much.

To which one can answer, so what. If you wanted the downsampled image to have its noise and visual characteristics match the natively lower resolution image, one has the freedom to perform noise reduction on it, which if made strong enough will deplete the noise power at the scales in which it was depleted by Bayer interpolation of the lower resolution image, and then downsampling will yield an image that looks the same as the natively lower resolution image.

Edit: What I am laboring to say in words would be much clearer with a few illustrative graphs; I may try to generate some later to reinforce the above analysis.

--
emil
--



http://theory.uchicago.edu/~ejm/pix/20d/

 

[Fred Briggs]

Emil - thanks for your efforts to quantify this effect - i.e. amount of noise reduction when down-sampling Bayer derived data versus non-Bayer data.

It will take me some time to try to understand and digest the new information!

However, one point stands out to me, which is the use of the green channel for the noise analysis in the simulated Bayer data. Might this be a bit more favourable than the red or blue - and to get a full picture (no pun intended) would it be better to take some kind of average?

I will have to do some careful studying of your argument about the significance or otherwise of the parts of the noise spectrum most affected.

However, my initial simplistic view is that area under the curve, which I would think represents the total noise energy, reduces considerably in the non-Bayer example, and much less in the Bayer example. I'm not sure if this is fully represented by quoting the relative widths of the curves.

Fred

 

[Kikl]

However, my initial simplistic view is that area under the curve,
which I would think represents the total noise energy, reduces
considerably in the non-Bayer example, and much less in the Bayer
example. I'm not sure if this is fully represented by quoting the
relative widths of the curves.

This is also my interpretation of the curves. The standard deviation - i.e. width - of the noise profiles is reduced to a lesser extent in the bayer-example.

Regards,

kikl

 

[Simon97]

If printing smaller sizes, say sizes that allow more than 300 pixels per inch or 120 pixes per cm, then the printing itself may be enough to, in effect, down sample to hide noise at viewing distance. This, of course, depends on the amount of noise you started with.

Under severe cases where there is a lot of chroma noise clumps of several pixels, it will be necessary to clean up with some NR before printing.

In other cases, I reduce images to 75% using an algorithm rather than just a pixel resize. Try this on any high megapixel camera, like the LX3 or G10, the images have better sharpness without halos and noise is reduced.

 

[ejmartin]

http://blog.dpreview.com/editorial/2008/11/downsampling-to.html#grainsize

--

Two things you might not be aware of:

1. Since the "coarse grain" image has its noise power depleted at Nyquist, and the std dev of the histogram is a measure of noise at Nyquist, the "coarse grain" image has in fact more noise at any spatial frequency below Nyquist.

2. Thus, after each image was downsampled, the width of the histogram measured the noise power at the new Nyquist frequency, and (surprise!) as is quite visually apparent the "coarse grained" image has more noise at that frequency.

You really need somebody on staff (or a technical consultant) who understands the math and science involved in the testing you do.

--
emil
--



http://theory.uchicago.edu/~ejm/pix/20d/

 

[Kikl]

Phil Askey wrote:

...

You really need somebody on staff (or a technical consultant) who
understands the math and science involved in the testing you do.

--
emil

Hi emil, are you looking for a job? ;-) Thanks for your detailed explanations!

Regards,

kikl

 

[Chato]

Much thanks to Phil for proving my own conclusions.

I almost always shoot at ISO 800 with a Nikon D2x, and it's relatively rare that I don't have to use noise reducing software.

And I often post on the net - downsizing to 800 wide. The chief "improvement" in terms of noise is that the image no longer fills the screen, which "apparently" reduces the noise.

So it's either Noise Ninja, or a hell of a lot of noise. Nor is my experience limited - We are talking plus 100K images.

I had read about downsizing dramatically lowering noise levels - Hasn't worked in the real world...
Yes there is "some" reduction - But not enough to discuss.

And I've tried every method known to humanity, and then even asked my dog for suggestions.

Dave

i'm sure phil has enough to do. It was nice of him to address the
issue.

Discussing digital imaging issues is his business.

How about you do your own research instead of asking Phil to
do it?

I'm not asking him to do something. I'm asking how he did what
he did. The only time he doesn't say "jpeg" is when he says "raw",
and I'm trying to figure out if that means he just bypassed jpeg
in the camera but still compared jpegs to jpges (he is plotting them
on the same graph after all, and jpeg should reduce noise keep
in mind).

Or ask questions to the forum, there are a lot of smart ppl
here.

My post was a question: does anybody know anything about
photosensor boundaries and microlenses in the context of the
current discussion. Phil doesn't seem to. Anyone else?

Emil compares a digicam to a DSLR and finds per-pixel results
are nearly tied, and that suggests that photosensor boundaries
and microlenses aren't a big deal.

Remember that electronics isn't a "perfect world". The smaller the
pixel site, the more challenging the electronics and manufacturing.
These introduce a big can of worms!

Emil's article concludes that the read noise is better on large pixels
at high ISO, but the reverse is true at low ISO, and overall the
difference isn't terribly significant, at least not compared to
sensor size.

Aside from that, there is the quantification effect of the smaller
pixel sites. The same number of photos are landing on the sensor,
but they are divided into smaller groups. This increases the amount
of error if you miss/add a photon

Per-pixel, yes, but assuming the same field of view, per-image it
doesn't. Phil may be making a mistake here as well -- it's not clear
to me how he's measuring the 'gray standard deviation'.

look at computers and More's law. The cpu chip speeds have reached a
peak, and now they are adding more and more cores. As part of the
chip design they add redundant circuits so that the chip can work
around problems - instead of throwing the entire chip away. On a
sensor, where/how would you put/use redundant photo sites?

Emil's got another article where he suggests we could do in-sensor
HDR. I don't know as much about sensors as Emil, but I don't think
you can run amps in parallel as he suggests without decreasing the
strength of the signal into each. But one could employ two classes
of photosensor on a chip -- one for high ISO and one for low. In
fact that sounds very much like what Fuji did with their SuperCCD HR
sensors, but again maybe somebody here knows more about them
than I.

-Carl

 

[Fred Briggs]

My other thought on this is that you possibly can achieve the same total amount of noise reduction in both cases, but only if you take account of any noise reduction due to the de-mosaicing process itself.

I kind of view noise cancellation in sets of image data as a bit like squeezing the juice out of an orange. There is only so much juice to be had, whether you give it one big squeeze or two smaller ones, given that you flatten the orange by the same total amount in both cases (corresponding to down-sampling to the same image dimensions).

I would liken down-sampling a data set regarded as non-Bayer to one big squeeze of the orange.

I would liken down-sampling the same data set treated as Bayer data to two squeezes - one when you de-mosaic it, and a second when you down-sample the result.

If de-mosaicing itself does some noise cancellation, then why would we expect there to be the same potential for further noise cancellation remaining as there is when down-sampling an equivalent but non-demosaiced data set?

If there is any merit in this argument, then I suppose the key to whether this significantly invalidates the DxOmark ratings is the relative amounts of noise reduction occurring in the de-mosaicing versus the subsequent downsampling, as I would assume that it is only the second of these that differentially affects the ratings.

Would it therefore be worth trying to separately measure the effects on noise of de-mosaicing, as well as the effects of down-sampling?

Fred

 

[Reverso]

A sensor that does not demosaice?

Any Sigma camera- Foveon X3s all have native pixel coverage, and if Phil is to be believed, these cameras would all cope with noise just by downsizing.

(My first post on DPR, but I've been a lurker for some time. )

 

[Henry Richardson]

Just for fun, here are some real life examples. They aren't tests, but still pretty interesting.

Take a look at these comparing high ISO images of the down-sized A900 images and full-size Nikon D700/D3:

http://www.photoclubalpha.com/2008/10/23/the-alpha-900-as-a-high-iso-body/

http://forums.dpreview.com/forums/read.asp?forum=1037&message=29804929

http://forums.dpreview.com/forums/read.asp?forum=1037&message=29795622

--
Henry Richardson
http://www.bakubo.com

 

[aruzinsky]

All of the examined reduction methods produce weighted rather than simple averages. Assuming white noise of zero mean (which is not realistic), the standard deviation, s, for weighted average is:

s = s0*sqrt( w1^2 + w2^2 + ... wn^2 )

where

s0 is the standard deviation of the noise

n is the number of pixels (terms)

wi are the weights

For example, for a 0.5X bilinear reduction, the weight matrix is:

(1/64)*

1 3 3 1
3 9 9 3
3 9 9 3
1 3 3 1

which gives

s = s0*0.3125

However, the white noise assumption is unrealistic because the noise in nearby pixels are correlated. Thus, experiment rather than pure analysis is warranted.

Since Bayer interpolation increases the correlation of noise, it would be a good idea to repeat with a Foveon sensor camera. I expect a greater noise reduction that is closer to the theoretical white noise case.

http://blog.dpreview.com/editorial/2008/11/downsampling-to.html

Phil's results look wonky, no doubt because he used jpeg and noise
reduction at some point in apparently every workflow in the test.
And I see (surprise!) the blog doesn't allow comments. What are we
going to do with you, Phil?

Averaging pixels 4:1 obviously increases the number of photons
collected per-pixel by a factor of 4, so the shot noise, sqrt,
goes down by a factor of 2, so the SNR goes up by about 3dB or 1
f-stop.

But that's per-pixel noise. It's even more obvious that, since
sensor sizes aren't changing, the average number of photons collected
per-image is constant from generation to generation. So more
megapixels just give the photographer the option of extended
resolution when plenty of light is available.

That is, assuming photosensor boundaries don't take up much room, or
restrict the angle of effective incoming light. Which I think they
do. Microlenses can compensate, but no doubt introduce their own
transmission loss. Anyone have any information on this?

--
(Author of SAR Image Processor and anomic sociopath)
Tell me your thoughts on Plato's allegory of the cave.

 

[ejmartin]

Emil - thanks for your efforts to quantify this effect - i.e. amount
of noise reduction when down-sampling Bayer derived data versus
non-Bayer data.

However, one point stands out to me, which is the use of the green
channel for the noise analysis in the simulated Bayer data. Might
this be a bit more favourable than the red or blue - and to get a
full picture (no pun intended) would it be better to take some kind
of average?

Since the test sample was grayscale and I treated it as if it were Bayer data, it would be as if one had a light source that had equal intensity in all three color channels, which indeed is not so realistic in practice -- for outdoor illumination the R/B channels can need a boost by a factor up to about 2 in order to achieve white balance. This would mostly affect those color channels, with a somewhat smaller effect on the green channel, which is the most finely sampled. Green also is closely related to luminosity (eg the L channel in Lab) and so I don't think my analysis would differ substantially for luminance noise in an actual raw image.

However, my initial simplistic view is that area under the curve,
which I would think represents the total noise energy, reduces
considerably in the non-Bayer example, and much less in the Bayer
example. I'm not sure if this is fully represented by quoting the
relative widths of the curves.

The area under the Gaussian curve is essentially the std dev times the number of pixels, up to a universal, fixed factor of order one. So we might as well discuss the std dev (width) of the curve. This quantity is a measure of the noise at the Nyquist frequency, and not any other frequency. Noise reduction and interpolation and Bayer interpolation are two varieties of filtering process; they affect the noise power at and near Nyquist, but not at lower frequencies. What needs to be understood is that the comparison of histogram widths before and after downsampling is a comparison of noise power at two different spatial frequencies; those two noise amounts are related by the scaling that DxO is using in the case that there hasn't been any filtering (ie the noise from pixel to pixel is uncorrelated), but not if there has been filtering.

I would argue that a better measure of the noise than the histogram width, is the slope of the noise power spectrum in the scaling region below Nyquist, which is unaffected by the filtrations applied by noise and debayering filters. By measuring noise only at Nyquist via the width of the histogram, one rewards noise reduction filtering that takes a high noise image and reduces the noise power at Nyquist by making a blotchy mess. Since the slope of the power spectrum in the scaling region is not affected by such trickery, it is a more accurate gauge of the noise in the image. The noise figure arrived at by analyzing raw data (as do Roger Clark, John Sheehy, I and others, including DxO) is closer to this measure of image noise than other methods commonly in use.

--
emil
--



http://theory.uchicago.edu/~ejm/pix/20d/