Noise data in reviews and luminosity

In Phil's D100 and S2 reviews he compares the camera's noise levels by using the Std.Dev of the luminosity channel. The problem was that I could not see the vastly different noise levels the number said should be there.

I then began to wonder if the luminosity channel is the most appropriate way to compare noise. Theoretically if you had a square made up of small red, green, and blue boxes, but carefully selcted the colors so that the luminosities were the same then there should be 0 Std.Dev in the lum histogram.

So I decided to test it!!



First I made two squares - both started at 50% grey, but to the first one I added 12.5% noise with monochromatic checked and the second I added the same 12.5% but in color.

Here is the histograms for the monochromatic version:



You can see the the Std.Dev is around 31 in the lum, red, green and blue channels.

Next we do the same to the color noise sample:



Wow! the luminosity Std.Dev drops to 21.5. BUT going further and looking at the color channels show that their Std.Dev are still around 31!

So although the Std.Dev of the luminosity shows that there is less noise in the color-noise sample there is actually the same amount of noise it is now "color noise"

How does this work in the real world? See next post!

To see how this might have affected the number in the review I grabbed the middle grey patches from the S2 and D100 ISO200 tests.

Here is the D100:



Here the Luminosity is 1.2 and the colors range from 1.25 to 1.36

The S2:



Here the luminosity is 1.08, but the colors range from 1.39 to 2.15

So what does this mean? (Aside from the fact that my family is visiting relatives and I am left with too much time on my hand?) It really on shows that the luminoisty is only part of the noise picture and the final comparison really needs to come down to what looks best on the printed page!

John,

Why wasn't the S2's ISO 100 compared to D100 ISO 200. To me, one should compare the best of both cameras, this is a more fair comparison.

Your comparisons by the way look impressive, way over my head.

Have a good one.

Wow.. impressive questioning of testing method and statistical results.. May be Phil should revise his tests..

Look at the D60 results too and you will get the same difference..

Michael

John Isgren said:

So what does this mean? (Aside from the fact that my family is
visiting relatives and I am left with too much time on my hand?)
It really on shows that the luminoisty is only part of the noise
picture and the final comparison really needs to come down to what
looks best on the printed page!

Click to expand...

Congratulations, John, on raising some fascinating issues and making a strong case.

It would be a wonderful resource to have the same comparison posted to this thread, comparing the D100 and the D60, and (since we can't quit making unreasonable requests of each on this forum ... ) comparing the raw modes of the D100, the S2 and the D60.

Anyone have the time and tools to run the numbers on hand this weekend? (Alas, I do not).

So, according to your analysis, the D100 is the "color noise" winner of the patch in question ... and perhaps across the board?

Best Regards

John Isgren said:

To see how this might have affected the number in the review I
grabbed the middle grey patches from the S2 and D100 ISO200 tests.

(Aside from the fact that my family is
visiting relatives and I am left with too much time on my hand?)
It really on shows that the luminoisty is only part of the noise
picture and the final comparison really needs to come down to what
looks best on the printed page!

Click to expand...

I would not say that any camera is the clear winner in the noise arena. The point of the analysis was not to show a winner or loser but rather to point out that there is not as much difference in the cameras as the luminosity numbers show. Which is better color noise or luminosity noise? Color noise may be easier to deal with with LAB adjustments, whereas luminoisty noise looks more like traditional film.

Also I hope that this dose not come off like I am attacking Phil's reviews! I believe Phil dose a fantasic job and love this forum. All I am suggesting tis that herxpand the noise comparison to take into account the noise in the color channels. When you take that into account I think you will see all three cameras ave a very simlar plot of noise versus ISO.

Cribbage said:

It would be a wonderful resource to have the same comparison posted
to this thread, comparing the D100 and the D60, and (since we can't
quit making unreasonable requests of each on this forum ... )
comparing the raw modes of the D100, the S2 and the D60.

Anyone have the time and tools to run the numbers on hand this
weekend? (Alas, I do not).

So, according to your analysis, the D100 is the "color noise"
winner of the patch in question ... and perhaps across the board?

Best Regards

John Isgren said:

To see how this might have affected the number in the review I
grabbed the middle grey patches from the S2 and D100 ISO200 tests.

(Aside from the fact that my family is
visiting relatives and I am left with too much time on my hand?)
It really on shows that the luminoisty is only part of the noise
picture and the final comparison really needs to come down to what
looks best on the printed page!

Click to expand...

Click to expand...

John Isgren said:

Also I hope that this dose not come off like I am attacking Phil's
reviews! I believe Phil dose a fantasic job and love this forum.
All I am suggesting tis that herxpand the noise comparison to take
into account the noise in the color channels. When you take that
into account I think you will see all three cameras ave a very
simlar plot of noise versus ISO.

Click to expand...

Nice post, and I agree. Especially if you control for exposure (the closest we can come to making sure the same number of light photons hit the photosite without some pretty darn sophisticated lab equipment).

There's far more to be tested here, too. Recently I've tried similar tests while controlling the color temperature of the light hitting the target (including near infrared). This quickly confirmed another thing I've been suspecting: there are some subtle noise response differences in the cameras at the very fringes of visible spectrum--get those doubled up green photosites out of the picture (pardon the pun), and I see slightly different noise results. Whether this shows up in pictures or not, I've yet to form any conclusion, but those of you who only shoot only candlelight...

--
Thom Hogan
author, Nikon Field Guide
author, Nikon Flash Guide
author, Complete Guide to the Nikon D100
author, Complete Guide to the Nikon D1, D1h, & D1x
http://www.bythom.com

Hi

Beside the fact that phils noise values over different review look curious to me, you gave a good explanation of my "feel" that there isn't anything interesting in noise level between D100 and S2 but that the noise looks different, from D100 more monochromatic and from S2 tending to be more coloured mottled... in the end simply different but not any better.

In addition I believe that temperature has a much more dramatic effect on all this then the difference between these two cameras are...

regards, A. Schiele.

I agree. I think that everyone, including Phil, would benefit from taking a more circumspect attitude, and refrain from drawing strong and/or absolutist conclusions from such a limited body of testing data.

Still interested in how the D60 looks under this analysis.

John Isgren said:

Also I hope that this dose not come off like I am attacking Phil's
reviews! I believe Phil dose a fantasic job and love this forum.
All I am suggesting tis that herxpand the noise comparison to take
into account the noise in the color channels. When you take that
into account I think you will see all three cameras ave a very
simlar plot of noise versus ISO.

Cribbage said:

It would be a wonderful resource to have the same comparison posted
to this thread, comparing the D100 and the D60, and (since we can't
quit making unreasonable requests of each on this forum ... )
comparing the raw modes of the D100, the S2 and the D60.

Anyone have the time and tools to run the numbers on hand this
weekend? (Alas, I do not).

So, according to your analysis, the D100 is the "color noise"
winner of the patch in question ... and perhaps across the board?

Best Regards

John Isgren said:

To see how this might have affected the number in the review I
grabbed the middle grey patches from the S2 and D100 ISO200 tests.

(Aside from the fact that my family is
visiting relatives and I am left with too much time on my hand?)
It really on shows that the luminoisty is only part of the noise
picture and the final comparison really needs to come down to what
looks best on the printed page!

Click to expand...

Click to expand...

Click to expand...

Nice post--a well conceived test and very clearly explained. I definitely agree with you and the others that there are quite a few variables that would have to be accounted for if one were to try to make very precise measurements of noise. Whether such precision makes any difference in image quality under real world photographic conditions is another question.

Point taken, and we may well add something about chroma noise to reviews in the future (although it will vastly increase the amount of work I will need to do). One point, you shouldn't be using the published crops to measure noise, they have been resaved as JPEG and thus will contain edge JPEG artifacts which will affect your measurements. Only the original JPEG's or TIFF's (from RAW) can be used for measurement.

John Isgren said:

In Phil's D100 and S2 reviews he compares the camera's noise levels
by using the Std.Dev of the luminosity channel. The problem was
that I could not see the vastly different noise levels the number
said should be there.

Click to expand...

Once more, you can not use the posted crops as they will contain edge JPEG artifacts, you need the original shots to measure noise levels. The crops are provided for visual inspection only.

Cribbage said:

It would be a wonderful resource to have the same comparison posted
to this thread, comparing the D100 and the D60, and (since we can't
quit making unreasonable requests of each on this forum ... )
comparing the raw modes of the D100, the S2 and the D60.

Anyone have the time and tools to run the numbers on hand this
weekend? (Alas, I do not).

So, according to your analysis, the D100 is the "color noise"
winner of the patch in question ... and perhaps across the board?

Best Regards

John Isgren said:

To see how this might have affected the number in the review I
grabbed the middle grey patches from the S2 and D100 ISO200 tests.

(Aside from the fact that my family is
visiting relatives and I am left with too much time on my hand?)
It really on shows that the luminoisty is only part of the noise
picture and the final comparison really needs to come down to what
looks best on the printed page!

Click to expand...

Click to expand...

--
Phil Askey
Editor / Owner, dpreview.com

Measuring noise objectively must be very difficult.

Mike Chaney pointed out a while ago already that the luminance noise on the S2 was very low, but that the color noise was higher than for D100.

Basing it on luminance noise makes sense. Luminance noise also affects all areas in a picture, the light, the average and the darker areas. Color noise, is less visible in light and dark areas, but mostly visible in grey toned mid areas.

Look at the comparison between the S2 and the D100, and you will notice the chrominance noise very clearly in the mid grey areas, but not in the white and black areas, however, luminance noise does show up everywhere. Look at the black areas of the D100, they are not even black anymore due to the noise.
I am not saying that one is more correct than another.

I also believe that from looking at the tests, I believe the D60 has the lowest noise in lower ISO ranges.

(In case you are interested my choise...I have an S2 on order.)
Regards,
Jos

I agree Phil that the crop could have a different amount of noise due to the compression, but I was more intersted in the difference between the luminosity and color noise which does not seem to be affected by the jpegs.

Phil Askey said:

John Isgren said:

In Phil's D100 and S2 reviews he compares the camera's noise levels
by using the Std.Dev of the luminosity channel. The problem was
that I could not see the vastly different noise levels the number
said should be there.

Click to expand...

Click to expand...

JPEG throws away information, especially chroma.

John Isgren said:

Phil Askey said:

John Isgren said:

In Phil's D100 and S2 reviews he compares the camera's noise levels
by using the Std.Dev of the luminosity channel. The problem was
that I could not see the vastly different noise levels the number
said should be there.

Click to expand...

Click to expand...

Click to expand...

--
Phil Askey
Editor / Owner, dpreview.com

Phil, unless you consider it valuable, proprietary intellectual property, perhaps you could make available your test patches in a lossless format.

There are obviously highly educated, experienced and intelligent people in this forum with good ideas and time to devote to these issues.

You said yourself that the color channel noise might make a good test for future reviews. It seems like coughing up the original image files might help people here give you additional good ideas and guidance.

Phil Askey said:

John Isgren said:

Phil Askey said:

John Isgren said:

In Phil's D100 and S2 reviews he compares the camera's noise levels
by using the Std.Dev of the luminosity channel. The problem was
that I could not see the vastly different noise levels the number
said should be there.

Click to expand...

Click to expand...

Click to expand...

--
Phil Askey
Editor / Owner, dpreview.com

Click to expand...

In real life test the D100 ISO 1600 shot looks much better than the S2:

D100:



S2:

Forget to include the shooting data:

AF-S 300/4D F/5.6 1/4 S2 shot is taken at 4258x2848 and resized to 3008x2013. All in Jpeg format.

And here are the ISO 400 shots.

D100:



S2:

John Isgren said:

In Phil's D100 and S2 reviews he compares the camera's noise levels
by using the Std.Dev of the luminosity channel. The problem was
that I could not see the vastly different noise levels the number
said should be there.

I then began to wonder if the luminosity channel is the most
appropriate way to compare noise. Theoretically if you had a
square made up of small red, green, and blue boxes, but carefully
selcted the colors so that the luminosities were the same then
there should be 0 Std.Dev in the lum histogram.

So I decided to test it!!

Click to expand...

Well done John, I actually came to the same conclusion after seeing the first high ISO beta shots a few months back, I looked at comparison images and the D100 shots always "look" less noisy, this I think is the case because of the lower color noise levels. I still think that looking at the noise levels between the D100, D60 and S2 to see any difference in noise in print at comparable ISO's would require huge prints, on the order of screen size equivalents (42" x 28") I have printed a beta D100 shot at 20" x 30" and it has a very film like grain to it, with zero color noise but very subtle luma noise.I think the color noise would betray the images digital origins if it were visible. I think the lack of color noise as you imply, leads to a film like print that would require post processing (to remove the color noise) to achieve using the D60 and S2 but again, a difference only to be seen at huge print sizes. I would estimate that for prints less than "huge" (I'd say 20 x 30" and smaller) the three announced DSLR's are close enough in noise performance to provide identical results, after that differences in resolution and noise quality (color versus luma noise) will become more apparent and may require handling to achieve optimal results depending on the camera used.

Regards,

--