Unbiased noise test.

Started Feb 28, 2009 | Discussions
landmarks4711 Contributing Member • Posts: 906
Brilliant !!!! Includes the Downsample

maluminas wrote:

Hello DPR!

I have been lurking here for many many years, but never created an
account because for some reason DPR did not allow registration with
web emails such as Gmail. But i checked recently and now they do so
here i am, with a gift.

An unbiased noise test. Or as unbiased as i could possibly make it.

As everyone else, i have been frustrated by disinformation stemming
from religious brand fanatism. So i decided to make a noise test that
everyone would agree on, though i know this is impossible, but here's
my best try, anyway.

I post it on the Nikon forums, because i own a Nikon camera, but i'm
sure the other camps will see it too.

It stars the latest high-MP cameras, the D3X, the A900 and the 5D2. I
also included the D3, to see if it is still the high-ISO king. The
1Ds3 makes an appearance, because i was just curious to see if there
is any visible improvement in the 5D2, which uses a very similar
sensor. The D300 insisted on having a cameo, because its the camera i
own, but it also has the densest sensor of the selection, just for
fun.

The test was done in two parts. A "pixel performance" test, using
crops from full-size images, to assess the performance of individual
pixels regardless of density. The second part is made using crops
from images resized to 12MP. I call it the "print performance" test
because it simulates a comparison of the final output on paper.

I consider both methods to be valid, they just provide different
conclusions.

Here is the description of my method.

METHODOLOGY:
1) Download "bottle test scene" RAW files from Imaging Resource.
(Many thanks! Sorry for the bandwidth spike!)
2) Load all of them in Bibble 5 (pre-release, see notes below).
3) Apply same profile to all RAWs. (no NR, no sharpening, portrait
color profile, no lens corrections, "As Shot" WB).
4) Carefully adjust exposure compensation of each file using
rightmost green spike on the histograms as a reference. Fine-tune by
eye, very painstakingly...
5) When exposure consistency across all files is satisfactory, batch
to JPEG (highest quality).
6) For the print simulation only: use IrfanView's batch conversion
function to downsample to D3 resolution (Lanczos filter used for best
quality). I reused the full-size crops for the D3 and D300 since the
differences would have been negligible, plus it saved time...
7) Crop two 100x100 squares on each files (save as PNG). One from the
WhiBal card at the top, one from the center of the black cup at the
lower right.
8) Assemble all crops in GIMP, along with proper labels.
9) Include method at the bottom because this is going on DPR and we
all know what happens to tests like these on DPR when there is any
hint of doubt...
10) Export to JPEG, 90% quality.
11) Post on DPR
12) Put on flame-proof suit and sit in a bath of flame retardant.

Now i understand the exposure isn't EXACTLY the same on each crop,
but i decided to stop adjusting before i lost my sanity.

Notes on Bibble 5 pre-release: Why did i choose it? Because it
doesn't perform hidden operations on the images, and because it is
the only converter i found that could convert all the RAWs from these
very recent cameras. It is very promising, but since its a
pre-release test edition it is plagued by bugs for some people. It
converted the 24MP images in 3 seconds. The interface is very nice.
The new functions are also very nice.

And now! I suspect no one read my elaborate description because they
wanted to see the tests, so here they are!

Draw your own conclusions!

maxz
maxz Veteran Member • Posts: 3,479
Invalid to begin with

The shots from imaging resource don't have same exposure to begin with. that's where most 'authorities' screw up. they don't understand in order to compare, you have to have exact a/s combination at same ISO setting. And that's the whole point of this ISO test.

Max

maluminas wrote:

Hello DPR!

I have been lurking here for many many years, but never created an
account because for some reason DPR did not allow registration with
web emails such as Gmail. But i checked recently and now they do so
here i am, with a gift.

An unbiased noise test. Or as unbiased as i could possibly make it.

As everyone else, i have been frustrated by disinformation stemming
from religious brand fanatism. So i decided to make a noise test that
everyone would agree on, though i know this is impossible, but here's
my best try, anyway.

I post it on the Nikon forums, because i own a Nikon camera, but i'm
sure the other camps will see it too.

It stars the latest high-MP cameras, the D3X, the A900 and the 5D2. I
also included the D3, to see if it is still the high-ISO king. The
1Ds3 makes an appearance, because i was just curious to see if there
is any visible improvement in the 5D2, which uses a very similar
sensor. The D300 insisted on having a cameo, because its the camera i
own, but it also has the densest sensor of the selection, just for
fun.

The test was done in two parts. A "pixel performance" test, using
crops from full-size images, to assess the performance of individual
pixels regardless of density. The second part is made using crops
from images resized to 12MP. I call it the "print performance" test
because it simulates a comparison of the final output on paper.

I consider both methods to be valid, they just provide different
conclusions.

Here is the description of my method.

METHODOLOGY:
1) Download "bottle test scene" RAW files from Imaging Resource.
(Many thanks! Sorry for the bandwidth spike!)
2) Load all of them in Bibble 5 (pre-release, see notes below).
3) Apply same profile to all RAWs. (no NR, no sharpening, portrait
color profile, no lens corrections, "As Shot" WB).
4) Carefully adjust exposure compensation of each file using
rightmost green spike on the histograms as a reference. Fine-tune by
eye, very painstakingly...
5) When exposure consistency across all files is satisfactory, batch
to JPEG (highest quality).
6) For the print simulation only: use IrfanView's batch conversion
function to downsample to D3 resolution (Lanczos filter used for best
quality). I reused the full-size crops for the D3 and D300 since the
differences would have been negligible, plus it saved time...
7) Crop two 100x100 squares on each files (save as PNG). One from the
WhiBal card at the top, one from the center of the black cup at the
lower right.
8) Assemble all crops in GIMP, along with proper labels.
9) Include method at the bottom because this is going on DPR and we
all know what happens to tests like these on DPR when there is any
hint of doubt...
10) Export to JPEG, 90% quality.
11) Post on DPR
12) Put on flame-proof suit and sit in a bath of flame retardant.

Now i understand the exposure isn't EXACTLY the same on each crop,
but i decided to stop adjusting before i lost my sanity.

Notes on Bibble 5 pre-release: Why did i choose it? Because it
doesn't perform hidden operations on the images, and because it is
the only converter i found that could convert all the RAWs from these
very recent cameras. It is very promising, but since its a
pre-release test edition it is plagued by bugs for some people. It
converted the 24MP images in 3 seconds. The interface is very nice.
The new functions are also very nice.

And now! I suspect no one read my elaborate description because they
wanted to see the tests, so here they are!

Draw your own conclusions!

 maxz's gear list:maxz's gear list
Nikon Coolpix P900 Sony RX10 III Olympus TG-5 Nikon D90 Nikon D800 +66 more
Iliah Borg Forum Pro • Posts: 25,844
Re: Invalid to begin with

in order to compare, you have to have exact a/s combination at same
ISO setting. And that's the whole point of this ISO test.

What is the value and point of ISO tests? Suppose you have two films, both rated ASA100. Are you interested in the performance at the rated speed, or in getting best possible quality from each of the films?

-- hide signature --
OP maluminas Regular Member • Posts: 178
Re: Invalid to begin with

Well, i admit i overlooked this important detail, i wrongly assumed IR would use the exact same conditions for every camera. But as Illiah says, even if a/s were the same for every camera, the stated ISO and actual ISO are different, which has me stumped. I am not a digital imaging expert, i hope it wasn't me you were referring to as an "authority", because i don't pretend to be.

BUT! I still hold on to my view that as imperfect as my test may be, we can still draw interesting conclusions from it.

stasvolik Contributing Member • Posts: 570
Tag (nt)
-- hide signature --

Cheers,
Stas.

 stasvolik's gear list:stasvolik's gear list
Nikon Coolpix 5400 Nikon D300 Nikon 1 J3 Nikon AF Nikkor 180mm f/2.8D ED-IF Sigma 17-50mm F2.8 EX DC OS HSM +6 more
erikstefan Contributing Member • Posts: 821
Tag too
n/t
JimPearce
JimPearce Veteran Member • Posts: 9,196
Very helpful...

whether the indicated ISOs are accurate or not.
--
Jim

 JimPearce's gear list:JimPearce's gear list
Nikon D7100 Nikon D500
Bob Quinn Senior Member • Posts: 2,204
Re: Invalid to begin with

Yes. The A900 doesn't look good at all - and I doubt that even if you had done this test taking Iliah's advice into consideration, that wouldn't change. Thank you for posting.

-- hide signature --

BUT! I still hold on to my view that as imperfect as my test may be,
we can still draw interesting conclusions from it.
--

Iliah Borg Forum Pro • Posts: 25,844
Re: Invalid to begin with

The A900 doesn't look good at all - and I doubt that even if
you had done this test taking Iliah's advice into consideration, that
wouldn't change.

Hm. Why do you think I use A900 than?

-- hide signature --
OP maluminas Regular Member • Posts: 178
Re: Invalid to begin with

In the hands of a knowledgeable photographer, any camera will produce good photographs, no doubt. Some cameras are easier. If i made an exposure mistake with the D3, it would be easier to "rescue" the photo than if i made the same mistake with the A900.

Iliah Borg Forum Pro • Posts: 25,844
Re: Invalid to begin with

If i made an
exposure mistake with the D3, it would be easier to "rescue" the
photo than if i made the same mistake with the A900.

If it is an under*exposure mistake - true.

-- hide signature --
headofdestiny Veteran Member • Posts: 9,226
Re: Invalid to begin with

maluminas wrote:

In the hands of a knowledgeable photographer, any camera will produce
good photographs, no doubt. Some cameras are easier. If i made an
exposure mistake with the D3, it would be easier to "rescue" the
photo than if i made the same mistake with the A900.
--

-- hide signature --

So much for "unbiased."

OP maluminas Regular Member • Posts: 178
Re: Invalid to begin with

hehe

I just drew a conclusion

headofdestiny Veteran Member • Posts: 9,226
Re: Invalid to begin with

I was just teasing. No biggie. Iliah's right, though, about the underexposure with the D3 vs. overexposure with the A900 and how easy it is to fix.

natureview Forum Member • Posts: 62
Re: Unbiased noise test.

No Iliah, but I am always willing to learn from a Pro. Thanks.

Iliah Borg wrote:

Can you share the methods/steps that you use to calibrate your camera
in details?

Yes of course I can. But a question before we start - are you
comfortable to use 4channels from LibRAW?

-- hide signature --

I wish I could capture all the beautiful things in this world.
-------------------------------------------------------------------------------

Iliah Borg Forum Pro • Posts: 25,844
Re: Unbiased noise test.

4channnels - a command-line utility that takes a raw file and splits it into four separate 16-bit grayscale TIFFs (1 per raw channel, R, G1, G2, B).

Command line switches:

  • -s N selects N-th image from RAW with multiple images (not needed for Nikon)

  • -g gamma correction (gamma 2.2) (I do not use it)

  • -A values autoscale by auto-calculated integer factor (not needed for this experiment)

  • -B turn off black subtraction (not needed for Nikon)

  • -N no RAW curve

You will find it included as pre-compiled sample (bin folder) into LibRAW distribution, downloadable from this page (choose your platform):
http://www.libraw.org/download#beta

Mount your camera onto a tripod, use a lens of about 150mm focal length, point the lens axis perpendicular to your monitor screen filled with grey from a 2-3 feet distance, set aperture to f/8, focus the lens to infinity, activate shutter priority, set the sensitivity so that the shutter speed falls below 1/30, shoot, use 4channels, open in Photoshop the G TIFF and note the resulting average raw value in the green channel in the central portion of the frame (I use picker in 101x101 pixels mode and Info in 16-bit mode). Next, increase the exposure by dialing in +5 eV compensation to fully blow out the green channel. Comparing the maximum value of green channel you got for the centre of the frame from the second shot to the value from the first shot you can calculate the grey value to which the camera meter is calibrated.

-- hide signature --
rhlpetrus Forum Pro • Posts: 25,861
to Iliah: midtone or saturation for ISO? Re: Unbiased noise test.

Iliah Borg wrote:

maluminas wrote:

I understand there is variation from model to model, but then where
do we draw the "standard" line?

I draw it with calibrating my cameras the way that they expose
consistently; so that while shooting I know how to compensate. The
idea that one camera has 4 stops between meter and saturation while
another has only 2.5 does not help shooting.

Do you use midtone for ISO calibration or saturation point?

Don't cameras also differ re what meter registers (midtone) and what's actually reproduced in terms of luminosity?

If you use saturation for calibration and ISO is lower than stated, it would mean range from midtone to saturation is wider, not exatly that ISO is actually lower, from midtone POV.

-- hide signature --

Renato.
http://www.flickr.com/photos/rhlpedrosa/
OnExposure member
http://www.onexposure.net/

Good shooting and good luck
(after Ed Murrow)

 rhlpetrus's gear list:rhlpetrus's gear list
Leica D-Lux (Typ 109) Nikon D7000 Nikon 1 V1 Nikon AF-S DX Nikkor 12-24mm f/4G ED-IF Nikon AF Nikkor 35mm f/2D +4 more
natureview Forum Member • Posts: 62
Re: Unbiased noise test.

Thank you Iliah,

I will try it out, thanks again:)

Iliah Borg wrote:

4channnels - a command-line utility that takes a raw file and splits
it into four separate 16-bit grayscale TIFFs (1 per raw channel, R,
G1, G2, B).

Command line switches:

  • -s N selects N-th image from RAW with multiple images (not needed

for Nikon)

  • -g gamma correction (gamma 2.2) (I do not use it)

  • -A values autoscale by auto-calculated integer factor (not needed

for this experiment)

  • -B turn off black subtraction (not needed for Nikon)

  • -N no RAW curve

You will find it included as pre-compiled sample (bin folder) into
LibRAW distribution, downloadable from this page (choose your
platform):
http://www.libraw.org/download#beta

Mount your camera onto a tripod, use a lens of about 150mm focal
length, point the lens axis perpendicular to your monitor screen
filled with grey from a 2-3 feet distance, set aperture to f/8, focus
the lens to infinity, activate shutter priority, set the sensitivity
so that the shutter speed falls below 1/30, shoot, use 4channels,
open in Photoshop the G TIFF and note the resulting average raw value
in the green channel in the central portion of the frame (I use
picker in 101x101 pixels mode and Info in 16-bit mode). Next,
increase the exposure by dialing in +5 eV compensation to fully blow
out the green channel. Comparing the maximum value of green channel
you got for the centre of the frame from the second shot to the value
from the first shot you can calculate the grey value to which the
camera meter is calibrated.

-- hide signature --

I wish I could capture all the beautiful things in this world.
-------------------------------------------------------------------------------

Iliah Borg Forum Pro • Posts: 25,844
Re: to Iliah: midtone or saturation for ISO? Re: Unbiased noise test.

Midtone is percentage of saturation. Say, I want midtone at 13%. For 12 bits it means 13% of 4095 (given the camera actually reaches that value in raw), or 532. Log base 2 (100/13) is the headroom to saturation; 2.94 eV for the 13%.

-- hide signature --
panos_m Senior Member • Posts: 1,507
Re: Unbiased noise test.

Hello Mr. Borg,

I think that I start to understand the procedure, 4 stops for the D3 is it right?
--
Panagiotis

Keyboard shortcuts:
FForum MMy threads