From DXOMark: More pixels offsets noise3

Started May 30, 2009 | Discussions
Steen Bay Veteran Member • Posts: 7,418
Re: Another possible issue..

bobn2 wrote:

Steen Bay wrote:

bobn2 wrote:

Steen Bay wrote:

John Sheehy wrote:

Steen Bay wrote:

I actually wondered a bit about that, because 40D only has ISO 3200
when "H", "ISO expansion" is activated. Is ISO 3200 a 'real' ISO on
40D, or just a 'pushed' ISO 1600? Doesn't that make a difference?

The actual exposure is what makes the real difference.

The 40D shot is 1/400 and the 50D shot is 1/500, so, unless something
in the setup gave different illumination, the 50D may be exposed 1/3
stop less.

Yes, that would be a disadvantage for the 50D, but the brightness
level seems pretty much the same. If anything the 50D image actually
seems a tab brighter. But suppose it's possible that 50D uses a
different tonecurve.

Not in this case, the images are from raw and the tone curve applied
in dcraw. It's the default straight line, hence PIX's comments about
the less than optimum conversion.

When comparing the detail/sharpness from high resolution cameras like
40D and 50D, then it's pretty important that the focus plane is
exactly the same, which (looking at the comparisons in the DPR
reviews) seems to be a bit difficult to achieve.

The time taken to set up a perfectly comparable test could be long.
Interesting to note, however that both the 50D/40D and D3/D3x
examples clearly show the detail advantage of the downsampled high
res camera over the native low res camera. I'm sure every such test
would do so - you'd just have to ignore the noise differences of very
different sensor technologies.

Yes, I'm sure that the 50D is a better 10mp camera than the 40D, as you put it in another post, but it's also possible that you could find another crop from the same IR images, that wouldn't show that.

bobn2
bobn2 Forum Pro • Posts: 61,446
Re: Regarding Pixmantra's stand on DXOMark..

photo nuts wrote:

I have to agree some of the quantitative results do not agree with
real world shooting experience.

For example, I have owned and used the Canon 400D and 450D quite
extensively. In their analysis, the 400D is equal, if not slightly
better than the 450D at high ISO. I call that BS. The 400D produces
white speckles in the image at high ISO. This type of random noise is
extremely hard to remove while the 450D is free from such ugly white
spots. Even the-digital-picture has the same findings when they
convert their RAW images. But somehow this was not captured in
DXOMark's quantitative tests.

I do not understand this.

But it does show some kind of serious flaw in DXOMark's tests.

No it doesn't. DxO's tests measure what they say they measure, and ugliness of the noise pattern is not one of them. Their low light ISO rating is the highest ISO that gives a 30dB normalised signal to noise ratio. It says nothing about the quality of the noise in that SNR. In any case, the quality of the noise changes enormously depending on the converter you use. DxO tests are done on the raw data, before conversion. DxO rates the 450D slightly better than the 400D (692-664). How much of that 'better' is your ugly white dots and how much your dots could be dealt with using a different converter, I don't know.
--
Bob

bobn2
bobn2 Forum Pro • Posts: 61,446
Re: Another possible issue..

Steen Bay wrote:

bobn2 wrote:

Steen Bay wrote:

bobn2 wrote:

Steen Bay wrote:

John Sheehy wrote:

Steen Bay wrote:

I actually wondered a bit about that, because 40D only has ISO 3200
when "H", "ISO expansion" is activated. Is ISO 3200 a 'real' ISO on
40D, or just a 'pushed' ISO 1600? Doesn't that make a difference?

The actual exposure is what makes the real difference.

The 40D shot is 1/400 and the 50D shot is 1/500, so, unless something
in the setup gave different illumination, the 50D may be exposed 1/3
stop less.

Yes, that would be a disadvantage for the 50D, but the brightness
level seems pretty much the same. If anything the 50D image actually
seems a tab brighter. But suppose it's possible that 50D uses a
different tonecurve.

Not in this case, the images are from raw and the tone curve applied
in dcraw. It's the default straight line, hence PIX's comments about
the less than optimum conversion.

When comparing the detail/sharpness from high resolution cameras like
40D and 50D, then it's pretty important that the focus plane is
exactly the same, which (looking at the comparisons in the DPR
reviews) seems to be a bit difficult to achieve.

The time taken to set up a perfectly comparable test could be long.
Interesting to note, however that both the 50D/40D and D3/D3x
examples clearly show the detail advantage of the downsampled high
res camera over the native low res camera. I'm sure every such test
would do so - you'd just have to ignore the noise differences of very
different sensor technologies.

Yes, I'm sure that the 50D is a better 10mp camera than the 40D, as
you put it in another post, but it's also possible that you could
find another crop from the same IR images, that wouldn't show that.

The choice of crop is quite important. PIX was basing his assessment on the teacup part, where the localised exposure is quite different due to differences in the set-up of the test. The background, where he was judging the noise, was much darker for the 50D. Moreover, the shot noise we're comparing is differently visible for different brightnesses. In the bright areas, the photon count is high enough that the signal masks the shot noise, even though it is high. In the very dark areas, the shot noise is very low, because it's the root of the number of photons. Therefore, the important areas for this assessment are the mid to dark areas. Of course, you can't judge detail where there is no detail to judge, but I can't think of a part of the image where the 40D would appear more detailed, and that crop is, I think, the one with most visible shot noise, so I don't think there are any crops which would reverse the order (whatever your own order is, as I said, I think they're too close to call in terms of noise at 10MP).

All these are the pitfalls of subjective assessment, which is why I prefer quantitative evaluation - if you know how to use it, you can gain a godd idea of how things will come out much quicker.

-- hide signature --

Bob

DSPographer Senior Member • Posts: 2,466
Re: Bob..

Kuivaamo wrote:

I'm aware of all this, hence me placing "outresolve" in inverted
commas. I would expect much higher lp/mm results in all the lens
tests if lenses really were diffraction-limited at f8. There's no
reason why an aberration-free optic would resolve less at f8 than
f5.6 on a 50D; with a high contrast target the 50D can resolve only
about 70% of what I'd expect an ideal lens to deliver at f8.

The only lens I listed that was reviewed by this site is the 50/1.8. It showed maximum resolution at f4.5 not f/8 and it had greater that Nyquist MTF50 on the 450D even though the camera anti-aliasing filter should ideally been removing all content above the Nyquist frequency.

 DSPographer's gear list:DSPographer's gear list
Canon EOS 5D Mark II Canon EOS M5 Canon EF 24mm f/2.8 Canon EF 50mm f/1.8 II Canon EF 200mm f/2.8L II USM +4 more
Steen Bay Veteran Member • Posts: 7,418
Re: Another possible issue..

bobn2 wrote:

Steen Bay wrote:

bobn2 wrote:

Steen Bay wrote:

bobn2 wrote:

Steen Bay wrote:

John Sheehy wrote:

Steen Bay wrote:

I actually wondered a bit about that, because 40D only has ISO 3200
when "H", "ISO expansion" is activated. Is ISO 3200 a 'real' ISO on
40D, or just a 'pushed' ISO 1600? Doesn't that make a difference?

The actual exposure is what makes the real difference.

The 40D shot is 1/400 and the 50D shot is 1/500, so, unless something
in the setup gave different illumination, the 50D may be exposed 1/3
stop less.

Yes, that would be a disadvantage for the 50D, but the brightness
level seems pretty much the same. If anything the 50D image actually
seems a tab brighter. But suppose it's possible that 50D uses a
different tonecurve.

Not in this case, the images are from raw and the tone curve applied
in dcraw. It's the default straight line, hence PIX's comments about
the less than optimum conversion.

When comparing the detail/sharpness from high resolution cameras like
40D and 50D, then it's pretty important that the focus plane is
exactly the same, which (looking at the comparisons in the DPR
reviews) seems to be a bit difficult to achieve.

The time taken to set up a perfectly comparable test could be long.
Interesting to note, however that both the 50D/40D and D3/D3x
examples clearly show the detail advantage of the downsampled high
res camera over the native low res camera. I'm sure every such test
would do so - you'd just have to ignore the noise differences of very
different sensor technologies.

Yes, I'm sure that the 50D is a better 10mp camera than the 40D, as
you put it in another post, but it's also possible that you could
find another crop from the same IR images, that wouldn't show that.

The choice of crop is quite important. PIX was basing his assessment
on the teacup part, where the localised exposure is quite different
due to differences in the set-up of the test. The background, where
he was judging the noise, was much darker for the 50D. Moreover, the
shot noise we're comparing is differently visible for different
brightnesses. In the bright areas, the photon count is high enough
that the signal masks the shot noise, even though it is high. In the
very dark areas, the shot noise is very low, because it's the root of
the number of photons. Therefore, the important areas for this
assessment are the mid to dark areas. Of course, you can't judge
detail where there is no detail to judge, but I can't think of a part
of the image where the 40D would appear more detailed, and that crop
is, I think, the one with most visible shot noise, so I don't think
there are any crops which would reverse the order (whatever your own
order is, as I said, I think they're too close to call in terms of
noise at 10MP).
All these are the pitfalls of subjective assessment, which is why I
prefer quantitative evaluation - if you know how to use it, you can
gain a godd idea of how things will come out much quicker.

I haven't downloaded the IR images, but in the DPR iso100 studio 40D vs. 50D comparison, for example the Kodak gray scale is much sharper in the 40D image. (Both upsized to 24mp, bicubic without gaussian blur :))

bobn2
bobn2 Forum Pro • Posts: 61,446
Re: Another possible issue..

Steen Bay wrote:

I haven't downloaded the IR images, but in the DPR iso100 studio 40D
vs. 50D comparison, for example the Kodak gray scale is much sharper
in the 40D image. (Both upsized to 24mp, bicubic without gaussian
blur :))

Somehow, that doesn't surprise me at all.
--
Bob

John Sheehy Forum Pro • Posts: 21,744
Re: Bob..

DSPographer wrote:

The only lens I listed that was reviewed by this site is the 50/1.8.
It showed maximum resolution at f4.5 not f/8 and it had greater that
Nyquist MTF50 on the 450D even though the camera anti-aliasing filter
should ideally been removing all content above the Nyquist frequency.

The 450D seems to have a much weaker AA filter than its predecessors.

The same is true of the 50D.

By weaker, I mean relative to the pixel pitch; not just absolute radius.

-- hide signature --

John

John Sheehy Forum Pro • Posts: 21,744
Re: Yes, a bit tricky..

Steen Bay wrote:

I still use my 'old' 40D. Can't afford, or justify, to update every
year, but I'm sure that the 50D is the better camera, and I'm looking
forward to a 18-20mp 60D, which I'll probably get. And yes, comparing
cameras and images can be a bit tricky, which all these long threads
should prove

Well, there are a lot of logical traps to fall into. Most people seem to have a very difficult time adapting one paradigm when they've already adapted another, and are incapable of recognizing the superioirity of the new one.

If we had 200MP monitors, we would be viewing the output of all cameras (and crops we make from them, unless very small and intended for small web images) full-screen on the monitor. Cameras which had many more pixels, but softer and noisier at the pixel level, would look more detailed in a more natural, non-aliased way, and would be less noisy or the same in most cases.

Printers get us closer to reality (when printing images at the same size), but they are still flawed in some ways.

Now, higher pixel counts suggest that you may be able to crop deeper or display/print larger, as is sometimes the case, but the ability to do so is above and beyond the standardized concept of image quality. People who expect the same quality doing so are unrealistic; only the most superior optics, focus, and camera stability, and shooting at low ISOs, allow that. And the results must be compared to cropping a lower-density camera just as deeply, or displaying the image just as large. To use the pixel level as the limit for comparing cropping or maximum display size is nonsense; the "unity" is an illusion.

Is there really a problem with camera B, because camera A looks better per unit of displayed area one foot from your face printed much smaller? That's ridiculous, yet that is the foundation of the pixel-centrics.

-- hide signature --

John

Stan Prevost Contributing Member • Posts: 650
Re: Resampling..

Been a while, but I believe the frequency response of the gaussian blur does not have a very steep rolloff. So to get significant attenuation at Nyquist and above, there will be significant attenuation at the higher spatial frequencies below Nyquist, thus the visible loss of detail sometimes reported when downsampling with blur. A higher order filter of a different type of transfer function having a steeper transition between passband and stopband would allow preserving more detail while preventing most aliasing, but possibly at the expense of some artifacts.

Stan

John Sheehy Forum Pro • Posts: 21,744
Re: Interesting..

Steen Bay wrote:

It's actually pretty interesting (and a bit confusing :)) to see
images compared at so many different output (or simulated print)
sizes. To see how much the impression of noise/detail in the image
depends on the 'magnification'.

In every case the 50D has more sharp detail. This means the 50D needs to be sharpened less, which also means that the 50D will have less noise relative to the 40D, which needs to be sharpened more, at high image frequencies.

-- hide signature --

John

Stan Prevost Contributing Member • Posts: 650
Re: Resampling..

When I posted, I forgot that Bryan van der Wolf's site that I referenced earlier had examples of different antialiasing prefilters used in ImageMagick. But nothing is given about the parameters of those filters. I guess one could dig through ImageMagick source code if sufficiently motivated. Nevertheless, they provide some off-the-shelf options to try on any given image.

bobn2
bobn2 Forum Pro • Posts: 61,446
Re: Resampling..

Stan Prevost wrote:

When I posted, I forgot that Bryan van der Wolf's site that I
referenced earlier had examples of different antialiasing prefilters
used in ImageMagick. But nothing is given about the parameters of
those filters. I guess one could dig through ImageMagick source code
if sufficiently motivated. Nevertheless, they provide some
off-the-shelf options to try on any given image.

I might give it a go if I find the time. What I've become interested in as a result of these threads is a resampling raw converter. I reckon you should be able to make a much better job of resampling if you do it direct from Bayer data, and include the HF roloff in the demosaicing cores. So, I might have a look around the ImageMagick functions as a prelude.
--
Bob

John Sheehy Forum Pro • Posts: 21,744
Re: Another possible issue..

bobn2 wrote:

All these are the pitfalls of subjective assessment, which is why I
prefer quantitative evaluation - if you know how to use it, you can
gain a godd idea of how things will come out much quicker.

It takes quite a bit of time, however, for the two to be properly related. Statistics don't always reflect what we are really interested in.

Look how recently people who should know better were relying on standard deviation to determine image DR and noise. Look at how much pattern noise is still ignored in camera analysis.

-- hide signature --

John

jeffkrol Veteran Member • Posts: 6,225
Re: Regarding Pixmantra's stand on DXOMark..

It does point to a need to use both though.
--
360 minutes from the prime meridian. (-5375min, 3.55sec) 1093' above sea level.

'The exposure meter is calibrated to some clearly defined standards and the user needs to adjust his working method and his subject matter to these values. It does not help to suppose all kinds of assumptions that do not exist.'
Erwin Puts

Steen Bay Veteran Member • Posts: 7,418
Re: Yes, a bit tricky..

John Sheehy wrote:

Steen Bay wrote:

I still use my 'old' 40D. Can't afford, or justify, to update every
year, but I'm sure that the 50D is the better camera, and I'm looking
forward to a 18-20mp 60D, which I'll probably get. And yes, comparing
cameras and images can be a bit tricky, which all these long threads
should prove

Well, there are a lot of logical traps to fall into. Most people
seem to have a very difficult time adapting one paradigm when they've
already adapted another, and are incapable of recognizing the
superioirity of the new one.

If we had 200MP monitors, we would be viewing the output of all
cameras (and crops we make from them, unless very small and intended
for small web images) full-screen on the monitor. Cameras which had
many more pixels, but softer and noisier at the pixel level, would
look more detailed in a more natural, non-aliased way, and would be
less noisy or the same in most cases.

Printers get us closer to reality (when printing images at the same
size), but they are still flawed in some ways.

Now, higher pixel counts suggest that you may be able to crop deeper
or display/print larger, as is sometimes the case, but the ability to
do so is above and beyond the standardized concept of image quality.
People who expect the same quality doing so are unrealistic; only the
most superior optics, focus, and camera stability, and shooting at
low ISOs, allow that. And the results must be compared to cropping a
lower-density camera just as deeply, or displaying the image just as
large. To use the pixel level as the limit for comparing cropping or
maximum display size is nonsense; the "unity" is an illusion.

Is there really a problem with camera B, because camera A looks
better per unit of displayed area one foot from your face printed
much smaller? That's ridiculous, yet that is the foundation of the
pixel-centrics.

John

Yes, lots of logical traps to fall into. Assuming that bigger pixels are always better, is a bit (just a bit) like assuming that bigger sensors are always better, and forgetting that when shooting at the same shutterspeed and DOF, all sensor sizes will have pretty much the same IQ, because with the bigger sensor, you'll have to use a higher ISO value and (f-number) to get the same DOF.

bobn2
bobn2 Forum Pro • Posts: 61,446
Re: Another possible issue..

John Sheehy wrote:

bobn2 wrote:

All these are the pitfalls of subjective assessment, which is why I
prefer quantitative evaluation - if you know how to use it, you can
gain a godd idea of how things will come out much quicker.

It takes quite a bit of time, however, for the two to be properly
related. Statistics don't always reflect what we are really
interested in.

I agree with that, the trick is to get the ones that do. Also, to learn how to deduce what we are interested in.

Look how recently people who should know better were relying on
standard deviation to determine image DR and noise. Look at how much
pattern noise is still ignored in camera analysis.

Was that me? Could have been, like many my understanding of these things has developed bit by bit, helped by contributions from your good self and several others. There's still more to be done. Pattern noise is an interesting one. There should be a way of quantifying it, probably not as a simple scalar. The interesting thing is, when a figure can be put on a quantity, and gains currency for assessment, manufacturers start improving it. That was one of the threats of DPR's bogus pixel density measure - if manufacturers had started 'improving' it the way DPR wanted, we'd all have lost. So far as pattern noise is concerned, once a simple figure of merit which has a good correlation with its visible objectionableness is developed, manufacturers will try to make sure they score well on that measure.

-- hide signature --

Bob

bobn2
bobn2 Forum Pro • Posts: 61,446
Re: Yes, a bit tricky..

Steen Bay wrote:

John Sheehy wrote:

Steen Bay wrote:

I still use my 'old' 40D. Can't afford, or justify, to update every
year, but I'm sure that the 50D is the better camera, and I'm looking
forward to a 18-20mp 60D, which I'll probably get. And yes, comparing
cameras and images can be a bit tricky, which all these long threads
should prove

Well, there are a lot of logical traps to fall into. Most people
seem to have a very difficult time adapting one paradigm when they've
already adapted another, and are incapable of recognizing the
superioirity of the new one.

If we had 200MP monitors, we would be viewing the output of all
cameras (and crops we make from them, unless very small and intended
for small web images) full-screen on the monitor. Cameras which had
many more pixels, but softer and noisier at the pixel level, would
look more detailed in a more natural, non-aliased way, and would be
less noisy or the same in most cases.

Printers get us closer to reality (when printing images at the same
size), but they are still flawed in some ways.

Now, higher pixel counts suggest that you may be able to crop deeper
or display/print larger, as is sometimes the case, but the ability to
do so is above and beyond the standardized concept of image quality.
People who expect the same quality doing so are unrealistic; only the
most superior optics, focus, and camera stability, and shooting at
low ISOs, allow that. And the results must be compared to cropping a
lower-density camera just as deeply, or displaying the image just as
large. To use the pixel level as the limit for comparing cropping or
maximum display size is nonsense; the "unity" is an illusion.

Is there really a problem with camera B, because camera A looks
better per unit of displayed area one foot from your face printed
much smaller? That's ridiculous, yet that is the foundation of the
pixel-centrics.

John

Yes, lots of logical traps to fall into. Assuming that bigger pixels
are always better, is a bit (just a bit) like assuming that bigger
sensors are always better, and forgetting that when shooting at the
same shutterspeed and DOF, all sensor sizes will have pretty much the
same IQ, because with the bigger sensor, you'll have to use a higher
ISO value and (f-number) to get the same DOF.

...and therein lies another trap. If you don't have a constraint on the shutter speed (like a really good tripod and no movement in the image) then in absolute terms you can gather more photons with the bigger sensor, and get a better image.

The corollary of the James equivalence principle that is also commonly misunderstood is that for a given output size, no format has an advantage with respect to diffraction blurring. You get the same blur for a given DoF with any format (and big pixels make it worse, not better).

-- hide signature --

Bob

Stan Prevost Contributing Member • Posts: 650
Some more filtering/resampling

http://www.dylanbeattie.net/magick/filters/result.html

http://www.digitalcamerasdatabase.com/forum/35mm-photography-forum/16026-downsampling-methods.html (interesting conclusions By Bryan van der Wolf that preblurring is not necessary in ImageMagick but is in Photoshop). Hmmm...

BTW, here is the Gaussian blurring done by CS3 for a single white pixel in a field of black, radius = 1, just one quadrant.

0
1 0 0
7 4 1 0
22 14 4 0
36 22 7 1 0

(sums to 248)

John Sheehy Forum Pro • Posts: 21,744
Re: Yes, a bit tricky..

Steen Bay wrote:

Yes, lots of logical traps to fall into. Assuming that bigger pixels
are always better, is a bit (just a bit) like assuming that bigger
sensors are always better, and forgetting that when shooting at the
same shutterspeed and DOF, all sensor sizes will have pretty much the
same IQ, because with the bigger sensor, you'll have to use a higher
ISO value and (f-number) to get the same DOF.

That's assuming that we want or need the DOF that a P&S forces upon us.

In real life shooting, we are not always aiming for equivalencies. Much of my wildlife photography looks for the sweet spot of lens sharpness vs exposure. This means 2/3 - 1.3 stops down from wide-open with my most frequently-used lens.

Also, DSLR tend to have lower read noise at high ISOs. Even when equalizing for DOF, my 5D2 has a few stops less read noise than my P&S cameras. The equivalency only applies to shot noise (which is actually slightly lower for the P&S in DOF equivalency).

-- hide signature --

John

Steen Bay Veteran Member • Posts: 7,418
Re: Yes, a bit tricky..

John Sheehy wrote:

Steen Bay wrote:

Yes, lots of logical traps to fall into. Assuming that bigger pixels
are always better, is a bit (just a bit) like assuming that bigger
sensors are always better, and forgetting that when shooting at the
same shutterspeed and DOF, all sensor sizes will have pretty much the
same IQ, because with the bigger sensor, you'll have to use a higher
ISO value and (f-number) to get the same DOF.

That's assuming that we want or need the DOF that a P&S forces upon us.

In real life shooting, we are not always aiming for equivalencies.
Much of my wildlife photography looks for the sweet spot of lens
sharpness vs exposure. This means 2/3 - 1.3 stops down from
wide-open with my most frequently-used lens.

Also, DSLR tend to have lower read noise at high ISOs. Even when
equalizing for DOF, my 5D2 has a few stops less read noise than my
P&S cameras. The equivalency only applies to shot noise (which is
actually slightly lower for the P&S in DOF equivalency).

If you're shooting at ISO 250 or above with your 5D2, then you could just as well use a 1,6x crop camera at 1,3 stops lower ISO (and 1,3 stops lower f-number), assuming that such a 21mp APS-C camera was available ..but of course, if you want to shoot at that particular f-stop ('sweet spot' of the lens), then it wouldn't work.

Keyboard shortcuts:
FForum MMy threads