Canon Line Offset Noise

Started Oct 20, 2009 | Discussions
Mark H
Mark H Veteran Member • Posts: 3,803
Re: Some info for cooling down

GaborSch wrote:

I don't see any reason to change this discussion into an argumentation regarding the persons instead of thew views.

I quite agree (surprising though it may seem to some people) - however, when a contributor starts throwing accusations of "shill" and "libel" and corporate "collusion" they should not be in the least surprised when the discussion heads off in that direction. [N.B. I'm not talking about Gabor here].

Now, on to your observation here...

I posted at the very beginning, that I experimented with this quite natural idea. Unfortunately the result is not good. When using a black frame as "template" for black level corrections for another shot with the same camera, same ISO, even if these shots have been made within seconds, the result is not satisfactory: the banding does not disappear completely, but it appears on patches, which did not show banding before.

This observation supports what I refer to in my most recent reply to John's repeated insistences - where I said... "A calibration as such, will not always be repeatable/successful to the degree that it would reduce the issues to the levels you seem to expect." .

Canon undoubtedly have many engineers and specialists in their particular fields of sensor development and signal/image processing, and invest millions in their development - it is truly absurd for some people (not Gabor here) to be suggesting that Canon could easily fix these issues, and that they don't simply because they are complacent and don't care.

Elsewhere in the thread someone else wonders why "...Canon has not implemented the easy fix for these issues" - well, in all probability, the simple answer is that the issues are not so easily fixed at all - otherwise Canon would probably have already done so.

GaborSch Veteran Member • Posts: 7,203
What we should wish from Canon

is, that they make a thorough test of their new products. It is unbelievable what crap they are releasing for sale. Think of the 5D2 black dots. Not a big deal for most users, but it was something Canon should not have left unaddressed.

Regarding the 7D banding, I have the feeling that it is the firmware's doing, at least partly. The top masked (?) area looks so wild and wilder with the 50D and the 5D2 as well, see below. Perhaps it serves for a pre-correction of the image area, and the 7D firmware is erroneous (though the degree of banding is dfifferent between the copies, and a few don't show it at all).

The raw data of the 7D is less raw than of any previous Canon's, i.e. the firmware does perform some heavier adjustment (the 5D2 and 50D too are doing that, but not that strong); perhaps the firmware "overcorrects" the data.

Sal Baker Forum Pro • Posts: 11,490
Re: 7D and 5D2 low ISO banding pretty much the same

GaborSch wrote:

erikstefan wrote:

But why people said 7D doesn't have Low Iso Banding problem like 5D2 ????????

Did someone really state this? Strange.

Sure they did, early on...even John...

"More than any Canon before, it will be safe to push high ISOs with big negative ECs, due to the almost total lack of line noise, and lower overall read noise."

http://forums.dpreview.com/forums/read.asp?forum=1019&message=33103877

Sal

 Sal Baker's gear list:Sal Baker's gear list
Ricoh GR Fujifilm X-E2 Fujifilm X-T100 Fujifilm XF 18-55mm F2.8-4 R LM OIS Fujifilm XF 23mm F1.4 R +3 more
GaborSch Veteran Member • Posts: 7,203
Observation

Mark H wrote:

This observation supports what I refer to in my most recent reply to John's repeated insistences - where I said... "A calibration as such, will not always be repeatable/successful to the degree that it would reduce the issues to the levels you seem to expect." .

It is definitive, that such a calibration would not work. After I found that the black frame substraction causes banding there, where there was none before, I went more after the banding. I found, that it is present only in the low ranges. This is not the visibility in normal display ; simply, the differences, which exist in the very dark areas are not present in the brighter ones.

I don't see any reason for myself in investing work into finding out if there is some firm correlation, i.e. perhaps the black frame's substraction should be weighted, based on the actual pixel intensity.

Canon undoubtedly have many engineers and specialists in their particular fields of sensor development and signal/image processing, and invest millions in their development - it is truly absurd for some people (not Gabor here) to be suggesting that Canon could easily fix these issues, and that they don't simply because they are complacent and don't care

I am not so sure. Think of the 5D2's black dots and the focusing problems with the 1DsMkIII. My feeling is, that the firmware is to blame at least partly.

xxD Contributing Member • Posts: 726
Re: Let me just say...

JimH wrote:

You've clearly shown that it's relatively easy to reduce the vertical pattern noise in a 7D file. I have yet to see anyone offer a technical rebuttal to that.

relatively easy for black frames. "Real" photographs are a different matter. I'm not convinced it's all that easy for images containing unpredictable, complex data and if he's actually got an algorithm that's robust enough to work on arbitrary data sets.

http://forums.dpreview.com/forums/read.asp?forum=1019&message=33436805

erikstefan Contributing Member • Posts: 821
Re: Some info for cooling down

GaborSch wrote:

I uploaded a new version of Rawnalyze, which can demonstrate this (although the result can not be passed to DPP or ACR). If someone wants to try it, I explain how to do that; it is not in the manual.

Ok I downloaded the Rawnalyze
could you spread the explanation of how to do that

Steen Bay Veteran Member • Posts: 7,418
John's 7D has arrived!

John is probably busy now, testing his new 7D. Sounds promising so far. Looking forward to hear more about his findings.

http://forums.dpreview.com/forums/read.asp?forum=1019&message=33464491

erikstefan Contributing Member • Posts: 821
Patchy Bandings and Camera Noise Reduction

GaborSch wrote:

Mark H wrote:

This observation supports what I refer to in my most recent reply to John's repeated insistences - where I said... "A calibration as such, will not always be repeatable/successful to the degree that it would reduce the issues to the levels you seem to expect." .

It is definitive, that such a calibration would not work. After I found that the black frame substraction causes banding there, where there was none before, I went more after the banding. I found, that it is present only in the low ranges. This is not the visibility in normal display ; simply, the differences, which exist in the very dark areas are not present in the brighter ones.

Regarding this patchy bandings that only exist on shadows area of "low iso" image,

and on the other hand there is camera "high iso" noise reduction capability that works "good".

Canon should be able to apply high iso noise reduction algorithm on shadows area of low iso image, directly in the camera, on the RAW. Selectively applied only to those patch of shadows area.

Although the shadows patch will be slightly soft due to the noise reduction but I prefer a banding free at low iso shadows area image.

And this should be applicable to 5D2, 7D, 1D4 and all other Digic4

Question: Is it possible to do this with firmware ?

I don't see any reason for myself in investing work into finding out if there is some firm correlation, i.e. perhaps the black frame's substraction should be weighted, based on the actual pixel intensity.

Canon undoubtedly have many engineers and specialists in their particular fields of sensor development and signal/image processing, and invest millions in their development - it is truly absurd for some people (not Gabor here) to be suggesting that Canon could easily fix these issues, and that they don't simply because they are complacent and don't care

I am not so sure. Think of the 5D2's black dots and the focusing problems with the 1DsMkIII. My feeling is, that the firmware is to blame at least partly.

Jaims Senior Member • Posts: 2,075
Re: John's 7D has arrived!

Steen Bay wrote:

John is probably busy now, testing his new 7D. Sounds promising so far. Looking forward to hear more about his findings.

And he has not posted anymore since then

http://forums.dpreview.com/forums/read.asp?forum=1019&message=33464491

-- hide signature --
 Jaims's gear list:Jaims's gear list
Canon EOS 5D Canon EOS-1D Mark III Canon EOS 7D Canon EF 16-35mm F2.8L II USM Canon EF 24-105mm f/4L IS USM +9 more
OP John Sheehy Forum Pro • Posts: 23,014
Re: Let me just say...

JimH wrote:

Let me just say that as someone who has spent a lot of time on this forum over a number of years, I know whose technical expertise I respect.

And John is one of those people.

John: You won't win this argument and it's totally off topic anyhow. Don't waste your energy worrying about attacks on your "tone". Who cares?

You've clearly shown that it's relatively easy to reduce the vertical pattern noise in a 7D file. I have yet to see anyone offer a technical rebuttal to that.

"There are people who know more than you or I" is not a technical rebuttal?

It's amazing how creative people can be, and how wide open their imagination gets when they are conjuring ways of discrediting someone who is saying something that they don't like.

Sometimes discussions have to assume a certain amount, and a certain amount of understanding, and the important thing missing in a few people's understanding is the fact that if you did any subtraction of banding that was unnecessary or incorrect, it would show up as worse or redistributed banding; not as a texture without the banding! I did not randomly replace some pixels from other columns to get rid of the banding; I did exactly what the banding does; I altered the values of entire lines by the same amount. If that can't be accepted, an argument is hard to have. Apparently, some people don't seem to understand that these corrections can't ruin other tonal levels, even if they were wrong, because the correction are infinitesimal compared to signal levels and even the shot noise, as you move up from black in the tonal range. The only mistakes possible of any consequence are near black.

Anyway, I am now a 7D owner, so I will investigate sometime soon to see if the low-ISO vertical offset banding is basically the same in every RAW of the same ISO. For my hand-converted color, and B&W images (who needs a converter for B&W?), I should be able to get very good results. Once you subtract the vertical banding, the 7D ISO 100 RAWs look pretty good. Not nearly as good as the D3X, perhaps, but about as good as any recent Canon at the pixel level, and better than any 12 MP or less Canon at the image level. Maybe I'll actually be able to shoot with that extended negative ISO that the 7D gives, or use HTP in auto-ISO manual mode (since Canon has given metering instead of EC in that mode - a bad decision, IMO).

I took the 7D out last night, right after opening the box and taking a few quick shots at home, and I must say that as far as high-ISO is concerned, Canon has taken a big step forward here. Not only is the RAW data low in read noise, but the JPEG engine, with the defaults the camera comes in, gives very reasonable detail despite the NR used, and the medium quality of the embedded JPEG. It looks like when pushed to the limits, the camera is struggling against a fine - grained noise that originates almost completely at the photosites.

Shooting at "H", and even at 6400 with -5 EC, I can read fine text and see reasonable detail. Apparently, not only the quantity of read noise has decreased, even at the pixel level, but the quality or character of it at high ISOs also has taken a big step forward.

-- hide signature --

John

erikstefan Contributing Member • Posts: 821
Re: John's 7D has arrived!

Jaims wrote:

Steen Bay wrote:

John is probably busy now, testing his new 7D. Sounds promising so far. Looking forward to hear more about his findings.

And he has not posted anymore since then

I hope he didn't kill him self
it's just a camera

http://forums.dpreview.com/forums/read.asp?forum=1019&message=33464491

-- hide signature --
OP John Sheehy Forum Pro • Posts: 23,014
Re: Let me just say...

xxD wrote:

JimH wrote:

You've clearly shown that it's relatively easy to reduce the vertical pattern noise in a 7D file. I have yet to see anyone offer a technical rebuttal to that.

relatively easy for black frames. "Real" photographs are a different matter. I'm not convinced it's all that easy for images containing unpredictable, complex data and if he's actually got an algorithm that's robust enough to work on arbitrary data sets.

I have a 7D now, so I will soon do a demo that includes a full range of tonal levels.

I have already explained in my reply to JimH that corrections on the level that I am doing can only mess up near-black ranges if done wrong. I doubt that anyone can dispute that if they really grasp what is going on. How much is a 0.5 to 2 ADU line-by-line correction going to harm a signal of 400 ADU, with a shot noise of 20 ADU? Or a signal of 11000 ADU with a shot noise of 105 ADU? The only consequences of doing it wrong, are near black.

-- hide signature --

John

OP John Sheehy Forum Pro • Posts: 23,014
Re: John's 7D has arrived!

erikstefan wrote:

Jaims wrote:

Steen Bay wrote:

John is probably busy now, testing his new 7D. Sounds promising so far. Looking forward to hear more about his findings.

And he has not posted anymore since then

I hope he didn't kill him self
it's just a camera

Almost. The best spots last night to photograph colorful NYC lights reflecting off the wet ground were in traffic lanes, and an MP3 player and a rain hood didn't help matters.

-- hide signature --

John

xxD Contributing Member • Posts: 726
Re: Let me just say...

John Sheehy wrote:

xxD wrote:

JimH wrote:

You've clearly shown that it's relatively easy to reduce the vertical pattern noise in a 7D file. I have yet to see anyone offer a technical rebuttal to that.

relatively easy for black frames. "Real" photographs are a different matter. I'm not convinced it's all that easy for images containing unpredictable, complex data and if he's actually got an algorithm that's robust enough to work on arbitrary data sets.

I have a 7D now, so I will soon do a demo that includes a full range of tonal levels.

What your correction algorithm is able to do on arbitrary images will be more telling than thousands or words in a forum debate. The proof's in the pudding, as they say. I, for one, am looking forward to see what you've got.

Mrs Reality Regular Member • Posts: 366
Re: Some info for cooling down

Mark H wrote:

GaborSch wrote:

I don't see any reason to change this discussion into an argumentation regarding the persons instead of thew views.

I quite agree (surprising though it may seem to some people) - however, when a contributor starts throwing accusations of "shill" and "libel" and corporate "collusion" they should not be in the least surprised when the discussion heads off in that direction. [N.B. I'm not talking about Gabor here].

Now, on to your observation here...

I posted at the very beginning, that I experimented with this quite natural idea. Unfortunately the result is not good. When using a black frame as "template" for black level corrections for another shot with the same camera, same ISO, even if these shots have been made within seconds, the result is not satisfactory: the banding does not disappear completely, but it appears on patches, which did not show banding before.

This observation supports what I refer to in my most recent reply to John's repeated insistences - where I said... "A calibration as such, will not always be repeatable/successful to the degree that it would reduce the issues to the levels you seem to expect." .

Canon undoubtedly have many engineers and specialists in their particular fields of sensor development and signal/image processing, and invest millions in their development - it is truly absurd for some people (not Gabor here) to be suggesting that Canon could easily fix these issues, and that they don't simply because they are complacent and don't care.

Elsewhere in the thread someone else wonders why "...Canon has not implemented the easy fix for these issues" - well, in all probability, the simple answer is that the issues are not so easily fixed at all - otherwise Canon would probably have already done so.

Blind FAITH in Canon is no substitute for reality.

OP John Sheehy Forum Pro • Posts: 23,014
Re: suggest a separate tech forum

on2art wrote:

Yes, I am very thankful for the generosity of the more technically astute as seen in various forums.

Their on going contribution helps to dispel so much wrong thinking that actually appears to be somewhat exploited by manufacturers in the marketing of digital cameras. Today I was at a popular well known camera store and the sales people were convinced that noise of any sort correlates directly to pixel density.

Of course, the only way in which the banding noise of the 7D at low ISOs could be related to pixel density is a combination effect with the burst speed. If it takes too long to correct the banding (or the system would have to be slowed to avoid it), then the density could be said to play a role, but it seems that the necessary math is done anyway, so the variables may just need to be made more precise.

It is more than a little exasperating to find how passionate they are at telling their little stories about how the G11 is so much better than the G10 because Canon "wisely upgraded" to a lower pixel count for that approximately 2/3" size of sensor found in many better compacts but passionate story telling sells cameras, yes.

Yeah, well the 5D2 gave the lowest high-ISO read noise per unit of sensor area with its slightly larger pixels when it came out, but it looks like the 7D has caught up to it in statistical quantity, and improved upon it in character, with Canon's smallest DSLR pixels yet. The full-frame images from my 7D at ISO 12800 resampled to a 1440*1050 screen slay those from my 5D2, unless they are high-key! At lower ISOs that are more shot-noise affected, the 5D2 should still be a bit better (and use more of the lens' resolving power, although I was getting aliasing last night with my 7D and 70-200 f/4L IS - showing that even for a good zoom, 18MP is nowhere near the practical limit).

-- hide signature --

John

Earthlight Veteran Member • Posts: 3,209
Re: Let me just say...

John Sheehy wrote:

Apparently, not only the quantity of read noise has decreased, even at the pixel level, but the quality or character of it at high ISOs also has taken a big step forward.

Hi John!

At what ISO would you say the pattern noise ceases to be a problem? I'm hoping ISO 800 is already free of it.

Earthlight

-- hide signature --

Mostly harmless

OP John Sheehy Forum Pro • Posts: 23,014
Re: Patchy Bandings and Camera Noise Reduction

erikstefan wrote:

Regarding this patchy bandings that only exist on shadows area of "low iso" image,

and on the other hand there is camera "high iso" noise reduction capability that works "good".

Based on the levels of vertical banding at various ISOs, it would seem that the 5D2 does have some kind of correction at ISO 800 (640 - 1000) and above. Banding increases from 200 to 400, and then drops at 800.

Canon should be able to apply high iso noise reduction algorithm on shadows area of low iso image, directly in the camera, on the RAW. Selectively applied only to those patch of shadows area.

I'd never want to see "filter-based" NR in the RAWs. Only simple offset and scalar corrections, which don't lose any subject detail.

-- hide signature --

John

OP John Sheehy Forum Pro • Posts: 23,014
Re: Let me just say...

Earthlight wrote:

John Sheehy wrote:

Apparently, not only the quantity of read noise has decreased, even at the pixel level, but the quality or character of it at high ISOs also has taken a big step forward.

Hi John!

At what ISO would you say the pattern noise ceases to be a problem? I'm hoping ISO 800 is already free of it.

Based on my brief experience with mine, and those links I've seen from others' cameras, pattern noise is only an issue in a typical 7D in the deepest shadows of low ISOs. To put it another way, the ratio of pattern noise to random noise is infinitesimal at high ISOs, and increases abruptly as you go from ISO 400 down to 100, but it is still low there, and you won't notice it unless you try to under-expose low ISOs for increased headroom or safety factor. If you combine HTP with a poor lighting color, like incandescent or deep foliage green, you might start seeing the vertical banding in the darker areas.

-- hide signature --

John

erikstefan Contributing Member • Posts: 821
Question John

John Sheehy wrote:

erikstefan wrote:

Jaims wrote:

Steen Bay wrote:

John is probably busy now, testing his new 7D. Sounds promising so far. Looking forward to hear more about his findings.

And he has not posted anymore since then

I hope he didn't kill him self
it's just a camera

Almost. The best spots last night to photograph colorful NYC lights reflecting off the wet ground were in traffic lanes, and an MP3 player and a rain hood didn't help matters.

With 7D normal shooting low iso Raw, (or maybe a slightly bit -EC),
and then with ACR/Lightroom put Fill-Light up to 50-70,
do you see banding at shadows area?

Thank's in advance

Any sample (100% crop) would be appreciated much

(My guess maybe the "sensitivity curve" applied to Raw on 7D is different from 5D2)

Keyboard shortcuts:
FForum MMy threads