Canon Line Offset Noise

Started Oct 20, 2009 | Discussions
ejmartin Veteran Member • Posts: 6,274
Re: The look of sharpened noise

Steen Bay wrote:

GaborSch wrote:

Steen Bay wrote:

What I meant is, that normal random noise doesn't appear as maze if sharpened

Well, here is an example, from one of the 7D raw files posted by bronxbombers, NR off, strong sharpening by DPP, 200% view:

Yes, it ain't pretty, I think we can agree about that, but Bronxbombers' 7D had this additional issue with unbalanced green gains, which confuses things a bit. I started this subthread wondering why mazing was so visible in Hulkster's 7D and (especially) 50D DPP test pics (sharpening 3, which isn't strong) at 100%, from ISO 400 and up, and I think it's (more or less) clear that the mazing must be created by the demosaicing algorytm in its attempt to maximize the amount of detail.

I agree.

Is the maze a real problem? Maybe not, it's probably just the visible effect of the compromises made in the demosaicing process, but it certainly isn't pretty at 100% view!

It's a problem in that the gain imbalance violates assumptions on which the demosaic algorithm is based. A checkerboard pattern of G values does not occur in real world images (it could only be caused by a texture at or beyond Nyquist in both vertical and horizontal directions , something quite rare in real world scenes, and would be suppressed by the AA filter anyway), so the algorithm should be allowed to assume that it will not be presented with such data. It would be nice if the demosaic algorithm is robust enough not to generate mazes, but let's put the blame for mazing where it belongs -- on the camera that generates RAW data not corresponding to any real world image.

-- hide signature --
OP John Sheehy Forum Pro • Posts: 23,014
Re: Absolutely agree...

Mark H wrote:

John Sheehy wrote:

Fine for whom?

Silly question - fine for them of course.

Who cares? Most folks are fighting to get their cat in focus.

You, on the other hand, seem to want to make the corporations look respectful, and want it to look like they are doing all they reasonably should to give us quality for our hard-earned dollars, at the expense of those who don't want to see patterns in the RAW data; unnecessary patterns whose existence you defend.

Absolutely - on the whole I think these corporations do an amazing job.

Then, why do I, a person who hasn't even gone to school for engineering, do a better job with calibrating their RAW data?

Maybe you should go it alone - make DSLRs better than they do - it should be dead easy with your superior knowledge and abilities.

I could write revolutionary firmware that makes cameras far more usable and customizable than they are now. They give us cr*p, because they are in lethargic collusion. You need lots of money behind you to compete against the inertial, inefficient monsters.

John Sheehy wrote:

Really? I said "few"; are you illiterate?

Clearly no more than you are - I am simply pointing out that to many readers your statement 'potentialy' misrepresents, and gives the impression that "a very high percentage [of 7ds] have poorly balanced gains" .

I clearly said "first few", and I also said, originally, which you snipped away, that hopefully this is not representative of future production. IOW, you are intentionally misrepresenting me, to discredit me. I am trying to help photography by raising awareness of corporate apathy, and you want to help corporations, and shoot me down.

P.S. Just in case you might have overlooked this - will you be answering the question/request I ask of you earlier in this thread... http://forums.dpreview.com/forums/read.asp?forum=1019&message=33443889 ...anytime soon?

I guess I'm not in a rush to prove that to you, because I know I'm right,..

Right, 'that' sounds really convincing - I will wait.

That was an explanation of why I didn't immediately reply; not my final word or proof. Again, you have chosen to misrepresent me, by cutting out what I referred to, after that, like Roger Clark's measurements, and the DxO screen comparison. Why do you pretend that I didn't offer these? That is pathetic.

Who do you think you're fooling with your libelous attacks?

-- hide signature --

John

OP John Sheehy Forum Pro • Posts: 23,014
Re: Hows 7D Low Iso Banding compared to 5D2 ?

erikstefan wrote:

But why people said 7D doesn't have Low Iso Banding problem like 5D2 ????????

Is that true

Well, perhaps some of the first few RAWs available didn't readily show it, but it is there nonetheless, to some degree, on every 7D. IOW, the offset banding is visible, which is usually the case when the random 2D noise is about 15x as strong as the banding noise, or less.

The 7D just has a periodic banding, though, of any strength. The random vertical and horizontal banding is very weak. With the banding subtracted, on both cameras, the read noise in both the 7D and the 5D2 at the pixel level is approximately the same. The 5D2 is more prone to random horizontal and vertical line offsets, only one pixel wide (concentrated at the nyquist). The 7D has an 8-pixel period, typically, and this causes a less chromatic banding than the finer 5D2 banding.

-- hide signature --

John

Mark H
Mark H Veteran Member • Posts: 3,803
Re: Absolutely agree...

John Sheehy wrote:

Mark H wrote:

John Sheehy wrote:

Fine for whom?

Silly question - fine for them of course.

Who cares? Most folks are fighting to get their cat in focus.

Here you go again - exaggerating and misrepresenting, yet again...

" Most folks" are not   "fighting to get their... in focus" - some may be, but not   " most " .

Absolutely - on the whole I think these corporations do an amazing job.

Then, why do I, a person who hasn't even gone to school for engineering, do a better job with calibrating their RAW data?

I don't see that you have done that at all.

You don't even know whether a calibration is already being applied.

A calibration as such, will not always be repeatable/successful to the degree that it would reduce the issues to the levels you seem to expect.

It's only too easy to look at something after the event and then 'claim' that "I could do better" - but you simply don't actually know what has previously been done, or whether there are any other mitigating or limiting issues.

Perhaps if you had "gone to school for engineering" then you might better appreciate this, rather than just conceitedly pronouncing that "I... do a better job with calibrating their RAW data"

Maybe you should go it alone - make DSLRs better than they do - it should be dead easy with your superior knowledge and abilities.

I could write revolutionary firmware that makes cameras far more usable and customizable than they are now.

I'm sure everyone thinks that they could make their own personal improvements.

It would be interesting to hear some of your "revolutionary" ideas.

They give us cr*p, because they are in lethargic collusion. You need lots of money behind you to compete against the inertial, inefficient monsters.

They don't give use "cr*p" - your distorted and paranoid view is the only "cr*p" around here.

I guess I'm not in a rush to prove that to you, because I know I'm right,..

Right, 'that' sounds really convincing - I will wait.

That was an explanation of why I didn't immediately reply; not my final word or proof. Again, you have chosen to misrepresent me, by cutting out what I referred to, after that, like Roger Clark's measurements, and the DxO screen comparison. Why do you pretend that I didn't offer these? That is pathetic.

There is no pretence - I am poking fun at the initial conceit in your response.. "I guess I'm not in a rush to prove that to you, because I know I'm right,.." .

Who do you think you're fooling with your libelous attacks?

There is no 'libel' in any of my messages - only reasonable criticisms, challenges, and rebukes where warranted.

Now your statements, however, really are frequently quite 'libellous' in nature - some recent 'John Sheehy' examples...

John Sheehy wrote:

"...Canon is full of you know what"

"...they write BS press releases and white-papers about improvements which are not realized"

"...I'll tell you right now; almost everything I've bought from Canon has been junk, in one way or another. "

"...The issue is ethics in manufacturing. Canon apparently has none"

"...They give us cr*p, because they are in lethargic collusion"

"...[Canon are] either stupid, apathetic, or malicious"

"...I tend to think you're a paid Canon shill,.."

...and you wonder why I previously referred to your 'agenda"?

springbock Senior Member • Posts: 2,263
Re: Absolutely agree...

Don't let it bother you Mark. John Sheehy appears to be a megalomaniac with delusions of grandeur.

I think he's a backyard physicist. That term drives him nuts

-- hide signature --

'The truth is rarely pure and never simple' Oscar Wilde

GaborSch Veteran Member • Posts: 7,203
7D and 5D2 low ISO banding pretty much the same

erikstefan wrote:

But why people said 7D doesn't have Low Iso Banding problem like 5D2 ????????

Did someone really state this? Strange.

Anyway, they are equal , except the 5D2 does this mostly horizontally, the 7D vertically.

I have raw files from about 20 copies of the 7D, but only two of them don't exhibit the vertical banding with the 8-column periodicity. One of these are suitable for a direct comparison.

Disclaimer: this is ONE copy of the 7D. The periodic banding overrides this kind of "banding" in the other images; who knows how they would look like if that banding were not present?

Following are captures from ISO 100 7D and 5D2 images; the green spots in the first pair, and the red vas. the blue in the second pair are at the same average intensity level, -10.5 EV respectively -10.9 EV, i.e. in the mid/at the end of the 11th stop of the DR; that's where the 5D2 becomes ugly.

JimH Forum Pro • Posts: 12,911
Let me just say...

Let me just say that as someone who has spent a lot of time on this forum over a number of years, I know whose technical expertise I respect.

And John is one of those people.

John: You won't win this argument and it's totally off topic anyhow. Don't waste your energy worrying about attacks on your "tone". Who cares?

You've clearly shown that it's relatively easy to reduce the vertical pattern noise in a 7D file. I have yet to see anyone offer a technical rebuttal to that.

-- hide signature --

Jim H.

Steen Bay Veteran Member • Posts: 7,418
Re: The look of sharpened noise

ejmartin wrote:

Steen Bay wrote:

GaborSch wrote:

Steen Bay wrote:

What I meant is, that normal random noise doesn't appear as maze if sharpened

Well, here is an example, from one of the 7D raw files posted by bronxbombers, NR off, strong sharpening by DPP, 200% view:

Yes, it ain't pretty, I think we can agree about that, but Bronxbombers' 7D had this additional issue with unbalanced green gains, which confuses things a bit. I started this subthread wondering why mazing was so visible in Hulkster's 7D and (especially) 50D DPP test pics (sharpening 3, which isn't strong) at 100%, from ISO 400 and up, and I think it's (more or less) clear that the mazing must be created by the demosaicing algorytm in its attempt to maximize the amount of detail.

I agree.

Is the maze a real problem? Maybe not, it's probably just the visible effect of the compromises made in the demosaicing process, but it certainly isn't pretty at 100% view!

It's a problem in that the gain imbalance violates assumptions on which the demosaic algorithm is based. A checkerboard pattern of G values does not occur in real world images (it could only be caused by a texture at or beyond Nyquist in both vertical and horizontal directions , something quite rare in real world scenes, and would be suppressed by the AA filter anyway), so the algorithm should be allowed to assume that it will not be presented with such data. It would be nice if the demosaic algorithm is robust enough not to generate mazes, but let's put the blame for mazing where it belongs -- on the camera that generates RAW data not corresponding to any real world image.

Thanks, but I'm not quite sure what it is you're saying here. Are you suggesting that the very visible maze artifacts in Hulkster's 7D and 50D test pics also is caused by an imbalance in gains, on both cameras? (That would be pretty disturbing!) Or is the maze a 'normal' thing, caused by the compromises made in the demosaicing, also on cameras with perfectly balanced gains?

(The maze pattern is very easy to see in the DPP 3.7 "sharpen=3" and "cropped by clock" pics, at ISO 400 and above. Link to the test in the post below)

http://forums.dpreview.com/forums/read.asp?forum=1019&message=33411995

springbock Senior Member • Posts: 2,263
Re: Let me just say...

Jim I'm not disputing everything John says... but he has a major chip on his shoulder. Short of being a Canon or Nikon engineer we can't refute what he says. But... do you think for one second that he would be a better engineer/sensor/software designer that what Canon has?

Or do you think that Canon is not interested in making the best product they can at a certain price point? Or do you think that they are are just trying to screw us? He thinks they are!!!

I used to respect John's posts.... but not anymore. It appears that he has an agenda and has a MAJOR CHIP on his shoulder.

How many of us know enough about digital photography to dispute him? He knows some things but that does not make him an expert, or his opinion relevant.

JimH wrote:

Let me just say that as someone who has spent a lot of time on this forum over a number of years, I know whose technical expertise I respect.

And John is one of those people.

John: You won't win this argument and it's totally off topic anyhow. Don't waste your energy worrying about attacks on your "tone". Who cares?

You've clearly shown that it's relatively easy to reduce the vertical pattern noise in a 7D file. I have yet to see anyone offer a technical rebuttal to that.

-- hide signature --

'The truth is rarely pure and never simple' Oscar Wilde

ejmartin Veteran Member • Posts: 6,274
Re: The look of sharpened noise

Steen Bay wrote:

ejmartin wrote:

Steen Bay wrote:

GaborSch wrote:

Steen Bay wrote:

What I meant is, that normal random noise doesn't appear as maze if sharpened

Well, here is an example, from one of the 7D raw files posted by bronxbombers, NR off, strong sharpening by DPP, 200% view:

Yes, it ain't pretty, I think we can agree about that, but Bronxbombers' 7D had this additional issue with unbalanced green gains, which confuses things a bit. I started this subthread wondering why mazing was so visible in Hulkster's 7D and (especially) 50D DPP test pics (sharpening 3, which isn't strong) at 100%, from ISO 400 and up, and I think it's (more or less) clear that the mazing must be created by the demosaicing algorytm in its attempt to maximize the amount of detail.

I agree.

Is the maze a real problem? Maybe not, it's probably just the visible effect of the compromises made in the demosaicing process, but it certainly isn't pretty at 100% view!

It's a problem in that the gain imbalance violates assumptions on which the demosaic algorithm is based. A checkerboard pattern of G values does not occur in real world images (it could only be caused by a texture at or beyond Nyquist in both vertical and horizontal directions , something quite rare in real world scenes, and would be suppressed by the AA filter anyway), so the algorithm should be allowed to assume that it will not be presented with such data. It would be nice if the demosaic algorithm is robust enough not to generate mazes, but let's put the blame for mazing where it belongs -- on the camera that generates RAW data not corresponding to any real world image.

Thanks, but I'm not quite sure what it is you're saying here. Are you suggesting that the very visible maze artifacts in Hulkster's 7D and 50D test pics also is caused by an imbalance in gains, on both cameras? (That would be pretty disturbing!) Or is the maze a 'normal' thing, caused by the compromises made in the demosaicing, also on cameras with perfectly balanced gains?

(The maze pattern is very easy to see in the DPP 3.7 "sharpen=3" and "cropped by clock" pics, at ISO 400 and above. Link to the test in the post below)

http://forums.dpreview.com/forums/read.asp?forum=1019&message=33411995

The maze artifacts of sharpened high ISO images are the same sort of thing as would be caused by demosaic of low ISO images with gain imbalance of greens. The demosaic is making straight line vertical or horizontal interpolations of RAW data that has a substantial (random, in the case of high ISO, due to noise) variation at Nyquist. The converter then sharpens those vertical or horiztontal bars. Now, low ISO data should hot have those sorts of bars, and indeed hulkster's low ISO images don't show any, even with sharpening. bronxbomber's 7D images did have strong variation at Nyquist, even at base ISO, due to the gain imbalance; thus he got the same sort of mazing pattern.

I have been wondering whether those who are disappointed with the low ISO behavior of their 7D in regard to noise, might have a copy with some gain imbalance, though perhaps not so strong as in BB's copy that would cause strong mazing but rather just an overall increase in artifacting in the conversions.

-- hide signature --
on2art Forum Member • Posts: 78
suggest a separate tech forum

Yes, I am very thankful for the generosity of the more technically astute as seen in various forums.

Their on going contribution helps to dispel so much wrong thinking that actually appears to be somewhat exploited by manufacturers in the marketing of digital cameras. Today I was at a popular well known camera store and the sales people were convinced that noise of any sort correlates directly to pixel density.

It is more than a little exasperating to find how passionate they are at telling their little stories about how the G11 is so much better than the G10 because Canon "wisely upgraded" to a lower pixel count for that approximately 2/3" size of sensor found in many better compacts but passionate story telling sells cameras, yes.

Anyhow what I would really like to see is a separate forum where more of these technical discussions might develop complete clarity without interruption from characters who are really trying so hard to focus on the rather more popular way of thinking.

Steen Bay Veteran Member • Posts: 7,418
Re: The look of sharpened noise

ejmartin wrote:

Steen Bay wrote:

ejmartin wrote:

Steen Bay wrote:

GaborSch wrote:

Steen Bay wrote:

What I meant is, that normal random noise doesn't appear as maze if sharpened

Well, here is an example, from one of the 7D raw files posted by bronxbombers, NR off, strong sharpening by DPP, 200% view:

Yes, it ain't pretty, I think we can agree about that, but Bronxbombers' 7D had this additional issue with unbalanced green gains, which confuses things a bit. I started this subthread wondering why mazing was so visible in Hulkster's 7D and (especially) 50D DPP test pics (sharpening 3, which isn't strong) at 100%, from ISO 400 and up, and I think it's (more or less) clear that the mazing must be created by the demosaicing algorytm in its attempt to maximize the amount of detail.

I agree.

Is the maze a real problem? Maybe not, it's probably just the visible effect of the compromises made in the demosaicing process, but it certainly isn't pretty at 100% view!

It's a problem in that the gain imbalance violates assumptions on which the demosaic algorithm is based. A checkerboard pattern of G values does not occur in real world images (it could only be caused by a texture at or beyond Nyquist in both vertical and horizontal directions , something quite rare in real world scenes, and would be suppressed by the AA filter anyway), so the algorithm should be allowed to assume that it will not be presented with such data. It would be nice if the demosaic algorithm is robust enough not to generate mazes, but let's put the blame for mazing where it belongs -- on the camera that generates RAW data not corresponding to any real world image.

Thanks, but I'm not quite sure what it is you're saying here. Are you suggesting that the very visible maze artifacts in Hulkster's 7D and 50D test pics also is caused by an imbalance in gains, on both cameras? (That would be pretty disturbing!) Or is the maze a 'normal' thing, caused by the compromises made in the demosaicing, also on cameras with perfectly balanced gains?

(The maze pattern is very easy to see in the DPP 3.7 "sharpen=3" and "cropped by clock" pics, at ISO 400 and above. Link to the test in the post below)

http://forums.dpreview.com/forums/read.asp?forum=1019&message=33411995

The maze artifacts of sharpened high ISO images are the same sort of thing as would be caused by demosaic of low ISO images with gain imbalance of greens. The demosaic is making straight line vertical or horizontal interpolations of RAW data that has a substantial (random, in the case of high ISO, due to noise) variation at Nyquist. The converter then sharpens those vertical or horiztontal bars. Now, low ISO data should hot have those sorts of bars, and indeed hulkster's low ISO images don't show any, even with sharpening. bronxbomber's 7D images did have strong variation at Nyquist, even at base ISO, due to the gain imbalance; thus he got the same sort of mazing pattern.

I have been wondering whether those who are disappointed with the low ISO behavior of their 7D in regard to noise, might have a copy with some gain imbalance, though perhaps not so strong as in BB's copy that would cause strong mazing but rather just an overall increase in artifacting in the conversions.

Think I've got it now, thanks. Sorry for being a bit slow

GaborSch Veteran Member • Posts: 7,203
Some info for cooling down

I don't see any reason to change this discussion into an argumentation regarding the persons instead of thew views.

I posted at the very beginning, that I experimented with this quite natural idea. Unfortunately the result is not good. When using a black frame as "template" for black level corrections for another shot with the same camera, same ISO, even if these shots have been made within seconds, the result is not satisfactory: the banding does not disappear completely, but it appears on patches, which did not show banding before.

I uploaded a new version of Rawnalyze, which can demonstrate this (although the result can not be passed to DPP or ACR). If someone wants to try it, I explain how to do that; it is not in the manual.

JimH Forum Pro • Posts: 12,911
Re: Let me just say...

I feel like things have gotten stirred up in this thread and the few others that lead up to it. And John's been outright attacked, with a few people questioning his knowledge and beating it to death. That seems to me to be what's caused his temper to flare (and, to me, understandably).

So I think the anger you're seeing in John's posts are uncharacteristic of him, and are the result of his having been badgered mercilessly on this point by a few people whose names I don't recognize from this forum.

So I think that's what's lead to this "argument".

John has mentioned this kind of problem on here for years. And he's right that with better use of the existing masked areas or perhaps with even more masked pixels, a lot could be done to fix some of the problems that have bedeviled (to my knowledge, and at the least), the 20D, 30D, 40D, 50D, and now 7D as well as, I guess, the 5DII.

My 20D suffers from pretty nasty horizontal pattern noise. Oddly, my old version of ACR which runs under CS2 automatically suppresses this banding for the 20D. DPP does not.

So for long exposures or shots taken in dark conditions, I use that old version of ACR to convert my 20D files, and they're great. I use DPP for everything else because I greatly prefer its colors.

So Adobe had this figured out for the 20D years ago. But as I understand it, they've dropped the ball on that in newer versions and for newer cameras. I find that interesting.

And John's talked about this all before, and he has not been very forgiving of Canon for not doing the obvious and easy things about it. I suspect that as time goes on and we see more and more cameras introduced, and still, nothing is done about these easy-to-fix issues, a lot of us have gotten a bit impatient with Canon.

I don't think anyone has to be a Nikon or Canon engineer to see this. And it's not logical to imagine that an "outsider" (to any company or organization) can see problems and offer good solutions. I think that the rudeness shown by some of the people who have been attacking John in these threads is unwarranted.

OK, maybe he's being a bit strident in his knocking of Canon. But this problem has persisted, year after year, model after model. It's kind of baffling and I think John's just been trying to get the point across that there doesn't seem to be any rational explanation for why something so obvious would go overlooked for so long.

I would say that while those participating in these threads (and I'm not just talking about John here, because there are a number of true experts in this field who have posted to this very thread) may not be fully informed as to all of the various tradeoffs and reasons why Canon does what they do, I do believe that they are capable of seeing faults and performing some fairly in-depth analysis of them.

I truly do believe that some of the posters to this very thread could offer quite a bit of insight to the engineers at Canon who are designing these cameras. You might be extremely surprised at how sharp some of these folks are.

As far as Canon's motivations for overlooking these things year after year, that really is baffling. One would imagine that they'd want to produce the very best products possible at the price points they've established so that they could compete better in the market.

I wonder if there are patent issues preventing them from implementing some of these seemingly obvious fixes. It's hard to know without being inside the companies.

I think what you're seeing is John's frustration with being hounded constantly by a few select folks who don't seem interested in the actual nuts and bolts of this, but instead are harping over and over on John's lack of faith in Canon.

Fine. Some people want to believe the best in Canon. John seems to have lost faith in that particular religion. So be it.

But I'm pretty sure that the vitriol you're seeing from John is his reaction to being constantly told that he's not qualified to make the observations he's made. That's simply not true.

He's proven how easy it is to fix the vertical pattern noise. He and others have established that the maze artifact is a result of the imbalance in the greens in even/odd vertical lines. All of that is easily fixed.

I'm not sure what the real reason is that Canon has not implemented the easy fixes for these issues. It's strange.

Don't just take John's word for it, read the posts by the others in this thread.

I don't think anything John has written on the technical details of this subject has been wrong.

And as for the flame war: That didn't exist until some people started attacking John over and over and over. That's the part of all of this that I'd like to see go away. It serves no purpose.

We know John's and Mark's opinions already. I, for one, don't need to hear it repeated over and over and over.

Let's get on with the technical and practical side of this and agree to disagree about Canon's motives, etc.

-- hide signature --

Jim H.

JimH Forum Pro • Posts: 12,911
What do you think...

What if the camera had more masked pixels to work from so that this could be done on a frame-by-frame basis? Or what about using the masked pixels that already exist?

That seems like what people have been asking for with the Canons for a number of years.

I think John's test showed that a correction factor, determined from just the bottom 100 rows, was sufficient to largely "fix" the problems all the way up to the top of the frame. That seems to me to indicate that if we just had a good strip of masked pixels to work from, this could be easily done in-camera or out-of-camera on a shot-by-shot basis as long as we have access to those masked pixels from each frame.

What do you think?

That's something that only the camera maker can really provide for us. It does seem odd that it's not done.

-- hide signature --

Jim H.

Wesayso Regular Member • Posts: 432
I'll support this view...

Thank you for this post. It says about all I could want to say on this subject. I've read all these analyses threads on the 7D and even the old ones on the 50D, because I own that camera (as does John I believe). For me the point in reading these threads is mostly curiousity and obtaining some knowledge and insights.

I've looked up the old threads on the 50D when I was shopping for that model to know what I could expect. I'm just glad I did read those views and bought the 50D anyway.

I really don't think John and others are saying the 7D is a bad camera. From what I've read the're saying it could have been better. IMHO the specific camera with the mazing effect at low ISO was a faulty camera and I do think our analists have turned every pixel up side down to find out why.

There will be other problem camera's out there but I think this thread is about making something good even better, that is if Canon wants to put their minds into it. Showing it is possible to do just that. How is that a bad thing? From what I've seen the 7D is better in than my 50D in read noise so progress is made and proves these cameras can get better over the years. Pointing out the weak spots isn't a bad thing is it? It may even help the next generation to improve.

Why make it personal, we can all learn from each other here. If I were having problems with my camera I'd love to have this team of analists check out my concerns.

To those who have bought the 7D and are happy with it don't doubt your choise and stay happy. You've bought a great camera (I envy many features). May the next one be even better ;).

GaborSch Veteran Member • Posts: 7,203
About options

JimH wrote:

What if the camera had more masked pixels to work from so that this could be done on a frame-by-frame basis?

I don't think it is worth of much thinking about a non-existent sensor.

Or what about using the masked pixels that already exist?

The 7D has the useful masked area at the left side. This means, that the corrections can be differentiated row-wise (or use the global average), but we need column-wise distinction.

There is a nonactive area at the top too, but it is useless for this purpose. That was the first I experienced with when seeing the banding; the result is worse than the banding.

Look at the masked areas (contrast enhanced):

Funny is, that the left masked area does not show the banding, which is present in the image area; following is from a black frame:

I think John's test showed that a correction factor, determined from just the bottom 100 rows, was sufficient to largely "fix" the problems all the way up to the top of the frame

John used the bottom of a black frame . That works well within the same image , i.e. the banding in the black frame can be removed that way (it is demonstrable by Rawnalyze).

However, we do not have that good strip of masked pixels.

ejmartin Veteran Member • Posts: 6,274
Re: About options

GaborSch wrote:

John used the bottom of a black frame . That works well within the same image , i.e. the banding in the black frame can be removed that way (it is demonstrable by Rawnalyze).

However, we do not have that good strip of masked pixels.

Different forms of pattern noise call for different treatments. The 7D sort seems to be a fixed repeating pattern; if so, then the preferred treatment is to generate a pattern removal template that would be applied as a subtractive offset to all images. dcraw for instance has this capability, though unfortunately most of the GUI raw converters based on it (eg RawTherapee) do not enable this capability.

A template for fixed pattern noise can be made by stacking multiple blackframes, the more the merrier; the more one stacks, the more the random read noise averages away, leaving only the fixed pattern noise. The template then with only the fixed pattern noise is subtracted from the image one wishes to convert, and removes the fixed pattern noise from it before conversion.

An example of the technique can be found at
http://theory.uchicago.edu/~ejm/pix/20d/tests/noise/#patternnoise

This technique will of course fail for random line noise which varies from frame to frame; stacking blackframes will not pick out this sort of noise, and there is no template one can build that will reliably remove this sort of noise from a given image. This is where it is useful to have a large masked border; as John has mentioned, with a sufficiently large border area one can detect the variable line noise sufficiently accurately to subtract it for the given image. However, it may not be worth it to Canon or other manufacturers to incur the cost of added sensor area that goes only to this purpose.

-- hide signature --
JimH Forum Pro • Posts: 12,911
Re: About options

GaborSch wrote:

JimH wrote:

What if the camera had more masked pixels to work from so that this could be done on a frame-by-frame basis?

I don't think it is worth of much thinking about a non-existent sensor.

I understand what you mean, but a large portion of this thread has been directed towards what we wish Canon would do. So it may have value to think about this in terms of what we'd request Canon to do in future models if we had any input.

Maybe they read these forums, maybe they don't. But my opinion is that if they'd give us a good batch of masked pixels along the bottom as well as along one side, then we could easily write software to put those masked pixels to use even if nothing was done with them in the camera.

Many manufacturers take suggestions from their users seriously. So we can still hope, I guess.

Or what about using the masked pixels that already exist?

The 7D has the useful masked area at the left side. This means, that the corrections can be differentiated row-wise (or use the global average), but we need column-wise distinction.

Right. We don't have what we'd really like to have ... in this model.

There is a nonactive area at the top too, but it is useless for this purpose. That was the first I experienced with when seeing the banding; the result is worse than the banding.

It makes one wonder what, if anything, they're doing with that area. It almost does seem as though things have been purposely made worse over the image portion of the frame when you compare it with the area on the left edge.

Look at the masked areas (contrast enhanced):

Funny is, that the left masked area does not show the banding, which is present in the image area; following is from a black frame:

It is very odd, indeed. Suspicious, actually.

I cannot think of a reasonable explanation for why we'd see this. I do see where John's thoughts of purposeful sabotage come from. (Don't flame me, Canon shills!) (Just kidding - sort of).

I think John's test showed that a correction factor, determined from just the bottom 100 rows, was sufficient to largely "fix" the problems all the way up to the top of the frame

John used the bottom of a black frame . That works well within the same image , i.e. the banding in the black frame can be removed that way (it is demonstrable by Rawnalyze).

I understand that entirely. That, again, was my point. His test using a black frame is a "proof of concept" kind of thing. Obviously, we don't really care about what we can do with black frames. We need masked pixels so that we can apply this to real frames.

However, we do not have that good strip of masked pixels.

But if we did , then it seems to me that a lot could be done. So it seems like a reasonable thing to request from Canon in future models. But then John has been doing that for years on here, so...

The idea even comes up about masking them off ourselves somehow. But you'd need to remove the entire "stack" over the sensor, and then you'd need a steady hand with the black paint to pull it off, too. Maybe some black electrical tape would do the job

Obviously, I'm kidding. But I'll bet that some place like the ones who remove the IR filters might be able to handle this. Just think of it. For $500, you could have your camera modified to have sufficient masked pixels to allow removal of most "banding".

Or... Canon could do this for us when they manufacture the sensors. I, like John and others, wonder why they haven't done this.

I'll bet there are a bunch of Canon engineers reading these threads and biting their tongues, unable to post for various reasons. Maybe there's a simple explanation. A flaw in our thinking that would make this unworkable. But I really don't see what it would be.

I tend to think that if we did have a masked area at the top or bottom that we could make use of, then some worthwhile improvements could be made in our image quality.

-- hide signature --

Jim H.

JimH Forum Pro • Posts: 12,911
Re: About options

If they can control the variations closely enough to keep this kind of pattern noise down to very low levels using other means, then it may well make sense for them to use every bit of precious sensor area for actual image capture.

But if they can't, then I suspect a lot of people would gladly give up some percentage of the sensor's active area in exchange for the improvements we could derive from the masked pixels.

The problem is especially sticky because the masked areas need to be outside of the already established "active" sensor area. Nobody wants their 1.6X sensor to suddenly be a 1.7X sensor. So the die would actually need to be larger, and of course that adds cost.

I suspect that they do agonize over these decisions.

Who knows, there may be some engineers in Japan, reading this, and saying: "See, See! Some of our customers wanted that, but you wouldn't do it!!"

And then there may be others saying: "Well, but most customers don't have a clue about all of this, and most of the cameras are working fine. So we're still making more money by leaving the masked areas off."

-- hide signature --

Jim H.

Keyboard shortcuts:
FForum MMy threads