Canon Line Offset Noise

Started Oct 20, 2009 | Discussions
Chez Wimpy
Chez Wimpy Veteran Member • Posts: 9,042
A joint effort would be great

Sylvain G wrote:

You could then go regularly to several fora of choice (fredmiranda, getdpi, etc...) and link to the blog regularly for interested people and to keep raising awareness of these issues I and many others take very seriously.

I remember a rather famous "somebody" that went on proselytizing... until he wound up getting kicked off of (many) boards for inciting conflict In this case though, lack of ulterior-motive (self-aggrandizement) or brand bashing (the contributors have Canon cameras, and genuinely want the product improved) make this an ideal topic for blogging. A repository of info such as the photon-collection counts for various cameras / isos, SNR, maximum DR, lists of "fake ISOs" ect... very useful stuff, only some of which I remember from stumbling upon these threads in the past.

-- hide signature --

-CW

よしよし、今日も生きのいい魂が手に入ったな

 Chez Wimpy's gear list:Chez Wimpy's gear list
Canon EOS 5D Mark II Canon EOS 20D Panasonic Lumix DMC-G1 Canon EOS 550D Panasonic Lumix DMC-GH2 +28 more
erikstefan Contributing Member • Posts: 821
Impossible to do Dark Frame Subtraction on banding

It is impossible to eliminate vertical banding with Dark Frame Subtraction.

The reson is because the "noise" in canon vertical banding is "dark", it's caused by the vertical wiring between the photocell.

While the dark frame subtraction is actually to subtract "illuminated noise" and "sensitivity difference" between pixels.

What you did in dark frame subtraction above is actually eliminating the "Signal" and illuminated noise on the "Signal", but it was not eliminating the vertical banding "Noise" which is the dark lines from the existance of wires in the design of Canon CMOS sensor.

Discussion are welcome
I am interested

Sylvain G Regular Member • Posts: 413
Re: On the other hand....

I'm not sure I read a hint of sarcasm or just another nay-sayer. In any case, you just made my point stronger.

Greetings,
S.

marcewicz wrote:

Esbutt wrote:

I think that is more likely to force Canon to do something.

Why would Canon want to do something about nothing? As the saying goes: if it ain't broke, don't fix it.

-- hide signature --

my 'review' of the Olympus PEN E-P1 - http://www.catbag.net/wp/?p=173

kolas Regular Member • Posts: 463
Re: It should work IMO

JimH wrote:

kolas wrote:

JimH wrote:

The question I've got is this:

Could a person come up with a good average value for these offsets for a given copy of the camera such that they'd be useful for other frames taken by the same camera?

I work in medical imaging field and exactly this is being done on digital X-ray detectors. A series of dark field and flat field (evenly exposed) images is taken to calibrate the detector for optimal results.

This seems like it would be extremely necessary for such systems. And it's got to make a huge difference given how the detection must be performed there.

Extremely low light, in photographic terms.

The medical "RAW converters" have been capable of correcting for uneven gains, barrell distortion and all kinds of artifacts and lens abberations quite some years ago. So the technology already exists and is developed to a great level of performance. I have seen some uncorrected images and could barely believe my eyes..

I'll bet they are pretty awful with no correction. With all of those separate detectors and the potential for geometric issues, I suppose that corrections are required before you get much of anything useful.

FYI most of the systems actually capture light, not X-rays, so the technology is very, very similar. One type of detectors is even based on optics and little CCD detectors, basically DSLR-size or MF-size sensors without Bayer RGB mask.

I remember looking into how the digital medical X-Ray detectors were implemented a while back. I do a lot of work with radiation detection equipment for laboratory and field use. It looked to me like arrays of scintillators coupled to light detector arrays was the standard way of doing this.

The so-called flat panels utilize a scintillator layer on TFT light detector matrix, CCD-based system have a separate scintillator screen with a WA lens pointing at it and a CCD detector. As the lens is basically a fisheye, there is a lot of geometric distortion that is automatically corrected by SW so that the output is a rectilinear image.

The same kinds of problems are present for both fields. Mainly, we want very high sensitivity so that low levels can be detected. After all, the medical folks don't want to roast the patients with any higher dose than necessary, and those of us doing environmental, prospecting, and research applications just want ever lower detection limits.

So you are familiar with the Radcal 1800ccm ionization chamber, right?

Btw. that is my dream, a dedicated BW DSLR, w/o the color filter array, with full luminance information in every pixel and ISO sensitivity shooting through the roof.

It's funny, but that's also been a dream of mine. Even though it seems like such a camera would be very cheap to build, I suppose that its limited appeal would still make it fairly expensive just due to the low demand.

That's a pity. Black and white is so wonderful. Imagine an 18 megapixel sensor with no CFA. Just a true 18 MP monochrome image with higher ISO sensitivity and a really weak (or no) AA filter. Maybe we're among the few who would like to see such a thing, but I'll bet there are at least a few others out there.

But I'd hate to have to pay a huge premium to get it.

DIY Basically all necessary parts are available off-the-shelf, they just won't fit into a SLR-sized body IMO.

-- hide signature --

Jim H.

-- hide signature --
Steen Bay Veteran Member • Posts: 7,418
DPP 3.7 maze artifacts?

Hulkster's very fine 7D vs 50D comparison (link in his post below) seems to show lots of maze artifacts in the DPP 3.7 pics, much more than with ACR, and it also seems to be worse with 50D than with 7D. A bit confused now. Wasn't this issue with maze supposed to be mostly a 7D/ACR thing?

http://forums.dpreview.com/forums/read.asp?forum=1019&message=33411995

rockbound Junior Member • Posts: 31
Banding Noise

This problem has been around for a long time. See this old thread and the response from Adobe.

http://forums.dpreview.com/forums/read.asp?forum=1019&message=12052096

Acording to this ACR should have this type of line by line noise reduction on RAW files for earlier Canon cams - maybe they dropped it for some reason or is does not work as well as expected.

Joe

Mrs Reality Regular Member • Posts: 366
Re: Your big 'wrong'...

Mark H wrote:

John Sheehy wrote:

Mark H wrote:

Your description makes it sound simple - but the underlying 'problem' probably isn't.

I seriously doubt that Canon are simply neglecting to do something that basic and simple.

As I'm sure you know, Canon's sensor technologies are quite sophisticated - supposedly with both 'fixed pattern noise' and 'random noise' reduction mechanisms built in to their CMOS sensor designs, on a per-pixel basis.

http://www.usa.canon.com/uploadedimages/FCK/Image/White%20Papers/Canon_CMOS_WP.pdf

You seem to have a quite a faith in corporate cohesiveness and integrity.

I don't need 'faith' - I've 'been there & done that'...

I've worked 24 years for a 'high technology' company, which does business with many other high tech' companies like, and including, Canon and Nikon, etc, etc. I've worked alongside R&D, design, patents, production, production test, quality assurance, returns and repairs, sales and marketing, customer support - I've seen 'a lot' (both the good and the bad).

... and yes there are, in general, highly commendable levels of 'cohesiveness and integrity', often especially so from our Japanese counterparts - which no doubt contributes greatly to why so much Japanese consumer technology and automobiles get sold to 'the west' and not so much vice versa.

Yeah, Canon's "integrity" really shows in the 1D3 fiasco.

Canon consistently says things which are not correct in their white-papers.

You really are 'full of it'...

For example, they said that the bigger pixels in the 5D and 1D2 make for better high-ISO performance than the 20D pixels, because the former, being bigger. collect more photons. The fact is, all three collect the same number of photons with the same exposure.

...nonsense, your "For example..." is completely wrong.

The bigger pixels do collect more light (photons) - and that is why they do have better high-ISO performance.

For any given exposure value the 'constant' at the sensor is the 'light per unit area' - a larger pixel size (area) therefore equates to more light collected per pixel.

For a given exposure value - the larger the pixel size, the more light collected per pixel - the larger the sensor size, the more light collected by the sensor.

You very recently asserted that you were "very intelligent" - unfortunately you're not doing a very good job of demonstrating it.

Perhaps 'the problem' here is quite simply that there is a fault and/or quality issue.

No kidding. And I solved it.

Wow - I guess you really should be working for Canon then, as you obviously know better than Canon's entire design team does.

You point out yourself that the 'masked' pixels in the top '~32 rows' don't work as an effective 'mask' for subtraction - there probably is an explanation for this, but we will probably never know it.

What difference does it make? 100 rows from the bottom works very well. If Ii were to have these pixels painted black in my camera, I could do this with every image. Why doesn't Canon do this right when anyone else can?

Well maybe you should just go right ahead and start painting your bottom pixels black - (that's sarcasm by the way, so perhaps don't actually).

Canon is either stupid, apathetic, or malicious. There is no noble excuse.

It seems more evident that you are the 'stupid' and 'malicious' one - not Canon.

Developing that idea though (purely hypothetically) - possibly, a subtraction has already been carried out on the main unmasked area, and you might therefore be trying to double subtract - but at this stage I/we are just extrapolating on speculation.

It doesn't matter what has already been done - the fact is, it wasn't done right, if there is still simple pattern left. The only mystery with the top border is how Canon managed to mess it up so badly.

I feel it's a waste of time talking to someone of your disposition and conceit.

You have no idea as to whether or not the 'top border' is being used, or how - so you are not in any position to claim that Canon have 'messed it up'.

There appear to be plenty of Canon DSLR cameras out there that do not need any additional noise subtraction process.

If someone thinks that they have a bad camera that does not perform to an acceptable standard, then they should either get a replacement, or repair, or if they prefer return it and buy a different brand.

The thing is, fewer people would have to return or exchange their cameras or send them in for repair if Canon would either design them better or have better quality control. There should be no such thing as a "bad copy" and no one should have to worry about getting a bad copy, or have to test copies to find a good one. It seems to me that this goes to the point that John and others are trying to make.

I can't fathom the mentality of someone who sees such solid evidence as this, and says there must be some insurmountable problem as an explanation. Get with Reality; Canon doesn't care - and they're not going to care if all their toadies keep defending them.

I don't defend Canon anymore than any balanced and fair minded person would defend any other company.

That's a laugh. A balanced person accepts reality.

For what it's worth, this "Canon toady" (and supposedly paid "Canon shill", as you've accused me of in your previous ranting) currently shoots with a Nikon D90.

You come across as a Canon shill.

erikstefan Contributing Member • Posts: 821
Re: Impossible to do Dark Frame Subtraction on banding

Of ccourse the "wires" are behind the photocells (pixels), not in between.

But the concept is still the same that the gaps between columns of photocell are dark, creating vertical dark lines as a banding noise.

I notice on 5D2 low iso, at shadows area the banding is vertical darker lines over lighter background.

erikstefan wrote:

It is impossible to eliminate vertical banding with Dark Frame Subtraction.

The reson is because the "noise" in canon vertical banding is "dark", it's caused by the vertical wiring between the photocell.

While the dark frame subtraction is actually to subtract "illuminated noise" and "sensitivity difference" between pixels.

What you did in dark frame subtraction above is actually eliminating the "Signal" and illuminated noise on the "Signal", but it was not eliminating the vertical banding "Noise" which is the dark lines from the existance of wires in the design of Canon CMOS sensor.

Discussion are welcome
I am interested

OP John Sheehy Forum Pro • Posts: 23,027
Re: Impossible to do Dark Frame Subtraction on banding

erikstefan wrote:

Of ccourse the "wires" are behind the photocells (pixels), not in between.

But the concept is still the same that the gaps between columns of photocell are dark, creating vertical dark lines as a banding noise.

I notice on 5D2 low iso, at shadows area the banding is vertical darker lines over lighter background.

With all due respect, what you wrote makes no sense in any way, shape or form.

The physical space between pixel columns or rows (however they may be wired) don't have anything to do with periodicity in RAW data. All the modulations of which you speak occur in a one-pixel period.

-- hide signature --

John

OP John Sheehy Forum Pro • Posts: 23,027
Re: DPP 3.7 maze artifacts?

Steen Bay wrote:

Hulkster's very fine 7D vs 50D comparison (link in his post below) seems to show lots of maze artifacts in the DPP 3.7 pics, much more than with ACR, and it also seems to be worse with 50D than with 7D. A bit confused now. Wasn't this issue with maze supposed to be mostly a 7D/ACR thing?

Mazing is a completely different issue than what I was discussing here, but the solution is also simple; the camera is already scaling the RAW data, it just needs to use different values than it is on the odd or even lines. There is no overhead in using a more correct value.

Mazing happens in ACR in any Canon or any camera with unequal gain in the odd and even lines. We have simply seen a very high percentage of poorly balanced gains in the first few 7D. I do not expect ACR to ever address this; it is not a "new camera" issue, IME. If enough people complain, they may address the problem, but they won't do it automatically. They may address it the wrong way, possibly - using the 4-color demosaic process, losing luminance resolution.

http://forums.dpreview.com/forums/read.asp?forum=1019&message=33411995

-- hide signature --

John

OP John Sheehy Forum Pro • Posts: 23,027
Re: Banding Noise

rockbound wrote:

This problem has been around for a long time. See this old thread and the response from Adobe.

http://forums.dpreview.com/forums/read.asp?forum=1019&message=12052096

Acording to this ACR should have this type of line by line noise reduction on RAW files for earlier Canon cams - maybe they dropped it for some reason or is does not work as well as expected.

Thomas Knoll was referring to the offset banding - the camera in question had lots of horizontal, non-periodic offset banding. You replied to Steenbay, who was talking about something else, mazing, which is caused by unequal gain.

It seems like many people are not making the distinction. Adobe has historically addressed offset banding on a camera-by-camera basis, based on perceived need. AFAIK, they have not addressed mazing. DPP never addresses offset banding, AFAIK, but always addresses mazing.

-- hide signature --

John

Mark H
Mark H Veteran Member • Posts: 3,803
Re: Your big 'wrong'...

bronxbombers wrote:

...nonsense, your "For example..." is completely wrong.

The bigger pixels do collect more light (photons) - and that is why they do have better high-ISO performance.

For any given exposure value the 'constant' at the sensor is the 'light per unit area' - a larger pixel size (area) therefore equates to more light collected per pixel.

i don't have the figures in front of me, but I believe the 1D2 was actually quite inefficient to the point that it might well have done less in that regard.

Regardless of camera specific variables - John's clear statement (regarding the 5D, 1D2 v 20D) was...

John Sheehy wrote:

The fact is, all three collect the same number of photons with the same exposure.

...which is complete nonsense - simply, wrong.

For a given exposure value - the larger the pixel size, the more light collected per pixel -

not true at all, it totally depends

Now what sort of a wooly statement is that - "...it totally depends"?

Even if we separate out possible differences in microlense filll factor, or CFA density (which are actually integral parts of the sensor/pixels) - even then, you would have to show a losses of almost -40% for yours or John's statements to be supported (that's almost a whole EV step).

I know exactly what John was saying, and he was clearly wrong - and even more foolishly wrong in using it to support his claim that "Canon consistently says things which are not correct in their white-papers" .

There appear to be plenty of Canon DSLR cameras out there that do not need any additional noise subtraction process.

well most do have have banding so of course they could benefit
not saying it is the end of the world

Undoubtedly - all cameras have noise, and many cameras' noise may well show some 'banding pattern' in the noise - particularly when staring into black shadows that have been 'pushed 3 stops'.

anyway canon did think it important since they did largely fix one of the three main types of banding with the 500D and moreso 7D

Well, you need tell John that - because he is convinced that Canon is 'stupid' and/or 'malicious' and couldn't care less.

Sylvain G Regular Member • Posts: 413
Re: DPP 3.7 maze artifacts?

John, please read my post somewhere above in thread view.

Thanks in advance,
S.
--
my 'review' of the Olympus PEN E-P1 - http://www.catbag.net/wp/?p=173

xxD Contributing Member • Posts: 726
Re: OK for black frames, what about "real" images

John Sheehy wrote:

xxD wrote:

John Sheehy wrote:

xxD wrote:

What complications are added to your algorithm if it's an photo with unpredictable subject and background detail?

None, if the bottom 100 rows don't receive light, or receive it evenly.

Is there a way for the algorithm to automatically determine if that's the case?

Sure, why not. If you're going to do an algorithm, it could look for flat dark areas throughout the image and try to make a mask from that, but it may be insufficient for the entire image. I only chose the bottom 100 rows because they could conceivably be covered over, guaranteed to always be blind.

I guess my real question is: How can you find "flat dark areas" automatically, in algorithm, if there's noise there (pattern or otherwise) that would make it appear to not be flat?

I might have missed something, but what's special about the bottom 100 rows?

rockbound Junior Member • Posts: 31
Re: Banding Noise

Guess I responded to the wrong thread. So what is the difference between:

"Canon line offset noise" - post title

and

"(Canon) offset banding (noise)"

The noise I experienced way back in with the S40 and 20D had horizontal line noise and it was somewhat periodic (in the vertical but) and could be removed by using the masked off areas of the sensor. It was line by line noise (horizontal) and was consistent over the whole line so could be removed by taking the average of the raw samples in the masked area at the beginning of a line and subtracting it from the image data in the remainder of the line.

What you seems to be saying is that the 7D has vertical lines (that could be called bands or offsets) that can't be removed by using the masked area at the top of the image.

Is this correct?

By the way I just checked an old S40 file with what I called horizontal line banding and CS3 removes the noise quite effectively

xxD Contributing Member • Posts: 726
Re: DPP 3.7 maze artifacts?

John Sheehy wrote:

We have simply seen a very high percentage of poorly balanced gains in the first few 7D.

That has not been sufficiently proven.

bill hansen Forum Pro • Posts: 10,016
intreresting discussion but.....

.....way above the heads of us amateurs. I like to read through things like this, and I do learn some interesting things. As I read this one, I wondered if there would ever be a practical way that we amateurs could use this information - short of complaining to Canon.

Even so - thanks for posting.

Bill

marcewicz Forum Member • Posts: 92
Re: intreresting discussion but.....

bill hansen wrote:

.....way above the heads of us amateurs. I like to read through things like this, and I do learn some interesting things. As I read this one, I wondered if there would ever be a practical way that we amateurs could use this information - short of complaining to Canon.

Complain to Canon about what , exactly?

Mark H
Mark H Veteran Member • Posts: 3,803
Completely unrealistic expectation...

Mrs Reality wrote:

Mark H wrote:

If someone thinks that they have a bad camera that does not perform to an acceptable standard, then they should either get a replacement, or repair, or if they prefer return it and buy a different brand.

The thing is, fewer people would have to return or exchange their cameras or send them in for repair if Canon would either design them better or have better quality control. There should be no such thing as a "bad copy" and no one should have to worry about getting a bad copy, or have to test copies to find a good one. It seems to me that this goes to the point that John and others are trying to make.

It shows great ignorance to say "There should be no such thing as a "bad copy" and no one should have to worry about getting a bad copy.." .

A DSLR design is incredibly complex and made of 1,000~2,000 individual, often precision, component parts.

Let's say there are 500 functionally 'quality critical parts' (and/or assembly processes), and the parts' 'quality' is such that just 1 part per 10,000 of each of those individual component parts might be 'faulty' - the probability of any one complete camera having a faulty part would be as high as 1 camera in every 20 (5% of production).

Hopefully, you can see that it is quite impossible for there to be "...no such thing as a bad copy." .

In reality, most manufacturers 'return/fault rate' will be far less - probably at least 10 times better (e.g. 0.5% or less).

Even with the highest quality standards, there will always be a very small percentage of problems - in most cases this will be only a fraction of a percent.

They manage to do all this at a cost that includes profit margins for all the manufacturer, distributors, retailers.

Camera manufacturers ship literally 'tens of thousands' of these DSLR models every single week.

If a manufacturer ships 'several million' or more DSLRs a year (and they do) - even when there are only 0.1% with problems, that still means there will be 'several thousand' with problems.

One problem with the 'noise' of the Internet, is that very often we only hear from the 1% who have problems - often the 99% who have no problems will never be heard from.

I can't fathom the mentality of someone who sees such solid evidence as this, and says there must be some insurmountable problem as an explanation. Get with Reality; Canon doesn't care - and they're not going to care if all their toadies keep defending them.

I don't defend Canon anymore than any balanced and fair minded person would defend any other company.

That's a laugh. A balanced person accepts reality.

So, perhaps you'll explain what your 'reality' is then?

For what it's worth, this "Canon toady" (and supposedly paid "Canon shill", as you've accused me of in your previous ranting) currently shoots with a Nikon D90.

You come across as a Canon shill.

That probably depends mostly on the particular disposition of who ever is reading, I think.

Mrs Reality Regular Member • Posts: 366
Re: Completely unrealistic expectation...

Mark H wrote:

Mrs Reality wrote:

Mark H wrote:

If someone thinks that they have a bad camera that does not perform to an acceptable standard, then they should either get a replacement, or repair, or if they prefer return it and buy a different brand.

The thing is, fewer people would have to return or exchange their cameras or send them in for repair if Canon would either design them better or have better quality control. There should be no such thing as a "bad copy" and no one should have to worry about getting a bad copy, or have to test copies to find a good one. It seems to me that this goes to the point that John and others are trying to make.

It shows great ignorance to say "There should be no such thing as a "bad copy" and no one should have to worry about getting a bad copy.." .

A DSLR design is incredibly complex and made of 1,000~2,000 individual, often precision, component parts.

Let's say there are 500 functionally 'quality critical parts' (and/or assembly processes), and the parts' 'quality' is such that just 1 part per 10,000 of each of those individual component parts might be 'faulty' - the probability of any one complete camera having a faulty part would be as high as 1 camera in every 20 (5% of production).

Hopefully, you can see that it is quite impossible for there to be "...no such thing as a bad copy." .

In reality, most manufacturers 'return/fault rate' will be far less - probably at least 10 times better (e.g. 0.5% or less).

Even with the highest quality standards, there will always be a very small percentage of problems - in most cases this will be only a fraction of a percent.

They manage to do all this at a cost that includes profit margins for all the manufacturer, distributors, retailers.

Camera manufacturers ship literally 'tens of thousands' of these DSLR models every single week.

If a manufacturer ships 'several million' or more DSLRs a year (and they do) - even when there are only 0.1% with problems, that still means there will be 'several thousand' with problems.

One problem with the 'noise' of the Internet, is that very often we only hear from the 1% who have problems - often the 99% who have no problems will never be heard from.

I can't fathom the mentality of someone who sees such solid evidence as this, and says there must be some insurmountable problem as an explanation. Get with Reality; Canon doesn't care - and they're not going to care if all their toadies keep defending them.

I don't defend Canon anymore than any balanced and fair minded person would defend any other company.

That's a laugh. A balanced person accepts reality.

So, perhaps you'll explain what your 'reality' is then?

My reality is believing my eyes when someone shows a flaw in a camera or lens, as John Sheehy and others have done with the 7D. As usual, some cameras have the problem and some others may not, or their owners may not notice it. Buying a Canon camera or lens is a risky gamble, and it shouldn't be.

Making excuses for Canon or any other brand is not going to get them to improve their design or QC.

For what it's worth, this "Canon toady" (and supposedly paid "Canon shill", as you've accused me of in your previous ranting) currently shoots with a Nikon D90.

You come across as a Canon shill.

That probably depends mostly on the particular disposition of who ever is reading, I think.

Keyboard shortcuts:
FForum MMy threads