Canon Line Offset Noise

Started Oct 20, 2009 | Discussions
OP John Sheehy Forum Pro • Posts: 22,950
Re: crudely removing pattern you loose the interaction term

Blastophaga wrote:

signal treatment is a compromise between removing noise while only removing limited information in signal. if you dont remove noise at all you end pretty badly, this is what you see with the banding. what you are doing in your rough approximation is trying to remove one source of noise.

What other noise do you think you can possibly remove? The random 2-D noise is not something that is going to change without a change in sensor/readout design.

I have no interest in, and see no use in, trying to get rid of other noises, after capture. You can't do so, without losing detail, unless you know for a fact what the artifact is (as I do in my demo).

much of current progress in hig iso capacity is in detecting which patterns are best explained by noise and removing them (look at how the treatment of old raw files has improved over time).

Whoah, you're talking about conversion. I'm talking about the RAW data.

it's a tough and complex analysis. it's a question about information theory. obviously if Fourrier analysis and convolution products and contact processes dont speak to you, well then ... maybe we should forget about it ...

You're talking a lot of general stuff, with no direct application to RAW data, while keeping it somewhat RAW. I am addressing THE most absurdly unnecessary artifacts in RAW data, which need to be removed.

but if you dont believe a minimum in what you can do with maths applied to physics then we should probably forget about it and about a bunch of technological progress. put differently, compare the old files with the old treatments with the same RAW files and modern treatment and ask, do you really see less detail? I see more.

NR algorithms and rendering styles have improved. What does that have to do with what I was talking about?

-- hide signature --

John

OP John Sheehy Forum Pro • Posts: 22,950
Re: OK for black frames, what about "real" images

xxD wrote:

Alright. I can see what you've done and it makes sense and seems to be a straightforward data manipulation.

But that's a black frame and you get to make a lot of assumptions that make the correction easy because it's a black frame.

What complications are added to your algorithm if it's an photo with unpredictable subject and background detail?

None, if the bottom 100 rows don't receive light, or receive it evenly.

-- hide signature --

John

OP John Sheehy Forum Pro • Posts: 22,950
Re: Canon Line Offset Noise

BSchultz wrote:

Additionally, what are the settings for your black frame? Exposure time?

I don't have the RAW with me, but I'm sure it was a very short exposure. The noise is bright only because I made it bright, but I did so equally for both (same blackpoint and whitepoint).

This is a "very good" 7D, BTW.

-- hide signature --

John

ejmartin Veteran Member • Posts: 6,274
Re: crudely removing pattern you loose the interaction term

Blastophaga wrote:

signal treatment is a compromise between removing noise while only removing limited information in signal. if you dont remove noise at all you end pretty badly, this is what you see with the banding. what you are doing in your rough approximation is trying to remove one source of noise.

much of current progress in hig iso capacity is in detecting which patterns are best explained by noise and removing them (look at how the treatment of old raw files has improved over time). it's a tough and complex analysis. it's a question about information theory. obviously if Fourrier analysis and convolution products and contact processes dont speak to you, well then ... maybe we should forget about it ...

but if you dont believe a minimum in what you can do with maths applied to physics then we should probably forget about it and about a bunch of technological progress. put differently, compare the old files with the old treatments with the same RAW files and modern treatment and ask, do you really see less detail? I see more.

I'm somewhat familiar with the mathematics of signal processing, and your mention of contact processes left me doing a web search. Why do you think they are relevant?

The "feature" that John wants to remove is not random noise, rather it is a fixed offset of the data that varies in a predictable way from column to column. One doesn't need a lot of fancy analysis, just a good estimate of the offset, which one or a few black frames could supply.

-- hide signature --
Sylvain G Regular Member • Posts: 413
John, Gabor, BB - please consider

John, thanks for this post...
To you, Gabor, Bronxbombers and other reporters.

I've been following your posts for some time now and have always thought they were a real bonus for the community.

Though, latest posts generated quite some heat that reminded me how forums are, in the end, rarely a good place to get a message across. It just gets too noisy. Your posts get polluted with "go out and shoot", "too much scrutiny","we don't care..." and I think the posts then become totally useless.

My question is...

Have you thought about setting up a quick "blog" with your findings?
I think it would bring several advantages:

-MAINLY : major increase of visibility for the target (Canon?) & users don't have to jump deep into a rather annoying, imho, forum design to find your posts.

-users who actually aren't interested in this issue are less likely to come to a page dedicated on these issues.
-A unique repository of your findings with more flexibility to publish them.
-A moderated space to exchange theories with key users.

Setting up a blog is now only a matter of minutes on wordpress (just an example).

You could then go regularly to several fora of choice (fredmiranda, getdpi, etc...) and link to the blog regularly for interested people and to keep raising awareness of these issues I and many others take very seriously.

The goal is to make some clean noise for Canon to hear and not to get stuck in that maze pattern forum noise (ok, bad pun)

Greetings,
S.

John Sheehy wrote:

In another thread, I mentioned the ease at which Canon can remove artifacts from RAW data. In this post I will demonstrate how simple the offset banding in Canon RAW data is, and how easily it is removed.

The following composite image is the upper left corner of a 7D ISO 100 RAW blackframe. The image is 1000*700 pixels, composed of two versions of a single 1000*350 pixel crop. The top is the RAW data as is, with color interpolation (simple demosaic). The first ~150 columns of pixels is masked in the camera, and sees no light. The top ~32 rows is the same, but something strange is done to that data. I tried using the 27 rows at the top to make a mask to subtract from the RAW image, and the data was not very effective. Too few samples, and some artifacts not seen elsewhere in the RAW. If Canon meant for these to be masked pixels, they really messed it up.

So, for the bottom half, being that the image is a blackframe, and is effectively just like a fully masked image, I made a mask from the bottom 100 rows of pixels (less than 3% of the image height). I took the average in each 100 pixel column, and subtracted it from every pixel in the line. As you can see, even though I took the data used to subtract the banding from the bottom of the image, and my crops are from the top, 3400 pixels away, the pattern cancelled almost perfectly, proving that most of the banding runs at an exact offset, consistently from one end of the image to the other, and is very easily removed.

Now maybe some of you can understand why it is so frustrating to those of us who understand the simplicity of these problems, that Canon leaves this garbage in their RAW files.

-- hide signature --

my 'review' of the Olympus PEN E-P1 - http://www.catbag.net/wp/?p=173

xxD Contributing Member • Posts: 726
Re: OK for black frames, what about "real" images

John Sheehy wrote:

xxD wrote:

Alright. I can see what you've done and it makes sense and seems to be a straightforward data manipulation.

But that's a black frame and you get to make a lot of assumptions that make the correction easy because it's a black frame.

What complications are added to your algorithm if it's an photo with unpredictable subject and background detail?

None, if the bottom 100 rows don't receive light, or receive it evenly.

Is there a way for the algorithm to automatically determine if that's the case?

xxD Contributing Member • Posts: 726
Re: my 7D ISO100 black frames don't look that bad

Sam_NYC wrote:

Hi,

Could you please share some of your black frames from your 7D?

Where can I upload them?

Esbutt Regular Member • Posts: 457
Great idea..

I think that is more likely to force Canon to do something. A blog that pops up any time someone searches "canon banding" is going to be far be more effective than a few dispersed forume posts.

-- hide signature --
 Esbutt's gear list:Esbutt's gear list
Canon EOS 7D Canon EF-S 10-22mm F3.5-4.5 USM Canon EF-S 17-55mm f/2.8 IS USM Canon EF-S 60mm f/2.8 Macro USM Canon EF 70-200mm F4L IS USM
Sylvain G Regular Member • Posts: 413
Re: Great idea..

Yes! I didn't even think about that... good one
--
my 'review' of the Olympus PEN E-P1 - http://www.catbag.net/wp/?p=173

OP John Sheehy Forum Pro • Posts: 22,950
Re: OK for black frames, what about "real" images

xxD wrote:

John Sheehy wrote:

xxD wrote:

What complications are added to your algorithm if it's an photo with unpredictable subject and background detail?

None, if the bottom 100 rows don't receive light, or receive it evenly.

Is there a way for the algorithm to automatically determine if that's the case?

Sure, why not. If you're going to do an algorithm, it could look for flat dark areas throughout the image and try to make a mask from that, but it may be insufficient for the entire image. I only chose the bottom 100 rows because they could conceivably be covered over, guaranteed to always be blind.

This repeating pattern, however, may be mostly removed with a simple formula, or just an 8-element lookup table.

Using masked areas helps mostly remove all full-line offsets, regardless of whether they are periodic, or fixed (repeating from frame to frame).

-- hide signature --

John

OP John Sheehy Forum Pro • Posts: 22,950
Re: Canon Line Offset Noise

BSchultz wrote:

Additionally, what are the settings for your black frame? Exposure time?

1/125, f/8 (not that f-stop means anything here, but some cameras have stretched the RAW data by 1/3 stop when wide open apertures are used)), ISO 100.

-- hide signature --

John

bronxbombers Forum Pro • Posts: 18,226
Re: John, Gabor, BB - please consider

you are not the only person to have suggested this

i haven't done it yet, but I do at least have any empty link up to my 'blog' now though

you do make a lot of good points

Sylvain G wrote:
John, thanks for this post...
To you, Gabor, Bronxbombers and other reporters.

I've been following your posts for some time now and have always thought they were a real bonus for the community.

Though, latest posts generated quite some heat that reminded me how forums are, in the end, rarely a good place to get a message across. It just gets too noisy. Your posts get polluted with "go out and shoot", "too much scrutiny","we don't care..." and I think the posts then become totally useless.

My question is...

Have you thought about setting up a quick "blog" with your findings?
I think it would bring several advantages:

-MAINLY : major increase of visibility for the target (Canon?) & users don't have to jump deep into a rather annoying, imho, forum design to find your posts.

-users who actually aren't interested in this issue are less likely to come to a page dedicated on these issues.
-A unique repository of your findings with more flexibility to publish them.
-A moderated space to exchange theories with key users.

Setting up a blog is now only a matter of minutes on wordpress (just an example).

You could then go regularly to several fora of choice (fredmiranda, getdpi, etc...) and link to the blog regularly for interested people and to keep raising awareness of these issues I and many others take very seriously.

The goal is to make some clean noise for Canon to hear and not to get stuck in that maze pattern forum noise (ok, bad pun)

Greetings,
S.

John Sheehy wrote:

In another thread, I mentioned the ease at which Canon can remove artifacts from RAW data. In this post I will demonstrate how simple the offset banding in Canon RAW data is, and how easily it is removed.

The following composite image is the upper left corner of a 7D ISO 100 RAW blackframe. The image is 1000*700 pixels, composed of two versions of a single 1000*350 pixel crop. The top is the RAW data as is, with color interpolation (simple demosaic). The first ~150 columns of pixels is masked in the camera, and sees no light. The top ~32 rows is the same, but something strange is done to that data. I tried using the 27 rows at the top to make a mask to subtract from the RAW image, and the data was not very effective. Too few samples, and some artifacts not seen elsewhere in the RAW. If Canon meant for these to be masked pixels, they really messed it up.

So, for the bottom half, being that the image is a blackframe, and is effectively just like a fully masked image, I made a mask from the bottom 100 rows of pixels (less than 3% of the image height). I took the average in each 100 pixel column, and subtracted it from every pixel in the line. As you can see, even though I took the data used to subtract the banding from the bottom of the image, and my crops are from the top, 3400 pixels away, the pattern cancelled almost perfectly, proving that most of the banding runs at an exact offset, consistently from one end of the image to the other, and is very easily removed.

Now maybe some of you can understand why it is so frustrating to those of us who understand the simplicity of these problems, that Canon leaves this garbage in their RAW files.

Redteg94 Senior Member • Posts: 1,546
Re: The question I've got is...

Good idea Jim, I'd definitely pay for such a program, even though I don't find my 7D banding much at all in the shots I've taken so far. I'm sure it could be very useful for more intense PP of higher ISO shots
--
-Scott
http://www.flickr.com/photos/redteg94/

ejmartin Veteran Member • Posts: 6,274
It would certainly solve the post editing problem (nt)
-- hide signature --
Mark H
Mark H Veteran Member • Posts: 3,803
Your big 'wrong'...

John Sheehy wrote:

Mark H wrote:

Your description makes it sound simple - but the underlying 'problem' probably isn't.

I seriously doubt that Canon are simply neglecting to do something that basic and simple.

As I'm sure you know, Canon's sensor technologies are quite sophisticated - supposedly with both 'fixed pattern noise' and 'random noise' reduction mechanisms built in to their CMOS sensor designs, on a per-pixel basis.

http://www.usa.canon.com/uploadedimages/FCK/Image/White%20Papers/Canon_CMOS_WP.pdf

You seem to have a quite a faith in corporate cohesiveness and integrity.

I don't need 'faith' - I've 'been there & done that'...

I've worked 24 years for a 'high technology' company, which does business with many other high tech' companies like, and including, Canon and Nikon, etc, etc. I've worked alongside R&D, design, patents, production, production test, quality assurance, returns and repairs, sales and marketing, customer support - I've seen 'a lot' (both the good and the bad).

... and yes there are, in general, highly commendable levels of 'cohesiveness and integrity', often especially so from our Japanese counterparts - which no doubt contributes greatly to why so much Japanese consumer technology and automobiles get sold to 'the west' and not so much vice versa.

Canon consistently says things which are not correct in their white-papers.

You really are 'full of it'...

For example, they said that the bigger pixels in the 5D and 1D2 make for better high-ISO performance than the 20D pixels, because the former, being bigger. collect more photons. The fact is, all three collect the same number of photons with the same exposure.

...nonsense, your "For example..." is completely wrong.

The bigger pixels do collect more light (photons) - and that is why they do have better high-ISO performance.

For any given exposure value the 'constant' at the sensor is the 'light per unit area' - a larger pixel size (area) therefore equates to more light collected per pixel.

For a given exposure value - the larger the pixel size, the more light collected per pixel - the larger the sensor size, the more light collected by the sensor.

You very recently asserted that you were "very intelligent" - unfortunately you're not doing a very good job of demonstrating it.

Perhaps 'the problem' here is quite simply that there is a fault and/or quality issue.

No kidding. And I solved it.

Wow - I guess you really should be working for Canon then, as you obviously know better than Canon's entire design team does.

You point out yourself that the 'masked' pixels in the top '~32 rows' don't work as an effective 'mask' for subtraction - there probably is an explanation for this, but we will probably never know it.

What difference does it make? 100 rows from the bottom works very well. If Ii were to have these pixels painted black in my camera, I could do this with every image. Why doesn't Canon do this right when anyone else can?

Well maybe you should just go right ahead and start painting your bottom pixels black - (that's sarcasm by the way, so perhaps don't actually).

Canon is either stupid, apathetic, or malicious. There is no noble excuse.

It seems more evident that you are the 'stupid' and 'malicious' one - not Canon.

Developing that idea though (purely hypothetically) - possibly, a subtraction has already been carried out on the main unmasked area, and you might therefore be trying to double subtract - but at this stage I/we are just extrapolating on speculation.

It doesn't matter what has already been done - the fact is, it wasn't done right, if there is still simple pattern left. The only mystery with the top border is how Canon managed to mess it up so badly.

I feel it's a waste of time talking to someone of your disposition and conceit.

You have no idea as to whether or not the 'top border' is being used, or how - so you are not in any position to claim that Canon have 'messed it up'.

There appear to be plenty of Canon DSLR cameras out there that do not need any additional noise subtraction process.

If someone thinks that they have a bad camera that does not perform to an acceptable standard, then they should either get a replacement, or repair, or if they prefer return it and buy a different brand.

I can't fathom the mentality of someone who sees such solid evidence as this, and says there must be some insurmountable problem as an explanation. Get with Reality; Canon doesn't care - and they're not going to care if all their toadies keep defending them.

I don't defend Canon anymore than any balanced and fair minded person would defend any other company.

For what it's worth, this "Canon toady" (and supposedly paid "Canon shill", as you've accused me of in your previous ranting) currently shoots with a Nikon D90.

marcewicz Forum Member • Posts: 92
On the other hand....

Esbutt wrote:

I think that is more likely to force Canon to do something.

Why would Canon want to do something about nothing? As the saying goes: if it ain't broke, don't fix it.

bronxbombers Forum Pro • Posts: 18,226
Re: Your big 'wrong'...

...nonsense, your "For example..." is completely wrong.

The bigger pixels do collect more light (photons) - and that is why they do have better high-ISO performance.

For any given exposure value the 'constant' at the sensor is the 'light per unit area' - a larger pixel size (area) therefore equates to more light collected per pixel.

i don't have the figures in front of me, but I believe the 1D2 was actually quite inefficient to the point that it might well have done less in that regard.

For a given exposure value - the larger the pixel size, the more light collected per pixel -

not true at all, it totally depends

There appear to be plenty of Canon DSLR cameras out there that do not need any additional noise subtraction process.

well most do have have banding so of course they could benefit
not saying it is the end of the world

anyway canon did think it important since they did largely fix one of the three main types of banding with the 500D and moreso 7D

FZ20_Alff Regular Member • Posts: 105
How do I apply this in real life?

Hi John,

Assume for a moment that I agree with your argument about the noise issue. How do I put your technique into practice? In the short run complaining to Canon does not help me because I want to take photographs now.

Assume I have ACR or DPP. Allow me to understand. What steps do I need to do in my workflow to reduce this type of noise and have a better overall image?

Thank you for all of your hard work and persistance.

Alff

zappa1976 Regular Member • Posts: 496
Pls John help many of us... How do I apply this in real life?

Me too!

John, please can you explain in a easy way how to apply this correction to solve the banding in our real pictures?

Which program are necessary, how to calculate the avarage from every column and then how to subtract these?
THANK for your help..
Saverio

FZ20_Alff wrote:

Hi John,

Assume for a moment that I agree with your argument about the noise issue. How do I put your technique into practice? In the short run complaining to Canon does not help me because I want to take photographs now.

Assume I have ACR or DPP. Allow me to understand. What steps do I need to do in my workflow to reduce this type of noise and have a better overall image?

Thank you for all of your hard work and persistance.

Alff

Blastophaga Regular Member • Posts: 109
heat is prone to diffusion and generates noise

If noise has partly something to do with heat, then heat has nice diffusion properties (and hence contact process properties of self organisation).

If banding has something to do with physical structure of the sensor then this physical feature may affect diffusion processes and then we want to keep the info in the raw before treatment.

Of course I may be completely totally uterly wrong. It is just a line of thinking.

A several million pixel grid makes for quite powerfull spectral analysis using Fourier decomposition to look a bit at the structure of the noise and would probably provide some interesting insights.

Cheers

Finn

Keyboard shortcuts:
FForum MMy threads