Canon Line Offset Noise

In another thread, I mentioned the ease at which Canon can remove artifacts from RAW data. In this post I will demonstrate how simple the offset banding in Canon RAW data is, and how easily it is removed.

The following composite image is the upper left corner of a 7D ISO 100 RAW blackframe. The image is 1000*700 pixels, composed of two versions of a single 1000*350 pixel crop. The top is the RAW data as is, with color interpolation (simple demosaic). The first ~150 columns of pixels is masked in the camera, and sees no light. The top ~32 rows is the same, but something strange is done to that data. I tried using the 27 rows at the top to make a mask to subtract from the RAW image, and the data was not very effective. Too few samples, and some artifacts not seen elsewhere in the RAW. If Canon meant for these to be masked pixels, they really messed it up.

So, for the bottom half, being that the image is a blackframe, and is effectively just like a fully masked image, I made a mask from the bottom 100 rows of pixels (less than 3% of the image height). I took the average in each 100 pixel column, and subtracted it from every pixel in the line. As you can see, even though I took the data used to subtract the banding from the bottom of the image, and my crops are from the top, 3400 pixels away, the pattern cancelled almost perfectly, proving that most of the banding runs at an exact offset, consistently from one end of the image to the other, and is very easily removed.

Now maybe some of you can understand why it is so frustrating to those of us who understand the simplicity of these problems, that Canon leaves this garbage in their RAW files.

I tried this (on the original, raw pixels), as a column-wise correction (average calculated for every column separatedly) and the result looks excellent; however, this requires the "knowledge" of the specific camera, i.e. a profile, per ISO.

I tried using the top masked area as well (not all rows are usable), but the result depends on the camera: it is sometimes over-corrected, sometimes not enough.

The top masked area became so strange starting with the 50D, then the 5D2. I suspect that the firmware is using that for some corrections.

Your description makes it sound simple - but the underlying 'problem' probably isn't.

I seriously doubt that Canon are simply neglecting to do something that basic and simple.

As I'm sure you know, Canon's sensor technologies are quite sophisticated - supposedly with both 'fixed pattern noise' and 'random noise' reduction mechanisms built in to their CMOS sensor designs, on a per-pixel basis.

http://www.usa.canon.com/uploadedimages/FCK/Image/White%20Papers/Canon_CMOS_WP.pdf

Perhaps 'the problem' here is quite simply that there is a fault and/or quality issue.

You point out yourself that the 'masked' pixels in the top '~32 rows' don't work as an effective 'mask' for subtraction - there probably is an explanation for this, but we will probably never know it.

Developing that idea though (purely hypothetically) - possibly, a subtraction has already been carried out on the main unmasked area, and you might therefore be trying to double subtract - but at this stage I/we are just extrapolating on speculation.

The question I've got is this:

Could a person come up with a good average value for these offsets for a given copy of the camera such that they'd be useful for other frames taken by the same camera?

If so, and if things don't drift or change with temperature, then a person might well be able to feed a few black frames into a converter program for analysis, store that camera's "black profile", and then have it automatically massage our CR2 files from that camera before we even suck them into DPP or ACR.

This could be automated to operate on batches of files (say, everything in a directory). So it'd be an easy thing to apply to all of our shots to remove this problem.

JimH wrote:

The question I've got is this:

Could a person come up with a good average value for these offsets for a given copy of the camera such that they'd be useful for other frames taken by the same camera?

Probably, I would say, for this particular pattern. A simple formula based on the modulo of the x value would probably work, with a factor for ISO if necessary. The approach I used would work for any whole-line offset, and need not know about ISO.

If so, and if things don't drift or change with temperature, then a person might well be able to feed a few black frames into a converter program for analysis, store that camera's "black profile", and then have it automatically massage our CR2 files from that camera before we even suck them into DPP or ACR.

Yep. Just like dust data, we could have gain and offset corrections.

This could be automated to operate on batches of files (say, everything in a directory). So it'd be an easy thing to apply to all of our shots to remove this problem.

Yes, but it would be nice if the camera calibrated itself, and this was all done before the RAW is written. 14 bits should be enough for doing the necessary corrections, but a mediocre 16-bit ADC with 13 bits of real precision (3 for dither - better to use noisy bits in an ADC than to "add" dithering noise) should be sufficient for the calculations, without causing histogram shifts.

As far as the other main issue, gain, is concerned, the 7D is already compressing the original RAW histogram, so just changing the factors to equalize the channels would add no further problems.

Mark H wrote:

Your description makes it sound simple - but the underlying 'problem' probably isn't.

I seriously doubt that Canon are simply neglecting to do something that basic and simple.

As I'm sure you know, Canon's sensor technologies are quite sophisticated - supposedly with both 'fixed pattern noise' and 'random noise' reduction mechanisms built in to their CMOS sensor designs, on a per-pixel basis.

http://www.usa.canon.com/uploadedimages/FCK/Image/White%20Papers/Canon_CMOS_WP.pdf

You seem to have a quite a faith in corporate cohesiveness and integrity. Canon consistently says things which are not correct in their white-papers. For example, they said that the bigger pixels in the 5D and 1D2 make for better high-ISO performance than the 20D pixels, because the former, being bigger. collect more photons. The fact is, all three collect the same number of photons with the same exposure. The 1D2's real benefit was lower banding noise, and higher contrast from the lenses, with a weaker AA filter and larger pixel pitch.

Perhaps 'the problem' here is quite simply that there is a fault and/or quality issue.

No kidding. And I solved it. Coarsely of course - I just did a quickie, but the accuracy can be improved, and splitting the blind pixels to both sides of the columns would eliminate ramping better, which, I believe, may account for some of the small amount of vertical banding remaining in the lower image.

You point out yourself that the 'masked' pixels in the top '~32 rows' don't work as an effective 'mask' for subtraction - there probably is an explanation for this, but we will probably never know it.

What difference does it make? 100 rows from the bottom works very well. If Ii were to have these pixels painted black in my camera, I could do this with every image. Why doesn't Canon do this right when anyone else can?

Canon is either stupid, apathetic, or malicious. There is no noble excuse.

Developing that idea though (purely hypothetically) - possibly, a subtraction has already been carried out on the main unmasked area, and you might therefore be trying to double subtract - but at this stage I/we are just extrapolating on speculation.

It doesn't matter what has already been done - the fact is, it wasn't done right, if there is still simple pattern left. The only mystery with the top border is how Canon managed to mess it up so badly.

I can't fathom the mentality of someone who sees such solid evidence as this, and says there must be some insurmountable problem as an explanation. Get with Reality; Canon doesn't care - and they're not going to care if all their toadies keep defending them.

I am not sure I agree with your simple way of pretreating a raw.

what you assume is that there is no interaction term between banding pattern and noise so that if you remove the banding pattern, you get the same power for treating residual noise as if you knew where you were on the banding pattern and used this info 1) to remove it but also 2) as an input for the remainder of noise treatment.

I suspect part of the noise in the raw files involves contact processes. if so you want the capacity to fully analyse the resulting noise patterns within a nice physical framework. the mathematical properties of contact processes are much fun and could be affected by the physical structure producing the banding.

but obviously I dont know if this is the case.

Cheers

Finn

none of my 7D photos look that bad.

Alright. I can see what you've done and it makes sense and seems to be a straightforward data manipulation.

But that's a black frame and you get to make a lot of assumptions that make the correction easy because it's a black frame.

What complications are added to your algorithm if it's an photo with unpredictable subject and background detail?

John Sheehy wrote:

In another thread, I mentioned the ease at which Canon can remove artifacts from RAW data. In this post I will demonstrate how simple the offset banding in Canon RAW data is, and how easily it is removed.

The following composite image is the upper left corner of a 7D ISO 100 RAW blackframe. The image is 1000*700 pixels, composed of two versions of a single 1000*350 pixel crop. The top is the RAW data as is, with color interpolation (simple demosaic). The first ~150 columns of pixels is masked in the camera, and sees no light. The top ~32 rows is the same, but something strange is done to that data. I tried using the 27 rows at the top to make a mask to subtract from the RAW image, and the data was not very effective. Too few samples, and some artifacts not seen elsewhere in the RAW. If Canon meant for these to be masked pixels, they really messed it up.

So, for the bottom half, being that the image is a blackframe, and is effectively just like a fully masked image, I made a mask from the bottom 100 rows of pixels (less than 3% of the image height). I took the average in each 100 pixel column, and subtracted it from every pixel in the line. As you can see, even though I took the data used to subtract the banding from the bottom of the image, and my crops are from the top, 3400 pixels away, the pattern cancelled almost perfectly, proving that most of the banding runs at an exact offset, consistently from one end of the image to the other, and is very easily removed.

Now maybe some of you can understand why it is so frustrating to those of us who understand the simplicity of these problems, that Canon leaves this garbage in their RAW files.

Just thought I'd add that my 7D ISO100 black frames don't look as bad as that top portion.

xxD wrote:

Alright. I can see what you've done and it makes sense and seems to be a straightforward data manipulation.

But that's a black frame and you get to make a lot of assumptions that make the correction easy because it's a black frame.

What complications are added to your algorithm if it's an photo with unpredictable subject and background detail?

John Sheehy wrote:

In another thread, I mentioned the ease at which Canon can remove artifacts from RAW data. In this post I will demonstrate how simple the offset banding in Canon RAW data is, and how easily it is removed.

The following composite image is the upper left corner of a 7D ISO 100 RAW blackframe. The image is 1000*700 pixels, composed of two versions of a single 1000*350 pixel crop. The top is the RAW data as is, with color interpolation (simple demosaic). The first ~150 columns of pixels is masked in the camera, and sees no light. The top ~32 rows is the same, but something strange is done to that data. I tried using the 27 rows at the top to make a mask to subtract from the RAW image, and the data was not very effective. Too few samples, and some artifacts not seen elsewhere in the RAW. If Canon meant for these to be masked pixels, they really messed it up.

So, for the bottom half, being that the image is a blackframe, and is effectively just like a fully masked image, I made a mask from the bottom 100 rows of pixels (less than 3% of the image height). I took the average in each 100 pixel column, and subtracted it from every pixel in the line. As you can see, even though I took the data used to subtract the banding from the bottom of the image, and my crops are from the top, 3400 pixels away, the pattern cancelled almost perfectly, proving that most of the banding runs at an exact offset, consistently from one end of the image to the other, and is very easily removed.

Now maybe some of you can understand why it is so frustrating to those of us who understand the simplicity of these problems, that Canon leaves this garbage in their RAW files.

Blastophaga wrote:

I am not sure I agree with your simple way of pretreating a raw.

what you assume is that there is no interaction term between banding pattern and noise so that if you remove the banding pattern, you get the same power for treating residual noise as if you knew where you were on the banding pattern and used this info 1) to remove it but also 2) as an input for the remainder of noise treatment.

I suspect part of the noise in the raw files involves contact processes. if so you want the capacity to fully analyse the resulting noise patterns within a nice physical framework. the mathematical properties of contact processes are much fun and could be affected by the physical structure producing the banding.

but obviously I dont know if this is the case.

Why would it matter? The banding is the main problem which has a solution. The real random 2d read noise can't be "corrected"; it has to be AVOIDED .

What else, besides the banding, do you see any hope of removing without removing detail?

Hi,

Could you please share some of your black frames from your 7D?

Thanks.

JimH wrote:

The question I've got is this:

Could a person come up with a good average value for these offsets for a given copy of the camera such that they'd be useful for other frames taken by the same camera?

I work in medical imaging field and exactly this is being done on digital X-ray detectors. A series of dark field and flat field (evenly exposed) images is taken to calibrate the detector for optimal results.

The medical "RAW converters" have been capable of correcting for uneven gains, barrell distortion and all kinds of artifacts and lens abberations quite some years ago. So the technology already exists and is developed to a great level of performance. I have seen some uncorrected images and could barely believe my eyes..

FYI most of the systems actually capture light, not X-rays, so the technology is very, very similar. One type of detectors is even based on optics and little CCD detectors, basically DSLR-size or MF-size sensors without Bayer RGB mask.

Btw. that is my dream, a dedicated BW DSLR, w/o the color filter array, with full luminance information in every pixel and ISO sensitivity shooting through the roof.
--
Marek
http://galerie.kolas.cz/

Additionally, what are the settings for your black frame? Exposure time?

xxD wrote:

Alright. I can see what you've done and it makes sense and seems to be a straightforward data manipulation.

But that's a black frame and you get to make a lot of assumptions that make the correction easy because it's a black frame.

What complications are added to your algorithm if it's an photo with unpredictable subject and background detail?

There are potentially several forms/sources of pattern noise:

1. Fixed pattern noise due to inconsistent black point in columns

2. Random pattern noise, line noise that doesn't repeat from shot to shot; can be both vertical and horizontal.

3. Vertical line noise due to mismatched ampilfier gains of different columns

John's method will deal with (1), if one simply records the data and applies it as a subtraction to every RAW image. It could deal with (2) if there were enough masked pixels on the border of the frame to measure accurately the random line noise in a given image; being random, it changes from image to image. Issue (3) is nasty and a camera that exhibits it (such as bronxbomber's) should be returned/repaired.

My Canon cameras (20D,40D,1D3) all exhibit both (1) and (2) to various extents. The 7D seems to be a improvement on (2), but has a bit of residual (1) that John is complaining about because it is the most easily dealt with of the three.

signal treatment is a compromise between removing noise while only removing limited information in signal. if you dont remove noise at all you end pretty badly, this is what you see with the banding. what you are doing in your rough approximation is trying to remove one source of noise.

much of current progress in hig iso capacity is in detecting which patterns are best explained by noise and removing them (look at how the treatment of old raw files has improved over time). it's a tough and complex analysis. it's a question about information theory. obviously if Fourrier analysis and convolution products and contact processes dont speak to you, well then ... maybe we should forget about it ...

but if you dont believe a minimum in what you can do with maths applied to physics then we should probably forget about it and about a bunch of technological progress. put differently, compare the old files with the old treatments with the same RAW files and modern treatment and ask, do you really see less detail? I see more.

cheers

finn

John Sheehy wrote:

This could be automated to operate on batches of files (say, everything in a directory). So it'd be an easy thing to apply to all of our shots to remove this problem.

Yes, but it would be nice if the camera calibrated itself, and this was all done before the RAW is written. 14 bits should be enough for doing the necessary corrections, but a mediocre 16-bit ADC with 13 bits of real precision (3 for dither - better to use noisy bits in an ADC than to "add" dithering noise) should be sufficient for the calculations, without causing histogram shifts.

As far as the other main issue, gain, is concerned, the 7D is already compressing the original RAW histogram, so just changing the factors to equalize the channels would add no further problems.

I guess my main point here was that given what we do have from Canon right now, and given that nothing we say here seems to ever have any effect on their cameras, it would be nice to develop a little program that photographers could use to reduce the vertical banding (and maybe other problems as well).

The program could easily store different calibrations for different ISOs, and it wouldn't take much effort on the part of the photographer to shoot a group of black frames to "feed" into the program to get it calibrated for their camera.

The EXIF from the files could be read by the program so it'd recognize the serial number of the camera and the ISO. Perhaps some other parameters could be used, too (shutter speed?).

So a photographer could then pre-process his or her shots with this program to clean them up prior to feeding them into their favorite RAW converter.

Let's not think of Canon's apathy as a bad thing. Let's think of it as an opportunity. An opportunity for some clever programmers (ahem) to make some money off of it all

I'll bet that even the Canon apologists might be our customers here (wink wink, nudge nudge).

If it was set up to work for any Canon DSLR, and became a hugely popular program, selling very well, maybe that would get Canon's attention. Then again, maybe not!

kolas wrote:

JimH wrote:

The question I've got is this:

Could a person come up with a good average value for these offsets for a given copy of the camera such that they'd be useful for other frames taken by the same camera?

I work in medical imaging field and exactly this is being done on digital X-ray detectors. A series of dark field and flat field (evenly exposed) images is taken to calibrate the detector for optimal results.

This seems like it would be extremely necessary for such systems. And it's got to make a huge difference given how the detection must be performed there.

The medical "RAW converters" have been capable of correcting for uneven gains, barrell distortion and all kinds of artifacts and lens abberations quite some years ago. So the technology already exists and is developed to a great level of performance. I have seen some uncorrected images and could barely believe my eyes..

I'll bet they are pretty awful with no correction. With all of those separate detectors and the potential for geometric issues, I suppose that corrections are required before you get much of anything useful.

FYI most of the systems actually capture light, not X-rays, so the technology is very, very similar. One type of detectors is even based on optics and little CCD detectors, basically DSLR-size or MF-size sensors without Bayer RGB mask.

I remember looking into how the digital medical X-Ray detectors were implemented a while back. I do a lot of work with radiation detection equipment for laboratory and field use. It looked to me like arrays of scintillators coupled to light detector arrays was the standard way of doing this.

The same kinds of problems are present for both fields. Mainly, we want very high sensitivity so that low levels can be detected. After all, the medical folks don't want to roast the patients with any higher dose than necessary, and those of us doing environmental, prospecting, and research applications just want ever lower detection limits.

Btw. that is my dream, a dedicated BW DSLR, w/o the color filter array, with full luminance information in every pixel and ISO sensitivity shooting through the roof.

It's funny, but that's also been a dream of mine. Even though it seems like such a camera would be very cheap to build, I suppose that its limited appeal would still make it fairly expensive just due to the low demand.

That's a pity. Black and white is so wonderful. Imagine an 18 megapixel sensor with no CFA. Just a true 18 MP monochrome image with higher ISO sensitivity and a really weak (or no) AA filter. Maybe we're among the few who would like to see such a thing, but I'll bet there are at least a few others out there.

But I'd hate to have to pay a huge premium to get it.

sharpshutter wrote:

none of my 7D photos look that bad.

These are pushed for easy comparison.