7D vertical banding - firmware error

[riknash]

The firmware seems to do something with the data in the borders, but whatever it does, it doesn't do it right, and overwrites some of the border areas in various ways. In earlier Canons, the masked borders were indistinguishable from the image area in a blackframe. In the 7Ds, the level of banding is sometimes different in the borders and in the blackframe image area. Canon has poor mathematicians, I think, because when I simulate what they do in firmware myself, I get better results. If I take a blackframe and treat some of it's image edges as if they were masked borders, I can subtract a lot of banding that Canon left in.

Can this simulated firmware processing be applied to actual photos using a specific sensors characteristics as the basis for the correction?

 

[jm67]

Very nice. Canon should hire you.
--
I am as ignorant as ever, and wiser than I've ever been.

 

[John Sheehy]

Very nice. Canon should hire you.
--
I am as ignorant as ever, and wiser than I've ever been.

Well, I didn't do anything unique or elaborate there; I just had the program (IRIS) subtract one blackframe from another. This is what the camera should have done.

This suggests that those deep thin bands are done after digitization. If they happened before or during digitization, they should have canceled themselves.

--
John

 

[John Sheehy]

OK, here is another visual. This is two strips, from the camera LENR (top) and from a single frame of the same duration, a couple of seconds later. They are from an 800-pixel-wide strip of each RAW, full height, downsampled 1% vertically. You can see that the vast majority of the deep notches are the same in both. Now, if they happened before or during digitization, they would have canceled out in the top strip, but they did not, and are the same as the single frame, which makes me think that this error is introduced before writing the RAW, but after digitization.



--
John

 

[dprsok]

Any comment on J Watkins observation...

http://forums.dpreview.com/forums/read.asp?forum=1019&message=34679103

--
http://www.flickr.com/photos/39169343@N04/

 

[Fred Dominic]

Very interesting! And nice discovery!

This next question is not entirely on point, but would it be possible to also examine what would happen if you subtract out an average (or even a median) of multiple black frames, rather than just one black frame (as we spoke about earlier in this thread). I.e., would it be easy for you to generate such an example? The reason is that even the one that you did still shows signs of periodic vertical banding, while of course not nearly as severe as the internal camera LENR subtraction. I.e., from the perspective of how to best correct the problem (since it is unlikely canon is going to do anything about it, and for those people who have 7Ds now), it'd be quite useful to know the extent to which the problem can be mitigated in post.

Sorry, I've been busy with other things, and the software has been a pain to work with, so some of my visual comparisons are not yet done, but here is the original issue I mentioned, the way the camera adds deep bands apparently after digitization.

The left side here is the camera doing the LENR; the right side is me doing it manually.

The values in the threshold slider have been multiplied by 100, so that I could get the mean close to zero in both. The real range here is 5 14-bit levels below black, to 5 above black:



--
John

 

[Fred Dominic]

Ok, so both of these strips are with camera LENR right? If so, while yes most of the notches are the same in both, there still seems to be some variability. Is the variability due to the LENR processing, or is the variability due to variability in the noise pattern in the original non-BF-subtracted images?

From the perspective of developing our own best method to remove this problem from our images in post, it would perhaps also be useful to see the same for the original, non-LENR images. I.e., the quest is to see the extent (both within the same camera and across different instances of different 7Ds) to which the pattern of noise is stable, so as to see how simple a universal fix at post might possibly be.

Again thanks!

OK, here is another visual. This is two strips, from the camera LENR (top) and from a single frame of the same duration, a couple of seconds later. They are from an 800-pixel-wide strip of each RAW, full height, downsampled 1% vertically. You can see that the vast majority of the deep notches are the same in both. Now, if they happened before or during digitization, they would have canceled out in the top strip, but they did not, and are the same as the single frame, which makes me think that this error is introduced before writing the RAW, but after digitization.



--
John

 

[John Sheehy]

Any comment on J Watkins observation...

http://forums.dpreview.com/forums/read.asp?forum=1019&message=34679103

I'll have to update DPP to see what's going on, but I've also read reports of people saying no difference in the new DPP. The examples I've seen where the black car is not banded when pulled up almost to gray look a bit NR-washed.

--
John

 

[John Sheehy]

Ok, so both of these strips are with camera LENR right? If so, while yes most of the notches are the same in both, there still seems to be some variability. Is the variability due to the LENR processing, or is the variability due to variability in the noise pattern in the original non-BF-subtracted images?

The top is the camera, using its LENR. The bottom is just a single frame of the same exposure time (1.3s, IIRC). BTW, other than the deep banding, the camera LENR seems to go beyond subtraction; it may do some outlier removal as well, since the camera LENR has a lower std dev than my software subtraction.

Between any two frames, the will be some unique banding. Any non-fixed noise increases 41% when you subtract one RAW from another. Subtracting a stack removes the fixed patterns, without introducing much extra noise.

From the perspective of developing our own best method to remove this problem from our images in post, it would perhaps also be useful to see the same for the original, non-LENR images.

Again, that's what the bottom is in these strips. Only the top is from one of the LENRs.

I.e., the quest is to see the extent (both within the same camera and across different instances of different 7Ds) to which the pattern of noise is stable, so as to see how simple a universal fix at post might possibly be.

There are a lot of combinations to try, but I haven't been spending much time on this lately.

--
John

 

[Fred Dominic]

Ok, so both of these strips are with camera LENR right? If so, while yes most of the notches are the same in both, there still seems to be some variability. Is the variability due to the LENR processing, or is the variability due to variability in the noise pattern in the original non-BF-subtracted images?

The top is the camera, using its LENR. The bottom is just a single frame of the same exposure time (1.3s, IIRC). BTW, other than the deep banding, the camera LENR seems to go beyond subtraction; it may do some outlier removal as well, since the camera LENR has a lower std dev than my software subtraction.

Oh, ok. That makes more sense then (top being but bottom not being LENR).

Between any two frames, the will be some unique banding. Any non-fixed noise increases 41% when you subtract one RAW from another. Subtracting a stack removes the fixed patterns, without introducing much extra noise.

From the perspective of developing our own best method to remove this problem from our images in post, it would perhaps also be useful to see the same for the original, non-LENR images.

Again, that's what the bottom is in these strips. Only the top is from one of the LENRs.

I.e., the quest is to see the extent (both within the same camera and across different instances of different 7Ds) to which the pattern of noise is stable, so as to see how simple a universal fix at post might possibly be.

There are a lot of combinations to try, but I haven't been spending much time on this lately.

You've already contributed quite a bit!

--
John

 

[ponytear]

Hello everyone,

I have been reading all your posts with great interest, especially since I noticed that vertical banding in my pictures recently, which were surprisignly noisy at ISO 200 with no apparent reason (I had been using a 350D for years and never had such a problem at low ISO).

It has probably been told in another topic, but I haven't seen it mentioned here: the "Highlight Priority Tone" option makes a BIG difference in the banding and noise in the shadows.

Here are 100% crops of two CR2 at ISO 200. They were developped in ACR and all the settings are identical for both images: WB as shot (5250 / +8), Exposure +4 (for testing purpose), Luminosity 50, Luminance noise 50 and color noise 25 (all other settings set to 0).

The ONLY thing that changes is the "Highlight Priority Tone" that is enabled in the first picture and disabled in the second one.

I don't think I will be using this option again in the future. I don't know if it helps...

 

[jm67]

Interesting. I wasn't aware that using htp had any effect whatsoever on the raw file, only the jpeg.
--
I am as ignorant as ever, and wiser than I've ever been.

 

[John Sheehy]

Interesting. I wasn't aware that using htp had any effect whatsoever on the raw file, only the jpeg.
--
I am as ignorant as ever, and wiser than I've ever been.

ISO 200 with HTP is just metering ISO 100 gain for ISO 200, so it is basically ISO 100 at -1 EC, under the hood. 250 with HTP is 100 at -1.33EC.

--
John

 

[landmarks4711589173]

unless if you can show me some strong valid samples

Coming soon.

I appreciate that

I doubt canon can fix it with firmware
The only way to fix it is in Mark III

 

[fanboy]

Any comment on J Watkins observation...

http://forums.dpreview.com/forums/read.asp?forum=1019&message=34679103

I'll have to update DPP to see what's going on, but I've also read reports of people saying no difference in the new DPP. The examples I've seen where the black car is not banded when pulled up almost to gray look a bit NR-washed.

--
John

You haven't even looked at the latest DPP? This brings all your claims and conclusions into question.

 

[NetMage]

DPP is irrelevant to the question of whether LENR is adding or subtracting the black frame when doing in-camera processing. It would seem that a RAW taken with LENR should actually save the original data and blackframe separately and then if DPP improved the picture it would be proof of the firmware bug, but I think John's proved it fairly well.

My conclusion is if you plan to do your own BF subtraction anyway (for astro) ou should leave LENR off, which is probably true even if it worked correctly, but on the 7D you should always leave it off.
--
EOS 50D, 20D, 10D, 630, A-1, FZ28, SD1000
-- Please remove the Quote option!
-- Why can't you edit more than once???
-- How about switching to real forum software?

 

[talu845]

There is a lot of detail and content on this thread, but I just wanted to add that I've noticed some evidence of vertical banding on my 7D at ISO 100. It's a pain in backside having spent a fair chunk of cash to find this.

It's not a chronic issue, I've only come to find it when I did some long exposures (20 to 30 seconds) using a 10 stop filter (LENR turned off). I could see some very slight banding in blue sky when viewing the image at 100%, although I managed to correct most of it in Noise Ninja whilst post processing, but I'd rather not have to use noise reduction to correct it if possible.

Fingers crossed for a firmware fix.

 

[photopsia]

I am new to this forum and new to Canon cameras.

I purchased a 7D in Feb 2010, with factor installed firmware version 1.1.0. I proceeded to take it for a vacation in Hawaii. All photos were recorded in RAW format. Shadow areas of photographs show vertical banding clearly. I assemble many panoramas, and after converting photos to tiff format using DPP batch conversion, assembling the panoramas the vertical banding is even more apparent. I have never seen this problem on Nikon D300. Sample images are shown below. Please help. I would like to rescue many of the vacation photos.





Above image is 100% crop of from a vacation photo from Hawaii, opened in DPP. The only change made in DPP default settings is Brightness set to 2.0 - to highlight the shadows.





This image shows vertical banding in a pure black image.





This is the average of 30 black images taken at ISO 400, speed 1/8000 - showing clear vertical bands. This vertical banding offset is present in all the black photos.

 

[Brian D. Schneider]

I am new to this forum and new to Canon cameras.

I purchased a 7D in Feb 2010, with factor installed firmware version 1.1.0. I proceeded to take it for a vacation in Hawaii. All photos were recorded in RAW format. Shadow areas of photographs show vertical banding clearly. I assemble many panoramas, and after converting photos to tiff format using DPP batch conversion, assembling the panoramas the vertical banding is even more apparent. I have never seen this problem on Nikon D300. Sample images are shown below. Please help. I would like to rescue many of the vacation photos.





Above image is 100% crop of from a vacation photo from Hawaii, opened in DPP. The only change made in DPP default settings is Brightness set to 2.0 - to highlight the shadows.





This image shows vertical banding in a pure black image.





This is the average of 30 black images taken at ISO 400, speed 1/8000 - showing clear vertical bands. This vertical banding offset is present in all the black photos.

Nik Dfine can remove banding. They have a free trial you can download.
http://www.niksoftware.com/dfine/usa/entry.php
--
Brian Schneider

 

[John Sheehy]

This is the average of 30 black images taken at ISO 400, speed 1/8000 - showing clear vertical bands. This vertical banding offset is present in all the black photos.

unfortunately, I know of no way to subtract the pattern in a "normal" converter. You can do it in IRIS, but you're still in the RAW color space, without any quality demosaicing. I've e-mailed the author of IRIS requesting that he offer DNG as an output format, so that images can be corrected and converted in a variety of converters, but he did not reply (and never has about anything else I've suggested, like an option to load RAWs with the black mask). The 7D still has a small amount of vertical banding after you subtract the black mask stack, but nothing like a single image, with much shallower modulation.

--
John