7D vertical banding - firmware error

Started Feb 12, 2010 | Discussions
John Sheehy
Forum ProPosts: 15,832
Like?
7D vertical banding - firmware error
Feb 12, 2010

I've alluded to this many times before, based on numerous isolated experiences and impressions, but today I did a controlled experiment to tie it all together.

The fact is, most of the periodic vertical banding in the 7D does NOT happen during readout, but happens after digitization!

I noticed a few months ago that if you subtracted one 7D ISO 100 RAW from another, that the periodic component mostly disappears, and you are left with a 5d2-like random burlap pattern of a lower magnitude with just a slight hint of soft periodic banding. I noticed a couple of weeks ago that long-exposure noise reduction does not eliminate the strong periodical vertical component. Tonight I did it both ways with blackframes, and compared the results. The periodic banding was present when the camera subtracted one blackframe from another, but when I did it, the periodic elementmostly disappeared. This can only mean that the periodic banding is done by the firmware, after readout. Either it is sabotage, to add value, perhaps to the 1D cameras (I hope not), or, the camera is incorrectly compensating for real banding, perhaps by adding instead of subtracting, or using an erroneous value.

Sloppy, sloppy Canon.

Edit: Actually, there is a very slight periodicity in my own black frame subtractions, but it is much lower contrast than the camera's

-- hide signature --

John

Esbutt
Regular MemberPosts: 457Gear list
Like?
Re: 7D vertical banding - firmware error
In reply to John Sheehy, Feb 12, 2010

Thanks for the info, let's see if Canon does anything about it.

-- hide signature --
 Esbutt's gear list:Esbutt's gear list
Canon EOS 7D Canon EF-S 10-22mm f/3.5-4.5 USM Canon EF-S 17-55mm f/2.8 IS USM Canon EF-S 60mm f/2.8 Macro USM Canon EF 70-200mm f/4L IS USM
Reply   Reply with quote   Complain
dprsok
Senior MemberPosts: 1,410
Like?
Re: 7D vertical banding - firmware error
In reply to John Sheehy, Feb 12, 2010

Is there a way to remove this periodic vertical banding in post?

Reply   Reply with quote   Complain
Clark666
Senior MemberPosts: 1,250Gear list
Like?
Re: 7D vertical banding - firmware error
In reply to John Sheehy, Feb 12, 2010

Is there a way to write a DPP "macro" or workflow that would take a dark frame exposure that has been saved to your computer and subtract it from all 7D raw files that you take? It seems that if you did that you would have solved the problem at least until Canon does something about it?
--
http://www.flickr.com/photos/75673106@N00/

 Clark666's gear list:Clark666's gear list
Canon PowerShot SX50 HS Olympus OM-D E-M5 Canon EOS M Olympus Zuiko Digital ED 12-60mm 1:2.8-4.0 SWD Panasonic Lumix G 20mm F1.7 ASPH +4 more
Reply   Reply with quote   Complain
Fred Dominic
Regular MemberPosts: 120
Like?
Re: 7D vertical banding - firmware error
In reply to John Sheehy, Feb 12, 2010

John Sheehy wrote:

I've alluded to this many times before, based on numerous isolated experiences and impressions, but today I did a controlled experiment to tie it all together.

The fact is, most of the periodic vertical banding in the 7D does NOT happen during readout, but happens after digitization!

I noticed a few months ago that if you subtracted one 7D ISO 100 RAW from another, that the periodic component mostly disappears, and you are left with a 5d2-like random burlap pattern of a lower magnitude with just a slight hint of soft periodic banding. I noticed a couple of weeks ago that long-exposure noise reduction does not eliminate the strong periodical vertical component. Tonight I did it both ways with blackframes, and compared the results. The periodic banding was present when the camera subtracted one blackframe from another,

Sorry, I'm not sure what you mean by this last statement. What I understood was, lets say you have two blackframes b1 and b2, where b1 and b2 are NxM blackframe matrices, and what you are forming is y = b1 - b2, and you are saying that banding is present when the camera subtracts one blackframe from anohter (i.e., performs the operation of matrix element subtraction "b1 - b2")?

In any event, might I ask if you could better explain how it was that you came to the conclusion that this is post read and after the ADC?

My understanding of what you did in the past is the following. You had a real image, lets call it x, and you had one sample of a blackframe, lets call it b, and then
you did

y = x - b

and then found that while x had the banding, y did not have the banding.

By the way, why would you only want one sample of blackframe to do the banding? It might work better to produce a set of blackframes, say b1, b2, ...,
bk, and then form an average b_avg = (1/k) \sum_{i=1}^k b_k, and then
do the subtraction as:

y = x - b_avg

so the blackframe subtraction does not add any extra spontaneous noise that might exist in only one particular blackframe, and only subtracts the noise that survives the aggregation of the blackframes.

but when I did it, the periodic elementmostly disappeared. This can only mean that the periodic banding is done by the firmware, after readout. Either it is sabotage, to add value, perhaps to the 1D cameras (I hope not), or, the camera is incorrectly compensating for real banding, perhaps by adding instead of subtracting, or using an erroneous value.

Sloppy, sloppy Canon.

Well, I've said in the past that what canon wants raw to be is raw, and that they give us the raw however poor a shape it is in, but if in fact it is caused by post ADC processing, then I would have to agree with you that they should offer a firmware update for this issue. But then I still don't understand precisely how you came to this conclusion.

thanks for reading this far!!

Edit: Actually, there is a very slight periodicity in my own black frame subtractions, but it is much lower contrast than the camera's

-- hide signature --

John

Reply   Reply with quote   Complain
Dmatter
Regular MemberPosts: 189
Like?
Re: 7D vertical banding - firmware error
In reply to Fred Dominic, Feb 12, 2010

To be honest vertical banding is a lot better than random noise pattern on my 50D. It helps noise structure to be more "film-like".

Reply   Reply with quote   Complain
agorabasta
Contributing MemberPosts: 926
Like?
Most likely wrong...
In reply to John Sheehy, Feb 12, 2010

Vertical aliasing at half-pixel freq is most prominent in the brighter parts of image. Moreover, the same pixel column may appear brighter at one part of the image and then turn darker at another.

It can mean a very simple thing, most likely, they have two or more full parallel pipelines with their own ADCs in each. Then all possible dark frame subtraction in camera happens for all pipelines separately and vertical aliasing stays intact.

BTW, demosaicing with VNG for four colours kills most of aliasing in the 7D images...

Reply   Reply with quote   Complain
John Sheehy
Forum ProPosts: 15,832
Like?
Re: 7D vertical banding - firmware error
In reply to Fred Dominic, Feb 12, 2010

Fred Dominic wrote:

In any event, might I ask if you could better explain how it was that you came to the conclusion that this is post read and after the ADC?

Because the very deepest modulations of the periodic banding only exists when the camera writes the RAW. Take shot 1 at 1 second, wait a little so the camera doesn't warm up, take shot 2 at 1 second, then, enable long exposure noise reduction for

My understanding of what you did in the past is the following. You had a real image, lets call it x, and you had one sample of a blackframe, lets call it b, and then
you did

y = x - b

and then found that while x had the banding, y did not have the banding.

Yes. That was the first time I noticed that much of the periodic component repeats in every frame. There is still vertical banding, but the periodic component is subliminal (no actual lines; just noise blob tendencies along lines).

By the way, why would you only want one sample of blackframe to do the banding? It might work better to produce a set of blackframes, say b1, b2, ...,
bk, and then form an average b_avg = (1/k) \sum_{i=1}^k b_k, and then
do the subtraction as:

y = x - b_avg

That would be a great way to get a final corrected RAW image. To compare to the camera's blackframe subtraction, however, the same number of exposures must be used. My whole point in conducting the test the way I did was to isolate firmware from hardware error. Doing what the camera is supposed to do accomplishes this.

so the blackframe subtraction does not add any extra spontaneous noise that might exist in only one particular blackframe, and only subtracts the noise that survives the aggregation of the blackframes.

Again, I am forced to compare to the camera, which does this for LENR.

but when I did it, the periodic elementmostly disappeared. This can only mean that the periodic banding is done by the firmware, after readout. Either it is sabotage, to add value, perhaps to the 1D cameras (I hope not), or, the camera is incorrectly compensating for real banding, perhaps by adding instead of subtracting, or using an erroneous value.

Sloppy, sloppy Canon.

Well, I've said in the past that what canon wants raw to be is raw, and that they give us the raw however poor a shape it is in, but if in fact it is caused by post ADC processing, then I would have to agree with you that they should offer a firmware update for this issue. But then I still don't understand precisely how you came to this conclusion.

RAW only needs to be RAW when cooking would lose information. Fixing offset and scalar problems do not lose information; they recover it. Clipping optically impossible pixel values is also another form of change with no loss of real information.

thanks for reading this far!!

-- hide signature --

John

Reply   Reply with quote   Complain
John Sheehy
Forum ProPosts: 15,832
Like?
Re: Most likely wrong...
In reply to agorabasta, Feb 12, 2010

agorabasta wrote:

Vertical aliasing at half-pixel freq is most prominent in the brighter parts of image. Moreover, the same pixel column may appear brighter at one part of the image and then turn darker at another.

You're talking about scalar line noise. That is a midtone/highlight issue mainly, but there could be a connection, since black really isn't at a signal level of zero; it's on an offset, so "black" could be scaled electrically.

It can mean a very simple thing, most likely, they have two or more full parallel pipelines with their own ADCs in each. Then all possible dark frame subtraction in camera happens for all pipelines separately and vertical aliasing stays intact.

I don't think canon would do the two frames from a LENR any differently than with regular exposures. The connection is hardware. The blackframe is taken after the image exposure is digitized.

BTW, demosaicing with VNG for four colours kills most of aliasing in the 7D images...

-- hide signature --

John

Reply   Reply with quote   Complain
jm67
Contributing MemberPosts: 912
Like?
Re: 7D vertical banding - firmware error-fix?
In reply to John Sheehy, Feb 12, 2010

So, other than nailing exposure (which so far seems to help for the most part), there is nothing we can do other than hope Canon graciously provides us with a firmware update?
--
I am as ignorant as ever, and wiser than I've ever been.

Reply   Reply with quote   Complain
John Sheehy
Forum ProPosts: 15,832
Like?
Re: 7D vertical banding - firmware error
In reply to Dmatter, Feb 12, 2010

Dmatter wrote:

To be honest vertical banding is a lot better than random noise pattern on my 50D. It helps noise structure to be more "film-like".

At what ISO? Vertical banding at high ISOs is very weak compared to random noise in most recent Canons.

-- hide signature --

John

Reply   Reply with quote   Complain
John Sheehy
Forum ProPosts: 15,832
Like?
Re: 7D vertical banding - firmware error-fix?
In reply to jm67, Feb 12, 2010

jm67 wrote:

So, other than nailing exposure (which so far seems to help for the most part), there is nothing we can do other than hope Canon graciously provides us with a firmware update?

That is possible for this specific component of line noise, since it is apparently poor math, but Canon considers patterns below a certain strength to be within their design specs. Their specs are cr@p, of course.

If I were designing the camera, I would dedicate more masked pixels (there are only about 13 usable rows of pixels on the top of recent canon RAWs), and have a mode where the camera takes a little more time before writing the RAW data (only if it really takes significant time, otherwise, make it default). The camera would digitize the RAW data, then perform analysis on the borders, and adjust line offsets. 14 bits gives plenty of precision for that, since the random noise alone makes 12 bits posterization-free, anyway.

The firmware seems to do something with the data in the borders, but whatever it does, it doesn't do it right, and overwrites some of the border areas in various ways. In earlier Canons, the masked borders were indistinguishable from the image area in a blackframe. In the 7Ds, the level of banding is sometimes different in the borders and in the blackframe image area. Canon has poor mathematicians, I think, because when I simulate what they do in firmware myself, I get better results. If I take a blackframe and treat some of it's image edges as if they were masked borders, I can subtract a lot of banding that Canon left in.
--
John

Reply   Reply with quote   Complain
agorabasta
Contributing MemberPosts: 926
Like?
Re: Most likely wrong...
In reply to John Sheehy, Feb 12, 2010

John Sheehy wrote:

agorabasta wrote:

It can mean a very simple thing, most likely, they have two or more full parallel pipelines with their own ADCs in each. Then all possible dark frame subtraction in camera happens for all pipelines separately and vertical aliasing stays intact.

I don't think canon would do the two frames from a LENR any differently than with regular exposures. The connection is hardware. The blackframe is taken after the image exposure is digitized.

All depends on how exactly you performed the black frame subtraction.

(Leaving alone the possibility you subtracted demosaiced bitmaps) I'd say they may do it for 'odd' and 'even' column subframes separately (with black subtraction values set in separately); they may even do it separately for every single column just in order to correct the black subtraction levels per column, and then the variations in pipelines' effective gain and their black offsets together could produce exactly what we are seeing.

Then again, their 'objective' measurements may say it's all nice and dandy... but our problem is that we perceive the images subjectively, and all the poorly designed 'objective' criteria may simply not work for us.

All in all, it may happen by design without a possibility to correct in firmware. But that depends.

P.S. There may exist still more exotic origins for the vertical patterns - what if the odd and even columns are read with some time delay interval between them, and the phonon field changes accordingly with speed of sound there so that the pixel pitch beats with the average phonon travel length over that time interval?

And then there still are present some non-thermalised photonic phonons with energies close to that of conductance electrons in the metals and so very far from that of grid phonons of the semiconductor lattice...

Reply   Reply with quote   Complain
agorabasta
Contributing MemberPosts: 926
Like?
Re: 7D vertical banding - firmware error
In reply to John Sheehy, Feb 12, 2010

John Sheehy wrote:

Clipping optically impossible pixel values is also another form of change with no loss of real information.

That's not true, especially so if done to raw data, for sure. Such clipping moves the local averages and readily creates the infamous shadow noise blobs.

Reply   Reply with quote   Complain
J Watkins
Forum MemberPosts: 67
Like?
Re: 7D vertical banding - firmware error
In reply to agorabasta, Feb 13, 2010

For what it's worth I had vertical banding in borad daylight at ISO100 with a properly exposed and even .5 to 1 stop over video I shot at a local park. It appeared in fairly calm water with a couple ducks floating across a pond. From there it only got worse and ISO1600 was horrible. The camera had already made one trip to canon and it wasn't fixed, focus was better but... I ended up swapping it for a newer one that is much better. I still have some minor banding in shadow areas that is worse in LR2.6 that it is in DPP for some reason. I haven't tried forcing any banding in video with this body but I'm with the masses hoping there is a 'fix' via firmware soon! Canon are you listening????? I didn't hop on the 5DMKII for the issues that it suffered out of the gate like the specular highlight black spots and it's banding. I figured (hoped) the 7D had enough R&D that Canon had learned from their mistakes but I'm thinking that's not the case. I guess I got spoiled but a flawless 40D and hoped the 7's would be as good or better.

Reply   Reply with quote   Complain
John Sheehy
Forum ProPosts: 15,832
Like?
Re: 7D vertical banding - firmware error
In reply to agorabasta, Feb 13, 2010

agorabasta wrote:

John Sheehy wrote:

Clipping optically impossible pixel values is also another form of change with no loss of real information.

That's not true, especially so if done to raw data, for sure. Such clipping moves the local averages and readily creates the infamous shadow noise blobs.

I'm not sure you are replying to what I meant or said. I am talking about something like hot pixel rejection, based on ridiculous RAW pixel values, which give something far worse than blobs, and the real value is probably beyond the clipped value, so it is "less" wrong after clipping, bu by how much, we don't know. The lessening of an artifact. Would it really help to have a tall zit in the middle of a blob?

-- hide signature --

John

Reply   Reply with quote   Complain
John Sheehy
Forum ProPosts: 15,832
Like?
Re: 7D vertical banding - firmware error
In reply to J Watkins, Feb 13, 2010

J Watkins wrote:

For what it's worth I had vertical banding in borad daylight at ISO100 with a properly exposed and even .5 to 1 stop over video I shot at a local park. It appeared in fairly calm water with a couple ducks floating across a pond. From there it only got worse and ISO1600 was horrible.

Definitely a lemon, with problems well beyond what I am talking about. The vertical banding that I am referring to is only significant up to ISO 500.

The camera had already made one trip to canon and it wasn't fixed,

That's why I have never sent a camera to Canon. Other people's experiences have told me Canon has very low standards in image quality, and you'd almost have to get a picture of a duck when shooting a deer to get a sensor or electronics swap.

focus was better but... I ended up swapping it for a newer one that is much better. I still have some minor banding in shadow areas that is worse in LR2.6 that it is in DPP for some reason. I haven't tried forcing any banding in video with this body but I'm with the masses hoping there is a 'fix' via firmware soon! Canon are you listening????? I didn't hop on the 5DMKII for the issues that it suffered out of the gate like the specular highlight black spots and it's banding. I figured (hoped) the 7D had enough R&D that Canon had learned from their mistakes but I'm thinking that's not the case. I guess I got spoiled but a flawless 40D and hoped the 7's would be as good or better.

It's really sad that low ISO seems to be getting worse as high ISO quality and pixel density increase. Low ISO should, in a properly designed camera without sloppiness, improve automatically with increased pixel density in a CMOS sensor, with limited per-pixel DR.

-- hide signature --

John

Reply   Reply with quote   Complain
agorabasta
Contributing MemberPosts: 926
Like?
Re: 7D vertical banding - firmware error
In reply to John Sheehy, Feb 13, 2010

John Sheehy wrote:

agorabasta wrote:

That's not true, especially so if done to raw data, for sure. Such clipping moves the local averages and readily creates the infamous shadow noise blobs.

Would it really help to have a tall zit in the middle of a blob?

-- hide signature --

John

I'm all for threshold filtering for stray values. I'm heavily against some Nikon-style clipping-out of the sub-zero values. I'm also very much against filtering that breaks the correlation between luma and chroma noise, like the ugly filtering seen in Nikon D700 raw - it makes the better new converters like Lr3 work wrong since the algorithm relies on luma/chroma correlation doing the chroma NR (same approach as in the Sony a500/550 cams internal conversion).

Your statement was too ambiguous, since the below-black raw values are exactly the "optically impossible pixel values" as per your parlance, and clipping the such to zero is exactly the ugly Nikon's habit with their cheaper cams.

Reply   Reply with quote   Complain
willyhill
Regular MemberPosts: 277
Like?
7D Vertical Banding is horrible
In reply to John Sheehy, Feb 13, 2010

I tried to ask for comments last time
http://forums.dpreview.com/forums/read.asp?forum=1019&message=34453723

Banding always looks horrible for high quality photography

Reply   Reply with quote   Complain
Dale Buhanan
Veteran MemberPosts: 3,202Gear list
Like?
Re: 7D vertical banding - firmware error
In reply to John Sheehy, Feb 13, 2010

Very interesting work, John. I don't know the answer, but I am sure that there is more to come on this issue. Anyway, thanks and good work!

John Sheehy wrote:

I've alluded to this many times before, based on numerous isolated experiences and impressions, but today I did a controlled experiment to tie it all together.

The fact is, most of the periodic vertical banding in the 7D does NOT happen during readout, but happens after digitization!

I noticed a few months ago that if you subtracted one 7D ISO 100 RAW from another, that the periodic component mostly disappears, and you are left with a 5d2-like random burlap pattern of a lower magnitude with just a slight hint of soft periodic banding. I noticed a couple of weeks ago that long-exposure noise reduction does not eliminate the strong periodical vertical component. Tonight I did it both ways with blackframes, and compared the results. The periodic banding was present when the camera subtracted one blackframe from another, but when I did it, the periodic elementmostly disappeared. This can only mean that the periodic banding is done by the firmware, after readout. Either it is sabotage, to add value, perhaps to the 1D cameras (I hope not), or, the camera is incorrectly compensating for real banding, perhaps by adding instead of subtracting, or using an erroneous value.

Sloppy, sloppy Canon.

Edit: Actually, there is a very slight periodicity in my own black frame subtractions, but it is much lower contrast than the camera's

-- hide signature --

John

-- hide signature --

kind regards
Dale

 Dale Buhanan's gear list:Dale Buhanan's gear list
Canon PowerShot SD500 Canon PowerShot G5 Canon PowerShot G6 Canon PowerShot G9 Canon PowerShot Pro1 +33 more
Reply   Reply with quote   Complain
Keyboard shortcuts:
FForum MMy threads