Banding Solved (just about)...

Started Dec 21, 2006 | Discussions thread
Flat view
GordonBGood Veteran Member • Posts: 6,312
Banding Solved (just about)...

I decided to start a new thread rather than add to any of the old ones as most are getting full and many may miss the good bits. Please feel free to link any of those that pop up to this one giving The Answer (42).

Even though most of you may be bored with discussions on vertical banding/pattern noise, this post confirms that it's real even though small in typical shooting situations, that it is systematic, and that thus comes about as a result of in camera processing of raw data. Because it is systematic, the effect can be "backed out" by post processing of raw.

First, as promised, the proof that the K10D Long Exposure Banding problem is systematic, due to a glitch in processing in the camera, and that I can back out the problem from a raw file:

The following image was developed from a raw file provided by mikerigel of this forum. It was shot with about a 10 minute exposure at ISO 100. It turns out that the banding problem is not limited to only the blue channel, but can also appear in either of the red or green channels, and the file contained instances of all of these. In the following image, I focus on the green band and display only the green channel with contrast boosted to clearly show the problem. This is a 100% zoom crop at origin (800,0) for a 640 X 480 crop saved with reasonable JPEG compression. Ignore the many hot pixels as they appear as a result of my requesting this image be shot without Dark Frame Subtraction (DFS) NR and this number of hot pixels is normal for an extreme time exposure such as this (in fact, this sensor is one of the good ones for this purpose). Hot pixels could be removed by a manual DFS process by subtracting a lens cap shot with the same camera parameters and about the same time exposure duration. The before image green channel looks like this:

Now after passing through my back out process with identical processing parameters, the identical crop looks like this:

If I showed you the red or blue banding in this image, they are resolved in exactly the same way.

Note that the back out process is in no way a noise reduction filter, as it searches out the root cause of this banding and the pattern it applies to the image and just reverses it.

So, banding is due to a systematic oversight on Pentax's part. I have duly sent Pentax a message with a link to this post through the channels that they have provided for me. If Pentax do not contact me about this, hopefully it means that they feel they already know what the problem is and a solution is on the way.

Along the way I discovered several interesting bits of information (speaking of the sensor in its normal portrait orientation):

1. The sensor is read in columns, likely from top to bottom, for both readout channels.

2. One readout channel reads the red and blue colour information in columns from left to right, first reading the red from the first even (zero) column followed by the blue from the adjacent odd column, etc.; the other readout channel reads the greens in columns from right to left. It's possible that the readout channels read two columns simultaneously with these columns being offset by two columns (this from the processing patterns), but it's very likely that PRIME was designed for compatibility with other sensor versions that have a four channel readout who's effect would be something like this.

3. It's very likely that the misapplied algorithm that causes this banding/vertical pattern noise is hardwired into the PRIME engine and likely can't be disabled in this model camera. That's not a deal breaker, just that a back out algorithm such as mine would have to be run in firmware in order to solve the problem in camera, which would add a processing overhead to every image (I estimate about 0.07 seconds per image based on what I infer about camera processor speed).

So what about vertical pattern noise seen at high ISO's or at somewhat lower ISO sensitivities push processed? From a raw file provided to me by Herb Chong, I see exactly the same patterns in the data and think that the same back out processing will solve this problem, too. The only thing that stops me from having before and after crops from that image showing that problem solved, is something about scale factors applied that I haven't worked out yet. I am actively trying to infer what these factors are from the data at hand.

From my limited data (three images), I have been able to estimate the appropriate scale factor to use for ISO 100 images, but it appears that when the problem is allowed full scale combinations of ISO range and anomaly magnitude, there is some sort of non-linear scaling going on, which may be as simple as clipping to a maximum. Not having a K10D in my possession, I am reliant on others to provide me with the data I need for analysis. I really need to be able to shoot, analyse, from that information re-shoot, analyse, etc.

It's very frustrating not being able to "put this problem to bed", as if I had a camera I could have it solved in a few days instead of the anticipated few weeks with others help. Of course, with access to Pentax's PRIME designer, a chief firmware engineer, and a good translator if they don't speak English, this could be solved in a day. This is because (upon signing a non-disclosure agreement), I would not have to infer from a succession of test shots but could ask directly.

Much thanks to Mike (mikerigel) and HerbChong for the raw images they provided.

Coming: A post showing "Vertical Pattern Noise" solved.

Do Pentax owe me a K10D and a complete set of lenses yet?

Regards, GordonBGood

Flat view
Post (hide subjects) Posted by
Keyboard shortcuts:
FForum PPrevious NNext WNext unread UUpvote SSubscribe RReply QQuote BBookmark MMy threads
Color scheme? Blue / Yellow