Nikon Z6 - Concentric Coloured Banding

[sharkmelley]

Let me say at the outset that what I'm about to discuss is very esoteric and probably has zero relevance to normal daylight photography. So please feel free to ignore.

I'm using the Nikon Z6 on a telescope for imaging very faint deep sky objects such as interstellar dust clouds. This involves very heavy post-processing such as calibration with flats, stacking of multiple exposures, subtraction of light pollution and then heavy stretching. Doing this I found some interesting coloured banding in the background:


60 stacked exposures of 120sec at ISO 800

Note that the camera is attached directly to the telescope so this is not some weird lens correction of the type seen on Sony mirrorless cameras

I then found I could isolate this pattern by taking a stacked flat exposure and dividing the red channel by the green and also the blue channel by the green:


Heavily stretched versions of red/green and blue/green

Further experiments showed that the spacing of the rings decreases with increased recorded light intensity i.e. longer exposures.

In fact the tonal variations can be shown using simple Photoshop manipulation of a single exposure:


Single ISO 400 flat frame, 12bit raw, 3 stops underexposed and boosted in post-processing

To produce the above I did the following:

Shoot a flat frame, 3 stops underexposed. For the raw conversion I did the following in Adobe Camera Raw:

• Adjust the white balance with the white balance tool
• Boost exposure by 3 stops
• Set vibrance and saturation to 100%

The effect is strongest when shooting 12bit at low ISO.

My first thought was that the tonal variations might be caused by the white preconditioning applied to the red and blue channels and the consequent histogram gaps. However these tonal variations do not align with the histogram gaps. So now I really don't know what is causing these artifacts.

Although I don't think these tonal variations have any practical impact on normal daylight photography, someone might find this discovery interesting, purely as a technical curiosity. On the other hand if you know what might be causing it, I would be delighted to know!

Mark

 

[SaltyPeanut]

Calling Jim Kasson

 

[sharkmelley]

After much experimentation I found that for a given exposure (in my case 2 minutues) then adjusting the ISO so the back-of-camera histogram peak of the background sky is more or less at the halfway point reduces the effect of these sharp tonal variations:


ISO800 vs ISO3200. 60 stacked exposures of 2 min.

The remaining (green to purple) background gradients are smooth enough to be easily dealt with by background subtraction.

Mark

 

[astrograph]

could it be that this is caused by the flat field box? Did you try with different flat-field illuminations? (white T-Shirt, Wall, ....)

 

[sharkmelley]

could it be that this is caused by the flat field box? Did you try with different flat-field illuminations? (white T-Shirt, Wall, ....)

You are correct of course to point out that these anomalous patterns might exist in the flat fields as well as (or instead of) the exposures of the night sky.

Underexposed flats would certainly cause a problem but my flats were well exposed and show virtually no evidence of these patterns even when stretched to a crazy degree:


Master Flat: Red and Blue channels divided by Green and stretched

You will notice that bands in the red channel are very faint and are much more closely spaced. This is consistent with what I have observed by experiment - the brighter the exposure, the closer the ring spacing.

Mark

 

[astrograph]

btw: compliments towards your image, very deep, showing gorgeous details of the dust, even though it was stretched extremely for this technical experiment.

 

[sharkmelley]

For the sake of interest, with the Z6 on my telescope, I took a series of sky flats at ISO 100 in quick succession at every shutter speed from severely underexposed (1/1250sec) to slightly overexposed (1/6sec). A sky flat is where the telescope is pointing at zenith in a featureless clear dusk sky.

Here is a montage where I've taken the pixel values from the raw data, divided the blue channel by the green channel and stretched the data by a factor of 25:


Blue divided by Green - Montage of shots at increasing exposures

For the same set of files here is the red channel divided by the green channel with the same data stretch of 25:


Red divided by Green - Montage of shots at increasing exposures

I'm still no nearer to understanding the cause. An additional point of interest is that the vertical discontinuity between the left-hand-side and right-hand-side of the sensor appears in some exposures but not in others.

Mark

 

[Jim Keye]

For the sake of interest, with the Z6 on my telescope, I took a series of sky flats at ISO 100 in quick succession at every shutter speed from severely underexposed (1/1250sec) to slightly overexposed (1/6sec). A sky flat is where the telescope is pointing at zenith in a featureless clear dusk sky.

Here is a montage where I've taken the pixel values from the raw data, divided the blue channel by the green channel and stretched the data by a factor of 25:


Blue divided by Green - Montage of shots at increasing exposures

For the same set of files here is the red channel divided by the green channel with the same data stretch of 25:


Red divided by Green - Montage of shots at increasing exposures

I'm still no nearer to understanding the cause. An additional point of interest is that the vertical discontinuity between the left-hand-side and right-hand-side of the sensor appears in some exposures but not in others.

Mark

Interesting, and thanks for posting. Have you tried posting this over in the Astro forum?

 

[bokemon]

what r u doin', recordin' at 12 bit instead of 14? yer gonna give yerself problems

 

[sharkmelley]

Interesting, and thanks for posting. Have you tried posting this over in the Astro forum?

There's a long thread on Cloudy Nights examining this issue and many other aspects of testing related specifically to astrophotography:

Loading…

www.cloudynights.com

But no-one knows the cause of this coloured rings issue.

Mark

 

[sharkmelley]

what r u doin', recordin' at 12 bit instead of 14? yer gonna give yerself problems

To be clear, only one of my examples was shot 12 bit. It was the example where I stretched an ISO 400 flat exposure in Photoshop. In fact the same effect can be shown by shooting a few stops underexposed at ISO 100.

Sorry if this caused any confusion.

Mark

 

[sharkmelley]

I've now done a whole load of experiments and received Nikon Z6 flat frame data from other sources (thanks Jerry!)

First familiarise yourself with my montages of rings: https://www.dpreview.com/forums/post/63096335

The tentative conclusions I have reached are these:

• The rings are caused by a scaling applied to the raw data of the red and the blue channels (but not the green)
• The scaling is some kind of fixed functional form that is applied to the red and blue channels even when vignetting corrections are switched off. It occurs with and without a recognised lens attached
• The main effect in the blue channel is to dim the corners (or brighten the centre) by a factor of around 1.03
• The main effect in the red channel is to brighten the corners (or dim the centre) by a factor of around 1.01
• For the avoidance of any doubt this scaling is in addition to the red and blue white pre-conditioning of approx 1.18 which causes the well-known histogram gaps
• Maybe the aim is to fix a corner colour cast?
• Very strangely, the functional form is not centred and not symmetrical, especially in the red channel. This asymmetry is hardwired into an individual camera but another Z6 shows a different asymmetry

Now the obvious question for an astrophotographer afflicted by the rings is the following: can this scaling be undone and therefore can the tonal variations be corrected?

I've made some progress here. I've tried different functional forms and have settled on this one as the closest match so far:

scale(x,y) = 1 + {(x-xc)^2 + (y-yc)^2}/k

where:

• (x,y) are the pixel coordinates
• (xc,yc) is the centre of the ring pattern (not the centre of the image)
• k is a constant. For the blue channel it is selected to scale the corner pixels by around 1.03

So I fitted the xc,yc and k parameters by iterations and applied this scaling to the blue channel. As before I display the blue channel divided by the green channel (then stretched). The before and after are as follows:


Red/Green - before and after applying functional form to raw data

I'm not quite there yet. The main problem is that the original rings are slightly elongated in the diagonal direction but my functional form is circular. So I need a slightly more complex function that takes generic elongations into account.

I have started with the blue channel because it looks like an easier problem than the red channel which has very elongated rings.

Mark

 

[sharkmelley]

There's one thing I should have made clear: the size of each step in those ring diagrams is precisely one digital unit. This makes it approximately as obvious as the PDAF banding.

In the shadow areas, where the image is really underexposed, is where you are likely to notice the PDAF banding and the rings. You are unlikely to spot the rings unless you can already see the PDAF banding.

Mark

 

[sharkmelley]

I spent a rainy Sunday afternoon fitting a functional form that would remove the rings when applied multiplicatively to raw data. Here is the result for the blue and red channel:


Functional form for blue and red channels

The one for the red channels needs some additional slight tweaking to remove the lumps and bumps. It's important to point out that these surfaces are completely smooth but I've drawn them as a contour map to show their shape.

Here's an example of it successfully applied to actual raw camera data (a flat field image):


Removing rings from the blue channel

Now for a quick explanation of how this works. The problems associated with applying any kind of scaling to digital data are well known. The data can only be shifted by integer increments and this causes "steps" to form in the resulting data values. It is these steps in the data that lead to the rings we are seeing. The purpose of these multiplicative surfaces is to introduce a set of equal and opposite steps into the data that cancel out the original ones.

In the blue channel, the function varies from 1.0 near the centre to approx 1.03 near the corners. In the red channel the function varies from 1.0 near the top right to approx 0.98 at the bottom corner.

Now the power of any model is in it's predictive ability. Can we use these functions to predict where and when the rings will appear? Take the blue channel for instance - if the pixel values are around 100 across the image then the model would predict that 3 rings would appear. So here is an underexposed image of a featureless sky:


Featureless blue sky ISO 100 with blue pixel values near 120

No rings are visible (yet). But if I divide the Blue channel by the Green and stretch then this is what I see:


Blue channel divided by Green

If I open the original image in Photoshop and apply an extreme stretch (Vibrance=100, Saturation=100) then I can see the same rings:


Photoshop confirms the rings really exist

So the model predicted the rings would exist in the image. Can it also remove them? Here's the before and after:


Blue divided by Green: Before and after functional form is applied

I doubt if all this has much practical use but I think it does demonstrate that there is something going on inside the camera's raw data processing that can be modelled.

Mark

 

[DPFranz]

Mostly only on the monitor of the Z7, and it hasn't been much of a problem in my images after ingest and processing, leading me to initially believe it was only an artifact of the camera's screen, but sometimes I still think I may be able to see it in smooth blue sky in finalized images, and with your analysis, it seems pretty likely that there really is an issue here. D850 is a bit smoother.

 

[Verto]

Let me say at the outset that what I'm about to discuss is very esoteric and probably has zero relevance to normal daylight photography. So please feel free to ignore.

I'm using the Nikon Z6 on a telescope for imaging very faint deep sky objects such as interstellar dust clouds. This involves very heavy post-processing such as calibration with flats, stacking of multiple exposures, subtraction of light pollution and then heavy stretching. Doing this I found some interesting coloured banding in the background:


60 stacked exposures of 120sec at ISO 800

Note that the camera is attached directly to the telescope so this is not some weird lens correction of the type seen on Sony mirrorless cameras

I then found I could isolate this pattern by taking a stacked flat exposure and dividing the red channel by the green and also the blue channel by the green:


Heavily stretched versions of red/green and blue/green

Further experiments showed that the spacing of the rings decreases with increased recorded light intensity i.e. longer exposures.

In fact the tonal variations can be shown using simple Photoshop manipulation of a single exposure:


Single ISO 400 flat frame, 12bit raw, 3 stops underexposed and boosted in post-processing

To produce the above I did the following:

Shoot a flat frame, 3 stops underexposed. For the raw conversion I did the following in Adobe Camera Raw:

• Adjust the white balance with the white balance tool
• Boost exposure by 3 stops
• Set vibrance and saturation to 100%

The effect is strongest when shooting 12bit at low ISO.

My first thought was that the tonal variations might be caused by the white preconditioning applied to the red and blue channels and the consequent histogram gaps. However these tonal variations do not align with the histogram gaps. So now I really don't know what is causing these artifacts.

Although I don't think these tonal variations have any practical impact on normal daylight photography, someone might find this discovery interesting, purely as a technical curiosity. On the other hand if you know what might be causing it, I would be delighted to know!

Mark

My Olympus EM1 Mark II does the same thing. I discovered it by accident, but it’s 100% identical. Keep in mind it’s 12bit as well.

 

[Tomx72]

Good job Mark, and BTW a very nice shot!

I think 12bit will be the keyword here, to my knowledge, banding arises when there is not enough bit depth OR the compression is overdone.

I think it is safe to recommend 14bit uncompressed.

I am unsure if electronic shutter has any influence here. By that I don't mean the "usual" ES banding with a fluctuating light source but any kind of readout/compression done differently than with mechanical shutter.

 

[sharkmelley]

Let me say at the outset that what I'm about to discuss is very esoteric and probably has zero relevance to normal daylight photography. So please feel free to ignore.

I'm using the Nikon Z6 on a telescope for imaging very faint deep sky objects such as interstellar dust clouds. This involves very heavy post-processing such as calibration with flats, stacking of multiple exposures, subtraction of light pollution and then heavy stretching. Doing this I found some interesting coloured banding in the background:


60 stacked exposures of 120sec at ISO 800

Note that the camera is attached directly to the telescope so this is not some weird lens correction of the type seen on Sony mirrorless cameras

I then found I could isolate this pattern by taking a stacked flat exposure and dividing the red channel by the green and also the blue channel by the green:


Heavily stretched versions of red/green and blue/green

Further experiments showed that the spacing of the rings decreases with increased recorded light intensity i.e. longer exposures.

In fact the tonal variations can be shown using simple Photoshop manipulation of a single exposure:


Single ISO 400 flat frame, 12bit raw, 3 stops underexposed and boosted in post-processing

To produce the above I did the following:

Shoot a flat frame, 3 stops underexposed. For the raw conversion I did the following in Adobe Camera Raw:

• Adjust the white balance with the white balance tool
• Boost exposure by 3 stops
• Set vibrance and saturation to 100%

The effect is strongest when shooting 12bit at low ISO.

My first thought was that the tonal variations might be caused by the white preconditioning applied to the red and blue channels and the consequent histogram gaps. However these tonal variations do not align with the histogram gaps. So now I really don't know what is causing these artifacts.

Although I don't think these tonal variations have any practical impact on normal daylight photography, someone might find this discovery interesting, purely as a technical curiosity. On the other hand if you know what might be causing it, I would be delighted to know!

Mark

My Olympus EM1 Mark II does the same thing. I discovered it by accident, but it’s 100% identical. Keep in mind it’s 12bit as well.

I regret including that 12bit example because it has made some people assume that it is a 12bit problem. Every other example in this thread is 14bit. Here's a real-world 14bit example, admittedly heavily post processed:

Loading…

www.dpreview.com

Mark

 

[Verto]

Good job Mark, and BTW a very nice shot!

I think 12bit will be the keyword here, to my knowledge, banding arises when there is not enough bit depth OR the compression is overdone.

I think it is safe to recommend 14bit uncompressed.

I am unsure if electronic shutter has any influence here. By that I don't mean the "usual" ES banding with a fluctuating light source but any kind of readout/compression done differently than with mechanical shutter.

From my brief experiments with the Z6 E-Shutter produces lower technical image quality.

 

[Tomx72]

I regret including that 12bit example because it has made some people assume that it is a 12bit problem. Every other example in this thread is 14bit. Here's a real-world 14bit example, admittedly heavily post processed:

https://www.dpreview.com/forums/post/63127228

Mark

I've missed that, sorry. Were the 14bit shots compressed?

From my brief experiments with the Z6 E-Shutter produces lower technical image quality.

Can you elaborate on that? Anything besides the "usual suspects" (fluorescent/LED banding, rolling shutter)?