Perplexing Nikon Pixel Shift implementation

[Horshack]

Yesterday I started analyzing Nikon's first pixel shift implementation, which is available on the recently released 24MP Zf. The camera offers four options:

• 4 shots
• 8 shots
• 16 shots
• 32 shots

The 4-shot and 16-shot operate as expected. The 4-shot exposes each "scene pixel" to each of the 4 surrounding bayer filters, resulting in a fully chroma-sampled image with the same resolution as a single exposure. The raw generated by Nikon's NX Studio is a non-bayered linear RGB, which means it doesn't require demosaicing. The 16-shot operates the same but adds a 1/2 pixel shift in each dimension, resulting in a non-bayered linear RGB file with 4x the output resolution (24MP -> 96 MP).

The 8-shot and 32-shot curiously add an additional 2x2 full-pixel shift in the vertical axis. This has the effect of exposing the same set of pixels to a redundant set of bayer filters but on different row positions of the sensors. I don't understand what could be gained by adding this redundant sampling other than perhaps noise reduction from the additional exposures, although I don't see why shifting to a different set of rows is necessary for this. Also, I don't see any difference in measured noise between the 4-shot vs 8-shot and 16-shot vs 32-shot.

Here is an animation showing a 300% nearest-neighbor crop comparing a single exposure vs 4-shot vs 8-shot. Notice the 8-shot eliminates a yellow artifact that was introduced in the 4-shot image by NX Studio.

Animation: Nikon Zf Single Exposure vs 4/8 shot pixel shift, 300% nearest-neighbor crop

Here are the sensor movements in the 8-shot shift, as determined by comparing the raw files to each other. All are full-pixel shifts.

• Exposure 1 - Base sensor position
• Exposure 2 - Sensor shifted right
• Exposure 3 - Sensor shifted down
• Exposure 4 - Sensor shifted left
• Exposure 5 - Sensor shifted down
• Exposure 6 - Sensor shifted right
• Exposure 7 - Sensor shifted down
• Exposure 8 - Sensor shifted left

 

[ipopov]

Can these 8/32 image modes be avoid any PDAF masking artifacts?

 

[Horshack]

Can these 8/32 image modes be avoid any PDAF masking artifacts?

Interesting thought, thanks. But that seems like overkill for the benefit gained, since those artifacts aren't visible in the vast majority of photos. It does gain the full sampling values for the blue-position PDAF pixels that would otherwise be interpolated but again, seems like overkill.

 

[Horshack]

Perhaps the extra frames are for motion correction but I'll have to do some experiments to see if that's the case.

 

[SrMi]

Perhaps the extra frames are for motion correction but I'll have to do some experiments to see if that's the case.

You do not need extra frames for motion correction. Motion artifact correction replaces image parts that show motion artifacts with a blown-up version of a single image. I have checked, and Nikon software does not apply motion artifact correction.

Nikon's in-camera help says the 32-image version should have less noise than a 16-image merge. I have not noticed any difference.

 

[Horshack]

Perhaps the extra frames are for motion correction but I'll have to do some experiments to see if that's the case.

You do not need extra frames for motion correction.

You do if you want to apply it without dropping the extra data that would've been provided from the shifted photos with motion.

Nikon's in-camera help says the 32-image version should have less noise than a 16-image merge. I have not noticed any difference.

Thanks. I'm going to recheck this to make sure I haven't overlooked something.

 

[J A C S]

It may try to find out if the pixel shift was accurate. Also, redundancy is good, in principle.

 

[Horshack]

Here's a noise comparison of a single exposure vs 4-shot shift vs 8-shot shift, all shot 5EV underexposed and pushed in ACR:

Animation: Zf Single Exposure vs 4-shot shift vs 8-shot shift, -5EV Exposure

Perceptually the 8-shot yields hardly any noise reduction vs the 4-shot but the noise measurements indicate it's there. Here are the stdev deltas measured in PS for a white patch pushed +5EV:

• Single Exposure -> 4-shot: 25.7% lower stdev
• 4-shot vs 8->shot: 31.1% lower stdev

Doesn't make much sense to me use extra exposures on a pixel shift to reduce noise since the scenario already requires you to be shooting locked-down on a static scene, so you could just as easily increase your exposure for the same result without all the extra shots and workflow penalty processing those extra shots during composite raw creation in NX Studio (it's very slow).

 

[SrMi]

Here's a noise comparison of a single exposure vs 4-shot shift vs 8-shot shift, all shot 5EV underexposed and pushed in ACR:

Animation: Zf Single Exposure vs 4-shot shift vs 8-shot shift, -5EV Exposure

Perceptually the 8-shot yields hardly any noise reduction vs the 4-shot but the noise measurements indicate it's there. Here are the stdev deltas measured in PS for a white patch pushed +5EV:

• Single Exposure -> 4-shot: 25.7% lower stdev
• 4-shot vs 8->shot: 31.1% lower stdev

Doesn't make much sense to me use extra exposures on a pixel shift to reduce noise since the scenario already requires you to be shooting locked-down on a static scene, so you could just as easily increase your exposure for the same result without all the extra shots and workflow penalty processing those extra shots during composite raw creation in NX Studio (it's very slow).

If you are shooting on a tripod, you probably have already maximized the single-shot exposure and cannot increase it anymore.

With frame averaging and pixel shift, you can reduce the noise, even more than the maximum exposure a single shot gives you, thus simulating the effect of a larger sensor.

I speculate that an 8-image pixel shift creates two 4-image pixel shifts and averages them into a single image for more DR.

 

[Horshack]

Here's a noise comparison of a single exposure vs 4-shot shift vs 8-shot shift, all shot 5EV underexposed and pushed in ACR:

Animation: Zf Single Exposure vs 4-shot shift vs 8-shot shift, -5EV Exposure

Perceptually the 8-shot yields hardly any noise reduction vs the 4-shot but the noise measurements indicate it's there. Here are the stdev deltas measured in PS for a white patch pushed +5EV:

• Single Exposure -> 4-shot: 25.7% lower stdev
• 4-shot vs 8->shot: 31.1% lower stdev

Doesn't make much sense to me use extra exposures on a pixel shift to reduce noise since the scenario already requires you to be shooting locked-down on a static scene, so you could just as easily increase your exposure for the same result without all the extra shots and workflow penalty processing those extra shots during composite raw creation in NX Studio (it's very slow).

If you are shooting on a tripod, you probably have already maximized the single-shot exposure and cannot increase it anymore.

How so? The exposures can extend well into multiples of seconds since the scene is static.

With frame averaging and pixel shift, you can reduce the noise, even more than the maximum exposure a single shot gives you, thus simulating the effect of a larger sensor.

How so? The magnitude of noise reduction is the same between longer exposures vs multiple exposures of shorter duration. That's for shot noise. Read noise would actually be higher for multiple shots and would offset some of the shot noise reduction vs longer exposures, although the longer exposures would incur additional thermal noise so they're probably net equa.

I speculate that an 8-image pixel shift creates two 4-image pixel shifts and averages them into a single image for more DR.

It's assumed the pixel-shift data will be mean or median averaged so it's a given that noise will be reduced, which will also increase DR.

 

[SrMi]

Here's a noise comparison of a single exposure vs 4-shot shift vs 8-shot shift, all shot 5EV underexposed and pushed in ACR:

Animation: Zf Single Exposure vs 4-shot shift vs 8-shot shift, -5EV Exposure

Perceptually the 8-shot yields hardly any noise reduction vs the 4-shot but the noise measurements indicate it's there. Here are the stdev deltas measured in PS for a white patch pushed +5EV:

• Single Exposure -> 4-shot: 25.7% lower stdev
• 4-shot vs 8->shot: 31.1% lower stdev

Doesn't make much sense to me use extra exposures on a pixel shift to reduce noise since the scenario already requires you to be shooting locked-down on a static scene, so you could just as easily increase your exposure for the same result without all the extra shots and workflow penalty processing those extra shots during composite raw creation in NX Studio (it's very slow).

If you are shooting on a tripod, you probably have already maximized the single-shot exposure and cannot increase it anymore.

How so? The exposures can extend well into multiples of seconds since the scene is static.

Still, there is a limit defined by max sensor saturation. You can go above that limit with pixel shift and/or frame averaging.

With frame averaging and pixel shift, you can reduce the noise, even more than the maximum exposure a single shot gives you, thus simulating the effect of a larger sensor.

How so? The magnitude of noise reduction is the same between longer exposures vs multiple exposures of shorter duration.

Assume you reach sensor saturation with a one second exposure. Using frame averaging or pixel shift you can increase that exposure to 4 or 8 seconds and thus reduce noise even more.

That's for shot noise. Read noise would actually be higher for multiple shots and would offset some of the shot noise reduction vs longer exposures, although the longer exposures would incur additional thermal noise so they're probably net equa.

I speculate that an 8-image pixel shift creates two 4-image pixel shifts and averages them into a single image for more DR.

It's assumed the pixel-shift data will be mean or median averaged so it's a given that noise will be reduced, which will also increase DR.

My point is that maybe Nikon combines pixel shift with frame averaging of two images, take two four shot pixel shifts and average them. Both pixel shift and frame averaging would reduce noise.

 

[Horshack]

Here's a noise comparison of a single exposure vs 4-shot shift vs 8-shot shift, all shot 5EV underexposed and pushed in ACR:

Animation: Zf Single Exposure vs 4-shot shift vs 8-shot shift, -5EV Exposure

Perceptually the 8-shot yields hardly any noise reduction vs the 4-shot but the noise measurements indicate it's there. Here are the stdev deltas measured in PS for a white patch pushed +5EV:

• Single Exposure -> 4-shot: 25.7% lower stdev
• 4-shot vs 8->shot: 31.1% lower stdev

Doesn't make much sense to me use extra exposures on a pixel shift to reduce noise since the scenario already requires you to be shooting locked-down on a static scene, so you could just as easily increase your exposure for the same result without all the extra shots and workflow penalty processing those extra shots during composite raw creation in NX Studio (it's very slow).

If you are shooting on a tripod, you probably have already maximized the single-shot exposure and cannot increase it anymore.

How so? The exposures can extend well into multiples of seconds since the scene is static.

Still, there is a limit defined by max sensor saturation. You can go above that limit with pixel shift and/or frame averaging.

With frame averaging and pixel shift, you can reduce the noise, even more than the maximum exposure a single shot gives you, thus simulating the effect of a larger sensor.

How so? The magnitude of noise reduction is the same between longer exposures vs multiple exposures of shorter duration.

Assume you reach sensor saturation with a one second exposure. Using frame averaging or pixel shift you can increase that exposure to 4 or 8 seconds and thus reduce noise even more.

Yeah, brain fart on my part. I'm so used to shooting in low-light that I always have room to increase my exposure time Tunnel vision on my part. But if you're at saturation then there isn't much need for further noise reduction. If it's greater DR that's needed then the additional median frames aren't going to provide that effectively as a manual bracket of the pixel shift.

That's for shot noise. Read noise would actually be higher for multiple shots and would offset some of the shot noise reduction vs longer exposures, although the longer exposures would incur additional thermal noise so they're probably net equa.

I speculate that an 8-image pixel shift creates two 4-image pixel shifts and averages them into a single image for more DR.

It's assumed the pixel-shift data will be mean or median averaged so it's a given that noise will be reduced, which will also increase DR.

My point is that maybe Nikon combines pixel shift with frame averaging of two images, take two four shot pixel shifts and average them. Both pixel shift and frame averaging would reduce noise.

It certainly is doing both since both the 4-shot shift-only mode and the 8-shot mode with four extra frames per axis (that don't contribute to increase in resolution) both yield approximately the same reduction in noise.

 

[Jack Hogan]

It may try to find out if the pixel shift was accurate. Also, redundancy is good, in principle.

In case someone wants pristine raw data without hot pixels?

 

[SrMi]

It may try to find out if the pixel shift was accurate. Also, redundancy is good, in principle.

In case someone wants pristine raw data without hot pixels?

Nikon recommends running pixel mapping before using pixel shift. It helped as it got rid of dead/stuck pixels.

 

[hjulenissen]

Could it combat some patterned noise/sensitivity unevenness?

 

[hjulenissen]

By encoding as a «non-bayered» raw file (i.e. each sensel site covered once for each subpixel), would they not have to throw away some data?

With 2x the green sensels as red or blue ones, by the time you have covered each sensel site with red and blue, you have taken 4 captures. That is 2x what is needed for the green channel, thus a «raw tiff» file would either discard 50% of the green information, or average to at least improve noise?