D7200 HPS (hot pixel suppression) news

Started Dec 8, 2015 | Discussions
Bernard Delley Senior Member • Posts: 1,008
D7200 HPS (hot pixel suppression) news
3

The D7200 appears to boast a new HPS method. Like as some previous models it kicks in at exposures longer than 1/5 s, and cannot be user controlled. It appears to allow for more peaky star images and it is quite effective at removing hot pixels.

This is a follow up on earlier HPS analysis by Marianne Oelund: In-camera processing of long-exposure RAW data and myself: hot-pixel suppression in D800 and D4  and after introduction of a nonzero black level in Nikon DSLR there was an issue with HPS The Story of One Hot D810 Pixel...   The present status for D810 is unknown to me, as I currently have no access to a sample of this camera.

The following graph shows cumulative statistics over all 24 M raw pixels of a black frame at ISO 1600 at 1/5s. There were very few hot pixels. Some come with brightness even way more than raw number 1000.

At 1/5s the HPS is active and removes the hot pixels. The cumulative noise statistics is the best I have seen so far at ISO1600 on any camera.

Nikon D4 Nikon D7200 Nikon D800 Nikon D810
If you believe there are incorrect tags, please send us this post using our feedback form.
bolador Contributing Member • Posts: 528
Re: D7200 HPS (hot pixel suppression) news

The D7200 appears to boast a new HPS method. Like as some previous models it kicks in at exposures longer than 1/5 s, and cannot be user controlled. It appears to allow for more peaky star images and it is quite effective at removing hot pixels.

This is a follow up on earlier HPS analysis by Marianne Oelund: In-camera processing of long-exposure RAW data and myself: hot-pixel suppression in D800 and D4  and after introduction of a nonzero black level in Nikon DSLR there was an issue with HPS The Story of One Hot D810 Pixel...   The present status for D810 is unknown to me, as I currently have no access to a sample of this camera.

The following graph shows cumulative statistics over all 24 M raw pixels of a black frame at ISO 1600 at 1/5s. There were very few hot pixels. Some come with brightness even way more than raw number 1000.

At 1/5s the HPS is active and removes the hot pixels. The cumulative noise statistics is the best I have seen so far at ISO1600 on any camera.

The Story of One Hot D810 Pixel...  seems to be a broken link.
So how does this new algorithm affects astro photography?

 bolador's gear list:bolador's gear list
Panasonic Lumix DMC-FZ200 Nikon D90 Nikon D7200 Tokina AT-X Pro 11-16mm f/2.8 DX
OP Bernard Delley Senior Member • Posts: 1,008
Re: D7200 HPS (hot pixel suppression) news

bolador wrote:

 Story of One Hot D810 Pixel... seems to be a broken link.

another try at the 810 link

I expect the a 810 firmware upgrade has fixed this  issue since.

So how does this new algorithm affects astro photography?

It should be better than the HPS since D4. But, as the D4,D800,D7100  etc  algorithm was very acceptable,

the improvement might be subtle.

With the non-zero digital black level cameras like D810 and D7200  offer more and  better options for multi frame  noise reduction methods . Thus they are preferable for astro- and some  other kinds of low light imaging.

bolador Contributing Member • Posts: 528
Re: D7200 HPS (hot pixel suppression) news

bolador wrote:

 Story of One Hot D810 Pixel... seems to be a broken link.

another try at the 810 link

I expect the a 810 firmware upgrade has fixed this  issue since.

So how does this new algorithm affects astro photography?

It should be better than the HPS since D4. But, as the D4,D800,D7100  etc  algorithm was very acceptable,

the improvement might be subtle.

With the non-zero digital black level cameras like D810 and D7200  offer more and  better options for multi frame  noise reduction methods . Thus they are preferable for astro- and some  other kinds of low light imaging.

Link now is working.. thanks.

With being "better" do you mean "not as bad"?
Or is this better to nothing..
As I suspect this kind of intervention to the data is not any good to astro photography. When a stack of dark frames should do a better job...

 bolador's gear list:bolador's gear list
Panasonic Lumix DMC-FZ200 Nikon D90 Nikon D7200 Tokina AT-X Pro 11-16mm f/2.8 DX
OP Bernard Delley Senior Member • Posts: 1,008
Re: D7200 HPS (hot pixel suppression) news
1

bolador wrote:

With being "better" do you mean "not as bad"?

Or is this better to nothing..
As I suspect this kind of intervention to the data is not any good to astro photography. When a stack of dark frames should do a better job...

hot pixels are very bright, even the Poisson noise from the hot part is likely to completely mask any low light detail still recorded on such a pixel.  Also a good stack of dark cannot do away with it, it only may give an improved average value for the hot pixel. See plot 1 in my OP post for illustration. So some image cleanup has to be done for these singular bad  pixels.

Practically no lens is capable producing a star image where a single (4 micron) pixel is X  times brighter than any of its neighbor pixels. The HPS of D4 D800 D7100  allows X= 2 times brighter, and that pretty much  cuts it, even for the sharpest lenses at their optimum aperture.  The new algorithm lets X= 6-8 times brighter pixels pass, anticipating incredibly sharp lenses: this implies MTF > 80% at 125 lp/mm .   As this HPS remains very effective in removing bad pixel data, it is really a very gad algorithm.  -- I agree with you that there is a minor gripe that one would like an option to turn it off altogether.  After having seen perhaps hundreds of false stars, I might turn it on again right away

As the hot pixels tend to be permanent, there is also user controlled hot pixel remapping . Contrary to the earlier models, it is activated by a single "clean sensor now" activation on the D7200.

bolador Contributing Member • Posts: 528
Re: D7200 HPS (hot pixel suppression) news

bolador wrote:

With being "better" do you mean "not as bad"?

Or is this better to nothing..
As I suspect this kind of intervention to the data is not any good to astro photography. When a stack of dark frames should do a better job...

hot pixels are very bright, even the Poisson noise from the hot part is likely to completely mask any low light detail still recorded on such a pixel.  Also a good stack of dark cannot do away with it, it only may give an improved average value for the hot pixel. See plot 1 in my OP post for illustration. So some image cleanup has to be done for these singular bad  pixels.

Practically no lens is capable producing a star image where a single (4 micron) pixel is X  times brighter than any of its neighbor pixels. The HPS of D4 D800 D7100  allows X= 2 times brighter, and that pretty much  cuts it, even for the sharpest lenses at their optimum aperture.  The new algorithm lets X= 6-8 times brighter pixels pass, anticipating incredibly sharp lenses: this implies MTF > 80% at 125 lp/mm .   As this HPS remains very effective in removing bad pixel data, it is really a very gad algorithm.  -- I agree with you that there is a minor gripe that one would like an option to turn it off altogether.  After having seen perhaps hundreds of false stars, I might turn it on again right away

As the hot pixels tend to be permanent, there is also user controlled hot pixel remapping . Contrary to the earlier models, it is activated by a single "clean sensor now" activation on the D7200.

Clear as water.
Thank you. .

 bolador's gear list:bolador's gear list
Panasonic Lumix DMC-FZ200 Nikon D90 Nikon D7200 Tokina AT-X Pro 11-16mm f/2.8 DX
dpthoughts Regular Member • Posts: 116
Re: D7200 HPS (hot pixel suppression) news

Bernard Delley wrote:

a black frame at ISO 1600 at 1/5s

To me it looks like, as if for the red channel, HPS is already effective at 1/5s. Else it could be pure luck, that the sensor doesn't have any hot red pixel, of course. Did you try shorter exposures than 1/5s to check, if red hot pixels emerge in the statistics then?

OP Bernard Delley Senior Member • Posts: 1,008
Re: D7200 HPS (hot pixel suppression) news

dpthoughts wrote:

Bernard Delley wrote:

a black frame at ISO 1600 at 1/5s

To me it looks like, as if for the red channel, HPS is already effective at 1/5s. Else it could be pure luck, that the sensor doesn't have any hot red pixel, of course. Did you try shorter exposures than 1/5s to check, if red hot pixels emerge in the statistics then?

You are right that it is not evident from the plot that the red HPS kicks in between 1/5 and 1/4s.

However, from the details of the HPS analysis, described in my previous post in this thread, I can see that also for the red channel HPS kicks in. For 1/5s there were 180 red pixels among the 23Mpix/4 that did not obey the HPS criterion.  At 1/4s there are none that defy the HPS criterion. So there is a clear indication of HPS acting on the moderately hot pixels in the fat tail.

Another good news in this is that this camera/sensor has fewer misbehaving pixels than any other sensor type,  I have "seen" before (about 10 Nikon, 3 Canon 2 Pentax). This comes in addition to the now very good HPS algorithm.

For the conditions of the test shown in the figure, we find 8 hot pixels exceeding raw number 250 among the 24 Mpix. The average expectation per channel is thus about 2. By consequence of the Poisson distribution, one has a 13.5% chance in each channel to just observe none such pixels, and a higher probability that no occurrence happens in any of the 4 color channels. What I try to say is that the appearance of above test result properties is fairly likely for this sensor. It is a very good sign.

dpthoughts Regular Member • Posts: 116
Re: D7200 HPS (hot pixel suppression) news

Thanks, that's interesting stuff.
Another interesting question somewhat related to HPS is, if there would be any forced RAW denoising kicking in at a certain ISO threshold, or not. 
If so, that serious users would want to avoid such high ISOs, because any RAW denoising would likely kill detail or introduce artefacts or destroy the statistical nature of the noise, on which our preferred denoising software likely depends to work efficiently.

A third interesting question concerning ISO choice would be, what's the ISO threshold where analogue amplification ends and digital multiplication kicks in.
I would use and recommend following ISO choice then:
- never exceed a RAW-denoising threshold, better is to underexpose then.
- only exceed the digital-gain-threshold, if this doesn't burn out highlights in the picture.
I believe that anwers to these have been made (or attempted) for the D7100, but since the D7200 has a new sensor and seemingly new refined algorithms, any D7100 insights may be obsolete for the D7200.

OP Bernard Delley Senior Member • Posts: 1,008
Re: D7200 HPS (hot pixel suppression) news

dpthoughts wrote:

Thanks, that's interesting stuff.
Another interesting question somewhat related to HPS is, if there would be any forced RAW denoising kicking in at a certain ISO threshold, or not.

to the best of my own + forum  knowledge there is no such high  ISO denoising with Nikon. (Sony cameras have been known  for such denoising applied to a then ironically called "cooked raw") . Nikon high ISO NR applies to jpg only, 'long exposure NR' is a built in black frame subtraction which can be turned off by the user. This subtraction requires a wait time equal to exposure time: potentially annoying. There are more sophisticated black frame subtraction methods possible in post, if needed.

If so, that serious users would want to avoid such high ISOs, because any RAW denoising would likely kill detail or introduce artefacts or destroy the statistical nature of the noise, on which our preferred denoising software likely depends to work efficiently.

A third interesting question concerning ISO choice would be, what's the ISO threshold where analogue amplification ends and digital multiplication kicks in.

digital multiplication is used differently on different camera models. It shows up in missing codes. With Nikon it always was chosen judiciously as to not noticeably increase quantization noise. The HPS has a favorable effect on 'salt" noise especially in deep shadows.

I would use and recommend following ISO choice then:
- never exceed a RAW-denoising threshold, better is to underexpose then.
- only exceed the digital-gain-threshold, if this doesn't burn out highlights in the picture.
I believe that anwers to these have been made (or attempted) for the D7100, but since the D7200 has a new sensor and seemingly new refined algorithms, any D7100 insights may be obsolete for the D7200.

Bill Claff has some interesting analysis and graphs. Comparing D7200 to D7100 shows that D7200 is an improvement in all aspects: as with d7100 read noise looks quite 'ISO-less' also the dynamic range  graph shows no hiccups. Means you have a lot of formal underexposure choices to preserve headroom without loosing much to noise.

Steve Bingham
Steve Bingham Forum Pro • Posts: 25,434
Re: D7200 HPS (hot pixel suppression) news

Bernard,

It really is a non-issue. Enough said.

-- hide signature --

Steve Bingham
www.dustylens.com
www.ghost-town-photography.com
Latest postings are always at the bottom of each page.

 Steve Bingham's gear list:Steve Bingham's gear list
Sony RX100 VI Nikon D810 Nikon D5500 Nikon D7200 Nikon AF-S DX Nikkor 18-105mm f/3.5-5.6G ED VR +31 more
bclaff Veteran Member • Posts: 6,876
Re: D7200 HPS (hot pixel suppression) news
1

Bernard Delley wrote:

dpthoughts wrote:

Thanks, that's interesting stuff.
Another interesting question somewhat related to HPS is, if there would be any forced RAW denoising kicking in at a certain ISO threshold, or not.

to the best of my own + forum knowledge there is no such high ISO denoising with Nikon. (Sony cameras have been known for such denoising applied to a then ironically called "cooked raw") . Nikon high ISO NR applies to jpg only, 'long exposure NR' is a built in black frame subtraction which can be turned off by the user. This subtraction requires a wait time equal to exposure time: potentially annoying.

Nikon does have High ISO Noise Reduction that can be controlled by the user and can be set to OFF.
When it is active it affects NEF as well as JPG files.

This is separate from the Long Exposure Noise Reduction (LENR) that you mention.

Furthermore, some Nikon models (not the D7200) perform Noise Reduction on the NEF (and JPG) that cannot be disabled based on the length of exposure.

Here's an example I documented for the D90: D90 Does Noise Reduction on NEFs at Longer Exposures

-- hide signature --

Bill ( Your trusted source for independent sensor data at http://www.PhotonsToPhotos.net )

bclaff Veteran Member • Posts: 6,876
Re: D7200 HPS (hot pixel suppression) news

Your cumulative distribution criteria is a good litmus test.

Using a more detailed analysis I have confirmed your 1/5s conclusion for all channels.

-- hide signature --

Bill ( Your trusted source for independent sensor data at http://www.PhotonsToPhotos.net )

azguy
azguy Veteran Member • Posts: 7,599
Re: D7200 HPS (hot pixel suppression) news

Steve Bingham wrote:

Bernard,

It really is a non-issue. Enough said.

-- hide signature --

Steve Bingham
www.dustylens.com
www.ghost-town-photography.com
Latest postings are always at the bottom of each page.

Agree with Steve on this one.

OP Bernard Delley Senior Member • Posts: 1,008
Re: High ISO Noise Reduction only for jpg

bclaff wrote:

Bernard Delley wrote:

to the best of my own + forum knowledge there is no such high ISO denoising with Nikon. (Sony cameras have been known for such denoising applied to a then ironically called "cooked raw") . Nikon high ISO NR applies to jpg only, 'long exposure NR' is a built in black frame subtraction which can be turned off by the user. This subtraction requires a wait time equal to exposure time: potentially annoying.

Nikon does have High ISO Noise Reduction that can be controlled by the user and can be set to OFF.
When it is active it affects NEF as well as JPG files.

I checked High ISO Noise Reduction "High" against "Off" on NEF black frames at ISO 1600 from 20s to 1/40s. I could not detect any significant difference. If you still think " High ISO Noise Reduction" would affect the NEF,, please give a hint at exposure conditions ISO, time +?? where one should be able to verify the effect.

This is separate from the Long Exposure Noise Reduction (LENR) that you mention.

Here's an example I documented for the D90: D90 Does Noise Reduction on NEFs at Longer Exposures

I have no direct access any more to a D90. The D90 HPS like for other cameras of that period was arguably too aggressive, and thus may have produced noticeable noise reduction as a side effect. I recall that its HPS was turned on near 1s exposure time, which would be consistent with your cited result.

The D7200 HPS only affects a relatively small number of very 'noisy' pixels. Here the HPS effect on noise is about 0.3% reduction of noise RMS.

bclaff Veteran Member • Posts: 6,876
Re: High ISO Noise Reduction only for jpg

Bernard Delley wrote:

bclaff wrote:

Bernard Delley wrote:

to the best of my own + forum knowledge there is no such high ISO denoising with Nikon. (Sony cameras have been known for such denoising applied to a then ironically called "cooked raw") . Nikon high ISO NR applies to jpg only, 'long exposure NR' is a built in black frame subtraction which can be turned off by the user. This subtraction requires a wait time equal to exposure time: potentially annoying.

Nikon does have High ISO Noise Reduction that can be controlled by the user and can be set to OFF.
When it is active it affects NEF as well as JPG files.

I checked High ISO Noise Reduction "High" against "Off" on NEF black frames at ISO 1600 from 20s to 1/40s. I could not detect any significant difference. If you still think " High ISO Noise Reduction" would affect the NEF,, please give a hint at exposure conditions ISO, time +?? where one should be able to verify the effect.

You seem to be right about the D7200; High ISO Noise Reduction does not affect the NEF.
I'm pretty sure this was not true on some early Nikon bodies but it's not something I've tracked over time.
(So in an abundance of caution I disable it when testing Nikon NEF files.)

Regards,

-- hide signature --

Bill ( Your trusted source for independent sensor data at http://www.PhotonsToPhotos.net )

Keyboard shortcuts:
FForum MMy threads