In-camera processing of long-exposure RAW data

Started Jan 21, 2010 | Discussions
cluna Senior Member • Posts: 1,403
Re: A couple of questions

William Carson wrote:

eNo wrote:

2) Do we have a guess as to the motivation to apply this irreversible "fix" to the RAW data itself rather than implementing it only in JPG?

One possible reason for this would be to lessen the number of owners seeing hot pixels and choosing to return the camera as flawed which would be costly for the company as, in addition to the replacement cost, the company might have to only take sensors cuts from the wafer that have less hot or flawed pixels. So, its all about the money.

Guys/gals, this is only on LONG exposures.... Don't get worked-up over something that the majority ppl never take advantage of.

-C

mozarkid Regular Member • Posts: 380
Re: Yes, it affects the NEF file data

This kinda makes me angry... seems Nikon is modifying our images to hide defects.

Nikon should, at the very least, provide a way to dissable this.

bob elkind Veteran Member • Posts: 5,815
gettig worked up... or not.

cluna wrote:

Guys/gals, this is only on LONG exposures.... Don't get worked-up over something that the majority ppl never take advantage of.

There is good reason for the excitement...

  1. this 'feature' has been hidden since... forever. definite shock/surprise factor.

  2. yes, this only applies for LONG exposures. on the other hand, in many or most situations when exposures are deliberately long, this 'feature' is most obtrusive and undesired.

-- hide signature --

Bob Elkind
Family,in/outdoor sports, landscape, wildlife
photo galleries at http://eteam.zenfolio.com
my relationship with my camera is strictly photonic

 bob elkind's gear list:bob elkind's gear list
Nikon D700 Canon EOS 500D Nikon D600 Canon EF-S 18-55mm f/3.5-5.6 Nikon AF Nikkor 50mm f/1.8D +5 more
William Carson
William Carson Veteran Member • Posts: 6,205
Example of long exposure with D3 -

The image from this link here seems to agree with you.
http://forums.dpreview.com/forums/read.asp?forum=1021&message=34194886

Will

cluna Senior Member • Posts: 1,403
Re: gettig worked up... or not.

bob elkind wrote:

cluna wrote:

Guys/gals, this is only on LONG exposures.... Don't get worked-up over something that the majority ppl never take advantage of.

There is good reason for the excitement...

  1. this 'feature' has been hidden since... forever. definite shock/surprise factor.

  2. yes, this only applies for LONG exposures. on the other hand, in many or most situations when exposures are deliberately long, this 'feature' is most obtrusive and undesired.

Hidden? Nawh. long exposure nr is mentioned in the manuals (the D300s says added to 8sec or more so may not be the same but I dont know--and u can turn it off). However. what do you expect, NR needs to be done, you have a fixed buffers size and data has to be stored.

Problem is, a perfect exposure like is being used for this algoritm testing is not indicative of a long exposure where gaussian noise distribution will typically average out.

-C

Yamo Junior Member • Posts: 43
Re: gettig worked up... or not.

cluna wrote:

bob elkind wrote:

cluna wrote:

Guys/gals, this is only on LONG exposures.... Don't get worked-up over something that the majority ppl never take advantage of.

There is good reason for the excitement...

  1. this 'feature' has been hidden since... forever. definite shock/surprise factor.

  2. yes, this only applies for LONG exposures. on the other hand, in many or most situations when exposures are deliberately long, this 'feature' is most obtrusive and undesired.

Hidden? Nawh. long exposure nr is mentioned in the manuals (the D300s says added to 8sec or more so may not be the same but I dont know--and u can turn it off). However. what do you expect, NR needs to be done, you have a fixed buffers size and data has to be stored.

Problem is, a perfect exposure like is being used for this algoritm testing is not indicative of a long exposure where gaussian noise distribution will typically average out.

-C

Er, two contexts of long exposure. Marianne was talking about an effect that initiates at 1/4 second. Not all that long... with VR I've taken some such handheld. The 8+ second NR is a different beast altogether.

Cheers,

-Yamo-

OP Marianne Oelund Veteran Member • Posts: 7,779
Kurt Vonnegut's "Harrison Bergeron" was correct (nt)

- - - - -

kyklops Forum Member • Posts: 66
Re: In-camera processing of long-exposure RAW data

Hey all of you,

reading this story I must raise the followup question (yes I do shoot Canon:-) if the "raw cooking" in Nikon cams is limitd to longer than 1/4 exposures. I have seen lots of claims of Nikon "cooking" black level clipping too in their cams, irregardless of shutter speed.

These clean D3s images, are they only the result of Nikon hex brews????

Of course if you eliminate all "" abborrations "" in images and call it raw, what have you got???

The Xroll

Raul Veteran Member • Posts: 8,504
Hello William

William Carson wrote:

The image from this link here seems to agree with you.
http://forums.dpreview.com/forums/read.asp?forum=1021&message=34194886

how is this relevant to the topic? i am missing something from your message.
regards

 Raul's gear list:Raul's gear list
Nikon Df Nikon D810 Nikon D5 Nikon D850
bob elkind Veteran Member • Posts: 5,815
Re: getting worked up... or not.

cluna wrote:

bob elkind wrote:

...

There is good reason for the excitement...

  1. this 'feature' has been hidden since... forever. definite shock/surprise factor.

  2. yes, this only applies for LONG exposures. on the other hand, in many or most situations when exposures are deliberately long, this 'feature' is most obtrusive and undesired.

Hidden? Nawh.

Hidden: yes. It's not mentioned in the manuals (D700).

long exposure nr is mentioned in the manuals (the D300s says added to 8sec or more so may not be the same but I dont know--and u can turn it off).

In the D700 manual, long exposure noise reduction is described thus:

  • ostensibly black frame subtraction

  • kicks in for 1sec (or longer) exposures

  • optional (it can be turned off).

What Marianne has brought to our attention is:

  • not optional

  • arguably more (differently) destructive (of certain detail) than black frame subtraction

  • it's threshold is 0.5sec rather than 1sec.

Other than that, it's entirely as described in the manual...

However. what do you expect, NR needs to be done, you have a fixed buffers size and data has to be stored.

You're saying all noise reduction, without user control, should be perfectly acceptable for everyone and anyone. Care to qualify that a bit?

Problem is, a perfect exposure like is being used for this algorithm testing is not indicative of a long exposure where gaussian noise distribution will typically average out.

I'm missing your point. Please try again with different wording. Long exposure noise does not exhibit gaussian distribution characteristics, if that's what you're trying to say.

-- hide signature --

Bob Elkind
Family,in/outdoor sports, landscape, wildlife
photo galleries at http://eteam.zenfolio.com
my relationship with my camera is strictly photonic

 bob elkind's gear list:bob elkind's gear list
Nikon D700 Canon EOS 500D Nikon D600 Canon EF-S 18-55mm f/3.5-5.6 Nikon AF Nikkor 50mm f/1.8D +5 more
jim stirling
jim stirling Veteran Member • Posts: 7,356
Re: In-camera processing of long-exposure RAW data

Whilst I am more than happy to accept the conclusions of the people with the knowledge specific to this form of analysis . The reality of the situation seems that it is only of relevance to those taking some form of photography where truly accurate image data was critical , perhaps astrophotography for scientific analysis as opposed to about 99.99% of photography that has aesthetics or capturing the moment as its objectives. And frankly unless you are in this probably microscopic niche of photographers who this would effect , the relevance to the vast majority of photographers is insignificant.

By directly comparing raw { using the admittedly low tech visual test} files across the makers it is very clear that with cameras such as the D3X Nikon manages to achieve excellent levels of detail. I would be interested in real life examples {beyond those more interested in scientific data than photography } where this would have any significant impact.
Jim

 jim stirling's gear list:jim stirling's gear list
Panasonic FZ1000 Panasonic Lumix DMC-GF1 Panasonic Lumix DMC-GX7 Panasonic Lumix DMC-GH4 Nikon D810 +12 more
OP Marianne Oelund Veteran Member • Posts: 7,779
Nikon have declared zero equal to zero
1

kyklops wrote:

reading this story I must raise the followup question (yes I do shoot Canon:-) if the "raw cooking" in Nikon cams is limitd to longer than 1/4 exposures. I have seen lots of claims of Nikon "cooking" black level clipping too in their cams, irregardless of shutter speed.

There is no processing of RAW data related to black level, i.e., there is no artificial black clipping added. Nikon set the DN=0 level to be at the analog 0V level. However, since noise is bipolar (averages to 0, but swings both above and below), half of the noise is suppressed (truncated) when the A/D converter assigns DN=0 to negative voltages, and there is no signal present (black frame).

In blacks or very dark shadows, the noise truncation produces a false signal since the remaining noise no longer averages to 0. We're talking about a very small signal, though, and most users would never notice it. It's of interest to those who study sensor performance, and a few critical users such as astrophotographers who are trying to stack images, etc.

These clean D3s images, are they only the result of Nikon hex brews????

Sensor data from the D3S is consistent across the ISO range, and the signal range, and does not show evidence of post-processing effects (for shutter speeds of 1/5 sec or faster).

Of course if you eliminate all "" abborrations "" in images and call it raw, what have you got???

In attempting to remove "hot" pixels, Nikon have actually created aberrations.

eNo
eNo Forum Pro • Posts: 11,744
So to get all this straight...

In reading this thread, there appears to be some confussion (certainly for me) as to what NR we are talking about. If I have deciphered things correctly, I see 3 types of NR described here:

1. Internal, non-user-configurable NR that kicks in somewhere between 1/4-1/2 sec exposures. This irreversibly affects RAW data.

2. Long Exposure NR, user-configurable that kicks in at exposures between 1-8 sec, depending on camera model. This applies black frame subtraction and is evident only at RAW-to-other_format conversion, either in camera, or in PP (NX2).

3. High ISO NR, user-configurable that kicks in at a given ISO level (800-1600 depending on camera model?). Also applies to RAW-to-other_format conversion, ...

The OP deals with #1. Do I have this right?

~~~~~~~~~~~~~~

It's easy to argue about equipment and technique, but hard to argue with a good photograph -- and more difficult to capture one .

Gallery and blog: http://esfotoclix.com
Special selections: http://esfotoclix.com/store
Flickr stream: http://www.flickr.com/photos/22061657@N03

OP Marianne Oelund Veteran Member • Posts: 7,779
The Sins of the Algorithm

jim stirling wrote:

The reality of the situation seems that it is only of relevance to those taking some form of photography where truly accurate image data was critical , perhaps astrophotography for scientific analysis as opposed to about 99.99% of photography that has aesthetics or capturing the moment as its objectives. And frankly unless you are in this probably microscopic niche of photographers who this would effect , the relevance to the vast majority of photographers is insignificant.

There are a great many details ignored by the vast majority of photographers. That does not make them insignificant.

The hot-pixel algorithm treats every single pixel in an image (except for the two border rows/columns, which never end up in camera JPEG or raw converter output anyhow), and adjusts every one it finds that has a higher value than any of its 8 same-channel neighbors. That means in a 12Mpix image, even one which contains no hot pixels at all, about 50,000 pixel values will be affected, on average. If an image contains many very fine details, that number will go up substantially.

If that doesn't matter to you, that's fine, but it annoys me to no end, especially when a simple change to the algorithm could avoid all of this mess. To me, the fact that Nikon have left this utterly simple-minded and destructive algorithm in their top-line cameras costing $5K US and up, is completely inexcusable.

Gerd Roppelt New Member • Posts: 16
Re: getting worked up... or not.

bob elkind wrote:

cluna wrote:

bob elkind wrote:

...

There is good reason for the excitement...

  1. this 'feature' has been hidden since... forever. definite shock/surprise factor.

Hidden? Nawh.

Hidden: yes. It's not mentioned in the manuals (D700).

May be it is not mentioned correctly in the manuals, but people doing astrophotography know this very well, and you can find descriptions of this behavior at astro websites for a long time. So for well informed people its absolutely no shock nor surprise.

Gerd

geo444
geo444 Contributing Member • Posts: 510
Re: getting worked up... or not.

bob elkind wrote:

In the D700 manual, long exposure noise reduction is described thus:

  • ostensibly black frame subtraction

  • kicks in for 1sec (or longer) exposures

  • optional (it can be turned off).

What Marianne has brought to our attention is:

  • not optional

  • arguably more (differently) destructive (of certain detail) than black frame subtraction

  • it's threshold is 0.5sec rather than 1sec.

Other than that, it's entirely as described in the manual...

i agree,
even the Mode 3 is described by NiKon in the D3(xs) manual, page 299(D3) :
" ... Long Exp. NR (Long Exposure Noise Reduction)
noise reduction will not be performed if the camera
is turned off before processing is complete...
"

do you know when NiKon decided to block this Mode 3 ??

thanx

OP Marianne Oelund Veteran Member • Posts: 7,779
Yes, that's a valid summary

eNo wrote:

In reading this thread, there appears to be some confusion (certainly for me) as to what NR we are talking about. If I have deciphered things correctly, I see 3 types of NR described here:

1. Internal, non-user-configurable NR that kicks in somewhere between 1/4-1/2 sec exposures. This irreversibly affects RAW data.

I try to avoid calling this NR, which I consider somewhat misleading. Its intended function is to identify and suppress hot pixels; the fact that it does somewhat more than that, is a side-effect of its inadequate design.

2. Long Exposure NR, user-configurable that kicks in at exposures between 1-8 sec, depending on camera model. This applies black frame subtraction and is evident only at RAW-to-other_format conversion, either in camera, or in PP (NX2).

2) starts at 10-sec exposures in the D3X.

3. High ISO NR, user-configurable that kicks in at a given ISO level (800-1600 depending on camera model?). Also applies to RAW-to-other_format conversion, ...

3) starts at ISO 2000 in the D3, and ISO 4000 in the D3S.

The OP deals with #1. Do I have this right?

Your confusion seems to have dissipated.

eNo
eNo Forum Pro • Posts: 11,744
Re: The Sins of the Algorithm

Marianne Oelund wrote:

jim stirling wrote:

The reality of the situation seems that it is only of relevance to those taking some form of photography where truly accurate image data was critical , perhaps astrophotography for scientific analysis as opposed to about 99.99% of photography that has aesthetics or capturing the moment as its objectives. And frankly unless you are in this probably microscopic niche of photographers who this would effect , the relevance to the vast majority of photographers is insignificant.

There are a great many details ignored by the vast majority of photographers. That does not make them insignificant.

The hot-pixel algorithm treats every single pixel in an image (except for the two border rows/columns, which never end up in camera JPEG or raw converter output anyhow), and adjusts every one it finds that has a higher value than any of its 8 same-channel neighbors. That means in a 12Mpix image, even one which contains no hot pixels at all, about 50,000 pixel values will be affected, on average. If an image contains many very fine details, that number will go up substantially.

If that doesn't matter to you, that's fine, but it annoys me to no end, especially when a simple change to the algorithm could avoid all of this mess. To me, the fact that Nikon have left this utterly simple-minded and destructive algorithm in their top-line cameras costing $5K US and up, is completely inexcusable.

That is truly disturbing. Would a fix be that the pixel should be declared "hot" only if it exceeds a certain threshold and it doesn't seem in line with its neighbors. If not, what fix would you propose to accomplish the intent (eliminate hot pixels) without unwanted/unnecessary averaging?

~~~~~~~~~~~~~~

It's easy to argue about equipment and technique, but hard to argue with a good photograph -- and more difficult to capture one .

Gallery and blog: http://esfotoclix.com
Special selections: http://esfotoclix.com/store
Flickr stream: http://www.flickr.com/photos/22061657@N03

Thom Hogan Forum Pro • Posts: 13,659
Re: In-camera processing of long-exposure RAW data

jim stirling wrote:

The reality of the situation seems that it is only of relevance to those taking some form of photography where truly accurate image data was critical

First: what form of photography is it where accurate image data wasn't critical? ;~)

But, no, I think you're making a mistake in dismissing this. It's something we Nikon users have been dealing with for most of DSLR-dom. I'm a landscape photographer that shoots at the edges of the day quite often. I'm very often at shutter speeds slower than 1/4 a second. The implications of the algorithm are that some high frequency data is being destroyed in my images, and yes, I've seen examples of it.

I've long written about trying to capture "optimal data" in the field. One thing I used to be able to do is set Long exp. NR to On, take a long exposure, then flip the camera off before NR had completed. This allowed me to bypass the "filter." Apparently, this is now "fixed" by Nikon's latest firmware and not possible.

-- hide signature --

Thom Hogan
author, Complete Guides to Nikon bodies (21 and counting)
http://www.bythom.com

jim stirling
jim stirling Veteran Member • Posts: 7,356
Re: The Sins of the Algorithm

Marianne Oelund wrote:

The reality of the situation seems that it is only of relevance to those taking some form of photography where truly accurate image data was critical , perhaps astrophotography for scientific analysis as opposed to about 99.99% of photography that has aesthetics or capturing the moment as its objectives. And frankly unless you are in this probably microscopic niche of photographers who this would effect , the relevance to the vast majority of photographers is insignificant.

There are a great many details ignored by the vast majority of photographers. That does not make them insignificant.

The hot-pixel algorithm treats every single pixel in an image (except for the two border rows/columns, which never end up in camera JPEG or raw converter output anyhow), and adjusts every one it finds that has a higher value than any of its 8 same-channel neighbors. That means in a 12Mpix image, even one which contains no hot pixels at all, about 50,000 pixel values will be affected, on average. If an image contains many very fine details, that number will go up substantially.

If that doesn't matter to you, that's fine, but it annoys me to no end, especially when a simple change to the algorithm could avoid all of this mess. To me, the fact that Nikon have left this utterly simple-minded and destructive algorithm in their top-line cameras costing $5K US and up, is completely inexcusable.

I would say that absolute critical image accuracy falls into a distant second place compared to delivering an aesthetically pleasing image . And in fact , images that are a more technically accurate when compared to the original scene are often much less aesthetically pleasing. And this can be evidenced by the extent of in camera or post processing adjustments that the vast majority of photographers do to ensure pleasing results as opposed to technical perfection.

And I would be genuinely interested in an a real life sample image where this issue shows significant degradation to the end result. If the problem is significant then it should be easily demonstratable . I do mainly event wedding shooting using D3s D700 and landscape work with the D3x and I am very happy with the results. And the D3x in particular seems to receive almost universal praise for its image quality { in direct comparison to the other 20mp+ models from the other makers} . I do wonder about the real life effects of this.

I am certainly not dismissing the validity of your observations only the relevance to the end result . And in which specific types of photography this would show a significant effect.
Jim

 jim stirling's gear list:jim stirling's gear list
Panasonic FZ1000 Panasonic Lumix DMC-GF1 Panasonic Lumix DMC-GX7 Panasonic Lumix DMC-GH4 Nikon D810 +12 more
Keyboard shortcuts:
FForum MMy threads