D7100 'streaking'

Started Apr 7, 2013 | Discussions
Jostian
Jostian Veteran Member • Posts: 3,421
Re: D7100 'streaking' - doesn't look like blooming or flare
1

AngryCorgi wrote:

Well, I'm feeling good about not being an early adopter on the D7100/D5200!!

+1 the QC issues of the original D7000 come to mind, I'd really hoped the D7100 would be 'sorted' QC wise...guess not, lets see, could be an isolated thing, time will tell.

-- hide signature --

Jostian

 Jostian's gear list:Jostian's gear list
Nikon D700 Panasonic Lumix DMC-GX8 Nikon D500 Tamron SP 24-70mm F2.8 Di VC USD Tamron 15-30mm F2.8 +5 more
krikman Regular Member • Posts: 419
Re: D7100 'streaking' - doesn't look like blooming or flare

Jack Hogan wrote:

krikman wrote:

Some reasoning:

ADR algorhytm works with native data (in 14-bit space?) by somehow analysing of large blocks of data,

If you mean ADL, we can exclude it as the culprit because I see streaking in RawTherapee and RawDigger which do not do ADL.

I presume that NEF files were shot with ADL in-camera enabled. (I forgot its canonic Nikon name). We cannot exclude it because it is already in NEF file. Nikon had 2 type of ADL, one simple as postprogram, such as in d3000 camera at least. Another is in senior cameras as in D300s, which involved data reading from chip.)

then RAW compression defines which data will cut

Raw compression is just a lookup table, no intelligence/definition involved

I mean compressian of data to be 12-bit wide, which are image specific. No

in conversion to 12-bit, but coders forgot to cut unusable bits from precise calculations so we got rounding error in green channel.

(i.e. pixel has value 10100011, we cut last 2 binary digits '11' but they appear in each and every cell in row and add regular 0,5% for whole row)

I doubt that Nikon would do this differently in the D7100 than in the D800 or that it would depend on whether part of the line is much brighter than another part.

No-no-no. Nikon do it in all cameras beginning from D200 or maybe from early beginning. But here at d7100 sample we found visual bug in these program code $)

But VERY IMPORTANT DISCOVERY:

Until ISO400 we presumable have clear raw data,

Yet streaking is visible at ISO100 without pushing

So at ISO 100-400 we have one data conversion path, but from ISO 800 image data gets another way...

but from ISO800 Nikon 'cooks' RAW data by different datapath with intensive onchip (?) NR. It is common sense but first time we had clear instrumental proving.

Can you provide details on your analysis and the data to support your claim?

You've already seen it on pictures. From these samples I conclude that these stripe artifacts are no lens defect, no photosites overloading, no sensor errors but round error in data calculations.

Jack

You, Jack, talk about RAW conversion software, but I talk about preprocesses before writing NEF.

DMillier Forum Pro • Posts: 20,884
Re: "I love how it forces me to become a better photographer"

Because the higher resolution exposes poor focusing and camera shake?

-- hide signature --
dougster1979 Regular Member • Posts: 106
Re: "I love how it forces me to become a better photographer"

So you don`t need good technique to use a 12mp camera

_sem_ Veteran Member • Posts: 5,032
Re: D7100 'streaking' - doesn't look like blooming or flare

krikman wrote:

If you mean ADL, we can exclude it as the culprit because I see streaking in RawTherapee and RawDigger which do not do ADL.

I presume that NEF files were shot with ADL in-camera enabled. (I forgot its canonic Nikon name). We cannot exclude it because it is already in NEF file. Nikon had 2 type of ADL, one simple as postprogram, such as in d3000 camera at least. Another is in senior cameras as in D300s, which involved data reading from chip.)

Nikon has only one type of ADL, but several generations of it. The other you refer to is the post-processing DL. But it was shown many times that the only thing ADL does to raw data is up to about a stop of underexposure. I haven't seen a confirmation that the latest generation does the same but I'd be surprised if it doesn't - although I believe this underexposure of raw data is in fact a bad idea and unnecessary, except perhaps for some limitations for instant in-camera image processing.

No-no-no. Nikon do it in all cameras beginning from D200 or maybe from early beginning. But here at d7100 sample we found visual bug in these program code $)

CCD sensors had a much worse vertical bleeding thing but that's a different technology.

Remember Fuji invented an entirely new form of sensor bleeding for the early X10 known as the white orbs.

You, Jack, talk about RAW conversion software, but I talk about preprocesses before writing NEF.

Issues in pre-cooking of raw data are not excluded...

Jack Hogan Veteran Member • Posts: 6,982
Re: D7100 'streaking' - doesn't look like blooming or flare

krikman, you have suggested that the streaking we see could be due to signal processing before the data is written to the Raw file.  I agree that it could be one of the reasons and said so in a previous post.

Unfortunately in the process you have gone off and made a number of unsubstantiated claims that are, imo, misleading and highly unlikely at best.  Rather than waste keystrokes challenging all of them, let's concentrate on the grandest, this one (I have highlighted it in bold and underlined it):

krikman wrote:

Jack Hogan wrote:

But VERY IMPORTANT DISCOVERY:

Until ISO400 we presumable have clear raw data, but from ISO800 Nikon 'cooks' RAW data by different datapath with intensive onchip (?) NR. It is common sense but first time we had clear instrumental proving.

Can you provide details on your analysis and the data to support your claim?

You've already seen it on pictures.

I have seen no such thing.   In fact I have checked : the D7100's full SNR curves do not appear to show intensive (or any) noise reduction at ISO800 or 1600 in the Raw data.  Perhaps I will see differently once you share how you arrived at that conclusion and the data that drew you to it.  Without this additional information, though, yours remains an unfounded, far-fetched claim without 'clear instrumental proving'.

Cheers,
Jack

krikman Regular Member • Posts: 419
Re: D7100 'streaking' - doesn't look like blooming or flare

Jack Hogan wrote:

krikman, you have suggested that the streaking we see could be due to signal processing before the data is written to the Raw file.  I agree that it could be one of the reasons and said so in a previous post.

Yes. My claims was only assumption and must be verified by shooting same picture

- with ADL off 12 bit.

- with and without ADL in 14-bit

- with very first settings but camera rotated 180 degrees.

>Unfortunately in the process you have gone off and made a number of unsubstantiated claims >that are, imo, misleading and highly unlikely at best.  Rather than waste keystrokes challenging >all of them, let's concentrate on the grandest, this one (I have highlighted it in bold and >underlined it):

English isn't my native language so I may speak in unclear manner. Sorry.

krikman wrote:

Jack Hogan wrote:

But VERY IMPORTANT DISCOVERY:

Until ISO400 we presumable have clear raw data, but from ISO800 Nikon 'cooks' RAW data by different datapath with intensive onchip (?) NR. It is common sense but first time we had clear instrumental proving.

Can you provide details on your analysis and the data to support your claim?

You've already seen it on pictures.

We've already seen than data processing paths are different before and after ISO800. It is very solid evidence of in-camera NR. And Nikon agreed many times that you cannot off Noise reduction completely with hi-ISO or long exposures.

I have seen no such thing.   In fact I have checked : the D7100's full SNR curves do not appear to show intensive (or any) noise reduction at ISO800 or 1600 in the Raw data.

DXO measuring is

Perhaps I will see differently once you share how you arrived at that conclusion and the data that drew you to it.  Without this additional information, though, yours remains an unfounded, far-fetched claim without 'clear instrumental proving'.

Are you fooling me? We've CLEARLY seen data processing artefacts at ISO100-400 and NO SUCH ARTIFACT after ISO800. If there was photosites saturation of any kind there would be no such borderline at ISO800.

Cheers,
Jack

Again, If we won't see stripes with ADL OFF in 14-bit RAW, there will prove my claims entirely.

Sincerely,

Mark

AdamT
AdamT Forum Pro • Posts: 58,922
Re: "I love how it forces me to become a better photographer"

dougster1979 wrote:

So you don`t need good technique to use a 12mp camera

You can get away with less precise focussing and more shake for 100% Viewing . this is why you can wave a D50 around with a crappy Sigma superzoom on it with wanton abandon and get 100% Keepers every time even pixel peeped - a very sharp 6Mp (in RAW) isn`t very demanding of glass or holding technique and as long as the cam has a decent AF system (which the D50 had) , you`re there .

-- hide signature --

** Please ignore the Typos, I'm the world's worst Typist **

 AdamT's gear list:AdamT's gear list
Canon PowerShot G1 X Sony RX100 III Sony RX10 III Fujifilm X-T100
krikman Regular Member • Posts: 419
Re: D7100 'streaking' - doesn't look like blooming or flare
I have seen no such thing.   In fact I have checked : the D7100's full SNR curves do not appear to show intensive (or any) noise reduction at ISO800 or 1600 in the Raw data.
Perhaps I will see differently once you share how you arrived at that conclusion and the data
that drew you to it.  Without this additional information, though, yours remains an unfounded, far-fetched claim without 'clear instrumental proving'.

DXO measures based on solid ground, but often have wrong interpretation. NR at hi ISOs performed not by blurring and smashing data from different photosites, (as works NR in post processing). It is kind of careful restoration and prediction of photosite measurings. It don't averages nearest pixels lightness, it averages color values by large blocks. So we had great color responce but no color details.

And if you need additional proving it would be tests at ISO 720 (with stripes and ISO800 with no stripes).

Cheers,
Jack

mosswings Veteran Member • Posts: 9,131
Re: D7100 'streaking' - doesn't look like blooming or flare

krikman wrote:

I mean compressian of data to be 12-bit wide, which are image specific. No

in conversion to 12-bit, but coders forgot to cut unusable bits from precise calculations so we got rounding error in green channel.

. From these samples I conclude that these stripe artifacts are no lens defect, no photosites overloading, no sensor errors but round error in data calculations.

Jack

You, Jack, talk about RAW conversion software, but I talk about preprocesses before writing NEF.

An interesting proposition, and definitely true that the D7100 should not normally be operated in 12 bit mode (Mattilla had a D90 previously, which had noise levels high enough to obivate the need for 14 bit recording, so only offered 12 bit recording).  The test definitely needs to be conducted at 14 bits.

However, I'm having a hard time ascribing the geometry dependence of the tonality in the deep shadows to strictly truncation sloppiness in the low-level RAW codec.  There would have to be a real change in the sensel charged stored in order for a converson truncation to emphasize any such shifts. Consider this discussion on truncation vs. noise levels from University of Chicago:

http://theory.uchicago.edu/~ejm/pix/20d/tests/noise/noise-p3.html

The essential figure is #18.

Clearly delineated steps emerge when the noise level is less than the quantization level, but there has to be a change in the average signal level to generate a change in the recorded data.

Note also that the transition region tonalities are not constant on either side of the step - which is not what I see in the pushed images.

Something else appears to be going on here.  So I agree that the following need be done:

1. Run tests at 14 bit as well as 12 - reduces posterization issues

2. Run tests with the camera in landscape mode, portrait mode, and at 45 degrees to both - reveals directional sensitivities to blooming.

 mosswings's gear list:mosswings's gear list
Olympus XZ-1 Olympus Stylus 1 Nikon D90 Nikon D7100 Sony a6400 +7 more
krikman Regular Member • Posts: 419
Re: D7100 'streaking' - doesn't look like blooming or flare

mosswings wrote:

krikman wrote:

I mean compressian of data to be 12-bit wide, which are image specific. No

in conversion to 12-bit, but coders forgot to cut unusable bits from precise calculations so we got rounding error in green channel.

. From these samples I conclude that these stripe artifacts are no lens defect, no photosites overloading, no sensor errors but round error in data calculations.

Jack

You, Jack, talk about RAW conversion software, but I talk about preprocesses before writing NEF.

An interesting proposition, and definitely true that the D7100 should not normally be operated in 12 bit mode (Mattilla had a D90 previously, which had noise levels high enough to obivate the need for 14 bit recording, so only offered 12 bit recording).  The test definitely needs to be conducted at 14 bits.

However, I'm having a hard time ascribing the geometry dependence of the tonality in the deep shadows to strictly truncation sloppiness in the low-level RAW codec.  There would have to be a real change in the sensel charged stored in order for a converson truncation to emphasize any such shifts. Consider this discussion on truncation vs. noise levels from University of Chicago:

http://theory.uchicago.edu/~ejm/pix/20d/tests/noise/noise-p3.html

The essential figure is #18.

Clearly delineated steps emerge when the noise level is less than the quantization level, but there has to be a change in the average signal level to generate a change in the recorded data.

Note also that the transition region tonalities are not constant on either side of the step - which is not what I see in the pushed images.

Something else appears to be going on here.  So I agree that the following need be done:

1. Run tests at 14 bit as well as 12 - reduces posterization issues

2. Run tests with the camera in landscape mode, portrait mode, and at 45 degrees to both - reveals directional sensitivities to blooming.

You guys turned me on. I've just purchased this camera only to prove my statements. Is any photographers left here?

Wait for my own tests by the end of the week!

Horshack Veteran Member • Posts: 7,490
Another set @ 14-bit lossless compressed

My original D7100 shots above were 14-bit lossy. Here are converted raws from 14-bit lossless, this time at both ISO 100 and 400. I don't see it at ISO 400.

ISO 100, pushed +5EV in LR4

ISO 400, pushed +5EV in LR4

ISO 100 and 400 NEFs (link good for 5 days)

fotolopithecus Senior Member • Posts: 1,699
Re: Another set @ 14-bit lossless compressed

Horshack wrote:

My original D7100 shots above were 14-bit lossy. Here are converted raws from 14-bit lossless, this time at both ISO 100 and 400. I don't see it at ISO 400.

ISO 100, pushed +5EV in LR4

ISO 400, pushed +5EV in LR4

ISO 100 and 400 NEFs (link good for 5 days)

I think I can see it in both shots Adam, but only just in the second shot.

 fotolopithecus's gear list:fotolopithecus's gear list
Nikon D610 Tamron SP AF 90mm F/2.8 Di Macro Nikon AF-S Nikkor 50mm f/1.8G
Jack Hogan Veteran Member • Posts: 6,982
Re: D7100 'streaking' - doesn't look like blooming or flare

krikman wrote:

I have seen no such thing.   In fact I have checked : the D7100's full SNR curves do not appear to show intensive (or any) noise reduction at ISO800 or 1600 in the Raw data.
Perhaps I will see differently once you share how you arrived at that conclusion and the data
that drew you to it.  Without this additional information, though, yours remains an unfounded, far-fetched claim without 'clear instrumental proving'.

DXO measures based on solid ground, but often have wrong interpretation. NR at hi ISOs performed not by blurring and smashing data from different photosites, (as works NR in post processing). It is kind of careful restoration and prediction of photosite measurings. It don't averages nearest pixels lightness, it averages color values by large blocks. So we had great color responce but no color details.

krikman, you may be able to but the rest of us can't eyeball this sort of stuff - we have to measure it.  If you feel that there is some noise reduction of whatever type happening at ISO 800, show us the numbers.  This is what I see in the green Raw channels near the spot of my previous post:

ISO 100 Mean (dark) 11.9 Stddev 1.25
ISO 100 Mean (light)  12.7 Stddev 1.20

ISO 800 Mean          97.2 Stddev 9.65

If I take the ISO 800 Mean and divide it by the average of the two ISO100 means I get an amplification of 7.9, pretty darn close to the expected 8.  If I now multiply the average noise at ISO 100 (1.25) by 7.9 I get 9.68 - pretty darn close to the measured value of 9.65 at ISO 800.

Where is the noise reduction of which you speak?

And if you need additional proving it would be tests at ISO 720 (with stripes and ISO800 with no stripes).

Yet again perhaps streaking has nothing to do with noise.  There are many operations that could be, and probably are, happening under the hood.

krikman Regular Member • Posts: 419
Re: D7100 'streaking' - doesn't look like blooming or flare

mosswings wrote:

krikman wrote:

I mean compressian of data to be 12-bit wide, which are image specific. No

in conversion to 12-bit, but coders forgot to cut unusable bits from precise calculations so we got rounding error in green channel.

. From these samples I conclude that these stripe artifacts are no lens defect, no photosites overloading, no sensor errors but round error in data calculations.

Jack

You, Jack, talk about RAW conversion software, but I talk about preprocesses before writing NEF.

An interesting proposition, and definitely true that the D7100 should not normally be operated in 12 bit mode (Mattilla had a D90 previously, which had noise levels high enough to obivate the need for 14 bit recording, so only offered 12 bit recording).  The test definitely needs to be conducted at 14 bits.

My whole point was in-camera ADL work. I can strongly conclude from my D300s behaviour that in-camera ADL works selective. It not just compresses data range but lengthen exposition for darks and somehow restores lights. At levels that impossible for written RAW data. All this required some 'area recognition' for dark and light regions.

From this I supposed that programmers forgot to reset 'pushed down' condition for bright areas setted by in-camera ADL, ant it is for unused or crippled data bits only.

Anyway it is always hard to find where is somebody's algorhytm fails, especially at 0.1% levels.

However, I'm having a hard time ascribing the geometry dependence of the tonality in the deep shadows to strictly truncation sloppiness in the low-level RAW codec.  There would have to be a real change in the sensel charged stored in order for a converson truncation to emphasize any such shifts. Consider this discussion on truncation vs. noise levels from University of Chicago:

Yes, if no ADL 14-bit NEF shows same behaviour all my statements will discarded and we'll blame sensor.

Something else appears to be going on here.  So I agree that the following need be done:

1. Run tests at 14 bit as well as 12 - reduces posterization issues

2. Run tests with the camera in landscape mode, portrait mode, and at 45 degrees to both - reveals directional sensitivities to blooming.

It is clear that in landscape mode and inclined we'll have no stripes at all. But what happens in opposite direction? I presume data reading from left to right in our cameras...

mosswings Veteran Member • Posts: 9,131
Re: Another set @ 14-bit lossless compressed

THAT's a relief.  I don't shoot lower than ISO 200 (both by choice and necessity) and am much more interested in 800-1600.

 mosswings's gear list:mosswings's gear list
Olympus XZ-1 Olympus Stylus 1 Nikon D90 Nikon D7100 Sony a6400 +7 more
Jack Hogan Veteran Member • Posts: 6,982
Re: Another set @ 14-bit lossless compressed

fotolopithecus wrote:

Horshack wrote:

My original D7100 shots above were 14-bit lossy. Here are converted raws from 14-bit lossless, this time at both ISO 100 and 400. I don't see it at ISO 400.

I think I can see it in both shots Adam, but only just in the second shot.

Horshack, you changed exposure (1/320 vs 1/1250), that would make the ISO 400 image noisier, and the streaking could get lost in the dithering.  I can just barely see some horizontal banding in the ISO 400 image against the texture of the wall, but I am not sure it is aligned with the blinds.

Horshack Veteran Member • Posts: 7,490
Re: Another set @ 14-bit lossless compressed

Jack Hogan wrote:

fotolopithecus wrote:

Horshack wrote:

My original D7100 shots above were 14-bit lossy. Here are converted raws from 14-bit lossless, this time at both ISO 100 and 400. I don't see it at ISO 400.

I think I can see it in both shots Adam, but only just in the second shot.

Horshack, you changed exposure (1/320 vs 1/1250), that would make the ISO 400 image noisier, and the streaking could get lost in the dithering.  I can just barely see some horizontal banding in the ISO 400 image against the texture of the wall, but I am not sure it is aligned with the blinds.

I had to change the exposure with the ISO increase, otherwise it would have clipped more on the ISO 400 after the push. I wanted to compare equivalent post-ADC ADUs.

Horshack Veteran Member • Posts: 7,490
ISO 100 vs 200 equiv exposure

Jack Hogan wrote:

fotolopithecus wrote:

Horshack wrote:

My original D7100 shots above were 14-bit lossy. Here are converted raws from 14-bit lossless, this time at both ISO 100 and 400. I don't see it at ISO 400.

I think I can see it in both shots Adam, but only just in the second shot.

Horshack, you changed exposure (1/320 vs 1/1250), that would make the ISO 400 image noisier, and the streaking could get lost in the dithering.  I can just barely see some horizontal banding in the ISO 400 image against the texture of the wall, but I am not sure it is aligned with the blinds.

ISO 100 pushed 5 stops (left), ISO 200 pushed 4 stops (right)

Jack Hogan Veteran Member • Posts: 6,982
Re: Another set @ 14-bit lossless compressed

You know what I find interesting is that in your new ISO100 NEF, the streaking seems to be more apparent next to the bottom half of the window, where there is less light coming through.  Perhaps the streaking is not a consequence of the bright areas, but of the dark ones.

Keyboard shortcuts:
FForum MMy threads