Gordon, did you see this (in the other forum)?

Started May 3, 2013 | Discussions
Kerusker
Contributing MemberPosts: 757Gear list
Like?
Gordon, did you see this (in the other forum)?
May 3, 2013

ewelot wrote:

.....I found - to my big surprise - that the black point issue didn't show up and dark frames exhibit a nice gaussian intensity distribution! It appears that accessing the in-memory raw image data of the camera via PC control software offers data at a less processed state and - most importantly - the image pixel mean intensity is not shifted to zero level. ......

This should be the link:

wwwDOTpentaxforumsDOTcom/forums/pentax-k-5/217407-blackpoint-issue-k5iis-raw-not-raw-3.html#post2366621

Gordon, I would like to know what you draw from the thread over there.

-- hide signature --

][.Kerusker
we don't see that we don't see (eye's blind spot)

 Kerusker's gear list:Kerusker's gear list
Pentax K-5 II Pentax K-5 Pentax smc DA 15mm F4 ED AL Limited Pentax smc DA* 16-50mm F2.8 ED AL (IF) SDM Pentax smc DA 55-300mm F4.0-5.8 ED
GordonBGood
Veteran MemberPosts: 6,276
Like?
Re: Gordon, did you see this (in the other forum)?
In reply to Kerusker, May 3, 2013

Kerusker wrote:

ewelot wrote:

.....I found - to my big surprise - that the black point issue didn't show up and dark frames exhibit a nice gaussian intensity distribution! It appears that accessing the in-memory raw image data of the camera via PC control software offers data at a less processed state and - most importantly - the image pixel mean intensity is not shifted to zero level. ......

This should be the link:

wwwDOTpentaxforumsDOTcom/forums/pentax-k-5/217407-blackpoint-issue-k5iis-raw-not-raw-3.html#post2366621

Gordon, I would like to know what you draw from the thread over there.

From my K-5, it seems to me that the black point is uncompensated up to ISO 1600, but it is a while since I have checked it and it may have changed with the latest firmware updates.  All of these cameras will definitely have a zeroed black point for ISO's of 3200 and higher where the ISO's are achieved by shifting of underexposed ISO 1600 captures, and I assume that is why the ISO 1600 and higher sensitivities are black point zero compensated:  to prepare for the shifting operations.

I didn't know that the K-5 II(s) would be any different and don't have access to one, but I guess I can check from the raw images posted on Imaging-Resource.

When I get some time in the next day or so, I will do a quick check of K-5 II(s) raw images and report back here.

Interesting that the PKTether type programs would be able to capture the raw data at a different stage in the processing pipeline.

Regards, GordonBGood

Reply   Reply with quote   Complain
GordonBGood
Veteran MemberPosts: 6,276
Like?
Re: Gordon, did you see this (in the other forum)?
In reply to GordonBGood, May 4, 2013

GordonBGood wrote:

Kerusker wrote:

ewelot wrote:

.....I found - to my big surprise - that the black point issue didn't show up and dark frames exhibit a nice gaussian intensity distribution! It appears that accessing the in-memory raw image data of the camera via PC control software offers data at a less processed state and - most importantly - the image pixel mean intensity is not shifted to zero level. ......

This should be the link:

wwwDOTpentaxforumsDOTcom/forums/pentax-k-5/217407-blackpoint-issue-k5iis-raw-not-raw-3.html#post2366621

Gordon, I would like to know what you draw from the thread over there.

From my K-5, it seems to me that the black point is uncompensated up to ISO 1600, but it is a while since I have checked it and it may have changed with the latest firmware updates.  All of these cameras will definitely have a zeroed black point for ISO's of 3200 and higher where the ISO's are achieved by shifting of underexposed ISO 1600 captures, and I assume that is why the ISO 1600 and higher sensitivities are black point zero compensated:  to prepare for the shifting operations.

I didn't know that the K-5 II(s) would be any different and don't have access to one, but I guess I can check from the raw images posted on Imaging-Resource.

When I get some time in the next day or so, I will do a quick check of K-5 II(s) raw images and report back here.

Interesting that the PKTether type programs would be able to capture the raw data at a different stage in the processing pipeline.

I took some time to download ISO 800 and ISO 1600 raw DNG K-5 II images from Imaging-Resource (IR) at K-5 II ISO 800 Raw  and K-5 II ISO 1600 Raw and checked the black offsets.  For these images I found they were at firmware version 1.00 and had an offset at ISO 800 of about 512 (out of a full scale of 16384) and at ISO 1600 image had a black offset of zero, just as for my K-5 (other than that the offset for low ISO's is at about level 64 out of 16384) for the reasons as explained in my last post above.  As the K-5 IIs is identical to the K-5 II except for no Anti-Aliasing filter, I didn't bother burdening IR's bandwidth in checking those images.

Now, it is an interesting question whether newer firmware versions have changed this black offset compensation behaviour, but I see no reason why they would.  If you would like to check this for yourself, you can use the Beta version of PhotoMe to see the Black Level (for each of the RGGB quad of channels) in the Additional Image Data (1) tab or just filter for Black Level or Black Point (same information contained in the Manufacturer notes section of the metadata).

So the interesting questions are"At what ISO level was this black point issue identified on the other forum experienced for normal raw captures?" and "Is the PKTether capture really any different given the ISO sensitivity constraints?"

I don't really think that the main CPU of the K-5's has access to the raw data pipeline as the black level offset compensation is likely more a function of the PRIME2 hardware imaging engine, but it is possible that black level determination and application are done using separate passes of the PRIME2 engine and that the uncompensated raw data may be available for access by the PKTether hook even for higher ISO sensitivities, especially if it is able to hook into some debug facilities for the camera(s).

For my use with the K-5, I haven't dabbled with PKTether or its variants, and find that if I want to have uncompensated black levels I just stick to ISO sensitivities below 1600, underexposing and boosting the raw brightness as necessary.

Regards, GordonBGood

Reply   Reply with quote   Complain
GordonBGood
Veteran MemberPosts: 6,276
Like?
Re: Gordon, did you see this (in the other forum)?
In reply to GordonBGood, May 4, 2013

GordonBGood wrote:

GordonBGood wrote:

Kerusker wrote:

ewelot wrote:

.....I found - to my big surprise - that the black point issue didn't show up and dark frames exhibit a nice gaussian intensity distribution! It appears that accessing the in-memory raw image data of the camera via PC control software offers data at a less processed state and - most importantly - the image pixel mean intensity is not shifted to zero level. ......

This should be the link:

wwwDOTpentaxforumsDOTcom/forums/pentax-k-5/217407-blackpoint-issue-k5iis-raw-not-raw-3.html#post2366621

Gordon, I would like to know what you draw from the thread over there.

From my K-5, it seems to me that the black point is uncompensated up to ISO 1600, but it is a while since I have checked it and it may have changed with the latest firmware updates.  All of these cameras will definitely have a zeroed black point for ISO's of 3200 and higher where the ISO's are achieved by shifting of underexposed ISO 1600 captures, and I assume that is why the ISO 1600 and higher sensitivities are black point zero compensated:  to prepare for the shifting operations.

I didn't know that the K-5 II(s) would be any different and don't have access to one, but I guess I can check from the raw images posted on Imaging-Resource.

When I get some time in the next day or so, I will do a quick check of K-5 II(s) raw images and report back here.

Interesting that the PKTether type programs would be able to capture the raw data at a different stage in the processing pipeline.

I took some time to download ISO 800 and ISO 1600 raw DNG K-5 II images from Imaging-Resource (IR) at K-5 II ISO 800 Raw  and K-5 II ISO 1600 Raw and checked the black offsets.  For these images I found they were at firmware version 1.00 and had an offset at ISO 800 of about 512 (out of a full scale of 16384) and at ISO 1600 image had a black offset of zero, just as for my K-5 (other than that the offset for low ISO's is at about level 64 out of 16384) for the reasons as explained in my last post above.  As the K-5 IIs is identical to the K-5 II except for no Anti-Aliasing filter, I didn't bother burdening IR's bandwidth in checking those images.

Now, it is an interesting question whether newer firmware versions have changed this black offset compensation behaviour, but I see no reason why they would.  If you would like to check this for yourself, you can use the Beta version of PhotoMe to see the Black Level (for each of the RGGB quad of channels) in the Additional Image Data (1) tab or just filter for Black Level or Black Point (same information contained in the Manufacturer notes section of the metadata).

So the interesting questions are"At what ISO level was this black point issue identified on the other forum experienced for normal raw captures?" and "Is the PKTether capture really any different given the ISO sensitivity constraints?"

I don't really think that the main CPU of the K-5's has access to the raw data pipeline as the black level offset compensation is likely more a function of the PRIME2 hardware imaging engine, but it is possible that black level determination and application are done using separate passes of the PRIME2 engine and that the uncompensated raw data may be available for access by the PKTether hook even for higher ISO sensitivities, especially if it is able to hook into some debug facilities for the camera(s).

For my use with the K-5, I haven't dabbled with PKTether or its variants, and find that if I want to have uncompensated black levels I just stick to ISO sensitivities below 1600, underexposing and boosting the raw brightness as necessary.

Regards, GordonBGood

Too late to edit:  It would be interesting if these tethering applications can obtain uncompensated black levels from the K-01 and K-30, which normally only output zero compensated black levels for all ISO levels - GBG.

Reply   Reply with quote   Complain
awaldram
Forum ProPosts: 10,801Gear list
Like?
Re: Gordon, did you see this (in the other forum)?
In reply to GordonBGood, May 4, 2013

I've known about this for some time and assumed its why there is no official tether app.I think it because the app reads from cached memory not the SD so is prior to any pp

 awaldram's gear list:awaldram's gear list
Pentax Q Pentax K-3 Pentax smc DA* 55mm F1.4 SDM Pentax smc DA* 16-50mm F2.8 ED AL (IF) SDM Pentax smc DA* 50-135mm F2.8 ED (IF) SDM +11 more
Reply   Reply with quote   Complain
Kerusker
Contributing MemberPosts: 757Gear list
Like?
settings for tests?
In reply to GordonBGood, May 4, 2013

Gordon,

thanks for your investigations and comments. I'll get back to it later on.

This afternoon I may have the opportunity to shoot a K-5 II or K-5 IIs.

Do I have to observe certain settings regarding dark frame subtraction or dynamic range expansion etc.?

-- hide signature --

][.Kerusker
we don't see that we don't see (eye's blind spot)

 Kerusker's gear list:Kerusker's gear list
Pentax K-5 II Pentax K-5 Pentax smc DA 15mm F4 ED AL Limited Pentax smc DA* 16-50mm F2.8 ED AL (IF) SDM Pentax smc DA 55-300mm F4.0-5.8 ED
Reply   Reply with quote   Complain
beholder3
Regular MemberPosts: 287
Like?
Re: Gordon, did you see this (in the other forum)?
In reply to Kerusker, May 4, 2013

For those interested into these types of topics I recommend this long essay on how Canon manipulates raw data before the pre-cooked stuff is written into CR2 files:

http://www.stark-labs.com/craig/resources/Articles-&-Reviews/CanonLinearity.pdf

"Canon is rescaling your data before it hits the CR2 file. Based on the thermal signal both shifting your histogram left (likely based on stats from the optical black portion), it is changing and scaling the intensity (i.e., subtracting a constant from the whole image) the contrast or gain."

Thsi is not about manufacturers. They all do it.

Here Nikon:

http://forum.photozone.de/index.php?/topic/162-how-does-nikon-do-it/

Here Leica:

http://www.dxomark.com/index.php/Publications/DxOMark-Reviews/DxOMark-review-for-the-Leica-M9

Reply   Reply with quote   Complain
GordonBGood
Veteran MemberPosts: 6,276
Like?
Re: settings for tests?
In reply to Kerusker, May 4, 2013

Kerusker wrote:

Gordon,

thanks for your investigations and comments. I'll get back to it later on.

This afternoon I may have the opportunity to shoot a K-5 II or K-5 IIs.

Do I have to observe certain settings regarding dark frame subtraction or dynamic range expansion etc.?

Kerusker, Dark Frame Subtraction (DFS) when it is used either by forcing or automatically might make a difference as in the subtraction of a black "lens cap" image from a normal image might result in zero black levels without any compensaton.  Highlight Compensation shouldn't make any difference as it is still using a basic capture other than for metering the exposure as if the ISO was set one stop higher; there is a slight question that the point at which the switch over to dark frame subtraction might start at ISO 3200 rather than 1600 as is usual.

Regards, GordonBGood

Reply   Reply with quote   Complain
GordonBGood
Veteran MemberPosts: 6,276
Like?
Re: Gordon, did you see this (in the other forum)?
In reply to beholder3, May 4, 2013

beholder3 wrote:

For those interested into these types of topics I recommend this long essay on how Canon manipulates raw data before the pre-cooked stuff is written into CR2 files:

http://www.stark-labs.com/craig/resources/Articles-&-Reviews/CanonLinearity.pdf

"Canon is rescaling your data before it hits the CR2 file. Based on the thermal signal both shifting your histogram left (likely based on stats from the optical black portion), it is changing and scaling the intensity (i.e., subtracting a constant from the whole image) the contrast or gain."

Thsi is not about manufacturers. They all do it.

Here Nikon:

http://forum.photozone.de/index.php?/topic/162-how-does-nikon-do-it/

Here Leica:

http://www.dxomark.com/index.php/Publications/DxOMark-Reviews/DxOMark-review-for-the-Leica-M9

This thread is not about lamenting any general raw manipulations that Pentax does for its raw output formats in its DSLR's such as compulsory Noise Reduction (NR) applied to the raw data at higher ISO sensitivities but rather the black level zero compensation that is a necessary part before raw conversion to view-able images, but for which it is preferred that full black level compensation be applied later rather than having black levels embedded in the recorded raw data as averaging to zero.  The K-5 series is the only model that has kept a positive black offset.

Yes, Canon save their CR2 raw files with a positive offset, which may be variable and which image signal may have variable gain tweaks applied to them as the first article to which you link states, but we Pentax users don'e care:  The Pentax K-5(s) models keep a positive black offset as we would prefer even though is offset may be variable for different ISO sensitivities and does not apply gain tweaks; the only problem we have with this is that the positive black level offset is not retained for higher ISO's (probably ISO 1600 and above) nor likely for raw images captured with Dark Frame Subtraction (DFS).

Yes, Nikon do even further manipulation for their NEF raw files made necessary my their option to save as virtually loss-less compressed files and also possibly some sort of sneaky NR to the dark tones, but having an available positive may be one of the reasons we prefer the K-5(s) models to their equivalent models from Nikon.

Yes, DxOMark has proven that Leica is using  a very slight compulsory raw NR for many of its ISO sensitivities, but why would we Pentax users care (or Leica users either for that matter considering it is so slight.  This is comparable to the compulsory raw NR applied by Pentax for the K-5(s) for ISO 3200 and above, and I don't even really care about that as it isn't really any more than one would likely apply anyway in post processing, and as it is applied to the raw data rather than to demosiaced raw data, it has less side effects on the resulting images.

In summary, yes, we know that raw data as saved into raw formats is rarely exactly as read from the sensor, but with the K-5 we have some modes where this data is very close to being "pure"; also, it seems that some tethering software may be capable of extracting the raw data at an earlier state in the processing pipeline where it is even more pure, at least as regards the black level offset.

Regards, GordonBGood

Reply   Reply with quote   Complain
Keyboard shortcuts:
FForum MMy threads