Highlight clipping, a case in point

[GordonBGood]

You have to note that the sony has 12 more ev of DR that can be exploited

As I pointed out in my earlier post above, there is not actually any difference between K-x and D5000. The difference shown by DPR come from its own test method but there are other tests (and plenty of threads here) that show that as far asK-x goes the DPR test is misleading.

Gerry, as I said before, I don't think that this OP receives instructions, is convinced that the K-x has a limited raw DR completely due to a slightly less "highlight headroom" and less shadow Dynamic Range (DR) because DPReview says so , and as proof has provided a single sample that is very likely flawed as to the reasons there is a clipped area and for which we have very few details about its provenance as to shooting conditions other than that it was likely shot in raw and developed with Silkypix (from the EXIF) with an 11-17 fisheye lens at 11 mm. and with the sun an something near 90 degrees to the angle of view.

Wouldn't you expect some lens flare in such a situation?

I think we are wasting our time other than for our own edification.

Best regards and top of the season to you, GordonBGood

 

[Gerry Winterbourne]

You have to note that the sony has 12 more ev of DR that can be exploited

As I pointed out in my earlier post above, there is not actually any difference between K-x and D5000. The difference shown by DPR come from its own test method but there are other tests (and plenty of threads here) that show that as far asK-x goes the DPR test is misleading.

Gerry, as I said before, I don't think that this OP receives instructions, is convinced that the K-x has a limited raw DR completely due to a slightly less "highlight headroom" and less shadow Dynamic Range (DR) because DPReview says so

Gordon - I fear you are right about the OP but I have a character trait (sometimes it shows as a flaw, sometimes as a virtue) that makes me want to help people understand things better.

... and as proof has provided a single sample that is very likely flawed as to the reasons there is a clipped area and for which we have very few details about its provenance as to shooting conditions other than that it was likely shot in raw and developed with Silkypix (from the EXIF) with an 11-17 fisheye lens at 11 mm. and with the sun an something near 90 degrees to the angle of view.

Wouldn't you expect some lens flare in such a situation?

Indeed, and there is no doubt that flare or some kind of reflection has contributed to the problem with this specific image. However, bright light of any sort would test the DR of any system. The OP was aware of this in principle but tried to blame the problem on K-x having only a limited DR. In this he was partly justified by the bizarre result shown by DPR. I concentrated on the DR aspect of his post because earlier respondents had already alluded to flare (with, as it seems, about as little success).

I think we are wasting our time other than for our own edification.

Being an optimist I hope that even if the OP gains nothing there will be others who read the thread and pick up some knowledge. Certainly I have learned a lot from this forum and I hope I can put something back. You don't need to hope - it is clear that many people rely on you for knowledge ...

Best regards and top of the season to you, GordonBGood

... and on that happy note I'll return my best wishes to you for some rewarding festivities

--
Gerry


First camera 1953, first Pentax 1983, first DSLR 2006
http://www.pbase.com/gerrywinterbourne

 

[YunZY655186]

I have to say that I really appreciate your discussions, actually I may explain more about the image. The fact is I personally haven't seen any trace of flare on ths photo. The essential post processing steps with the raw image are: dynamic range expansion and blue sky emphasis.

The dynamic range expansion is effective in pulling back the near saturated pixels. Then plus the blue sky emphasis I saw this bizzare pattern. Hope the swift change is normal, looking forword more experience in highlight raw processing related with k-x.

Merry Christmas to everyone!

 

[GordonBGood]

You have to note that the sony has 12 more ev of DR that can be exploited

As I pointed out in my earlier post above, there is not actually any difference between K-x and D5000. The difference shown by DPR come from its own test method but there are other tests (and plenty of threads here) that show that as far asK-x goes the DPR test is misleading.

Gerry, as I said before, I don't think that this OP receives instructions, is convinced that the K-x has a limited raw DR completely due to a slightly less "highlight headroom" and less shadow Dynamic Range (DR) because DPReview says so

Gordon - I fear you are right about the OP but I have a character trait (sometimes it shows as a flaw, sometimes as a virtue) that makes me want to help people understand things better.

As you perhaps know from my previous posting history, I mostly have the same virtue/flaw.

I think we are wasting our time other than for our own edification.

Being an optimist I hope that even if the OP gains nothing there will be others who read the thread and pick up some knowledge. Certainly I have learned a lot from this forum and I hope I can put something back. You don't need to hope - it is clear that many people rely on you for knowledge ...

Which is also why I persevere also in many of these discussions even in the face of many who doubt what I try to explain: for others who may gain from the discourse.

Best regards and top of the season to you, GordonBGood

... and on that happy note I'll return my best wishes to you for some rewarding festivities

Oh, the festivities are pretty much over here as far as Xmas day goes, not that Xmas is such a major deal in a Thailand anyway.

Best regards, GordonBGood

 

[Gerry Winterbourne]

I have to say that I really appreciate your discussions

Good. I hope they are helpful.

actually I may explain more about the image. The fact is I personally haven't seen any trace of flare on ths photo.

Fair enough. Flare shows irself in several ways: one of them is the sort of fuzzy transition from very bright to bright in the top right of this picture. That doesn't mean that it definitely is flare but explains why some people have queried it.

The essential post processing steps with the raw image are: dynamic range expansion and blue sky emphasis.

I don't really understand what you mean by this. The RAW file already has the full DR of the shot so you can't expand it. If some parts of the image are clipped to white (that is, they are overexposed) then you have irretrievably lost part of the scene's DR. In this case you can stretch the histogram out to the right-hand end: is this what you mean by DR expansion? If it is it is the source of some of your problem.

The dynamic range expansion is effective in pulling back the near saturated pixels. Then plus the blue sky emphasis I saw this bizzare pattern. Hope the swift change is normal

In this situation it is. A digital image is, by definition, filed as a set of numbers. What this means is that rather than a continuous spectrum of colour shades there are discrete steps. An 8-bit file like JPG (which is what your screen or printer display regardless of the format you shoot in) has 2^8 = 256 colour steps. A 12-bit file (most RAW files) has 2^12 = 4096 steps. In the RAW conversion, whether in camera or in PP, the many steps of the RAW file are mapped to the fewer steps in the JPG file.

A bit of simple theory: raising the powers of 2 is just doubling; a photographic stop is also a doubling: so each stop is represented by 1 bit of data. this has an apparently odd effect - the top stop of exposure has half the steps I mentioned. This then halves as you go down.

So in JPG the top stop has 128 steps, the next brightest 64 and so on; in RAW it is 2046, 1024 etc. In practice files viewed like this would be very odd and the converters adjust the data (the "tone curve" I mentioned in an earlier post). Normally the top stop is something like 1000 in RAW and 50 in JPG, so the converter has to map 1000 steps into 50 - a 20-fold squeeze.

This looks like a lot but our eyes can cope with the effect quite easily. But now think of the effect of stretching the histogram. What you are doing, in effect, is throwing away some of your RAW file's 4000 steps. Remember that half of the data are in the top stop - moving the end of the histogram out by 1 stop throws away a full half of the available steps so you are down to 2000. The histogram always shows as JPG on your viewing device so 1 stop is 1/8th of what you see. With a badly clipped file you can lose even more.

This means that instead of 20 steps to map to 1 JPG step you only have 10. Next consider your blues: emphasising them means, in effect, reducing their exposure (think about it - if you want a whole picture to look darker you expose it less). This also entails reducing the available steps - darkening the blues by 1 stop will take the already reduced number by another half. So now we are trying to map only 5 RAW steps on to 1 JPG.

What this means in practice is that local variations of colour density start to show very obvious changes. The same thing happens if you shoot JPG except that the effect (it's caled posterzation) is very much more obvious. Indeed, this is one of the big reasons to shoot RAW. But however you shoot it will show in extreme cases and your shot is one of them.

looking forword more experience in highlight raw processing related with k-x.

Experience in PP is always useful but the real thing to learn here is exposure: no space here to explain but do a search on this forum for ETTR (expose to the right). You'll find some people query it but the theory is perfectly correct.

Merry Christmas to everyone

And to you.

--
Gerry


First camera 1953, first Pentax 1983, first DSLR 2006
http://www.pbase.com/gerrywinterbourne

 

[GordonBGood]

A bit of simple theory: raising the powers of 2 is just doubling; a photographic stop is also a doubling: so each stop is represented by 1 bit of data. this has an apparently odd effect - the top stop of exposure has half the steps I mentioned. This then halves as you go down.

So in JPG the top stop has 128 steps, the next brightest 64 and so on; in RAW it is 2046, 1024 etc. In practice files viewed like this would be very odd and the converters adjust the data (the "tone curve" I mentioned in an earlier post). Normally the top stop is something like 1000 in RAW and 50 in JPG, so the converter has to map 1000 steps into 50 - a 20-fold squeeze.

Whoops, Gerry, you forgot (again) that raw is linear whereas JPEG is not and already has a "gamma" Tone Response Curve (TRC) applied such that the top stop only occupies about 69 levels (gamma 2.2, which is close to the sRGB TRC), else JPEG could only contain 8 stops of Dynamic Range when it can actually represent over 10 to 12, with quite poor definition in the last few stops. If JPEG were linear, 18% gray would be at level 46 rather than about level 117 where we actually find it.

The relationship between numerical gray levels and actual luminance levels for JPEG is away too often misstated in documents; let's not add to that here.

Regards, GordonBGood

 

[Richard Murdey]

It's not Penax's problem unless the in-camera JPEG has the same issue. Getting stuff like that in ACR just means the highlight compensation (applied in ACR) was maxed out. It did it's best up until the point where the image suddenly goes white, that's the limit where the 12 bit RAW data maxes out. There's not more headroom to recover highlights above that.

For the record you can do exactly the same thing with DCU.

The in camera JPEG would have probably just clipped he entire sky to white, but it would have been relatively graceful. if you had used extended dynamic range chances are it would have managed to capture a complete blue sky but the foreground would have been quite dark. (The metering changes when the setting is applied in order to keep highlight detail)

The K-x tends to max out it's JPEGS, in the sense that the in camera processing uses as much of the RAW data as possible. There is less highlight headroom than some other cameras, perhaps.

But returning to your photo: frankly, it sucks. You should have known better than to take it against such a bright sky! Hardware can only do so much to correct for poor technique...

 

[Model Mike]

The dynamic range expansion is effective in pulling back the near saturated pixels.

Bingo! Your 'dynamic range expansion' tool sounds just like Highlight Recovery in Lightroom/ACR. And you've been too enthusiastic in its use!

Highlight Recovery shifts the RHS of the histogram to the left. If you overdo it, your blown highlights get shifted too far, and simply end up as a grey spike. This is what's happening in your image: the 'blown' highlights in your image have become a 235 grey, not pure white. The image also exhibits the tell-tale narrow boundary of recovered colour.

IMO this is a far more likely explanation than flair. On closer examination, the image doesn't display any real flair artifacts.

So: your image is overexposed and poorly PP'd - sorry . I think you're asking too much of any camera to meter a scene perfectly for a scene like this. You don't need to look at DR figures, just use your eyes, the histogram, and the EC adjustment.
--
Mike
http://flickr.com/rc-soar

 

[Model Mike]

Highlight Recovery shifts the RHS of the histogram to the left. If you overdo it, your blown highlights get shifted too far, and simply end up as a grey spike.

Whoops - correction. I just double checked this in Lightroom in order to recreate the o/p's problem using a blown image of my own (why didn't I do this before posting!). Actually Highlight Recovery doesn't turn blown areas uniformly grey, so I couldn't reproduce the exact effect at the top right of his image. LR/ACR does however produce harsh transitions where colour info is only partially recovered.

This leaves a couple of possibilities: Perhaps flair is a factor as originally thought (but why is there an absence of flair artifacts?). Or else it could be that the 'dynamic range expansion' tool in the o/p's editor works more crudely than Recovery in Lightroom/ACR.
--
Mike
http://flickr.com/rc-soar

 

[Jarda_Houdek]

Don't be sad, this image is ugly anyway. No sensor would save it from that.
--

Through a Pentax Limited prime things may appear sharper than they actually are...
http://www.pentaxphotogallery.com/jaroslavhoudek

 

[Gerry Winterbourne]

A bit of simple theory: raising the powers of 2 is just doubling; a photographic stop is also a doubling: so each stop is represented by 1 bit of data. this has an apparently odd effect - the top stop of exposure has half the steps I mentioned. This then halves as you go down.

So in JPG the top stop has 128 steps, the next brightest 64 and so on; in RAW it is 2046, 1024 etc. In practice files viewed like this would be very odd and the converters adjust the data (the "tone curve" I mentioned in an earlier post). Normally the top stop is something like 1000 in RAW and 50 in JPG, so the converter has to map 1000 steps into 50 - a 20-fold squeeze.

Whoops, Gerry, you forgot (again) that raw is linear

Well, not quite, Gordon (I hope). This time I think I'm guilty of oversimplifying the explanation rather than forgetting it.

whereas JPEG is not and already has a "gamma" Tone Response Curve (TRC) applied such that the top stop only occupies about 69 levels (gamma 2.2, which is close to the sRGB TRC), else JPEG could only contain 8 stops of Dynamic Range when it can actually represent over 10 to 12, with quite poor definition in the last few stops. If JPEG were linear, 18% gray would be at level 46 rather than about level 117 where we actually find it.

What I was trying to convey was what you describe ( my "... something like ... 50 in JPG ..." was a rounding of your 69 to simplify the maths (coupled with the fact that I couldn't remember the exact figure anyway). Then the idea that the RAW converter has to map into the available levels: but I wanted to avoid "gamma". What I think would have been correct is something like:-

"The RAW data itself is linear; if our display devices were 12-bit the linear data would nominally need mapping so that the brightest stop's information would use the top half (2048) of the available luminance steps but this looks wrong because our eyes don't respond like that; to look natural the data are adjusted by a curve (whose power, called "gamma", is 2.2) which means the brightest stop only gets a bit over 1000 steps. As it happens our display devices are 8-bit devices so the "RAW" levels have to be mapped to a smaller number of steps: about 50 [about =69 for my simple maths] ..." and then continue as I originaly wrote.

My unarticulated thinking was that this would be confusing to a newb; it actually doesn't seem so bad now. So is it sufficiently close to the truth (accepting the desire to be simple) bearing in mind my next para?

There is, of course, the question of how an 8-bit device can ever show an "original" DR of over 8 stops. The answer, of course, is that the RAW data are "squeezed" so that (to adopt another arihtmetical simplification without any power functions at all) if we pretend that a 12-stop DR is represented by 3840 luminance levels (320 levels/stop) then those 3840 levels are divided by 480 with each division mapped to 1 stop of JPG. I think this concept is a step too far for the concerns of the OP.

The relationship between numerical gray levels and actual luminance levels for JPEG is away too often misstated in documents; let's not add to that here.

Indeed. My main reason for discussing only the brightest stop was because of the OP's problem, but it did also keep me away from grey.

Regards, GordonBGood

--
Gerry


First camera 1953, first Pentax 1983, first DSLR 2006
http://www.pbase.com/gerrywinterbourne

 

[ozdean]

IMO this is simply bad photographic technique I wouldn't shoot that shot with film.
--
Regards Dean - Capturing Creation

 

[juleha]

Don't be sad, this image is ugly anyway. No sensor would save it from that.
--

IMO this is simply bad photographic technique I wouldn't shoot that shot with film.
--

agree

 

[borno]

lens flare, delete it, try again, that's about all I have to say about that.