Raw recording FACTS - debunking some myths

Following should clear up misunderstandings re the advantages of recording the image data in raw format.

The very first thing I have to say, is that I am recording exclusively in raw format. I am saying this, because one could think that I am just arguing against recording in raw. That's not the case; I am arguing against the myths , which will be spread here. (The first time I turned on JPEG additionally to raw in my camera was for the sake of the following tests)in the past months

Anyway.

The most important point to understand is, that the issue is not raw vs JPEG but raw vs converted format (i.e. raw vs not raw). Converting the data into JPEG in the camera involves the interpretation of the raw data, and that's the main issue, not the conversion in JPEG format. There are certain kinds of loss of image detail, for which the JPEG conversion is repsonsible. These losses can be avoided by recording the data in TIF format (if the camera offers this option).

However, there are other issues, for which NOT the JPEG conversion is responsible, but principally the processing of the raw data, most notably the application of the settings for white balance, saturation and contrast.

These problems (caused by the raw processing) have nothing to with the format the image data finally recorded: the same problems exist when converting in TIF, BMP, PSD or whatever format (except when converting in DNG format, which is a raw format as well, but independent of the camera maker).

Applying the same settings (WB, saturation, contrast), which would be applied during the in-camera conversion causes the same effect (loss of some details) in the in-computer raw conversion (IF the in-camera conversion results in such losses).

I post the facts in the first "answer" to this post, so that people don't need to download the images if they "accidentally" come to this thread.

If someone does not recognize, that there is ZERO stop to gain by recording in raw per se , then we can speculate if the problem is with my communicating skill or with his comprehension.

One could justifiedly ask what the source of the myth about "gaining x stops in raw compared to JPEG" is. I can only speculate, for I have not spread it, but I think the misconception comes from the fact, that the in-camera histogram of the 20D (and probably of the 10D) is totally unreliable, for it is based only on the green-filtered pixels. If someone adjusts the exposure according to the history and re-shoots the image, the result can easily blow the highlights of red and blue. However, this has nothing to do with recording in raw or not.

Now, someone could (in fact should) ask: "then why record raw instead of JPEG"?

That will be (perhaps) a separate post.

--
Gabor

http://www.panopeeper.com/panorama/pano.htm

Notes:

• I made custom white balance setting sheet before starting.

• I recorded raw + the highest quality JPEG.

• I shot a serie with standard custom settings, then one serie with contrast +2, one with saturation +2 and one with contrast +2 AND saturation +2.

• I shot a serie with -2 EV, -4/3 EV, -2/3 EV, 0EV, +2/3 EV, +4/3 EV, +2 EV; however, the result relating the comparison between conversions does not change with under- or overexposure

• I shot a serie in tungsten halogen light, that's what I am posting. I shot another serie in white fluorescence; I didn't prepare the same comparisons, for these ones took several hours work, and my findings are the same anyway.

• I have used Digital Photo Professional and Canon EOS Viewer in the conversions. I am working normally with RSE, but that does not honour the in-camera settings, except the WB.

• The RGB histogram DPP shows from the raw file is not really identical to the one, what DPP creates from the raw in the raw-> JPEG conversion. Canon sux in software (like Nikon); Japanese will never learn how to program.

• I used following composition (the flower is silk, don't except the red of real roses):

Now, to the facts:

1. The raw data is identical in the shots, independently of the settings (perhaps this is not natural for everyone):



2. although the application of white balance influences the image a lot (it can cause clipping of highlights), but that is the nature of the beast. If that setting will be applied, the effect is the same with the in-camera raw conversion (to JPEG) and in the in-computer conversion . In this example, the application of white balance "pushes the histogram" to the right (without causing clipping), but that is equal in the in-camera and in-compuer processing . The following comparison shows the histograms of the files with standard contrast and saturation setting (The small histogram underneath is that from the raw file; this is the truth in the raw data ):

See



2. The effect of saturation +2 is (in this image), that the histogram will be "pushed to the right", causing clipping at the right; the effect of contrast +2 is, it will be "stretched", i.e. the right side "goes out of the space" (clipping), and the left side is doing the same (losing shades).

The following comparisons show the histogram without adjustment, with saturation +2, with contrast +2 and with saturation +2 AND contrast +2.

First the in-camera generated JPEGs:



then the DPP-generated JPEGs:



and the EOS Viewer generated JPEGs:



There is no relevant difference, no matter if the JPEG was created in the camera of afterwards from raw data

I put the histograms together in another grouping for easier comparison; I post only those with contrast +2 saturation +2; the first one is from the raw data (but RGB), the senond one is from the camera-generated JPEG, the third one is from the DPP generated JPEg and the last one is generated by EOS Viewer:



--
Gabor

http://www.panopeeper.com/panorama/pano.htm

That is exactly the point I try to make to people when they ask why I shoot in "RAW Only" instead of "RAW + JPG".

Since I can later batch process the files in DPP or EVU and get identical-looking JPGs, I lose absolutely nothing by shooting in "RAW Only" and I save some room on the memory cards.

Further, since I'm shooting in "RAW Only", and have no reason to worry about the camera's parameter settings since they won't affect the RAWs anyway, I am free to set the contrast to -2 which gives me a much more relavent on-camera histogram to work with.

The on-camera histogram is based on the JPG the camera "would have" made even if you don't let it actually do it, and with contrast set to -2, that histogram is closer to what the RAW file will capture. It still isn't great, but it's a lot better.

Unless you have some serious time constraints and will need processed JPGs as soon as you're done shooting, then why save a JPG along with the RAW? And unless you're saving memory card space, or unless you're shooting long bursts why shoot in "JPG Only"?

--
Jim H.

JimH said:

Unless you have some serious time constraints and will need processed
JPGs as soon as you're done shooting, then why save a JPG along with
the RAW?

Click to expand...

IIRC from previous use of RAW, the image review is based on the embedded JPEG thumbnail and it is much more difficult to assess accurate focus when zooming in on this.

David

it doesn't help to save a JPG since the preview is still based on the embedded JPG in the RAW.

I've tried it both ways and it is equally bad whether or not you shoot in RAW+Large JPG or just plain RAW.

I'd consider saving JPGs while shooting if it did improve the preview since that is one of my peeves about the 20D. When you zoom in on the preview, you really can't tell if a shot is sharp or not.

I suppose they're just trying to save space in the .CRW files by keeping the preview JPG small. I guess that's a noble cause, but I'd like it to be a bit larger if it meant that the preview would be more useful for judging sharpness.

--
Jim H.

JimH wrote:

That is exactly the point I try to make to people when they ask why I shoot in "RAW Only" instead of "RAW + JPG".

Since I can later batch process the files in DPP or EVU and get identical-looking JPGs, I lose absolutely nothing by shooting in "RAW Only" and I save some room on the memory cards.

Further, since I'm shooting in "RAW Only", and have no reason to worry about the camera's parameter settings since they won't affect the RAWs anyway, I am free to set the contrast to -2 which gives me a much more relavent on-camera histogram to work with.

The on-camera histogram is based on the JPG the camera "would have" made even if you don't let it actually do it, and with contrast set to -2, that histogram is closer to what the RAW file will capture. It still isn't great, but it's a lot better.

Unless you have some serious time constraints and will need processed JPGs as soon as you're done shooting, then why save a JPG along with the RAW? And unless you're saving memory card space, or unless you're shooting long bursts why shoot in "JPG Only"?

My question:
I have problems with the quote "would have" in Jimh's answer because....

Isn't it true that the camera always internally makes a "raw"--> "jpg" conversion for the little "preview "image" that's included in the ".CR2" file?

Happy Newyear to you all,
HappyWim

HappyWim said:

My question:
I have problems with the quote "would have" in Jimh's answer
because....
Isn't it true that the camera always internally makes a
"raw"--> "jpg" conversion for the little "preview "image" that's
included in the ".CR2" file?

Click to expand...

Even when you turn off all JPG production, the camera does still do a RAW -> JPG conversion to generate the thumbnail file which is embedded into the CR2. And, unfortunately, the histogram is also based on that conversion. So the parameters you've got set will affect the histogram.

Thus, if I want a histogram that more accurately reflects what the RAW file is capturing, I want the contrast turned all the way down.

It's unfortunate that the 20D doesn't give us a three-color, RAW-Based histogram

Sorry for the confusing way I put that.

--
Jim H.

I take it that you're arguing that there is ZERO gain in dynamic range when shooting RAW compared to JPEG. If so, you're mistaken, and it's very easy to demonstrate it.

(1) Shoot a scene in RAW+JPG, in such a way that the highlights are clearly blown, but the transition to highlights is gradual rather than abrupt. Use any JPEG conversion setting you like (I would suggest lowest saturation and contrast, daylight lighting and daylight white balance).

(2) Pull the JPEG into Photoshop. Find the point in the transition where the highlight starts to read 255,255,255.

(3) Open the RAW with Adobe Camera RAW. Pull down the digital exposure compensation. Watch detail emerge from the highlight. Pull it down until the brightest area is below 255,255,255, convert.

(4) Find the RGB value in the conversion where the blown highlight flattens out. Say it's 240,240,240.

(5) Find the zone in the transition where the levels drop below that.

I predict that the RAW conversion will show detail that extends further into the highlights than the JPEG.

That's what people mean when they say that "RAW gives you an extra stop of dynamic range." It may not be all of a stop, and it may not be all the time, but there's definitely extra latitude there -- the amount varies depending on lighting and JPEG settings. (For example, you'll get more "extra" latitude in tungsten than in daylight.) Also in the shadows, although the problems are different (posterization).

Petteri
--
Me on photography: [ http://www.prime-junta.net/ ]
My RSS feed: [ http://www.prime-junta.net/pont/rss/whatsnew.xml ]
My flickr page: [ http://www.flickr.com/photos/primejunta/ ]

GaborSch said:

The most important point to understand is, that the issue is not
raw vs JPEG but raw vs converted format (i.e. raw vs not raw).
Converting the data into JPEG in the camera involves the
interpretation of the raw data, and that's the main issue, not the
conversion in JPEG format.

Click to expand...

So what? This is why you are banging heads with so many people; you go out on a meaningless semantic limb so that you seem to have a point.

You started out talking about "recording". Recording is what you get from the format you have the camera set to when you press the shutter button. The choices are RAW, JPEG, and RAW+JPEG. If you choose just JPEG, your RECORDING is of a lower quality, and that is the only issue; it matters not that you could shoot RAW and botch up a conversion, or clip highlights or shadows with it, etc. That is not the RECORDING . You have discarded options when you choose JPEG only, and you have a lower quality RECORDING. It doesn't matter one bit whether or not the losses are due to generic characteristics of JPEGs, or the specific in-camera conversion. Not recorded is not recorded.

--
John

The entire message of this post sequence is just a restatement of the obvious ... that converted images originate from RAW data regardless of whether they're formed in-camera or on a PC and thus look the same in both cases if similar settings are used. Who ever argued otherwise? This is not the basis of why people claim, for example, extra headroom when shooting RAW or the ability to maintain higher dynamic range. Not at all, in fact.

Rather, the popularity of RAW stems from the combination of three factors. The first is that the in-camera processing, which, true, can be reproduced in the PC to create an identical end result, is limited to specific options. You have a limited set of tone curves from which to choose, with contrast ranging from -2 to 2, for example, and little control over the shape of those curves. Thus, when faced with a scene with difficult tonality (lot of info clumped into specific tones rather than a pleasing progression) or wide dynamic range, you're forced to use the in-camera conversion and then manipulate an 8-bit, gamma compensated image back on your computer. If the image is clipped on either end, there's no recovery. If you have to expand contrast locally within some range of tones, you're stuck with a limited number of tone values due to the 8-bit nature of the file, thus creating gaps and possibly making visible posterization effects. The canned conversion options and final 8-bit output limit your choices.

If the same image is shot in RAW, you wouldn't simply apply a few canned options back at your PC. The situation isn't just one of delayed processing, but of different processing. With in-camera conversion, capturing a wide dynamic range might require using contrast = -2, for an overall flat result. Shooting in RAW, you could create a tone response curve with a toe in the shadows and shoulder in the highlights to reduce contrast in these areas and increase it where it counts most in the midtones, maintaining an overall pleasing image while maintaining good contrast. Or, you could convert twice with different EC and selectively blend. All the while, you're manipulating data that started at 12-bits/pixel of grayscale and thus avoiding posterization issues. The end result is that the full dynamic range of the RAW file can be compressed into the final result (at the expense of contrast in some or all tonal ranges) or a portion of it can selectively mapped to the final file based on knowing what's important and what's not in the composition. You're not stuck with a few canned contrast curves combined with the possibility of multiple exposure compensations if you have a tripod on hand and static subject.

Finally, delaying processing until you're back at your PC is itself an advantage regardless of the conversion options selected. Not only can you make settings choices in an unrushed manner in the comfort of your home and studio, but you're not committed to them. You can also take advantage of any proprietary advantages present or future converters provide over the sophistication of the in-camera processing algorithms. RSe/p provides control over detail vs artifacts in its demosaicing algorithm, for example, and Bibble provides a highlight recovery feature that reconstructs highlights smoothly if a channel is clipped based on the info in the remaining ones.

These factors, and not belief that a PC-converted image somehow differs from the in-camera's interpretation of the very same settings, is what drives (or is what should drive) the photographer toward RAW. It's about the flexibility of level and curve control (very fine control over contrast within tonal ranges and not just overall), applied while at a higher bit depth, and the flexibility of delayed processing.

David

Furthering Peter's point, it's already been demonstrated that even the lowest contrast setting on the 20D uses a curve that clips highlights before the RAW data actually clips. Doug Kerr showed a simple curve mapping at various contrast settings in this forum not too long ago, for example, and the experiment is easy to reproduce. Even the original poster's data shows the effect ... the final file data (JPEG in this case) always has less shadown and highlight headroom. He shows this headroom disappear when the RAW is converted using settings similar to in-camera, but doens't address what happens when, say, a custom curve or even a linear tone curve is used instead of in-camera setting duplications (something not available in camera).

At least on the 20D and competitors, there's no way to fine-tune the in-camera processing to state which RAW tone value maps to a final file value of 0 and which to a final file value of 255 and no way to upload custom curves (a few cameras such as the D70 excepted on this point). This automatically makes it impossible to fit the entire dynamic range of the RAW data into the final file without clipping on one end or the other.

David

Petteri Sulonen said:

That's what people mean when they say that "RAW gives you an extra
stop of dynamic range." It may not be all of a stop, and it may not
be all the time, but there's definitely extra latitude there -- the
amount varies depending on lighting and JPEG settings. (For
example, you'll get more "extra" latitude in tungsten than in
daylight.) Also in the shadows, although the problems are different
(posterization).

Click to expand...

Posterization of the highlights is an issue if you set contrast to -2, to get more DR into the JPEG. The camera will come close to 4095 in the green RAW channel before it clips the JPEG completely, but a lot of RAW levels are crammed into the top 10-15 8-bit levels just below 255. If you try to expand the contrast back to normal in this range, you will have chromatic noise, and the JPEG artifacts will gain contrast. It is OK as a final render; it is not very good as a "recording" to work from.

--
John

GaborSch said:

Now, to the facts:

Click to expand...

Now to the real fact that makes your experiment a house of cards that falls and is remembered as an excercise in making pretty, psuedo-scientific posts:

You RAW captures did not use the top stop of RAW dynamic range.

--
John

JimH said:

The on-camera histogram is based on the JPG the camera "would have"
made even if you don't let it actually do it, and with contrast set
to -2, that histogram is closer to what the RAW file will capture.
It still isn't great, but it's a lot better.

Click to expand...

The camera always makes such a JPEG. It is embedded in the RAW file.

I shoot RAW only, and I review my 20D images in irfanview, which pulls out the 1.5MP JPEG and displays it.

--
John

David R Stout said:

IIRC from previous use of RAW, the image review is based on the
embedded JPEG thumbnail and it is much more difficult to assess
accurate focus when zooming in on this.

Click to expand...

When I first bought my 20D, I thought there was someting wrong with the lens or camera, because the zoom-in wasn't as sharp as my previous 10D, which had 6MP JPEGs embedded in its RAW files.

--
John

JimH said:

It's unfortunate that the 20D doesn't give us a three-color,
RAW-Based histogram

Click to expand...

It is pathetic and inexcusable that any camera that records RAW doesn't give RAW feedback as to exposure. It would be very simple to implement; when the image is digitized, save a buffer of a subsample of pixels in a pseudo-random, almost equal distribution across the frame (to prevent aliasing with man-made geometrical subjects), and use that as a basis for the histogram (if using all the pixels would slow the camera down).

RAW "exposure" is all that really matters if you're shooting RAW. The JPEG histogram has almost nothing to do with exposure, unless the subject is greyscale (or all the highlight range is white), the WB is set the same as it will be in the final render, and the contrast is set to -2.

--
John