Does RAW include a tonal curve as mentioned in http://www.cambridgeincolour.com/tutorials/photoshop-curves.htm or is non linear because of sensor non linearity. What is the meaning of 'linearization, in Adobe's DNG standard?
-
Welcome to the new forums! Please read this first. For known issues we are working to resolve, click here.
RAW linearity
- Thread starter adadad
- Start date
Jay Turberville
Forum Pro
Every raw file I've looked at is very linear. There is typically some slight non-linearity in the shadows, but I suspect that is probably an artifact of a slight amount of light scatter - probably from my test apparatus and from the lens and camera. Here is an example of the raw file response file from an E-330. This is very typical. Note the slight deviation from linearity at the bottom of the graph.
This is the response of the same exposure after internal JPEG processing and the application of a typical gamma correction to fit the response into an 8 bit file. This is also very typical. Note how the response is now more film-like with its gradual "S" curve.
I don't know.
--
Jay Turberville
http://www.jayandwanda.com
Thanks Jay for the interesting information. So the link I referred to, where it is being explicitly stated that RAW files include (photographic) tonal curves is wrong. Does adobe's DNG standard refer to the non-linearity you have, when it is about the linearisation function included in the standard?
Barry Pearson
Veteran Member
It is just a way of making DNG files smaller than they otherwise would be.
http://www.barrypearson.co.uk/articles/dng/specification.htm#linearizationtable
The Leica M8 is the only camera I know of that uses a DNG LinearizationTable. It holds 1-byte per sensor sample. Then each M8 DNG file contains a look-up table indexed by that 1-byte, giving a 2-byte value. (So this table is 512 bytes long). The result is that sensor values become 16-bit values, although only (at most) 256 of all those values are actually used.
In fact, the M8's raw image values start as 14-bit. Then they get converted to 16-bit in-camera, and the square-root is taken so that they fit into 1-byte, and these are stored in the DNG file. The LinearizationTable in the case of the M8 is just a way of squaring the number.
But it could be some other algorithm. For example, if you open a JPEG in ACR 4.x or Lightroom, and save it as DNG, you get a Linear DNG file holding 3 1-byte values per pixel. Then there is a LinearizationTable in the DNG file that can be used to convert each 1-byte value to up to 16-bit. In that case, it ISN'T a simple squaring like the M8.
Jay Turberville
Forum Pro
No. I think it is just a language problem. When the camera makes a JPEG (or TIFF) it applies the curve as shown in the JPEG example. I suspect that is what the article meant.
I don't know what DNG is referring to. I haven't studied up on the format. The basics are that raw is inherently linear and that attractive/usable output is accomplished by altering that linearlity.
--
Jay Turberville
http://www.jayandwanda.com
GordonBGood
Veteran Member
I suspect that this use of the word "linearization" should really be "normalization", as normal raw data from digital cameras is very linear as Jay has shown. However, our eyes and our monitors are not linear, both having a logarithmic power curve response where there is less response to linear light level changes at the bright end than at the dark end of the range, which requires a "normalization" step in order to convert the linear raw data to the logarithmic data we can view properly.
The use of tone curves in the referenced article refers exactly to this "normalization" as explained above.
Raw files, including DNG, sometimes include a "linearization table" so that the raw values can be saved in a file size reducing format where a tone reponse curve of some sort has already been applied, with most of the linear raw data being able to be recovered by the inverse transformation implied in the "linearization table".
However, this linearization table could also be used to linearize raw data that wasn't as linear as the data seen in Ray's charts, such as would happen if the sensors were run over a brighter range where they more closely approached the full saturation filling of the electron well of the photosites, meaning that less amplification would be used and noise would be reduced. This is really why Fuji CCD sensors have less noise and more dynamic range than the competition; they run their sensors much closer to saturation levels and take care of any required linearization with processing.
In fact, raw image format files from modern Nikon DSLR cameras also record the linear raw data with a Tone Response Curve (TRC) already applied and most of the original linear data is recovered by use of a linearization table, whether embedded in Nikon's NEF files or in DNG (Nikon's term is near lossless). In this way the original data with a range of zero to 4095 decimal is reduced to only have a range of zero to 683 decimal for a slight reduction in file size (12 vs. something less than 10 bits), although the reduced codes also compress a little better using the Lossless JPEG compression both NEF and DNG use.
The Leica M8 and the new NEF files are the only two raw formats of which I am aware that use this, which is really an application of a nonlinear curve to the raw data which requires a "linearization" in order to retrieve the original generally linear data.
Regards, GordonBGood
Barry Pearson
Veteran Member
GordonBGood wrote:
[snip]
[snip]
I wasn't aware that NEF used JPEG Lossless. dcraw says:
/*
Not a full implementation of Lossless JPEG, just
enough to decode Canon, Kodak and Adobe DNG images.
*/
[snip]
True. If the values were still held as 12-bit values, (because it isn't deliberate compression, just non-linear), this is still only 4K 2-byte values, not much for a raw file.
[snip]
[snip]
I wasn't aware that NEF used JPEG Lossless. dcraw says:
/*
Not a full implementation of Lossless JPEG, just
enough to decode Canon, Kodak and Adobe DNG images.
*/
Eric28757
Senior Member
Actually , to disagree Barry.
WHile the M8 usese the linrarization table to make tis files smaller, that really isn't the intent of the spec.
Some cameras (Mostly older models, and more consumer then professional) exhibit quite a bit of non linearity in their signals. Some more so in shadows, Others as the signal appoaches saturation.
Using a matrix for the cam-> XYZ transformation requirets a linear signal for it to work correctly, and thats why the spec includes it. Many of the more modern cameras have very linear signals in the file itself and thus the table is't used a lot.
Eric
http://www.bibblelabs.com
WHile the M8 usese the linrarization table to make tis files smaller, that really isn't the intent of the spec.
Some cameras (Mostly older models, and more consumer then professional) exhibit quite a bit of non linearity in their signals. Some more so in shadows, Others as the signal appoaches saturation.
Using a matrix for the cam-> XYZ transformation requirets a linear signal for it to work correctly, and thats why the spec includes it. Many of the more modern cameras have very linear signals in the file itself and thus the table is't used a lot.
Eric
http://www.bibblelabs.com
Barry Pearson
Veteran Member
[snip]
DNG specification version 1.1.0.0, (the latest), page 20:
LinearizationTable:
"LinearizationTable describes a lookup table that maps stored values into linear values. This tag is typically used to increase compression ratios by storing the raw data in a non-linear, more visually uniform space with fewer total encoding levels".
Eric28757
Senior Member
Barry,
My most humble apologies.
I sit corrected.
Eric
My most humble apologies.
I sit corrected.
Eric
Barry Pearson
Veteran Member
I think we have come to a consensus in this thread that, whatever the intention, (which probably doesn't matter much), this is a useful feature that can be used either as a compression aid or to correct for non-linearity in the raw image data. (Or both).
This discussion has been useful to me. I'll update what I have written about this feature, which I had limited to its use for compression:
http://www.barrypearson.co.uk/articles/dng/specification.htm#linearizationtable
It will be interesting to see if camera manufacturers exploit it for this extra use in future.
The whole thread is very informative to me. I learned a lot. Thanks all. Still one more practical question in http://www.dpreview.com/reviews/canoneos30d/page21.asp
under 'RAW headroom', there are e few curves published for EOS 30D. The last one is called 'RAW ACR best' and the setting includes 'Curve linear'. Does this curve represent the native RAW curve?
under 'RAW headroom', there are e few curves published for EOS 30D. The last one is called 'RAW ACR best' and the setting includes 'Curve linear'. Does this curve represent the native RAW curve?
GordonBGood
Veteran Member
As follows:
Regards, GordonBGood
DCRAW is not a full implementation of Lossless JPEG as a base decoder just because it doesn't handle all the possible variations of the JPEG Lossless specfication; the Canon, Nikon, and Kodak versions of Lossless JPEG are all close enough to the spec to be able to be read by a base Lossless JPEG decder if there was one.
Regards, GordonBGood
GordonBGood
Veteran Member
What is referred to on that page as "curve linear" is just the curve by that name in ACR, which means that there is no extra boost or cut of the highlight/shadow lumininances but rather the raw development Tone Response is set to what Adobe considers "normal" neutral. In fact, there still is some contrast boost for even the "linear curve" and a TRC is applied to bring the developed images into a visual colour space, so this is not the native raw data curve.
In fact, Phil's DR curves aren't really log/log curves where a straight line would represent perfectly linear raw response because he plots the JPEG-type 8 bit luminance levels on the vertical axis after the images are converted to visual colour space, and these levels out of 256 possible states are not exactly logarithmic, which is why the curves change with different raw/JPEG contrast setting processing.
Regards, GordonBGood
Barry Pearson
Veteran Member
GordonBGood wrote:
[snip]
[snip]
I was commenting more on the absence of "Nikon" in that list in dcraw. I've never known what lossless compression Nikon used, but assumed it might be something other than that.
Jay Turberville
Forum Pro
The actual curve of a raw file is a nearly perfectly straight line. The curves you reference are the results after the linear response has been adjusted by some program. In this case, different options were used in ACR that result in different adjustments to the linear sensor response.
--
Jay Turberville
http://www.jayandwanda.com
GordonBGood
Veteran Member
An oversight by Dave Coffin as far as his comment goes. If you read a few pages further down in his dcraw source code you will see that the nikon_compressed_load_raw() function calls the JPEG decompression routines just as much as the Canon one, and is written as a special function because he incorporates the linearization curve look-up in the routine. For Canon, he uses the standard lossless JPEG decompression function, lossless_jpeg_load_raw(), but he puts special cases in it for the CR2 format so that it handles "slicing" the image vertically. He could have used the same routine to handle NEF's with some more special "if statement" cases so that it would use the linearization table, but chose to write a separate routine instead.
In fact, Dave's code has so many special cases that many of the routines should really be written as special for each camera image format in the interests of making the routines more time efficient in not having to handle all the extra logic in determining what case to handle.
Regards, GordonBGood
Similar threads
Keyboard shortcuts
- f
- Forum