X100 + Lightroom = DNG (and a hint for Windows XP users)

Like several other people on this forum, I've been trying to figure out the best way to manage the image files from my new X100.

For now, I've settled on using Lightroom to import the files from the camera, and asking Lightroom to convert the files to DNG format upon import. The reason for this is that the DNG files are about half the size of the corresponding raw files, but my understanding is that they contain all of the original information. I presume that the DNG converter does some kind of lossless compression, and I think a factor of 2 is not unreasonable if the camera does not compress the files.

The one problem I had found was that if I visit a folder full of DNG files on Windows XP, Windows Explorer does not show thumbnails for those files, although it does for Fuji raw files. There are several DNG previewers available for a price, and Adobe has a free one available that does not yet work on Windows XP.

Fortunately, hunting around online I found the following workaround. I'm going to describe it in general terms rather than giving you a file to run, because I figure that if you know enough to figure out how to do what I'm suggesting, you also know that if you screw up while you're editing your registry, you can trash your entire machine. Before I go on, I want to make it clear that if you follow these suggestions and your machine turns into an expensive doorstop, it's not my fault. If you don't understand what I'm suggesting well enough to evaluate the risk for yourself, don't do it.

That said, what you do is install the registry key HKEY_CLASSES_ROOT\.dng\ShellEx\{BB2E617C-0920-11d1-9A0B-00C04FC2D6C1} with default value {3F30C968-480A-4C6C-862D-EFC0897BB84B} (including the curly braces in both cases). Basically, this tells Windows to use a built-in previewer for DNG files that knows enough about their format to display thumbnails.

I believe that by storing DNG files instead of raw files, I will not lose any editing capability in Photoshop or Lightroom, and as those are my two main editing tools anyway, I don't see any reason to keep the Fuji raw files that are twice the size.

If anyone knows any reason not to use this procedure, I'd appreciate knowing about it.

Fuji RAF files have always been huge in relation to other camera makers with the same Megapixels. This was true even before they had the expanded DR capability of the S3 and S5.

However, to think that you aren't losing valuable info when the files come out half the size I believe is wishful thinking. I'm not saying that you aren't still getting a good raw data file, but what you are losing is likely what gives the RAF it's signature Fuji look and possibly the expanded DR. If theses aren't important you and if you are satisfied with the results that is truly all that matters. but I wouldn't want other to be mislead that this has no impact at all on the raw data.

Trust me, the large RAF files have always been a sore spot for Fuji users as it not only impacts your hard drive, but it increases write times, transfers times and processing times. If Fuji thought they could shrink the size without impacting IQ I truly believe they would have done it by now.

My two cents.

--

Dave

Do you have evidence to support this belief, or is it just what you believe?

I'm not trying to be snarky here. Rather, I would hope that Lightroom would write everything into its DNG files that it could obtain from reading the raw file in the first place. Moreover, I have no difficulty in believing that it is possible to reduce the size of an uncompressed image file by a factor of 2 without losing any information--i.e. while being able to get back exactly the same file as the original.

In other words, as long as one uses Lightroom or Photoshop for editing, I do not understand why there should be any difference between

Reading a Fuji raw file into lightroom, editing it, and printing the result,

and

Ronverting a Fuji raw file to DNG format, reading the DNG into Lightroom,
editing it, and printing the result.

Now... If Lightroom doesn't actually write all the information it has into a DNG file, then your belief is correct. My question, then, is whether you can find concrete evidence one way or the other.

I've had Fujifilm cameras since the S2. Had the S3 and S5 now the x100. EVEN when using DNG converter to convert the S3 and S5 RAF files into DNG files, I'm comfortable no data was lost. I feel the same way with the x100 RAF files being converted to DNG. With the S3 and S5 the main thing you lost was the ability to work with the file with Fujifilm's software which was slow, but created pretty good pictures.

DNG uses a lossless compression I believe to get you the smaller size. Regardless, I've done tests and could not see any loss in converting to DNG.

Wishing you well!

Paul
--
http://www.okpablo.com

Here's what I did:

Start with an X100 raw file.
Open the file in Lightroom.
Export the file as a TIFF file.

Now ask Lightroom to import the raw file and convert it to DNG.
Exit Lightroom, just in case.
Launch Lightroom again.
Open the DNG file.
Export the file as a TIFF file.

The two tiff files, each one about 70 MB, are EXACTLY identical except for 9 bytes that inspection revealed to be timestamps written into the file.

This experiment confirms that at least for this particular image, converting to DNG does not change any information that Lightroom uses to produce a TIFF file.

Of course it doesn't prove that the conversion never loses information, but it does at least suggest that using DNG format doesn't wantonly discard important information.

No evidence. Simply postulating.

And at this point since you both have an X100 and I don't I have no way to verify. However, it might be interesting to see now that both Lightroom and Aperture support the X100 RAF how different they profile the instant results between Fuji DNG and the original RAF. Do they produce the same output? If so, I love the idea of saving space. It just seems odd to me that Fuji would continue to create needlessly bloated files because write speed to the card has always been a bugaboo for them.

--

Dave

I automatically back up the raf files upon import to Lightroom to an external HD. I convert to DNG as I import. This places raf files on my external HD as an immediate backup and gives me DNG files to use on my iMac. I really don't care about using up space on my back up HD. When it fills up I just move it off-site as an emergency archive.

If it turns out there is some value to the raf format (which I seriously doubt), then I can always resort to the external HD files.

I also back up by external HD daily. So I actually have another copy of the raf files as well.

The iMac internal drive is backed up using Time Machine. This gives me an hourly back up of the DNG files and my Lightroom catalog file.

I end up with four copies of the raw files two raf and two DNG (not counting those off site).

D a v e said:

However, it might be interesting to see now that both Lightroom
and Aperture support the X100 RAF how different they profile
the instant results between Fuji DNG and the original RAF.
Do they produce the same output?

Click to expand...

I'm afraid I don't understand the question. It looks to me like you are describing the test I just tried:

1) Use Lightroom to produce a TIFF file from a RAF file.

2) Use Lightroom to produce a DNG file from the same RAF file.

3) Use Lightroom again to produce a TIFF file from the DNG file.

The two resulting TIFF files are identical except for timestamps. Not just visually identical, but bit-for-bit identical.

I can't do any tests with Aperture because I'm on a PC.

So what do you mean by "how different they profile the instant results" ?

I'm sorry, but some of the technical disinformation in this forum is astounding.

In the case of any uncompressed format, it's nearly always possible to get a substantial compression ratio. Adobe wouldn't create a lossless format like DNG just to throw information away, that's preposterous. Some algorithms are very computational to do the compress, for example bzip2 (based on block compression) vs. gzip, or even L7. In the case of visible data, it's also common to do trellis encoding and other types of compression. JPEGs use Huffman encoding (lossless) on top of the DCT transform (lossy) to further reduce the size. It's way more likely that Fuji is being lazy at the camera and not bothering to compress raw images at all or resorting to Run length encoding than it is that a DNG is lossless. Why? Well because it doesn't make sense to do the more advanced compression algorithms on the camera, especially if you need software to process the images anyways. They do JPEG because they've developed highly specialized ASICs for it that perform their task within some specified amount of power requirements, and JPEG doesn't need specialized software to process, as it's universal.

__1 said:

Adobe wouldn't create a lossless format like DNG just to throw
information away, that's preposterous.

Click to expand...

The point of my original question was that I did not know for sure whether DNG was always lossless, in practice even if so in theory, and hoped that someone else would be able to tell me definitively.

Hi, I've just read this thread with some interest and though I might add what I found out when I investigated this for Nikon raws (i.e. NEF vs DNG). Interestingly, there is a similar size difference, and this is ultimately mainly due to the fact that NEF files also store a JPEG image inside the file as well!

D a v e wrote:

If Fuji thought they could shrink the size without impacting IQ I truly believe they would have done it by now.

This is the basis of economic reasoning.

Two economists walking down the street.

"Do you suppose that $100 bill on the sidewalk is real?"

"Nah. If it were real, someone would have picked it up by now."

If converting RAF to DNG cuts the file size in half, then the conversion most definitely looses image information. It could be reducing the number of bits per pixel (eg., from 14 to 12), removing the EXIF metadata (including a high resolution JPG that's including in the RAF file), and/or using lossy compression instead of lossless compression. Personally I wouldn't do it.

One test for loss of bit depth is to take an overexposed image and an underexposed image, convert them to DNG, and then try to fix the original RAF files and the converted DNG files, and then compare the fixed images. If they look different after exposure compensation, then you know the conversion is loosing information.

Another test for lossy compression is to take a picture of a very detailed scene (eg., a resolution chart) and pixel peep on the RAF vs. DNG. Note that you need a good RAW viewer that will actually show you the RAF pixels instead of the JPG that is embedded in the RAF file. (Personally I use FastPictureViewer, which IMO is required for anyone who works with RAW files on Windows machines.)

--
http://www.flickr.com/photos/noirist/

Another issue to consider -- are you converting to DNG RAW files or DNG linear files? As I understand it, DNG linear files are like JPGs and certainly loose information.
--
http://www.flickr.com/photos/noirist/

I just tried converting a 16.2MB Nikon RAW file (14 bit NEF file with lossless compression) to a DNG file using DxO Optics Pro. The DNG file is in a "16-bit linear DNG format" that DxO documentation states provides no loss of information. The DNG file is 48.7MB! So its highly unlikely that your 1/2 size DNG files are lossless.
--
http://www.flickr.com/photos/noirist/

Unfortunately this test says more about the limitations of TIFF files than it does about the amount of image information contained in the RAF vs. DNG files. You didn't say what kind of TIFF you used... let's say its an 8-bit TIFF. Then the conversion to TIFF is just taking the high 8 bits of each color. Converting from a 16-bit RAW file to an 8-bit DNG file would result in two files that would both convert to the same 8-bit TIFF file. But obviously the original 16-bit RAW file has a lot more information than the 8-bit DNG file.

Andrew Koenig said:

Here's what I did:

Start with an X100 raw file.
Open the file in Lightroom.
Export the file as a TIFF file.

Now ask Lightroom to import the raw file and convert it to DNG.
Exit Lightroom, just in case.
Launch Lightroom again.
Open the DNG file.
Export the file as a TIFF file.

The two tiff files, each one about 70 MB, are EXACTLY identical except for 9 bytes that inspection revealed to be timestamps written into the file.

This experiment confirms that at least for this particular image, converting to DNG does not change any information that Lightroom uses to produce a TIFF file.

Of course it doesn't prove that the conversion never loses information, but it does at least suggest that using DNG format doesn't wantonly discard important information.

Click to expand...

--
http://www.flickr.com/photos/noirist/