The longevity of Raw files - a reconsideration

The longevity of Raw files - a reconsideration

Twenty years from now, will you be able to open Raw files from some of today's new camera models? For years it's been taken for granted that archiving Raw files was a safe practice. For some of the latest camera models, though, maybe it's not so safe.


History

I made the transition to digital SLR back in 2005. At the time, there was considerable concern about the archival longevity of Raw files. JPEG was so ubiquitous that there was really little question that in twenty years you'd still be able to open them, and to convert them to whatever new and wonderful format might have taken over. But what about Raw files? Nikon triggered some angst by (allegedly*) dropping support for some of its earliest cameras from its own software, and then starting to encrypt the "as shot" white balance information in new cameras (here's Nikon's initial response to the furore).

There were those who said that the solution was simple: hold onto whatever software you currently use to open the Raw file. That isn't a solution. Image software has specific needs from the operating system — Windows, OS X, Linux, or whatever — and the operating system has specific hardware requirements. I used to play a lot of computer games, but most of the ten-year-old game software I have won't run even on my aged Windows XP system because too much has changed... and that's with the same basic computer chip. Macs have used three totally different computer chips over the years — Motorola, Power PC, and now Intel. And as we've seen with LightZone™, activation issues can keep you from installing old software on a new system even when the software itself will run fine. No, relying on being able to run ancient software to read ancient Raw files isn't something you can count on.

In 2005, OpenRaw was formed to try to convince camera manufacturers to publicly document their Raw file formats. A year later, it faded away. To some extent, the problem might've been that it wasn't a solution to the longevity issue. It's a long way from having a documented Raw file format to having workable software that can decode that format into a quality RGB image, and few photographers have the programming skills to undertake that task. In the end, support for Raw file formats was still dependent on the software manufacturers.

Besides, most people didn't notice a problem. Adobe began working agreements with camera manufacturers to get access to the proprietary information about their Raw formats, and other software developers continued to reverse-engineer the formats without concern for the legal implications. Adobe had introduced DNG as a publicly-documented universal format for Raw files in 2004, and although it wasn't yet widely accepted in 2005 (specifications for version 1.1 had just been released), DNG promised a way forward that would require decoding only one format — rather than hundreds of formats, both old and new.

Significant among the unauthorized  reverse-engineered Raw software was dcraw. It was working software, not just documentation, that handled almost every Raw format ever produced. Furthermore, it was available as C-language source code, which just about any knowledgeable computer nerd could turn into a working program for just about any computer. More than anything else, dcraw quieted the anxiety that today's Raw files might not be readable tomorrow.

Over the following years, Raw file formats became even less controversial. With a few exceptions, they were based on the TIFF standard, so pulling the data out of the file wasn't an issue — the only question was making sense of the data. And except for Fuji and Sigma, sensors in the new models were basically all rectangular Bayer arrays designed with repeating 2x2 squares that each contained two green pixels on a diagonal, one red pixel, and one blue pixel. The only big question was the color response of the Bayer filter on the sensor: what we might think of as the color space for the Raw data. Most of the model-specific part of dcraw now consists of color space information.


Present day

From a Raw-file point of view, the Golden Age of DSLRs is drawing to a close. An increasing number of compact cameras now produce Raw files, and the new "mirrorless" cameras all do [I think that's true; I haven't fact-checked it]. In some cases, these cameras include innovative sensor designs. In an increasing number of cases, these cameras include custom processing of the Raw files to remove artifacts like distortion and chromatic aberration. Those corrections aren't constant for a given camera; they depend on the lens, the focal length, and often on aperture.

Adobe has managed to stay ahead of this situation. Their contracts with the camera manufacturers give them early access to the information they need to update Adobe Camera Raw and their DNG converter. In 2009, Adobe published the specification for version 1.3 of DNG, which includes a kind of programming model for describing corrections to distortion, vignetting, chromatic aberration, and bad pixels. Few (if any) Raw converters other than Adobe's currently take advantage of these new features of DNG. In January 2012, Adobe issued new Camera Raw and DNG Converter implementations based on DNG 1.4, but eight months later still has not published the specification for DNG 1.4 — although it seems like lossy compression is the primary addition so not of immediate concern to this discussion. Update: a few days after this article was originally published, Adobe finally released the DNG 1.4 specification.

Other software manufacturers are falling behind, as it takes more and more effort to "profile" a new camera model. It used to be all they needed to do was to take a few shots of a color reference to determine the color response of the red, green, and blue photosites. With some of the new camera models, the software manufacturers now have to determine the effects of various lenses at various focal lengths and apertures, then produce and test code to correct for those effects. All of which generally occurs after they manage to get a production camera unit. Photo Ninja sidesteps the matter by having the user develop and store their own correction profiles, but I don't think that approach is going to win them many friends.

Now we come to dcraw. Dave Coffin generally doesn't get cameras to test with. He typically gets color space coefficients from Adobe's DNG Converter. As of now, dcraw has no ability to correct for various lens artifacts, and my guess is that it's not going to be coming.


The future

So here's the problem: for camera models that expect the Raw software to compensate for lens artifacts, dcraw can't provide the assurance that Raw files will be able to be processed in the future. For those camera models, Adobe controls the only reasonably-safe bets on the future. Either plan to use Adobe software if necessary, no matter how expensive it might become, or use Adobe DNG.

But is DNG a safe bet if you don't want to commit to using Adobe software forever? I'm not too sure. Given that Adobe took so long to publish the specifications for DNG 1.4, I have to wonder just how "open" DNG is going to continue to be.

For now, I don't have any answers. I only have questions. But I do think it might be time to revisit the issue of archival longevity for this new breed of Raw file.



footnotes

This article was originally published on the LightZombie Project web site. Minor revisions have been made for clarity.

* I've never seen a specific statement as to which camera models Nikon quit supporting in which software, but the allegations were widespread and appear in Peter Krogh's The DAM Book (1st ed), in the even more watered-down statement, "we have already seen manufacturers drop support for certain RAW file formats." Krogh changed that statement in the second edition of the book to, "As the cost of supporting older formats grows larger than the reward, software manufacturers will eventually drop this support," then cites Raw files from Kodak's DSLRs as an example. Certainly, creating  support for older models is problematic for new software like Photo Ninja, because the software company needs appropriate reference Raw files from those cameras. But retaining support that's already provided shouldn't be an issue.

The views and opinions expressed in this article are those of the author and do not necessarily reflect the views and opinions held by dpreview.com or any affiliated companies.

Comments

Total comments: 12
Glen Barrington
By Glen Barrington (5 months ago)

I still don't see much of a problem. "Done" photos invariably wind up as Tiffs or jpgs. Who cares about raw? it is what photos are made of, not what they ARE.

1 upvote
Victor Engel
By Victor Engel (Apr 15, 2013)

I can't get Agfa Rapid film or process my exposed Kodachrome film anymore either.

0 upvotes
wchutt
By wchutt (Apr 4, 2013)

There is no reason not to take the opposite view. Every DNG standard has been published so far. Why would future DNG standards not be published? Where is the evidence Adobe will not publish future standards? How does an eight month delay translate into fear about a complete reversal of Adobe's strategy? I wish Pardee's would elaborate on how Adobe will make more profit abandoning open DNG than staying the course.

Instead of fear and loathing, optimism and confidence are equally justified. I do not share any of the Pardee's concerns. Pessimistic speculation is no more or less valid than optimistic speculation. But I do know this. If DNG failed for sny reason, there are millions of DNG files right now and millions more will be recorded. Opportunity for ample profit will provide incentive for DNG data to be converted into something useful no matter what.

The sky is not falling.

Of course one prints photographs to increase the odds they will be around for a long time.

0 upvotes
wchutt
By wchutt (Feb 12, 2013)

Are you saying Adobe plans to take DNG out of the public domain? As long as DNG is open, people can convert DNGs to something else. In principle the raw data within a DNG from 2010 could be converted to any other two-dimensional space array in the future. This could be done by highly motivated amateurs decades from now.

I am not concerned about on-board lens correction either. Are you saying the lens correction parameters written to raw fils are proprietary or encrypted so only Adobe can read them? Even if that were true, it would be possible to reverse engineer the corrections empirically. This too could be done by highly motivated amateurs decades from now.

0 upvotes
Osvaldo Cristo
By Osvaldo Cristo (Jan 14, 2013)

Too much preocupation for an inexistent problem... so far. My Nikon D100 raw files from 2002 are completely workable after more than ten years. I can open and work on them with NX2 or ACR (PS CS5 or LR) on Windows 7.

If in the future the format is not supported for the mainstream applications certainly you will have time to convert to another one, then supported. In this meantime, I have no interest on DNG.

1 upvote
Doug Pardee
By Doug Pardee (Jan 22, 2013)

Well, "so far" was precisely my point. We've been lulled by how easy it's been to keep processing the Raw files from DSLRs. But the new "mirrorless" brigade is breaking new ground, both with innovative sensor designs and with optics that demand extensive custom corrections during Raw processing.

The popular "mirrorless" cameras come from smaller manufacturers. There are too many of those manufacturers crowding the field, and it's a safe bet that many of them won't be making cameras come the end of this decade. Those who aren't still around also won't be supplying updated software to decode the Raw files from their old cameras. (Think Kodak.)

As for the third-party Raw converters, which do you trust to survive? Adobe? Probably. But will you always be able to afford them?

I'm not offering any answers. I'm just suggesting that the relatively carefree experience we've had with DSLR Raw files won't necessarily be applicable to some of these mirrorless models.

0 upvotes
hjulenissen
By hjulenissen (Dec 13, 2012)

This is most easily solved by the market. If hardcore camera customers refused to purchase any camera without DNG output, the manufacturers would get the hint. The fact that we don't do this, suggests that it is really not that important to us.

You still can convert any acr-supported raw file to dng? This will be a well-documented file that anyone with access to old DNG specification printouts, the raw dng file and an interest in image processing and programming will be able to dechifre?

Did anyone notice that Adobe Lightroom adds lots of meta-info to exported jpegs? Seems to me to be a complete "cookbook" of how the user edited the file?

-h

0 upvotes
hjulenissen
By hjulenissen (Dec 13, 2012)

From the article: "But retaining support that's already provided shouldn't be an issue."

This is not true. Keeping old, hard-to-test, seldom-used code forever is in itself a problem for software.

I think that for important stuff it makes sense to render a jpeg version. You loose the ability to reprocess the raw file, but if the image was that important for you, it seems safe to assume that you put lots of energy into making the settings look good.

0 upvotes
Barry Pearson
By Barry Pearson (Oct 6, 2012)

1. DNG is the only archival raw file format. It is the only format that contains sufficient camera details for raw conversion. Other formats need examination, by a raw converter developer, of a sample camera to obtain camera details needed for raw conversion, which accounts for the delay in support when a new camera model that doesn't use DNG is launched. Or, like dcraw, they can copy details from DNGs. (And of course DNG is the only openly-specified raw file format suitable for submission to ISO, which has been done).

2. The Adobe DNG Converter (there are others) is a raw converter in its own right. It can generate and store a full-sized JPEG of the image based on the converted date. I think (I'm not certain) that it uses lens profiles to do so. In which case the size of the lens profiles is added to the converter size.

(I'm writing this just after Adobe published the new specification).

1 upvote
CAcreeks
By CAcreeks (Oct 3, 2012)

If cameras wrote 16-bit per color lossless files, such as JPEG 2000, we would have no need for RAW. Also now that cameras produce too many megapixels for most purposes, downsampling is common practice. JPEG is sufficient for that, so perhaps we no longer need RAW anyway.

0 upvotes
hjulenissen
By hjulenissen (Dec 13, 2012)

The main problem with JPEG/JPEG2k is not that the formats are lossy. The main problem is that those formats discards information necessary to do proper after-the-fact white-balancing, recovery of clipped highlights etc.

What you need (for full flexibility in editing) is a lossless representation of what the camera sensor recorded + any relevant camera settings. This means non-standard colorspace, bayer (or not) pattern etc.

0 upvotes
Mako2011
By Mako2011 (Oct 3, 2012)

An excellent article. I currently have NEF's from as far back as the Nikon D70 and currently shoot RAW only. With that in mind, you pose excellent questions. I currently make extensive use of Nikon's ADL (Active D-Lighting) technology and only Nikon software can produce an accurate TIFF/JPEG from a RAW file that includes ADL. Until ADL can be supported by DNG....then TIFF is the only real option. My plan is then to watch and as soon as D70 NEF's aren't supported...batch convert to TIFF for archive or whatever might be a better option. In the future, I honestly think it will be easier to convert my NEF's to something useful than it has been getting my super 8mm film converted to Blu-Ray. Can be done but expensive and a big pain :)

0 upvotes
Total comments: 12