The longevity of Raw files - a reconsideration

The longevity of Raw files - a reconsideration

Twenty years from now, will you be able to open Raw files from some of today's new camera models? For years it's been taken for granted that archiving Raw files was a safe practice. For some of the latest camera models, though, maybe it's not so safe.

---

History

I made the transition to digital SLR back in 2005. At the time, there was considerable concern about the archival longevity of Raw files. JPEG was so ubiquitous that there was really little question that in twenty years you'd still be able to open them, and to convert them to whatever new and wonderful format might have taken over. But what about Raw files? Nikon triggered some angst by (allegedly*) dropping support for some of its earliest cameras from its own software, and then starting to encrypt the "as shot" white balance information in new cameras (here's Nikon's initial response to the furore).

There were those who said that the solution was simple: hold onto whatever software you currently use to open the Raw file. That isn't a solution. Image software has specific needs from the operating system — Windows, OS X, Linux, or whatever — and the operating system has specific hardware requirements. I used to play a lot of computer games, but most of the ten-year-old game software I have won't run even on my aged Windows XP system because too much has changed... and that's with the same basic computer chip. Macs have used three totally different computer chips over the years — Motorola, Power PC, and now Intel. And as we've seen with LightZone™, activation issues can keep you from installing old software on a new system even when the software itself will run fine. No, relying on being able to run ancient software to read ancient Raw files isn't something you can count on.

In 2005, OpenRaw was formed to try to convince camera manufacturers to publicly document their Raw file formats. A year later, it faded away. To some extent, the problem might've been that it wasn't a solution to the longevity issue. It's a long way from having a documented Raw file format to having workable software that can decode that format into a quality RGB image, and few photographers have the programming skills to undertake that task. In the end, support for Raw file formats was still dependent on the software manufacturers.

Besides, most people didn't notice a problem. Adobe began working agreements with camera manufacturers to get access to the proprietary information about their Raw formats, and other software developers continued to reverse-engineer the formats without concern for the legal implications. Adobe had introduced DNG as a publicly-documented universal format for Raw files in 2004, and although it wasn't yet widely accepted in 2005 (specifications for version 1.1 had just been released), DNG promised a way forward that would require decoding only one format — rather than hundreds of formats, both old and new.

Significant among the unauthorized  reverse-engineered Raw software was dcraw. It was working software, not just documentation, that handled almost every Raw format ever produced. Furthermore, it was available as C-language source code, which just about any knowledgeable computer nerd could turn into a working program for just about any computer. More than anything else, dcraw quieted the anxiety that today's Raw files might not be readable tomorrow.

Over the following years, Raw file formats became even less controversial. With a few exceptions, they were based on the TIFF standard, so pulling the data out of the file wasn't an issue — the only question was making sense of the data. And except for Fuji and Sigma, sensors in the new models were basically all rectangular Bayer arrays designed with repeating 2x2 squares that each contained two green pixels on a diagonal, one red pixel, and one blue pixel. The only big question was the color response of the Bayer filter on the sensor: what we might think of as the color space for the Raw data. Most of the model-specific part of dcraw now consists of color space information.

---

Present day

From a Raw-file point of view, the Golden Age of DSLRs is drawing to a close. An increasing number of compact cameras now produce Raw files, and the new "mirrorless" cameras all do [I think that's true; I haven't fact-checked it]. In some cases, these cameras include innovative sensor designs. In an increasing number of cases, these cameras include custom processing of the Raw files to remove artifacts like distortion and chromatic aberration. Those corrections aren't constant for a given camera; they depend on the lens, the focal length, and often on aperture.

Adobe has managed to stay ahead of this situation. Their contracts with the camera manufacturers give them early access to the information they need to update Adobe Camera Raw and their DNG converter. In 2009, Adobe published the specification for version 1.3 of DNG, which includes a kind of programming model for describing corrections to distortion, vignetting, chromatic aberration, and bad pixels. Few (if any) Raw converters other than Adobe's currently take advantage of these new features of DNG. In January 2012, Adobe issued new Camera Raw and DNG Converter implementations based on DNG 1.4, but eight months later still has not published the specification for DNG 1.4 — although it seems like lossy compression is the primary addition so not of immediate concern to this discussion. Update: a few days after this article was originally published, Adobe finally released the DNG 1.4 specification.

Other software manufacturers are falling behind, as it takes more and more effort to "profile" a new camera model. It used to be all they needed to do was to take a few shots of a color reference to determine the color response of the red, green, and blue photosites. With some of the new camera models, the software manufacturers now have to determine the effects of various lenses at various focal lengths and apertures, then produce and test code to correct for those effects. All of which generally occurs after they manage to get a production camera unit. Photo Ninja sidesteps the matter by having the user develop and store their own correction profiles, but I don't think that approach is going to win them many friends.

Now we come to dcraw. Dave Coffin generally doesn't get cameras to test with. He typically gets color space coefficients from Adobe's DNG Converter. As of now, dcraw has no ability to correct for various lens artifacts, and my guess is that it's not going to be coming.

---

The future

So here's the problem: for camera models that expect the Raw software to compensate for lens artifacts, dcraw can't provide the assurance that Raw files will be able to be processed in the future. For those camera models, Adobe controls the only reasonably-safe bets on the future. Either plan to use Adobe software if necessary, no matter how expensive it might become, or use Adobe DNG.

But is DNG a safe bet if you don't want to commit to using Adobe software forever? I'm not too sure. Given that Adobe took so long to publish the specifications for DNG 1.4, I have to wonder just how "open" DNG is going to continue to be.

For now, I don't have any answers. I only have questions. But I do think it might be time to revisit the issue of archival longevity for this new breed of Raw file.

---

---

 

footnotes

This article was originally published on the LightZombie Project web site. Minor revisions have been made for clarity.

* I've never seen a specific statement as to which camera models Nikon quit supporting in which software, but the allegations were widespread and appear in Peter Krogh's The DAM Book (1st ed), in the even more watered-down statement, "we have already seen manufacturers drop support for certain RAW file formats." Krogh changed that statement in the second edition of the book to, "As the cost of supporting older formats grows larger than the reward, software manufacturers will eventually drop this support," then cites Raw files from Kodak's DSLRs as an example. Certainly, creating  support for older models is problematic for new software like Photo Ninja, because the software company needs appropriate reference Raw files from those cameras. But retaining support that's already provided shouldn't be an issue.