If anyone understands the ins and outs of RAW, it's Dave Coffin, he has reverse engineered the RAW formats of almost every digital camera on the market and provides his code (dcraw.c) freely for anyone to use. He recently posted a note on his web page pointing out that the encryption of metadata (in the current Nikon vs. Adobe situation) is nothing new and that it's fairly common for manufacturers to apply some kind of protection to their RAW formats. We decided to ask him some of the questions this information raises and also those which have been asked by our readers.
From Dave's page (talking about metdata encryption):
A note about metadata encryption
A firestorm of controversy recently erupted when Thomas Knoll of Adobe accused Nikon of encrypting the white balance data in the D2X and D2Hs cameras, thus preventing Adobe from fully supporting these cameras.
I cracked this encryption on April 15, and updated dcraw.c and parse.c on April 17. So "dcraw -w" now works correctly with all Nikon cameras.
This is not a new problem. Phase One, Sony, Foveon, and Canon all apply some form of encryption to their RAW files. Dcraw decodes them all -- you can easily find decryption code by searching for the ^ operator.
Compression is not encryption. Phase One and Sony do encryption only. Kodak does compression only. Canon, Nikon, and Foveon compress the image data and encrypt some of the metadata.
dpreview.com interview with Dave Coffin
We decided to try and get a bit more background to this and conducted a brief interview with Dave to discuss his work and the encryption / obfuscation of RAW data. It seems clear to us that while it's a concern that manufacturers are making it harder for the RAW decoders, this isn't something new and certainly (not at the moment at least) nothing that can't be cracked.
1. Can you just give us a short history of dcraw and how it got started?
It started in February 1997, when I bought a Canon PowerShot 600. Decoding the RAW data was more difficult than I had expected, knowing nothing about filter arrays, colorspace conversion, etc. But in August 1997, I found a decent interpolation technique, and finally was able to create images comparable in quality to Canon's.
Word slowly spread, and people asked me to do other cameras, sending me sample images to decode. I added support for the PowerShot A5 in May 1999 and the PowerShot A50 and Pro70 in May 2000.
In late September 2001, after months of effort, I finally figured out the lossless compression algorithm used by the PowerShots Pro90, G1, G2, S30, S40, and EOS D30/D60 cameras.
I solved the Canon EOS-1D on Jan 28, 2002 and the Nikon compressed NEF format on March 24, 2002. Olympus ORF format is not compressed, so it's much easier to decode.
On November 19, 2002, I was laid off. During that month, I added nineteen Kodak cameras, the PowerShot G3/S45, the Canon EOS-1DS, the Fuji S2, and the Minolta DiMAGE 7. In early December, I replaced the whole color-interpolation system, yielding sharper images for all cameras.
On December 10, I attacked the Sigma SD9. I solved the compression algorithm on December 31, then spent another six weeks constructing a Foveon-specific interpolation routine to enhance color and reduce noise.
2. As we know none of the manufacturers openly document their RAW formats, how long does it typically take for you to reverse-engineer a format?
It can take minutes or months, depending on the complexity of the format.
3. Are you ever concerned about the legal implications of reverse-engineering proprietary file formats?
If anyone sued me, I'd be the biggest free software hero since Jon Johanson. It's better for the camera makers to ignore me and hope I lose interest.
4. I take it that reverse-engineering the metadata out of the RAW file is just as complicated (if not more so) than the actual sensor data itself, is this correct?
Yes, the metadata is much more complicated. That's why dcraw reads only metadata necessary to decode the image, and ignores the rest.
5. Which RAW format was the first you worked on which showed signs of having its metadata deliberately encrypted / obsfuscated? Can you give us examples of other formats which have been made 'hard to decode' by the manufacturers?
The Canon PowerShot G6, S60, S70, and Pro1 apply a trivial XOR to the metadata related to color balance. Phase One encrypts the entire image in a slightly more complicated way.
6. I understand that Sony's SRF file format is encrypted, does this include the actual RAW data or just metadata?
Both are encrypted with a hard cipher. My sony_clear program decrypts the entire SRF file.
7. Do you believe manufacturers are doing this to protect their own RAW converters or simply as a method of compressing the metadata?
Encryption is not compression. XOR'ing cleartext with a key does not change the size of a file -- it only makes the contents harder to read.
8. It's clear that many photographers are concerned over the current situation between Adobe and Nikon because they feel it may be an indicator of worse to come (harder encryption, more 'locking down' of file data). So is this a storm in a teacup or a sign of more to come?
Photographers have reason to feel scared. Not being computer hackers, they feel powerless to stop Nikon from asserting property rights over their images.
I'm not so worried. Whatever scheme Nikon tries next, I'll just reverse-engineer it.
9. Is there a place for a standard 'Open' RAW format or does that raise too many issues to do with the sharing of proprietary image processing between competitive manufacturers?
Adobe Digital Negative (DNG) is a great format -- I totally redesigned dcraw for maximum DNG compatibility. But you won't see much enthusiasm from the camera makers. This Joel essay explains why:
Photoshop and digital cameras are complements. Adobe wants to commoditize the digital camera, and the camera makers want to stop them.
10. Manufacturers (Canon for example) claim that only they know how to use the RAW data - along with their knowledge of the sensor's characteristics - to squeeze the best possible quality out of their cameras. Our tests indicate this often isn't the case, with 3-rd party converters often getting better results. Is there any advantage a manufacturer might have when producing a RAW converter?
Whatever advantage the manufacturer has, it disappears when a camera reaches the market. Then anyone is free to buy the camera, shoot test patterns, and analyze the RAW data.
11. Are you aware of significant differences in the way the various manufacturer's converters process RAW files, as we see huge differences in the quality of output, or are they all basically the same thing, just some better optimized?
I don't know -- I usually trace the manufacturer's code just far enough to extract the RAW data.
12. How raw is a RAW file, are there any formats in which the sensor data has actually been modified before it's recorded?
Some Nikon cameras have gaps or spikes in the raw histogram, indicating that the colors were multiplied before being saved to the RAW file. Most cameras leave the RAW data alone, and write color multipliers into the metadata.
|2014_1211_140657AA by old shutter bugger|
from The Bride
|Overloaded by NZ Scott|
from Your City - Delivery Boy
|Petals by Flor Tempra|
from Petal Portraits
|Barley by Will B Milner|
|APPLE & ROACH by TX Photo Doc|
from Delicious - Unpalatable
Take a quick tour of Nikon's new D850 in our 'First Look' video and find out what makes this new pro-level DSLR so exciting. Hint: a lot of things.
Nikon appears to have pulled out all the stops for its D850. It combines high resolution and speed: a full-frame 45.7MP BSI CMOS sensor capable of 7 fps bursts. The D5's 153-point AF system, a tilting touchscreen and 4K/24p video are also on board. It arrives in September for $3300.
The Nikon D850 is a 45.7MP full-frame DSLR that can shoot at seven frames per second. Supporting this is an autofocus system lifted wholesale from the company's pro-sports D5 model. Add in a bigger viewfinder and full-width 4K capture and you've got a lot more than a warmed-up D810.
Cinema equipment powerhouse Arri has introduced a new line of full spectrum neutral density filters that drawn on its years of experience creating internal filtration systems for its ALEXA Mini and the AMIRA movie cameras.
The 'pocket powerhouse' Godox Wistro AD200 flash just got a new accessory. Now, you can swap out the speedlight or bare bulb heads for a 3.6-watt LED lamp.
Photographers Andrew Studer and Ted Hesser captured some of the most iconic images of this week's eclipse, showing a climber standing in the middle of the glowing corona. This is the story behind those images.
Intrigued by those ultra-cheap, fast lenses coming out of China? This video review of the $550 Sainsonic Kamlan 55mm F1.2 lens points out some of the performance you sacrifice to save an almighty dollar (or $800).
Canon has expanded its PIXMA TS-series Wireless Inkjet printer lineup with five new models, two of which contain an improved ink system that adds a sixth color ('Photo Blue') to help reduce graininess and improving overall quality.
Oprema Jena is on a roll. After a wildly successful Kickstarter to bring back the legendary Zeiss Biotar 75mm F1.5 lens, the company is sweetening the pot by resurrecting the Biotar 58mm F2 as well.
Nikon has issued a delay and apology regarding their 100th Anniversary D5, D500, and Triple Lens sets. Due to a logo issue, the company is being forced to delay shipments until October.
Yet another reason to always shoot Raw. These two shots are actually the same photo, photographer Dan Plucinski simply pulled up the shadows in post.
The Galaxy Note 8 is the first Samsung smartphone to feature a dual-cam setup. The 2x tele lens allows for a background-blurring portrait mode and comes with optical image stabilization.
Cloud backup service CrashPlan has announced that it will permanently shutter it's "for home" service by the end of October. If you use CrashPlan to back up your photos, you'll want to find an alternative ASAP.
Equivalence is much-discussed, but still often misunderstood. Here's a simplified explanation of the concept of equivalent apertures, which is just another way of talking about light received by your camera.
Try your hand at this blind portrait shootout between the Canon 1DX Mark II, Nikon D5 and Sony a9. With all bias removed, you might just rank your favorite camera brand worst.
Photo sharing site 500px has just added support for wide-gamut color profiles such as AdobeRGB and ProPhotoRGB, even allowing users to filter their searches by color profile.
DJI just released a mandatory firmware update for the DJI Spark. If you own a Spark and don't update your firmware by September 1st, DJI will remotely ground your drone.
Affordable flash manufacturer Godox has updated its smartphone app so that it can be used to control all of its wireless X flash units, not just the A1 smartphone flash.
Western Digital's new My Book Duo external desktop storage system offers up to 20TB of storage capacity, and comes with RAID-optimized WD Red hard drives.
Version 1.04 of the Sony a6500 firmware can be downloaded from the Sony Support website now.
Not sure how to choose your first drone? In this article, the second of a 3-part series, we discuss what factors you should consider when deciding what drone is right for you.
NASA photo editor Joel Kowsky didn't just capture the solar eclipse from his vantage point in Wyoming, he also managed to capture the ISS buzzing across what remained of the sun.
In these videos, talented photographer and filmmaker Daniel DeArco breaks down several tips that will help flash photography newbies start experimenting with artificial light.
Photographer and master potter Steve Irvine makes incredibly intricate, functional ceramic pinhole cameras that look like robots and monsters.
Chinese gimbal manufacturer Gudsen has released a firmware update for its Moza Air that lets you control the direction and angle of the head remotely just by moving a small handlebar-mounted control unit.
Curious how the Sony a9 performs underwater? Our friends at Backscatter took the camera diving off the Baja California coast, to find out how it handled shooting great white sharks.
While most of the DPReview crew put away our cameras and just watched the celestial event, Rishi decided last-minute to hack together a rig and capture a few shots.
Defunct Russian camera maker Zenit is making a comeback, and they're planning to release a full-frame mirrorless camera in 2018.
The days where you're more or less locked into premium or first-party flash units has gone. They're less than $50 now, so there's one less excuse not to get one. Here's our case for adding one to your kit, and a few pointers to get you going.
If you're shooting the solar eclipse here's a hint: don't fry your camera's sensor. Use a proper solar filter that offers at least 16 stops of light filtration, along with UV and IR filtering. More important? Don't look at it unless you've got solar filters. Sensors can be replaced, your retinas can't.