Reading mode: Light Dark

DNG versus NEF for archiving

Go to last post
egrivel wrote:
[snip]

Rather than comment on your DNG statements, I'll comment on the following about NEF:
egrivel said:
On the other hand, the NEF format is not really a proprietary
format. The way the current Nikon cameras store data in the NEF
file is pretty well documented in the "dcraw.c" program, which is
publicly available. I looked around in that program as well and
found that I could always use that to access my NEF files. As a
software developer, I could always use that to write my own program
to access my NEF files. Of course, non-programmers don't have that
luxury, but there are plenty of free and low-cost programs around
that will support the NEF file format to feel confident of support
for this format in the years to come.
Click to expand...
[snip]

According to the webpage for "exifprobe", NEF is a "TIFF & TIFF- derived "raw"" format. In other words, what you said about DNG pretty well applies to NEF too. In fact, many raw formats are based on TIFF (including TIFF/EP). I suspect that many cameras actually even use the same TIFF tags for some of their data as DNG. The difference is that DNG is openly documented and licensed for use compared with the others.

(I'm not sure whether I am allowed to post URLs to products like exifprobe here, so I'll leave you to investigate further).

Dave Coffin was so enthusiastic about DNG that he restructured dcraw around DNG. However, he only reverse-engineers other formats sufficiently to process the images. Not to extract everything from the raw file, such as extra metadata. So his code is not full documentation of other raw formats, and indeed could possibly have issues that might show up later.

DNG isn't a different sort of thing from other raw formats. It really is "just" a raw format, and one that is now used by a number of cameras. But it has extra metadata to enable different sensor configuations to be described, a good version control scheme, and no unnecessary differences from one camera to another. It holds sensor data in the same way that camera makers use.
 
Robert Fronckowiak said:
Barry, do you work for Adobe?
Click to expand...
No. I simply pay them lots of money for Photoshop! (I'm not sure if I'm allowed to provide a URL here, but you should have no trouble using my name to find my main web site to find out more).

I spent decades helping to design large-scale multi-vendor computer systems. Since getting a dSLR, I have only shot raw. It was absolutely obvious to me last year that a proper raw file format was needed, just by analogy with the standard interfaces in complex computer systems.

When Adobe launched DNG, I spent 2 weeks examining the specifications and everything I could find on the topic. I became convinced that DNG was the sort of specification that photographers needed. So, on 10th October 2004, I downloaded the DNG Converter and started to use it. I have never been disappointed.

There is a sort of "chicken and egg" problem with DNG. If no one uses it, because they are waiting for others to, or are afraid of it, it would fail, and we would be worse off. But if it achieves "critical mass", so that everyone wants to use it because everyone else appears to be using it, and products suppliers jump aboard, raw shooters will ALL benefit in the medium and long term.

(I am a supporter of OpenRAW, which lobbies for openly documented raw formats. But I push a bit harder than some supporters, and believe that a common raw format is also needed, not just open documentation).

I don't want to bully people into using DNG if it isn't suitable for them. And they shouldn't use it if they are uncomfortable with it. But some people are wary because of lack of knowledge, rather than specific problems. I feel I can help by posting what I know. If people still reject it, at least it will be an informed rejection. That is OK - I respect that.
 
Russ Sprouse said:
Barry, I guess I just don't get it. What does DNG do for me that
I'm not getting with my NEF files?
Click to expand...
There are two matters here:

1. What would any individual photographer get from adopting DNG now?

2. What would photographers, and users of photographs, gain, if most/all products used DNG?

In the post your responded to, I was addressing "2". Imagine a world in which, when you bought a new camera, all existing raw-handling products (raw converters, image viewers and management systems, etc) could already support it with proven software. And people developing new products for photographers only had to cater for a single raw format. That is how I want the future to be. (Obviously, things never get that perfect! But things could be an improvement over where we are now).

(When Leica released the DMR digital back, which uses DNG as its raw format, they could immediately be processed In Photoshop CS & ACR 2.4, which was released months before the Leica back. In fact, Leica didn't release their own software with the back, but shipped it with Photoshop Elements 3, saving themselves some work).

The answer to "1" depends on the tools you use and your workflow. If you use tools that don't accept DNG, such as Nikon Capture, then you HAVE to continue to use NEFs, of course. You may decide to archive using DNG for extra security, or you may have confidence that you will always be able to access your NEFs later. I would never recommend to anyone that they use DNG without understanding their workflow. In fact, DON'T use it unless you are confident enough!

But I am at the other extreme. In spite of the thread here, I actually use Pentax & PEFs, not Nikon. I use Photoshop CS2 & ACR 3.2. So, by converting to DNG directly from the card, I get the advantage of raw files that are less than half the size, and ACR (since 3.1) offers the option of holding its settings (non-destructively) in the DNG files instead of in sidecars. (Nikon Capture can save settings in NEFs. The Pentax software can't do the same in PEFs).

People who use just Canon, or just Nikon, and who don't want to use ACR 3.x to save settings in the file, tend to have least to gain from switching to DNG. Some professionals who use Canon or Nikon and ACR 3.x use DNG because it enables them to save their settings in the file, but they either archive DNGs with embedded NEFs, (an option that enables them to extract the original NEF if they ever need it), or they archive both NEFs & DNGs. People who use cameras of more than one manufacturer may use DNG to rationalise their workflow.

Some people are switching to DNG because they want to avoid upgrading from Photoshop CS to CS2, yet they have a new camera. (D2X, D70s, 350D, etc). So they convert their NEFs or CR2s to DNGs, (using the 3.1 or 3.2 converter), and these can then be handled by CS/ACR 2.4.

DNG is a year (+ 2 days) old, and we are really just getting a glimpse of the advantages. Niche camera makers are the first to adopt it (Leica, Hasselblad-Imacon, Ricoh, Samsung). A few major raw converters either don't yet support it or only do so tentatively. Some products may actually support it but don't know it, because many products use Dave Coffin's dcraw, which accepts DNG. It is a bit of a mess! Canon and Nikon may be the last to support it, in that order. (Some people would like their camera to add it in as an option, even if they don't use it as their native format. I would happily forego TIFF support in favour of DNG).

If you search for "Products supporting DNG" you will see that it had made useful progress in its first year, but there is a long way to go.
 
I would never archive in DNG over NEF. What is the point? Who knows what information might be lacking in the DNG over the original NEF? Even if the claim is that no info is lost, I still wouldn't be convinced (for a start, ACR ignores most of the info in the NEF, so who is to say that this info is not lost or compromised in the conversion to DNG?). One of the points in shooting in the NEF format is that you have the maximum information available to work with at a later date, so why risk throwing some of that away? The NEF format is here forever, and any fears of not being able to access your image later are completely unfounded IMHO. Already people have reverse engineered the format, and should Nikon go belly-up (which won't happen, buf if...), then that won't mean that full details of the NEF format will die with them. There are just too many NEFs out there for that to happen. If I need my NEFs in DNG for whatever reason, I'll convert them to DNG when I need to (retaining the originals)... why do it now?

Mike

--
http://www.threeboks.com/
My smugmug coupon code: lcvdCTMPuQYro
 
... to piggyback on what Barry has stated, there are some people (like me) that must process RAW files from varying sources (Nikon, Canon, Pentax, Leica, Hasselblad, etc). for them, DNG introduces a consistancy to their workflow. this applies to publishers, libraries, image banks, and any other entity that must archive their assets. for these groups, the advantages are clear:

consistancy in production
consistancy in optimizing file attributes
consistancy in format

this may not affect you now, in which case there may be no burning need to adopt DNG. but as some camera manufacturers have stopped supporting earlier RAW formats, how can one be sure the next one, two, or three generations of OS's later you'll still be able to open up your old files? are you prepared to archive your current applications, OS, and hardware in perpetuity?

DNG is an open format. NEF, ORF, CRW, etc are not.

if you're not concerned about any of this, then there is no compelling reason to insure your RAW negs by way of converting them to DNG. and that's ok too. i view DNG as a way of hedging my bets that today's photos will be available for production both today and tomorrow. some camera manufacturers are recognizing that too.

the born 2 design
design guy
 
drmrbrewer said:
... for a start, ACR ignores most of the info in
the NEF, so who is to say that this info is not lost or compromised
in the conversion to DNG?
Click to expand...
er, it doesn't ignore the data. it just can't be found because that information hasn't been standardized, or revealed. this is one of the hazards of keeping proprietary formats so close to the camera maker's chest.
drmrbrewer said:
One of the points in shooting in the NEF format is that you have the maximum information
available to work with at a later date, so why risk throwing some of that away?
Click to expand...
that's assuming all NEF's continue to be supported. you could have maximum data...or you might have none. for those who are concerned, DNG has the capability to buy you extra insurance by way of embedding the entire file inside itself so that if the location of the information is ever revealed by the camera makers, or found by way of hacking, it can be plucked out. DNG's can even be disassembled to give you back your original RAW file should you ever need it.

an NEF file is the original artifact-- while DNG is your insurance. whether you feel you need to insure is up to you. if you aren't a commercial photographer, this issue might never affect you.
drmrbrewer said:
The NEF format is here forever, and any fears of not being able to
access your image later are completely unfounded IMHO.
Click to expand...
forever? that's a long time. do you mean 10 years? one hundred years? one thousand years? there is no NEF format. there are only NEF formatS. which are you referring to? the one for slide scans, or ALL the others for each camera Nikon ever made? are there any other programs which recognize Nikon scan NEFs? who is ready to gamble that all future production tools will recognize every flavour of NEF--or perhaps only a few popular ones? who will care about 6MP files in a world of 120MP files?
drmrbrewer said:
Already people have reverse engineered the format, and should Nikon go
belly-up (which won't happen, buf if...), then that won't mean that
full details of the NEF format will die with them.
Click to expand...
you are only partially correct. it's only the image data that has been reverse engineered and not the maker notes (i.e. GPS data, etc). the reason for this is that the maker notes are well hidden, if not encrypted, inside some proprietary formats. also, how will that data be retrieved? will those reverse engineered algorithims render the files into fully expanded TIFF's, JPEG's etc, or save them to a "common open RAW format"--such as DNG-- so they can continue to be processed with RAW tools like ACR before production?
drmrbrewer said:
There are just too many NEFs out there for that to happen. If I need my NEFs
in DNG for whatever reason, I'll convert them to DNG when I need to
(retaining the originals)... why do it now?
Click to expand...
perhaps. that's your choice. but if you're a commercial vendor, a publisher, an archivist or other person of inherent responsibility, you'd probably prefer to purchase your hurricane insurance before it hits you.

DNG is not a necessity. it's an insurance policy that can be amortized cost-effectively into your workflow before something is lost. it's up to you how much you value your work.

the born 2 design
design guy
 
Barry and nunatak:

Thank you both for your thoughtful and detailed responses.

I can see advantages to DNG for larger operations - commercial shops with multiple photographers, multiple camera brands, etc.

At this point, as an independet free-lance photographer with only one brand of digital camera, I don't see any advantage short- to mid-term. If something changes that creates a compelling advantage, I will convert at that time.

Maybe when I'm laying on my deathbed, I'll feel really bad that I didn't "preserve" my images by getting them into DNG format. But I'm just not worried about it now.

Enough flogging of the expired equine. Anybody wanna go on a shooting outing?

-- Russ
 
nunatak said:
er, it doesn't ignore the data. it just can't be found because that
information hasn't been standardized, or revealed.
Click to expand...
If it can't be found, surely it can't be included in the DNG file in any meaningful way? I'd rather leave it in the NEF file where it belongs and where at least somebody (i.e. Nikon) knows what to do with it.
nunatak said:
that's assuming all NEF's continue to be supported. you could have
maximum data...or you might have none.
Click to expand...
Erm. NEFs will continue to be supported. If not, there will be such a clamour that Nikon will be forced to reveal all to a third party who can support it. In any case, there will still be a version of Nikon Capture 4.3 that will work, or be made to work with an emulator on whatever platform there is in years to come.

The bottom line is... in the unlikely absence of any support or information from Nikon, you'll be able to reverse-engineer the NEF format(s) in 100 years' time just as well as (or better than) Adobe can today to produce their NEF-> DNG converter, so what's the point in doing producing a poor man's reverse-engineered NEF (i.e. DNG) file now?

And Nikon have a publically-available SDK that allows anyone to read everything in the NEF file to their heart's content.

I just don't get it.
nunatak said:
DNG has the capability to buy you extra insurance by way of
embedding the entire file inside itself
Click to expand...
Excellent. I've always wanted to store data twice, unneccessarily.

Mike

--
http://www.threeboks.com/
My smugmug coupon code: lcvdCTMPuQYro
 
drmrbrewer said:
If it can't be found, surely it can't be included in the DNG file
in any meaningful way?
Click to expand...
it can be entirely embedded and excavated at some future date-- if, that is, the information can be pried from nikon's cold-dead-hands.
drmrbrewer said:
Erm. NEFs will continue to be supported. If not, there will be
such a clamour that Nikon will be forced to reveal all to a third
party who can support it.
Click to expand...
previous clamours have not always resulted in success. that would lead one to conclude that the customer does not always come first.
drmrbrewer said:
In any case, there will still be a
version of Nikon Capture 4.3 that will work, or be made to work
with an emulator on whatever platform there is in years to come.
Click to expand...
i'm highly skeptical of this, but if this were overlooked for the moment ... what would it be converted to ... a processed file? nikon doesn't yet support open RAW formats that software of the future can and will process. RAW processing software will be as much improved and optimized as codecs have become.

an "already processed" file will have significantly lesser value than one that can be reprocessed indefinitely.
drmrbrewer said:
And Nikon have a publically-available SDK that allows anyone to
read everything in the NEF file to their heart's content.
Click to expand...
they have published a black box. no more, no less. support for that black box is as scarce as support for a nikon scan NEF -- and/or hen's teeth. :-)
drmrbrewer said:
Excellent. I've always wanted to store data twice, unneccessarily.
Click to expand...
forgetting for the moment that it needs to be retrieved in some useful format, perhaps you don't value your RAW data enough to back it up. that's fine ... but i do.

as previously stated, the value of DNG depends on your workflow and how much you value your data. while you may suppose access to all NEFs will be infinitely available, others need greater surety. DNG is a form of surety and it comes in very cheap--actually for free.

the born 2 design
design guy
 
nunatak said:
nunatak said:
If it can't be found, surely it can't be included in the DNG file
in any meaningful way?
Click to expand...
it can be entirely embedded and excavated at some future date-- if,
that is, the information can be pried from nikon's cold-dead-hands.
Click to expand...
Sounds a bit messy to me. So you extract things you understand and write them to one part of the DNG. Then extract other bits you don't understand and write them to another part of the DNG. Then later on Adobe figures out how to understand part of the latter part, so you run another converter over your converted file to extract that bit and write it to a new part of a new file that... geez I could go on, but frankly I value my raw data enough to mess with it in that way.
nunatak said:
nunatak said:
Erm. NEFs will continue to be supported. If not, there will be
such a clamour that Nikon will be forced to reveal all to a third
party who can support it.
Click to expand...
previous clamours have not always resulted in success. that would
lead one to conclude that the customer does not always come first.
Click to expand...
Previous clamours have not been as a result of Nikon withdrawing support for their own raw format (like that would happen).
nunatak said:
nunatak said:
Excellent. I've always wanted to store data twice, unneccessarily.
Click to expand...
forgetting for the moment that it needs to be retrieved in some
useful format, perhaps you don't value your RAW data enough to back
it up. that's fine ... but i do.
Click to expand...
My NEFs are safely backed up, thank you. I don't count backing up as being unneccessary.

I note that you ignored my point about a DNG produced today from a NEF being a mere reverse-engineered version of that NEF, nothing more, nothing less. You can reverse engineer that same NEF in the same way in 10 years time to produce the same DNG file, only in 10 years time you can probably do it better (or Nikon may have decided to produce their own, perfect, NEF> DNG converter, wouldn't that be gutting for you?). So why do it now, and throw away the original NEF? I'm sorry, if that's an insurance policy, I'm not buying.

Mike

--
http://www.threeboks.com/
My smugmug coupon code: lcvdCTMPuQYro
 
drmrbrewer said:
Sounds a bit messy to me. So you extract things you understand and
write them to one part of the DNG. Then extract other bits you
don't understand and write them to another part of the DNG.
Click to expand...
that's not the way the spec works. it's much simpler. a full non-lossy (compressed) version can be stored along with the extracted data if that's what you need or want. no fuss--no muss. whether you choose to do this depends entirely on your production workflow and how much you value your files.
drmrbrewer said:
Previous clamours have not been as a result of Nikon withdrawing
support for their own raw format (like that would happen).
Click to expand...
neither the Canon nor Kodak communities have been successful in having older RAW formats supported with newer software. it's wishful thinking that every NEF format ever produced will be supported indefinitely--may i remind you that you have ignored my point about nikon scan NEFs.

not everyone needs to be on board, or should be if it doesn't suit them. conversley, it wasn't long ago that people (in this same forum) could fathom using RAW over JPEG. there were countless threads entitled RAW or JPEG? today it's DNG or NEF?

open standards are to the benefit of everyone--except those that profit by impeding it. barry was correct in asserting it's a chicken and egg scenario. many people, who are the only ones who will ever touch their files, will see no necessity.
drmrbrewer said:
My NEFs are safely backed up, thank you. I don't count backing up
as being unneccessary.
Click to expand...
necessity is relative. for entities that have responsibilities beyond today, surety is a necessity.
drmrbrewer said:
I note that you ignored my point about a DNG produced today from a
NEF being a mere reverse-engineered version of that NEF, nothing
more, nothing less.
Click to expand...
do you? i thought i've explained how a DNG can embed the entire RAW file (e.g. NEF) into a compressed non-lossy format. if i've misunderstood please elaborate.
drmrbrewer said:
or Nikon may have decided to produce their own, perfect, NEF> DNG converter,
wouldn't that be gutting for you? ...
Click to expand...
not at all ... I'd welcome that development. however as I live in the here and now, and we've witnessed the proliferance of new and undocumented formats for every camera, some deliberately fabricated to make integration with other products more difficult, i'm not optimistic about this scenario.
drmrbrewer said:
So why do it now, and throw away the original NEF?
Click to expand...
who said throw away the original NEF? that's entirely your choice. how you setup your workflow depends very much on how important integration is to your photography. there are still a lot of people shooting JPEGs.

the born 2 design
design guy
 
nunatak said:
drmrbrewer said:
Sounds a bit messy to me. So you extract things you understand and
write them to one part of the DNG. Then extract other bits you
don't understand and write them to another part of the DNG.
Click to expand...
that's not the way the spec works. it's much simpler. a full
non-lossy (compressed) version can be stored along with the
extracted data if that's what you need or want. no fuss--no muss.
whether you choose to do this depends entirely on your production
workflow and how much you value your files.
Click to expand...
So if you value your files, you basically embed the original NEF within the DNG, is that what you're saying? Why not save space and just keep the NEF, without also storing Adobe's current guess at what they think is in the NEF?
nunatak said:
nunatak said:
Previous clamours have not been as a result of Nikon withdrawing
support for their own raw format (like that would happen).
Click to expand...
neither the Canon nor Kodak communities have been successful in
having older RAW formats supported with newer software. it's
wishful thinking that every NEF format ever produced will be
supported indefinitely--may i remind you that you have ignored my
point about nikon scan NEFs.
Click to expand...
When Nikon withdraw support for D70 NEFs and burn all details of the D70 NEF format and destroy all the companies that bought their SDK and rid the web of applications that can display and tweak NEFs using either the official SDK or some clever reverse engineering of their own, then I will gladly convert my NEFs to DNG. I don't see any point in doing so before that unlikely concurrence of events.
nunatak said:
open standards are to the benefit of everyone
Click to expand...
I'd welcome a standard RAW format, but until that happens, I'm not going to try and hack my round pegs into square ones so that they can fit into square holes. I'll just keep my round pegs intact, thank you, and see what the future holds.
nunatak said:
nunatak said:
I note that you ignored my point about a DNG produced today from a
NEF being a mere reverse-engineered version of that NEF, nothing
more, nothing less.
Click to expand...
do you? i thought i've explained how a DNG can embed the entire RAW
file (e.g. NEF) into a compressed non-lossy format. if i've
misunderstood please elaborate.
Click to expand...
No, that was perfectly clear. It just doesn't make sense to do that.
nunatak said:
nunatak said:
So why do it now, and throw away the original NEF?
Click to expand...
who said throw away the original NEF? that's entirely your choice.
Click to expand...
So given the fact that you are, it seems, not entirely trusting of the DNG version of your NEF, you would store both? If you can easily and automatically produce a DNG from a NEF at any time you want, why store both the DNG and the NEF? The original poster was talking about archiving NEFs vs archiving DNGs. Why in the world would you ever archive a DNG as well as or instead of a NEF?

Mike

--
http://www.threeboks.com/
My smugmug coupon code: lcvdCTMPuQYro
 
drmrbrewer said:
So if you value your files, you basically embed the original NEF
within the DNG, is that what you're saying? Why not save space and
just keep the NEF, without also storing Adobe's current guess at
what they think is in the NEF?
Click to expand...
contrary to the spirit of how this discussion started, you're framing it to how you interpret your own immediate needs. you say if the sky is blue--what hurricane? therefore your argument becomes more narrowly focused and rhetorical as you exclude the need for continuity in archiving, production, and collaboration.

adobe does a pretty good job of extracting the data to be used, as does DxO (which also saves to DNG format). the only thing missing is some of the metadata nikon doesn't want you to share. i suspect they don't want you to know that's all that's missing (admittedly conjecture on my part).

perhaps you should clamour about that as it's quite possible to obtain superior results when programs like DxO and ACR work together in a common portable format. they can only do this through DNG--not NEF. if your workflow depends on superior imaging, and continuity in production, you use the tools that best serve a greater purpose.

NEF's are loners that don't work well with collaborative processing methods.
drmrbrewer said:
When Nikon withdraw support for D70 NEFs and burn all details of
the D70 NEF format and destroy all the companies that bought their
SDK and rid the web of applications that can display and tweak NEFs
using either the official SDK or some clever reverse engineering of
their own, then I will gladly convert my NEFs to DNG. I don't see
any point in doing so before that unlikely concurrence of events.
Click to expand...
your data may not be valuable enough to react until the horse has already bolted out the barnyard door. others feel it's important enough to insure their data before that happens. you may gamble with your livlihood--others will not.
drmrbrewer said:
I'd welcome a standard RAW format, but until that happens ...
Click to expand...
it has happened. it's called DNG. you don't sound very welcoming. others may come in time.
drmrbrewer said:
No, that was perfectly clear. It just doesn't make sense to do that.
Click to expand...
once again, that's from a narrower perspective.
drmrbrewer said:
So given the fact that you are, it seems, not entirely trusting of
the DNG version of your NEF, you would store both?
Click to expand...
it really depends on the criteria for archiving. there are different levels of surety and portability. DNG offers the best of both worlds. NEF is tied to nikon's SDK -- which can't even recover highlight data.

the born 2 design
design guy
 
Cutting to the chase... my basic point is simple: if you can easily and automatically produce a DNG from a NEF at any time you want from now to eternity, why archive the DNG and the NEF, and especially why archive the DNG instead of the NEF? If the DNG project is so open and transparent, they're hardly going to withdraw the ability to convert D70 NEFs to DNG in X number of years hence. If you can do it now, you sure will be able to do it in 100 years' time. If, in 50 years' time, you have an application that can understand DNGs but not NEFs, just get that NEF out of your archive, run it through the DNG converter (which, by then, will take a nanosecond), and use it as a DNG. No horses will bolt. No options are closed. Archiving a DNG as well as or instead of a NEF just makes no sense to me. But it does to you. Fair enough. You can have the final word.

Mike

--
http://www.threeboks.com/
My smugmug coupon code: lcvdCTMPuQYro
 
You must log in or register to reply here.

Keyboard shortcuts

f
Forum
m
My threads
Mobile site
About
Editorial content
Cameras & Lenses
Community
All content, design, and layout are Copyright © 1998–2025 Digital Photography Review All Rights Reserved.
Reproduction in whole or part in any form or medium without specific written permission is prohibited.
When you use DPReview links to buy products, the site may earn a commission.
©GPS Media - Guides, Products, Services.
Back
Top