DNG open or not?

Dan Zemke said:

Barry Fitzgerald -

Eventually, ISO seems like it would be a good choice. But an ISO
working group isn't even likely to be considered until at least one
of the two largest vendors in the market support DNG. IMO,
insisting that ISO take ownership at this stage, even if well
intentioned, pragmatically amounts to a delaying tactic.

On the Microsoft and Open Office format analogy - in my mind there
is a very significant difference. MS Office dominates the market
for PC office software. By opening up their specs and even
providing source code for file access, they remove an objection and
reduce the desire for their customers to go through the pain of
conversion.
In the case of a cross-industry RAW format, there isn't any
dominant standard.

I'm personally disappointed in the lack of leadership exhibited by
Nikon and Canon. Each knows that there is a rapidly growing
interest in RAW shooting by professional and presume photographers,
but neither has stepped up to the plate to do anything about it.
Both continue to cling to their own proprietary formats and "manage
any PR issues" with statements like we don't see the need or the
classic delaying technique of - we're studying it.

Dan

Click to expand...

The crucial difference if you talk about open office, is that Sun has released the code to the community. They cannot withdraw it...developement of OO, is with the team/community, not sun microsystems. Sure maybe it looks good for Sun to do this..and yup they offer a slightly enhanced version at a cheap price. Of course they are trying to dent MS and office, and MS do fear this..they should do.

Until adobe releases a basic free image editor...there is a world of difference.

I agree there is no dominant standard..but that doenst mean adobe should control it. Why should makers bow to adobe? Why doesnt adobe release control to a neutral 3rd party or community..to ensure open developement? They wont do it...

DO you want the same situation we have with MS and office docs..with RAW? No thanks....even MS have had to adopt support for open document, they have no choice. Lets have a transparent format, controlled by a non commercial organisation, with the full support of camera makers..then we are talking business..until this happens..DNG is an Adobe dream. and one which has a commercial long term interest.

If adobe cared so much about users, why then did they gobble up pixmantec and RSE..sure you can download it for a while...until their own lightroom is out for sale..then kiss goodbye to a free RAW convertor...

And make no mistake it will be gone, in the same way Symantec bought out Sygate and their "free firewall" (which happened to be vastly superior to any of the junk Symantec produce)...same story...

Adobe could easily make ACR available as a small download free for users..but they wont..

--

DNG has everything to do with business, not photography or altruism. Adobe is a huge corporation run by suits not photographers, and their focus is profit – as it should be.

The only reason DNG was introduced to prevent Adobe from being locked out of processing raw files from Canon and Nikon, both of which want their users to utilise their own software. Smart move on Adobe’s part.

If you have paid $500 for Photoshop then ACR is free. But if you want to use the later updates to ACR then you must upgrade to CS2 – a good use of dominant market position to force upgrade and maintain income. Of course you could buy Elements for $90 and then you can use all ACR updates. If you don’t want to do this then go through conversion to Adobe DNG, and then you can still use Photoshop CS. A case of heads Adobe win, Tails Adobe win. An excellent text book example of market manipulation.

Another modern classic marketing tool is software piracy. Make no real attempt to protect your software, effectively encouraging piracy. This gets your softtware used by everyone, establishing a standard, then lock the product with activation etc. When Lightroom is launched it will be interesting to see if it requires activation, my guess is not.

If a camera manufacturer wants to lock me into using their, software then I accept this as part of the “package” of facilities offered by the camera hardware / software, when choosing where to spend my money. If I don’t like it I don’t buy it. As regards access to my images in the future then I am already well served by TIFF or JPEG. Before we could post process the raw file, processing took place in the camera and we took our JPEG and stored it for posterity.

--
Ian

Agree that Adobe's motivation is not altruism. I sort of agree with your assertion that if I don't like the totality of what a vendor is offering, I don't buy into their line (much like lenses).

But to me, the point isn't about making a decision on what vendor's system to buy into. It's about an emerging new 12 bit file format that hasn't been standardized yet. No portible container to hold the 12 (or greater) channel bit depth, and no portable way to associate even standard metadata.

What ever emerges as a standard (and there will be one, which incorporates most, but not all of the data of interest) will simplify software tools and facilitate interoperability amongst them.

As a customer, why on earth would anyone not prefer this to a bable of incompatible proprietary formats that add no value.

Dan

IanSeward said:

DNG has everything to do with business, not photography or
altruism. Adobe is a huge corporation run by suits not
photographers, and their focus is profit – as it should be.

The only reason DNG was introduced to prevent Adobe from being
locked out of processing raw files from Canon and Nikon, both of
which want their users to utilise their own software. Smart move
on Adobe’s part.

If you have paid $500 for Photoshop then ACR is free. But if you
want to use the later updates to ACR then you must upgrade to CS2 –
a good use of dominant market position to force upgrade and
maintain income. Of course you could buy Elements for $90 and then
you can use all ACR updates. If you don’t want to do this then go
through conversion to Adobe DNG, and then you can still use
Photoshop CS. A case of heads Adobe win, Tails Adobe win. An
excellent text book example of market manipulation.

Another modern classic marketing tool is software piracy. Make no
real attempt to protect your software, effectively encouraging
piracy. This gets your softtware used by everyone, establishing a
standard, then lock the product with activation etc. When
Lightroom is launched it will be interesting to see if it requires
activation, my guess is not.

If a camera manufacturer wants to lock me into using their,
software then I accept this as part of the “package” of facilities
offered by the camera hardware / software, when choosing where to
spend my money. If I don’t like it I don’t buy it. As regards
access to my images in the future then I am already well served by
TIFF or JPEG. Before we could post process the raw file,
processing took place in the camera and we took our JPEG and stored
it for posterity.

--
Ian

Click to expand...

I agree with what you say...but I dont think adobe "accepts" piracy. It isnt much fun for them to have people steal/use their software..and I bet that lightroom does need activation. not that this will matter, it is well know that PSCS2 and its activation process has been defeated for some time. As will CS3 when it is out...(not that I condone this action of course)

Let us not forget even Corel require activation for PSP and that is a far cheaper software package...PS is damn expensive..they dont want people using it for nothing. But your points about subtle marketing, and pretty much forcing users into an upgrade (ACR element), is a valid one..though many would argue that the cost of upgrade isnt that bad compared to an off the shelf version.

--

I have no illusion that Adobe is promoting DNG because they care so much about thier users.

But you missed my dominant point. Until Canon and/or Nikon get behind a portable RAW format, there won't be one. Do you disagree?

Dan

Dan Zemke said:

I have no illusion that Adobe is promoting DNG because they care so
much about thier users.

But you missed my dominant point. Until Canon and/or Nikon get
behind a portable RAW format, there won't be one. Do you disagree?

Dan

Click to expand...

I already pointed this out. Some seem very happy to overlook this crushing fact....maybe they enjoy converting all their RAW files to DNG...!
--

Barry Fitzgerald said:

Dan Zemke said:

I have no illusion that Adobe is promoting DNG because they care so
much about thier users.

But you missed my dominant point. Until Canon and/or Nikon get
behind a portable RAW format, there won't be one. Do you disagree?

Dan

Click to expand...

I already pointed this out. Some seem very happy to overlook this
crushing fact....maybe they enjoy converting all their RAW files to
DNG...!
--

Click to expand...

Yep, I agree. Nikon and Canon are crushing RAW shooters by not supporting a portable RAW format like DNG.

Dan

All file formats open or not die eventually. Yep even JPG and tiff will die.

The thing is so long as you can run old software (the old raw converters) you will be ok.

Yes universal compatibility would be better, but the fact is its hardly a big deal to add a 10 MP EXE file to every backup media you put raw files on. So long as windows/x86 comp exists that will be ok.

If one day you note a transition to a new paradigm (that is x86/windows is dieing) then you can bet that you and the 10000 other people who are in that boat will be a market exploited by a software company that converts your format into something usable in the future.
--
http://www.flickr.com/photos/bbryce/

Godfrey said:

What, don't you trust Adobe? ];-)

Click to expand...

I trust Adobe to do what is best for their shareholders as is their legal obligation. If you want an open format, use PNG. In conjunction with linux and other tools, the cost is zero, and the risk of corporate shenanigans is also zero. An OS, raw-converter, 16-bit graphics software, etc. will pay for a substantial part of a fairly good new lens instead of paying for gas for Steve Job's jet or what Steve Ballmer buys with his $$$.

PNG is lossless, compressed, and 16-bit-capable. The files are smaller than tiff, and can be displayed in a browser without conversion. If I used Microsoft / Apple and/or Adobe software, I would not be buying a new tripod this week as soon as I decide which one to get...

Here's a png 16-bit file I sharpened automatically using a 'generic' bash script, optimized for 92 dpi. I save my files in RAW and also save an xml file with the file that can easily be converted to any format. If I need to process, I use CinePaint. Total cost to me: $0. Future upgrade costs? $0.

Barry Fitzgerald said:

We see that the pentax raw format is compressed, and the DNG one is
not.

Leaving aside openraw arguments and vested interests...it works out
at a monster 16-17mb for DNG compared to 10mb ish for the pentax
raw.

If someone who is up on DNG can suggest a reason for this file size
difference..I wouild be interested...

Click to expand...

Barry, the reason for the file size difference is as you state: the Pentax PEF raw file format is compressed and the DNG one is not.

Any reasonable lossless compression for a typical raw image will reduce the raw data from a 10 MP camera to something averaging a little less than 10 MBytes, about 9.3 MBytes in my experience. To that, manufacturers typically add a thumbnail of some size (160 X 120 is common), EXIF data, and other metadata as required. So a "proper" losslessly compressed raw file size for a 10 MP camera such as the K10D is something under 10 MBytes.

In previous Pentax PEF formats, they used packed uncompressed raw formats of 12 bits per photosite, or 1.5 bytes each, for a data file size of 15 MBytes for a 10 MP camera. To this they added the usual metadata, etc., plus a maximally compressed (* quality) full size JPEG for another approximate 1.5 Mbytes for a 10 MP camera for a grand total of something like 16.5 MBytes.

The currently implemented DNG format provided by the Pentax K10D is essentially the same as the "old" PEF format, just that it has been reformatted to fit into the DNG 1.1.0.0 version standard as an uncompressed file. Note that this DNG standard includes provision for compression or not and the inclusion of any number of other images and images sizes, including thumbnails. If Pentax had chosen not to embed the full size maximually compressed JPEG and used the Lossless Huffman JPEG compression standard specified as an option in the DNG standard, file size would also have been something under 10 MBytes.

If you are asking why Pentax could produce a small file size in PEF and not in DNG using the available techniques, we are moving into speculation territory as I have not seen the "new" PEF files. But, if you wish, my thoughts are as follows:

a) Pentax may be using a different compression algorithm than the Lossless JPEG one specified by DNG (similar to what Sony does in the Alpha 100) so the compression they use cannot be applied to DNG files.

b) Given that the new PEF files average 10 MBytes for a 10 MP camera, they have either eliminated the embedded maximually full size JPEG, in which case they may as well have used Lossless JPEG which could have been applied to both formats or

c) they still embed the maximally compressed full size JPEG (about 1.5 MBytes) leaving about 8.5 MBytes for the compressed raw data. If the typical compressed file sizes are this small, they most likely do not use Lossless JPEG and must use some other algorithm, and may use something like:

d) Nikon's "virtually lossless compression" which does result in smaller file sizes but loses some resolution in the highlights. However, this could be implimented in DNG using the Linearization table tag so it does not look likely.

e) Or they may use some other more mathematically intensive lossless compression algorithm (perhaps not "virtually lossless") which produces even smaller file sizes but is not compatible with the DNG standard. If implemented in hardware in the PRIME graphics engine, this does not necessarily imply that it would take longer.

I am happy to see that they have finally addressed the issue of raw compression, and will be happy to use the new PEF compressed format in the K10D provided that it does not slow the response speeds of the camera. I would have liked to see a properly implemented compressed DNG format, but understand that it would still have been about 11 MBytes in size if the embedded full size JPEG is a requirement, either for speed of in-camera playback or as a convenience in quick viewing image files. This embedded JPEG is not a requirement for my use, as I have written software to generate a quick full size view of a raw file in about a half second for a 2 GHz PC.

Hope this helps, GordonBGood

PNG is not a RAW image data format. It is an RGB image data format.

It is useless as an "open standard for RAW format image data" on that basis, regardless of what other useful qualities it might have are.

Godfrey

This is a faulty thesis:

IanSeward said:

...
If you have paid $500 for Photoshop then ACR is free. But if you
want to use the later updates to ACR then you must upgrade to CS2 –
a good use of dominant market position to force upgrade and
maintain income. Of course you could buy Elements for $90 and then
you can use all ACR updates. If you don’t want to do this then go
through conversion to Adobe DNG, and then you can still use
Photoshop CS. A case of heads Adobe win, Tails Adobe win. An
excellent text book example of market manipulation.
...

Click to expand...

The reason that later versions of ACR require CS2 is that ACR v3 and later have advanced features that are dependent upon functions resident only in CS2's libraries. ACR v3.x, used in PSE, does not permit access to the advanced features ... the basic features interfaces and dependencies have not changed. CS loads ACR without checking feature requirements based on version, the result is that ACR exits when it tries to access functions in CS that are not implemented. In other words, the reason that ACR v3 and later doesn't work in PS CS is due to technical issues unforeseen when CS was developed.

Remember that CS was the first version of Photoshop that was delivered with Camera Raw included. Photoshop 7 didn't have Camera Raw included at all, it was a $100 optional purchase.

DNG is not some ridiculous conspiracy to lock people into Adobe products. It's an attempt by the market leader in image processing software to define a standard RAW format as standard which will ease working with RAW data for their customers. If you're not an Adobe user and you have other software tools which do what you want acceptably for your needs, lord have mercy, keep on using what you are using and forget about DNG format.

Godfrey

Godfrey said:

The reason that later versions of ACR require CS2 is that ACR v3
and later have advanced features that are dependent upon functions
resident only in CS2's libraries. ACR v3.x, used in PSE, does not
permit access to the advanced features ... the basic features
interfaces and dependencies have not changed. CS loads ACR without
checking feature requirements based on version, the result is that
ACR exits when it tries to access functions in CS that are not
implemented. In other words, the reason that ACR v3 and later
doesn't work in PS CS is due to technical issues unforeseen when CS
was developed.

Click to expand...

You are correct on this Godrey. However, it would have only taken a couple of lines of code to enable ACR 3 to load into CS without enabling the relevant features. Microsoft said IE was imbedded in the Windows operating system until, with enough complaints, suddenly it wasn't.

Godfrey said:

DNG is not some ridiculous conspiracy to lock people into Adobe
products. It's an attempt by the market leader in image processing
software to define a standard RAW format as standard which will
ease working with RAW data for their customers.

Click to expand...

If that is their real intention then why don't Adobe release the source code to the open source community, as people have asked for. This would go a very long way to establishing their stated goal. Kudos to Adobe, it releases them from any further development costs; absolutely no disadvantages for them - except perhaps their future business plans for DNG and related future developments, which only Adobe know about. There is no conspiracy, Adobe is a very professional well run corporation maximising shareholder value by maximising their dominant market position similar to Microsoft. This is good for their shareholderds but may not be good for the general public in the longer term.
--
Ian

Well, I plan to use DNG. Since I only shoot RAW when I think I will need it the size is of no importance. Since I have 5-1/2GB of SD Cards and a 60GB portable hard drive I could careless. Besides it will save me the trouble of having to convert the PEF's to DNG when/if Adobe should update ACR and the Converter to support it.

I think it is interesting that no one has even stopped to think that since the K10D supports DNG that Adobe may not bother supporting its PEF RAW files. I doubt it would happen, but one can never know.

Robert

Adobe said they wouldn't ask for fees and that it would remain free. Should they fall back and start charging it would do more damage to Adobe's reputation and how companies and consumers trust them. I seriously doubt that Adobe would do that.

Now if we were talking Microsoft, well I would just tell everyone to bend over now, something unpleasant is coming.

Robert

Godfrey said:

PNG is not a RAW image data format. It is an RGB image data format.
It is useless as an "open standard for RAW format image data" on
that basis, regardless of what other useful qualities it might have
are.

Click to expand...

My only point was intended to be that it is 'open', which was what the poster was questioning and the discussion evolved (digressed?) into a discussion of formats and business priorities and software costs. If a format is truly 'open', why can't it also be used for RAW data? Doesn't a camera only capture three colours? Is there a 'true', 'open' RAW format? Why couldn't PNG be that format?

As I stated in my post, I keep everything in RAW, and quickly process a corresponding xml file that can be processed into any other format when needed. The original poster wanted to know about being 'open'. For those of us who will not be tied to a proprietary format, I contributed a solution. That is all. I applaud Pentax for any additional formats but I wish they would support a truly open format if, as this thread is showing, apparently does not exist.

IanSeward wrote:
[snip]

IanSeward said:

If that is their real intention then why don't Adobe release the
source code to the open source community, as people have asked for.

Click to expand...

[snip]

What source code?

DNG is a file format specification, not a product.

Dan Zemke wrote:
[snip]

Dan Zemke said:

Eventually, ISO seems like it would be a good choice. But an ISO
working group isn't even likely to be considered until at least one
of the two largest vendors in the market support DNG. IMO,
insisting that ISO take ownership at this stage, even if well
intentioned, pragmatically amounts to a delaying tactic.

Click to expand...

[snip]

Remember that there is already an ISO standard raw file format - ISO 12234-2 (TIFF/EP). It was developed by TC42 WG18, with a draft in 1998, acceptance as an agreed standard in 2001, and scheduled for review 2006.

This reveals something important: ISO, and other standards development organisations, don't react at "digital photography industry" speeds! Adoption by ISO might result in DNG becoming less useful as a working raw file format within years, because it wouldn't adapt fast enough. New versions might be needed every couple of years, then what would happen?

I believe that DNG is in good hands at the moment. It is being taken up at a healthy rate, and criticisms about it not being open appear to be a feature of forums like this, not of the industry as a whole. I see no evidence that take up is being inhibited because Adobe controls the specification, except perhaps by those forum critics! Something to remember is that merely being an ISO standard doesn't force anyone to adopt a specification.

I believe that what ISO should do is develop an ARCHIVAL standard raw file format. That would need a slower rate of evolution. It wouldn't need buy-in from camera manufacturers, (although that would be a bonus), just software developers. There is a need for an agreed archival raw file format, and this would resolve a problem that librarians and archivists have been discussing. (The Library of Congress has already decided that DNG is its prefered raw file format, but others are waiting for more evidence).

I believe that what TC42 WG18 should do is develop DNG/A, an archival standard for raw files, not as a stand-alone specification, but as a commentary on the DNG specification. It might only be (say) 15 pages long, with statements about what is required, what is banned, and conformance rules. This is what ISO did for ISO 19005-1, PDF/A, its archival standard document format. The ISO standard is a 36 page commentary on Adobe's 978 page specification of PDF 1.4.
http://www.barry.pearson.name/articles/dng/speculate.htm#00

Dan Zemke wrote:
[snip]

Dan Zemke said:

But you missed my dominant point. Until Canon and/or Nikon get
behind a portable RAW format, there won't be one. Do you disagree?

Click to expand...

I don't know what you mean by "portable RAW format". (Why "portable"?)

But the benefits of having a common raw file format don't all depend on acceptance by Canon and Nikon. Many people and organisations are already seeing those benefits from DNG.

I think people who believe that DNG somehow relies on Canon and Nikon to support it have too narrow a view of how people can benefit from it. Even users of Canon and Nikon can benefit - and this is a Pentax forum, where many people don't feel constrained by what Canon and Nikon do!

What matters most is software support, not camera support. At the moment, the main obstacles are Bibble and Capture One, because they prevent some photographers adopting a full DNG-based workflow. Capture One should support it in v 4.

Joe Barnhart said:

Certainly in the classic sense of the word. "Open" software has a
process for the software to grow and evolve over time -- a process
that includes everyone who uses the product. Open software
includes packages such as Linux (e.g. Debian distro) and Python.

Click to expand...

This is an argument that says that only "Open source" software is open standards. That's utter rubbish. HTTP is an Open standard. If you're using Internet explorer or Safari to read this you're not using open source software. It's deliberate FUD which comes only from the most bigotted of open source advocates.

Joe Barnhart said:

DNG is "open" only inasmuch as Adobe's current license allows other
companies to use the same format without paying a fee.
Since the
license is owned by Adobe, who exerts sole control over it, it may
be changed at any time to include any provision that Adobe may want
in the future.

Click to expand...

You can publish a protocol , or a file format, or an Application Programming interface and give people a royalty free license to use it. You can't put the genie back in the bottle. You're arguing that Microsoft could come along and charge everyone who wants to write any kind of Window program, or the W3C could charge anyone who wants to write a web browser or server ....

You can only talk about the "current" license and changing it at any time if you haven't read it. Go here, http://www.adobe.com/products/dng/license.html and read it, it isn't in legalease. It says Adobe "grants all individuals and organizations the worldwide, royalty-free, non-transferable, non-exclusive right under all Essential Claims to make, have made, use, sell, import and distribute Compliant Implementations".

THEY CAN'T TAKE THAT BACK.

A 'compliant implementation" reads or writes computer files compliant with the DNG Specification. And the 'DNG specification' is "any version of the Adobe Digital Negative (DNG) Specification made publicly available by Adobe"

All they can do is come up with a future version which they don't publish, and who is going to adopt tat.

Joe Barnhart said:

Don't make the error of believing that the DNG
format is free of license fees forever. If the DNG format takes
off, license fees could be demanded at any time.

Click to expand...

Read the license. One should ask why someone would post a statement which is so blatantly false. Given the posters muddling of open source and open standards, it seems reads like a statement of "Adobe is a comercial organization, and as such is out to screw you"

Joe Barnhart said:

Those of you old enough to remember GIF and the license brouhaha it
created will know what I'm talking about.

Click to expand...

And who uses GIF these days. Everything is JPG.

Joe Barnhart said:

At one time, GIF was
free of license fees and Compuserve encouraged its use. When it
became ubiquitous the license was changed and fees demanded.

Click to expand...

There never was a license which said people could use it.

Joe Barnhart said:

It
threw the graphics market into a turmoil. Only now, after the
patents have expired, is GIF becoming respectable again. The same
could happen with DNG at any time.

Click to expand...

Read the license. When you've read it see if you conclude it says that Adobe may or may not have patents around DNG, but they license them to anyone who wants support DNG (but not for any other purpose.)

They have beem very open, they may have intellectual property claims which relate to parts of DNG. They are not giving those away,

If you look at Adobe's behaviour with PDF, where they have used all kinds of threats to protect their near monopoly in authoring tools, so I don't think they have a right to be automatically trusted.

In case people don't know I work for Microsoft: a company that a lot of people don't trust. It's just as foolish to say a company can NEVER be trusted as it is to automatically trust it - cases should be taken on merit.

DNG is a different case from PDF; and Adobe's behaviour is different.