What is Future of DNG File Format due to Shifts at Adobe

[theswede #100093]

There are also no limitations stopping, eg, a NEF file, from containing a full camera profile and sensible default values (and documentation by Nikon how to read them) leading to as good results as the default settings in a typical raw converter which contains a dedicated profile for the camera used.

With the rather huge difference that if you have a nice third party software you like which happens to be a few years old it can't open the NEF file, whether or not it has embedded profile information.

It can open the DNG file, and if the DNG file has no profile information you can quickly use an optimization script to get an image you like out of it.

Thus, if my grandmother had wheels she'd be a wagon. You seem to have a problem distinguishing between what is possible in principle and what is reality.

You're the one who set up the scenario. Now you're running around with the goalposts. I have consistently, from the get go, stated that it's possible but not necessary to embed profile information in a DNG to make a whole hog switch to that format worth our while. The more information it contains the better, of course, but the premise in no way stands and falls with embedded profiles. They're just a bonus.

Except for Adobe where you users are given a tool to match colours automatically in order to create a full conversion profile with relative ease, this is normally next to impossible because trying to match all colours on a test chart manually with only the usual controls like WB (temp and tint), tone response curve, saturation and individual colour adjustments is just not possible.

And who's decided that those are the only tools you're allowed in a raw conversion software? And Adobe still provides that tool, right? What's to stop every photo interested coder on github from making similar functionality? Right now the horror of multiple raw formats are stopping them.

You'd have to match potentially dozens of colours with dozens of sliders. That is an optimisation problem that is simply impossible to solve by hand.

Take a photo of a reference chart. Click the button "optimization script" which will be one of the first functions available in the dozens (if not hundreds) of programs that will pop up when there is only one raw standard used by everyone. Heck, I can make such a script today - except there is absolutely no point, since there are hardly any cameras where raw software can open the file but have no profile - because the formats keep changing all the time.

Rational includes reasonable, which in opposition to my arguments yours have not been.

If that's how you feel I can't change your mind. But do try to come up with some rational arguments, it would be wonderful to see where that takes us.

Jesper

 

[Andy Hewitt]

theswede wrote:

So, there's no choice of 'progressive' or 'standard' compression then?

No.

Well there is in the software I've used! When saving a JPEG you can choose either.

Which is what most people use if they want highest quality images. Or of course there's JPG2000 .
It's not what people should use if they want highest quality images. It's what people should use if they want maximum portability. You should not use TIFF, except to get images into whatever you're going to edit with. If you're just after high quality save it as PNG, or if you're editing as PSD or XCF depending on your preferred poison.

Hmm, having looked that up, I agree. However, I also see a note that PNG doesn't support EXIF.

JPG2000 suffers from splintered standard (it's a lot like TIFF in this regard), high processing demands, suspected submarine patents and small gains over well established and much less complex standards. It'll never catch on.

The comment about JPG2000 was tongue in cheek.

Hmm apart from this quote from the Wiki page on Raw files:

"Several major camera manufacturers, including Nikon, Canon and Sony, encrypt portions of the file in an attempt to prevent third-party tools from accessing them"

A fancy way of saying they're scaling the parameters oddly. It's no big deal to figure out

Hmmm, how do you know?

Providing the conversion is actually clean, and works properly in all software - if you read the link I posted about Aperture and DNGs, you'll see that this is not always the case.

Considering that there's be thousands (at least) applications which can handle raw files if they were a single standard it doesn't much matter if a few of them are broken. They'll get lost in the dust of the best few hundred soon enough.

You're still thinking in artificial scarcity created by ever changing raw formats creating a limited playfield. Standardized raw would make mincemeat of Aperture, or force it to get its act together.

No, I'm just indicating that there are optional ways to save the format, which can still cause problems in some software.

To me, that is not yet a true 'standard', as there are ways to save a file that cannot be opened by software that supports DNG files.

The usefulness of raw images would become much greater, and that would benefit every raw image ever taken which can be converted to DNG - which is almost all of them.

No, it isn't. For example, I could not convert my Olympus images to DNG and continue using them in Aperture, as it does not support the format that they're converted to.

This is a problem with Aperture. Not with DNG. And definitely not if DNG was the standard for all cameras and software going forward.

I'd say it's a bit of both. There are two ways to save a DNG file during conversion, one is more 'standard' than the other. This is forced on the user by the Adobe software, and is not, as I see it, forced because of the rules of DNG.

Strange as it seems, I do agree with you that a single image format would be much better, but I disagree that it wouldn't affect existing Raw users.

Oh, it would affect them all right! It would make their lives so much easier and better in the digital dark room it's almost impossible to imagine!

Apple would hate it though. They'd be forced to fix Aperture.

Like noir, I think we're slightly at cross purposes in this discussion. I don't disagree with you at all in the concept of a single standard file format for photos. However, in the real world, this just doesn't happen, and never has really. Someone will always come up with a new standard. The fact we have so many image file formats means there is no such thing as a 'standard' anyway, just look at the number of file formats that Graphic Converter handles, and the variations that are possible with some of them.

And indeed, the fact that Graphic converter still supports and converts file formats that existed almost 30 years ago, means that the worry about obsolescence is pretty much unfounded.

Even if Raw was no longer being supported tomorrow, we still have computers and software that can open them now, and will continue to work for many years - I still have a 1989 Mac here that could open old files if I really needed it to.

The trouble with a 'standard' is that it can also stifle progress too, and prevent, or at least delay, progress into something better.

 

[theswede #100093]

Well there is in the software I've used! When saving a JPEG you can choose either.

There is no difference in what compression algorithm or storage format is used. The same software can read it in the same way. JPG is very good this way, and that is one reason it is so popular.

Hmm, having looked that up, I agree. However, I also see a note that PNG doesn't support EXIF.

PNG supports metadata, but there is no standardization. EXIF is not quite standardized either though, beyond basic information (which PNG also has). That's one area PNG lags behind.

The comment about JPG2000 was tongue in cheek.

My answer was with a straight face, but my tongue wasn't really out of my cheek either. ^^

Hmmm, how do you know?

Mostly because raw software makers have reverse engineered the values. But it's easy enough to look inside a few files with varying values and see that there's no real encryption going on, just values that seem out of whack.

No, I'm just indicating that there are optional ways to save the format, which can still cause problems in some software.

There is no software without bugs. The best way to solve this issue is to open the door for more competing software, weeding out the incompetently implemented ones.

To me, that is not yet a true 'standard', as there are ways to save a file that cannot be opened by software that supports DNG files.

If I stream /dev/random to a file and call it "test.dng" it can't be opened. Failure to create files according to spec are not failures of the spec. They're bugs.

I'd say it's a bit of both. There are two ways to save a DNG file during conversion, one is more 'standard' than the other. This is forced on the user by the Adobe software, and is not, as I see it, forced because of the rules of DNG.

Are you referring to raw vs linear? Basically linear is a "half cooked" raw, but neither is more or less standard than the other. This is as close to variants DNG gets, but the difference in reading them is small and in sane implementations (which it seems Aperture isn't? I don't know as I use LR and Ufraw) have no issues at all. And the standard is clear on how DNG works; there is no room for interpretation there.

However, what the raw converter does with the read raw data from the file is of course not standardized, or there would be no use for different raw converters. And some are better than others at handling variety. This is something which would vastly improve if the ecosystem got more diverse.

Like noir, I think we're slightly at cross purposes in this discussion. I don't disagree with you at all in the concept of a single standard file format for photos. However, in the real world, this just doesn't happen, and never has really. Someone will always come up with a new standard. The fact we have so many image file formats means there is no such thing as a 'standard' anyway, just look at the number of file formats that Graphic Converter handles, and the variations that are possible with some of them.

Some image formats are a lot more common than others though. JPG is the king when it comes to photographs (and many other types of images, whether they're suited for it or not) and PNG has come very strongly since it was introduced. Unfortunately Adobe are not trusted enough for DNG to take off properly. It takes "critical mass" for formats to become de facto standards, and so far that hasn't happened to any "uncooked" format.

And indeed, the fact that Graphic converter still supports and converts file formats that existed almost 30 years ago, means that the worry about obsolescence is pretty much unfounded.

There are no proprietary raw formats which are 30 years old. And most converter software can't handle raws anyway. That's the change I would love to see come about; that raw becomes just another image format.

Even if Raw was no longer being supported tomorrow, we still have computers and software that can open them now, and will continue to work for many years - I still have a 1989 Mac here that could open old files if I really needed it to.

Obsolescence is not the large worry. Lack of application ecosystem is. There are what, a baker's dozen raw converters actively maintained? That's way too few to get much progress going, especially since they're generally special purpose and lacking the most powerful image manipulation tools, only allowing sophisticated editing after locking the conversion down.

That's very last millenium. We can do a lot better, but it takes competition to reach there.

The trouble with a 'standard' is that it can also stifle progress too, and prevent, or at least delay, progress into something better.

Image capture using bayer arrays is well understood. There is very little to improve on the current raw formats, if anything. DNG already allows for much larger capture latitude than cameras will get in many years. The reason for proprietary raw formats is not superiority or "special sauce" tags with magic information, but a quaint desire to lock the consumer in.

Of course the file format does no locking in. Lenses and familiarity with the cameras does that. Raw formats are reverse engineered rapidly anyway, and camera makers want third party software to support their formats as it increases the appeal of the system.

Thus, the logical next step would be a unified raw format without weird scalings, allowing for a larger raw converter ecosystem. This would be excellent for the camera manufacturers. But trying to make a big corporation make such a change is like trying to bootleg turn an oil tanker. If it happens it will be because someone of vision and near psychotic strength (like Steve Jobs) gets in charge of a camera manufacturer. Thus, not likely. But I can dream!

Jesper

 

[noirdesir]

theswede wrote:

There are also no limitations stopping, eg, a NEF file, from containing a full camera profile and sensible default values (and documentation by Nikon how to read them) leading to as good results as the default settings in a typical raw converter which contains a dedicated profile for the camera used.

With the rather huge difference that if you have a nice third party software you like which happens to be a few years old it can't open the NEF file, whether or not it has embedded profile information.

That assumes that the NEF format from today is different from the one a few years ago. But we are digressing, there is little point in discussing what the DNG format could be or do or what the NEF format could be or do, what counts is what they actually do in today's reality

And your claim was that the DNG file as saved by cameras contains everything raw converters need and that the whole 'supporting new cameras updates' (for cameras which save as DNG) is a charade.

It can open the DNG file, and if the DNG file has no profile information you can quickly use an optimization script to get an image you like out of it.

Can you show me where I could find such a script for Aperture (Apple does not explicitly support any Ricoh camera)?

Thus, if my grandmother had wheels she'd be a wagon. You seem to have a problem distinguishing between what is possible in principle and what is reality.

You're the one who set up the scenario. Now you're running around with the goalposts. I have consistently, from the get go, stated that it's possible but not necessary to embed profile information in a DNG to make a whole hog switch to that format worth our while.

You have also stated that all these things are already embedded in the DNG files generated directly by cameras. So, which is it: Do the Leica, Ricoh, etc. DNGs have all the necessary information or would just be possible for them if they wanted to?

Except for Adobe where you users are given a tool to match colours automatically in order to create a full conversion profile with relative ease, this is normally next to impossible because trying to match all colours on a test chart manually with only the usual controls like WB (temp and tint), tone response curve, saturation and individual colour adjustments is just not possible.

And who's decided that those are the only tools you're allowed in a raw conversion software?

Because these are the only tools most raw converters expose to the user.

And Adobe still provides that tool, right? What's to stop every photo interested coder on github from making similar functionality? Right now the horror of multiple raw formats are stopping them.

As long as raw converters don't have the option to use this kind of information (exception Adobe) it doesn't matter whether this data are available or not.

You'd have to match potentially dozens of colours with dozens of sliders. That is an optimisation problem that is simply impossible to solve by hand.

Take a photo of a reference chart. Click the button "optimization script" which will be one of the first functions available in the dozens (if not hundreds) of programs that will pop up when there is only one raw standard used by everyone.

Since essentially all raw formats have been successfully reverse-engineered in the open-source community (dcraw) why aren't these scripts already here?

Rational includes reasonable, which in opposition to my arguments yours have not been.

If that's how you feel I can't change your mind. But do try to come up with some rational arguments, it would be wonderful to see where that takes us.

Show me anything that proves that the existing DNG files created by cameras result in the exactly the same conversion before and after the raw started to officially support by a raw converter.

And I'm afraid, any conspiracy theories (eg, Adobe holding back support for marketing reasons) by default fall into the 'unreasonable' bucket. Because almost all of them aren't true. And something that is unreasonable is not a rational argument for me.

 

[theswede #100093]

That assumes that the NEF format from today is different from the one a few years ago.

It is. That's why older software can't open newer camera files. It's enough that the number of MP in a camera differs from what was available previously and the file format has to be reverse engineered since the dimensions are not provided in an openly specified place in the file but usually have been reverse engineered.

Software which is licensed and has access to the trade secret spec, or Nikon's own software, could ignore these difficulties of course. In practice that doesn't seem to happen.

But we are digressing, there is little point in discussing what the DNG format could be or do or what the NEF format could be or do, what counts is what they actually do in today's reality

We're discussing a hypothetical development in what DNG is used for. But sure, we can stick to what the format contains today. Doesn't change what becomes possible if the format becomes the new standard from all (or even several) major camera manufacturers.

And your claim was that the DNG file as saved by cameras contains everything raw converters need and that the whole 'supporting new cameras updates' (for cameras which save as DNG) is a charade.

If that had been my claim it would indeed have been.

Can you show me where I could find such a script for Aperture (Apple does not explicitly support any Ricoh camera)?

No. No-one cares enough about proprietary software and closed formats to put that amount of effort in. Nor do I, I wouldn't waste my time writing such a script for Aperture. Even if I was an Aperture user and owned the camera I wouldn't. I'd just download someone else's profile or convert the manufacturer provided one. That is, if Aperture will allow me to use it and have an open format for it, something I don't know nor care enough about to find out.

You have also stated that all these things are already embedded in the DNG files generated directly by cameras.

I have stated that if the camera manufacturer wants to, they can embed color profiles in the DNG files. I have also stated it is in their interest to do so since optimal support for third party converters increases the appeal of their system.

Since very few manufacturers do what's best for them this information is usually missing these days. Like with so much else in the world of consumer technology it will take a visionary paving the path for the others to follow.

So, which is it: Do the Leica, Ricoh, etc. DNGs have all the necessary information or would just be possible for them if they wanted to?

Haven't examined the DNG files from these makers in detail so I don't know. And frankly, since the question of universal acceptance of DNG is academical at this point in time I have no desire to find out. It suffices that the standard allows it, and if the data is present it's a boon to both manufacturers, image conversion developers and customers.

When DNG files become the standard for camera output I will of course revisit this issue and (if no-one beats me to it) make a detailed list of which camera manufacturers include what in their files.

Because these are the only tools most raw converters expose to the user.

And who has decided that this will be what happens when there is a use for other tools exposed to the user? You have already thought of a need for more. If a standard format removes the current extreme barrier of entry to create image converters accepting raw, why would everyone just blindly copy the most limited of today's tools?

And regardless, the open source converters of today have API'd allowing access to much more functions than the HMI exposes. There is no point in making a color matching UI when there's no way to read the file without already color matching anyway.

By the by, open source tools usually use manufacturer provided profiles. They're already being handed out for most cameras, so the changes required to go to a standard format are very small.

As long as raw converters don't have the option to use this kind of information (exception Adobe) it doesn't matter whether this data are available or not.

As long as Adobe can use it it matters very much. All it takes is one person to create and post a color profile and anyone can use it.

These so called problems are non-issues even today, much less if the formats become completely standardized across the board.

Since essentially all raw formats have been successfully reverse-engineered in the open-source community (dcraw) why aren't these scripts already here?

1. Have you checked the dcraw repository and mailing list?

2. Most camera profiles are simply those handed out by the manufacturer, making the whole problem moot.

3. What use would anyone have of these scripts anyway, given that if the file can be opened by dcraw it already has a profile in dcraw?

Show me anything that proves that the existing DNG files created by cameras result in the exactly the same conversion before and after the raw started to officially support by a raw converter.

Of course they do not. That's not expected nor necessary, and a complete non-sequitur unrelated to the point at hand. The camera manufacturers profile which is what is usually used in "official support" is never going to be completely identical to any other generated profile, no matter the method.

What matters is that the images can be acceptably converted either way. And they can, although at present it takes more work than it should before official support is added. This is precisely what would be helped by acceptance of fully open standards.

And I'm afraid, any conspiracy theories (eg, Adobe holding back support for marketing reasons) by default fall into the 'unreasonable' bucket. Because almost all of them aren't true. And something that is unreasonable is not a rational argument for me.

"Conspiracy theories!?" What on earth are you smoking, and can I have some? It's the reality of the market today that if the camera is not listed in the "supported" list, the software manufacturer gets slammed in media and forums for it. Just read the dpreview forum and see.

It's as much reality that getting a change from development through testing to delivery in a large corporation is a nightmare of red tape, legal audits and CYA operations by every manager involved.

Where does "conspiracy" enter into anything?

Jesper

 

[noirdesir]

Let's cut things down to the two key points.

theswede wrote:

But we are digressing, there is little point in discussing what the DNG format could be or do or what the NEF format could be or do, what counts is what they actually do in today's reality

We're discussing a hypothetical development in what DNG is used for.

That is not what I am discussing here. My very first point in this subthread was:

At best, cameras using DNG files save you a roundtrip via the DNG converter, but apart from this there is zero advantage if a camera uses the DNG format for its raw files [as observed from the current use of DNG files by camera makers].

Where I meant with 'cameras using DNG files' the real-life cameras that currently use DNG files. I made that clearer in my next post when discussing how existing 'DNG cameras' are supported by existing raw converters.

And your claim was that the DNG file as saved by cameras contains everything raw converters need and that the whole 'supporting new cameras updates' (for cameras which save as DNG) is a charade.

If that had been my claim it would indeed have been.

You said that Apple and Adobe only for marketing purposes add support for new cameras using the DNG format. If it is only done because of marketing purposes it means that the DNG file already contains everything these raw converters need.

Your reply to my question:

Me: "Why do you think Apple adds support for raw files from cameras that save their raw data natively in DNG files?"

You: "Because [of] marketing"

 

[bravozulu]

I confess to being the OP on this thread, and I hope that the responses offered productive information to others. It all went over my head quickly.

Given the news from Adobe this week, I wouldn't have ever asked about DNG in the first place. On a human dimension what I really asked was about 'faith'. What DSLR protocol can we invest our trust in? What will endure?

At the moment, my future strategy is to save everything in RAW. In my case, NEF, because I own Nikon equipment. I'm not going to expect DNG to ensure universality or survival of my data. I am impressed with the knowledge of folks here and the time and effort they've invested in learning these formats and applications.

Thanks to all. And please, don't fight.

Gary

 

[theswede #100093]

We're discussing a hypothetical development in what DNG is used for.

That is not what I am discussing here.

Then we have have nothing to discuss. I am in the context of what the thread is about and have zero interest in hashing the current problems with raw files; they are well known.

You said that Apple and Adobe only for marketing purposes add support for new cameras using the DNG format.

That is the primary reason for Apple and Adobe to add any camera to their software, yes.

If it is only done because of marketing purposes it means that the DNG file already contains everything these raw converters need.

Nothing is done only for one purpose, so this does not necessarily follow. Besides, how things are today is immaterial to the context of the thread, which you claim you do not care to discuss, so we're done here too.

Jesper

 

[Andy Hewitt]

theswede wrote:

Well there is in the software I've used! When saving a JPEG you can choose either.

There is no difference in what compression algorithm or storage format is used. The same software can read it in the same way. JPG is very good this way, and that is one reason it is so popular.

That's not true though, I do know that JPGs saved using 'progressive' mode can cause issues with opening in some software, and particularly in web use.

No, I'm just indicating that there are optional ways to save the format, which can still cause problems in some software.

There is no software without bugs. The best way to solve this issue is to open the door for more competing software, weeding out the incompetently implemented ones.

I'm not sure I entirely agree with that as a concept. Historically what has been one of the Mac platform's strengths, it that there have usually been fewer application choices, but generally of better quality. On the whole I feel that although we have seen fewer choices, we have seen less of the drivel that has been available on Windows.

Of course that is over simplified, but from what I have seen of software choices over the years, I still believe this to be fairly true. Give me a few good choices, rather than a bargain box of rubbish.

To me, that is not yet a true 'standard', as there are ways to save a file that cannot be opened by software that supports DNG files.

If I stream /dev/random to a file and call it "test.dng" it can't be opened. Failure to create files according to spec are not failures of the spec. They're bugs.

Of course, but having options within the spec that allow such failures are surely a failing of the spec.

I'd say it's a bit of both. There are two ways to save a DNG file during conversion, one is more 'standard' than the other. This is forced on the user by the Adobe software, and is not, as I see it, forced because of the rules of DNG.

Are you referring to raw vs linear? Basically linear is a "half cooked" raw, but neither is more or less standard than the other. This is as close to variants DNG gets, but the difference in reading them is small and in sane implementations (which it seems Aperture isn't? I don't know as I use LR and Ufraw) have no issues at all. And the standard is clear on how DNG works; there is no room for interpretation there.

But why the option? is it really necessary?

The whole point of a standard is to eliminate problems opening files, if there are problems opening files, then the standard has failed. Of course I agree that Aperture should be keeping up, but again, if it can open one 'version' of a 'standard' file, and not another, then that is not a 'standardized' file.

However, what the raw converter does with the read raw data from the file is of course not standardized, or there would be no use for different raw converters. And some are better than others at handling variety. This is something which would vastly improve if the ecosystem got more diverse.

Again, I don't agree. We would get more choices, perhaps, but would they be 'better'? Looking at other types of media or communication handling, I would say not.

Hypothetically you are right, but in the real world experience tells me that we're more likely to see a myriad of rubbish apps, that are more likely to confuse the issue.

How many decent mail clients are there, yet IMAP and POP have been pretty standard for years (even decades). And web browsers, HTTP is pretty standard, yet we only have two rendering engines, and a handful of browsers that are any good (if that).

We will still end up with a few decent image processors, with a bunch of others that do other jobs, but nothing that will do everything.

Like noir, I think we're slightly at cross purposes in this discussion. I don't disagree with you at all in the concept of a single standard file format for photos. However, in the real world, this just doesn't happen, and never has really. Someone will always come up with a new standard. The fact we have so many image file formats means there is no such thing as a 'standard' anyway, just look at the number of file formats that Graphic Converter handles, and the variations that are possible with some of them.

Some image formats are a lot more common than others though. JPG is the king when it comes to photographs (and many other types of images, whether they're suited for it or not) and PNG has come very strongly since it was introduced. Unfortunately Adobe are not trusted enough for DNG to take off properly. It takes "critical mass" for formats to become de facto standards, and so far that hasn't happened to any "uncooked" format.

I think that is their problem, people would trust it more if it really was a proper independent open standard, such as JPG is, and not trust it to the care of a corporate monster such as Adobe.

Personally I don't care for Adobe software anymore, or their business policies, and even without any technical limitations, I'd not want to support them by using DNG.

And indeed, the fact that Graphic converter still supports and converts file formats that existed almost 30 years ago, means that the worry about obsolescence is pretty much unfounded.

There are no proprietary raw formats which are 30 years old. And most converter software can't handle raws anyway. That's the change I would love to see come about; that raw becomes just another image format.

Graphic Converter handles Raws, and that is the only converter software I can think of that exists right now.

Even if Raw was no longer being supported tomorrow, we still have computers and software that can open them now, and will continue to work for many years - I still have a 1989 Mac here that could open old files if I really needed it to.

Obsolescence is not the large worry. Lack of application ecosystem is. There are what, a baker's dozen raw converters actively maintained? That's way too few to get much progress going, especially since they're generally special purpose and lacking the most powerful image manipulation tools, only allowing sophisticated editing after locking the conversion down.

That will still probably happen anyway, each app will have it's own way of processing the data, and saving it in different ways. And someone will always come up with a non-standard way to do it, because they'll say it's a better way.

That's very last millenium. We can do a lot better, but it takes competition to reach there.

There's always a battle between competition, and someone trying to monopolise, that goes back as far as business has been a concept.

The trouble with a 'standard' is that it can also stifle progress too, and prevent, or at least delay, progress into something better.

Image capture using bayer arrays is well understood. There is very little to improve on the current raw formats, if anything. DNG already allows for much larger capture latitude than cameras will get in many years. The reason for proprietary raw formats is not superiority or "special sauce" tags with magic information, but a quaint desire to lock the consumer in.

That's a concept that's been going on for decades in any kind of business. However, history does show us that new technologies have often been held back because of existing 'standards'.

I can remember that Atari nearly ruined themselves in the early 90's by spending so much time developing the CD as a format for data storage. They were too early, because it couldn't compete with existing standards at the time. Of course a couple of years later everybody was using them, but by then it was too late for Atari.

Of course the file format does no locking in. Lenses and familiarity with the cameras does that. Raw formats are reverse engineered rapidly anyway, and camera makers want third party software to support their formats as it increases the appeal of the system.

They do now, yes. At first they didn't, they wanted you to buy their proprietary software - most of them charged you extra for them when they first came out (at least for an upgraded version if you wanted more than basic editing tools).

Thus, the logical next step would be a unified raw format without weird scalings, allowing for a larger raw converter ecosystem. This would be excellent for the camera manufacturers. But trying to make a big corporation make such a change is like trying to bootleg turn an oil tanker. If it happens it will be because someone of vision and near psychotic strength (like Steve Jobs) gets in charge of a camera manufacturer. Thus, not likely. But I can dream!

Indeed, the theory is perfect, but the real world does not always follow - indeed, invariably it won't as long as someone thinks they can make more profit.

 

[theswede #100093]

That's not true though, I do know that JPGs saved using 'progressive' mode can cause issues with opening in some software, and particularly in web use.

The only issue I know of is a bug in Flash where Flash could not open progressive files. Otherwise I know of no issues of this kind. And that issue is a bug, not a spec issue.

I'm not sure I entirely agree with that as a concept. Historically what has been one of the Mac platform's strengths, it that there have usually been fewer application choices, but generally of better quality. On the whole I feel that although we have seen fewer choices, we have seen less of the drivel that has been available on Windows.

This is also true of the FOSS world (Linux, BSD etc.) where there is lots of software available and lots of healthy competition. When the type of software is utility and highly technical FOSS tends to come out on top, even if the GUI's may be a bit sparse.

Of course that is over simplified, but from what I have seen of software choices over the years, I still believe this to be fairly true. Give me a few good choices, rather than a bargain box of rubbish.

In the aftermath there will be a few good choices. When the Mac was new there was also an explosion of software, and a few very good contenders survived and became what we have today.

This needs to be shaken up now and then. Photoshop is completely stagnated and Aperture appears abandoned. Something has to happen.

Of course, but having options within the spec that allow such failures are surely a failing of the spec.

The spec does not allow this. That is the point. Both DNG and JPG are strict, but no rigor in the spec can protect against buggy implementation.

But why the option? is it really necessary?

Yes, linear is necessary. It's to work around the issue with incompatible sensor arrays.

The whole point of a standard is to eliminate problems opening files, if there are problems opening files, then the standard has failed. Of course I agree that Aperture should be keeping up, but again, if it can open one 'version' of a 'standard' file, and not another, then that is not a 'standardized' file.

Again, how can a spec protect against bugs, sloppiness and bad programming? Apple are not following the spec; how is that the fault of the spec?

Again, I don't agree. We would get more choices, perhaps, but would they be 'better'?

Most definitely. The existing FOSS libraries for converting raw files would become improved as the effort now spent on new file formats can instead be spent on improving functionality. The barrier of entry will be lower, meaning any PhD in image management can hack together an experimental converter in nearly no time, and if it is good commercialize it or make it FOSS.

Lowering barriers to entry is always a good ting in the end, especially if combined with free software philosophy.

Looking at other types of media or communication handling, I would say not.

Other media and communication handling is incredibly fragmented. There is no way to learn how stronger standardization will come out from looking at that.

Hypothetically you are right, but in the real world experience tells me that we're more likely to see a myriad of rubbish apps, that are more likely to confuse the issue.

To start with yes. But if the format holds as a standard we'll soon be in the same boat as with JPG; the support will become solid, the rubbish will be washed away by the quality apps, and even open source tools will be competitive, forcing the quality up. It's a very rare phenomenon, most all software is splintered by competing standards and proprietary cruft.

How many decent mail clients are there, yet IMAP and POP have been pretty standard for years (even decades).

Lots. I use mutt and gmail mostly. But once more, the issue isn't the standard. Email isn't IMAP and POP, it's headers and MIME. And the big players break both. Try comparing the headers Outlook writes to the RFC. It's not pretty. The issue is a lack of following standards.

And web browsers, HTTP is pretty standard, yet we only have two rendering engines, and a handful of browsers that are any good (if that).

HTTP is not the issue. Splintered standards (again) is the issue. The problem is HTML, XHTML, CSS, proprietary browser extensions, flash, silverlight, jscript (in many flavors) - again, the exact opposite of standardization.

We will still end up with a few decent image processors, with a bunch of others that do other jobs, but nothing that will do everything.

I don't see why this would be the case. We're not talking a transfer protocol with dozens of embedded competing standards. We're talking one standard format (with the linear variety). I know of other field which is that focused, except PDF and Postscript. And there it's working fantastically.

I think that is their problem, people would trust it more if it really was a proper independent open standard, such as JPG is, and not trust it to the care of a corporate monster such as Adobe.

Definitely. Adobe are not known for playing nice.

Personally I don't care for Adobe software anymore, or their business policies, and even without any technical limitations, I'd not want to support them by using DNG.

Cutting off the nose to spite the face can give some satisfaction.

Graphic Converter handles Raws, and that is the only converter software I can think of that exists right now.

ImageMagick, XnView, Xv, x2y, Pixillion, etc., etc. ... there's tons.

That will still probably happen anyway, each app will have it's own way of processing the data, and saving it in different ways. And someone will always come up with a non-standard way to do it, because they'll say it's a better way.

Of course, but if the major camera manufacturers support DNG like they now support JPG, the pressure will be on, and manufacturers refusing to provide DNG will lose sales over it.

That's very last millenium. We can do a lot better, but it takes competition to reach there.

There's always a battle between competition, and someone trying to monopolise, that goes back as far as business has been a concept.

And when the competition stagnates because there are no credible threats to the market dominators around something has to change to lower the barrier of entry.

That's a concept that's been going on for decades in any kind of business. However, history does show us that new technologies have often been held back because of existing 'standards'.

While that is correct, what new technology would be held back by a universal raw format?

They do now, yes. At first they didn't, they wanted you to buy their proprietary software - most of them charged you extra for them when they first came out (at least for an upgraded version if you wanted more than basic editing tools).

Once camera manufacturers realize they're manufacturing cameras and not selling proprietary raw converters they will have incentive to change to a unified raw format. I hope they do, even if it ends up not being DNG. All the same advantages apply to any open, strictly standardized raw format.

Indeed, the theory is perfect, but the real world does not always follow - indeed, invariably it won't as long as someone thinks they can make more profit.

And when someone thinks they can make more profit by following, we will have started a major shift in power towards the consumer without causing harm to the camera manufacturers bottom line. That's a worthy goal in my world.

Jesper

 

[Andy Hewitt]

theswede wrote:

In the aftermath there will be a few good choices. When the Mac was new there was also an explosion of software, and a few very good contenders survived and became what we have today.

This needs to be shaken up now and then. Photoshop is completely stagnated and Aperture appears abandoned. Something has to happen.

Well, being as this is a speculative subject, I think we'll leave it at that. We don't really know what would happen, we can only hope for the best. History does not leave me optimistic though.

Of course, but having options within the spec that allow such failures are surely a failing of the spec.

The spec does not allow this. That is the point. Both DNG and JPG are strict, but no rigor in the spec can protect against buggy implementation.

Buggy, or simply lack of resource to develop?

But why the option? is it really necessary?

Yes, linear is necessary. It's to work around the issue with incompatible sensor arrays.

Ok, that to me suggests that DNG is not capable of being the standardised format we need it to be. A 'workaround' for some sensor arrays is not compatibility if it's not able to access all the data that those arrays have available (which is shown on the Apple bulletin I provided the link to).

Of course we are distracting from the original point here, which was whether it's worth converting existing Raw files to DNG. I think that it is probably isn't, as there's currently no guarantee that all the information available will be usable.

The whole point of a standard is to eliminate problems opening files, if there are problems opening files, then the standard has failed. Of course I agree that Aperture should be keeping up, but again, if it can open one 'version' of a 'standard' file, and not another, then that is not a 'standardized' file.

Again, how can a spec protect against bugs, sloppiness and bad programming? Apple are not following the spec; how is that the fault of the spec?

Because the spec does not allow for all the sensors available properly. That surely is the whole point of it, no?

Again, I don't agree. We would get more choices, perhaps, but would they be 'better'?

Most definitely. The existing FOSS libraries for converting raw files would become improved as the effort now spent on new file formats can instead be spent on improving functionality. The barrier of entry will be lower, meaning any PhD in image management can hack together an experimental converter in nearly no time, and if it is good commercialize it or make it FOSS.

Lowering barriers to entry is always a good ting in the end, especially if combined with free software philosophy.

Hmm, we would possibly get more choices - if anyone though it adding any more to the ones already available.

My own recent searches has already found there are more choices for Raw conversion than I thought there were. I've been trying out darktable and RawTherapee, just to see if they could really be replacements for the commercial options. They do a good job for sure, but IMHO, no, they're not ready for that, but for someone starting out, wanting a cheap way to edit and organise Raw images, yeah, they are a good start.

But, they do show that having more choices does not mean better choices.

Looking at other types of media or communication handling, I would say not.

Other media and communication handling is incredibly fragmented. There is no way to learn how stronger standardization will come out from looking at that.

Hypothetically you are right, but in the real world experience tells me that we're more likely to see a myriad of rubbish apps, that are more likely to confuse the issue.

To start with yes. But if the format holds as a standard we'll soon be in the same boat as with JPG; the support will become solid, the rubbish will be washed away by the quality apps, and even open source tools will be competitive, forcing the quality up. It's a very rare phenomenon, most all software is splintered by competing standards and proprietary cruft.

It might if it's allowed to mature into a proper standard, and one that isn't controlled by a large corporate.

FWIW, I do agree that DNG as a concept is very much a good thing, but only for new cameras that support it natively, and we can trust that all the data is universally available in all software. Personally I'm not as optimistic as you seem to be.

Currently, I can't see any point in converting if you already have software that already processes your existing Raw files. Why bother adding workload?

How many decent mail clients are there, yet IMAP and POP have been pretty standard for years (even decades).

Lots. I use mutt and gmail mostly. But once more, the issue isn't the standard. Email isn't IMAP and POP, it's headers and MIME. And the big players break both. Try comparing the headers Outlook writes to the RFC. It's not pretty. The issue is a lack of following standards.

Exactly my point.

And web browsers, HTTP is pretty standard, yet we only have two rendering engines, and a handful of browsers that are any good (if that).

HTTP is not the issue. Splintered standards (again) is the issue. The problem is HTML, XHTML, CSS, proprietary browser extensions, flash, silverlight, jscript (in many flavors) - again, the exact opposite of standardization.

Yup, my point again.

We will still end up with a few decent image processors, with a bunch of others that do other jobs, but nothing that will do everything.

I don't see why this would be the case. We're not talking a transfer protocol with dozens of embedded competing standards. We're talking one standard format (with the linear variety). I know of other field which is that focused, except PDF and Postscript. And there it's working fantastically.

Even PDF has odd anomalies that sometimes cause apps to trip up though.

I think that is their problem, people would trust it more if it really was a proper independent open standard, such as JPG is, and not trust it to the care of a corporate monster such as Adobe.

Definitely. Adobe are not known for playing nice.

Personally I don't care for Adobe software anymore, or their business policies, and even without any technical limitations, I'd not want to support them by using DNG.

Cutting off the nose to spite the face can give some satisfaction.

That depends. I do the same with Microsoft too, and I'm not suffering too badly for it

Adobe have screwed us for too long, particularly here in the UK where their pricing policies have shown they are profiteering in an unreasonable manner. They do upgrades that cost more than full versions, and their software is about as buggy as anyone elses. How do we trust them with DNG?

Graphic Converter handles Raws, and that is the only converter software I can think of that exists right now.

ImageMagick, XnView, Xv, x2y, Pixillion, etc., etc. ... there's tons.

Name one with a standard Mac OS GUI!

That will still probably happen anyway, each app will have it's own way of processing the data, and saving it in different ways. And someone will always come up with a non-standard way to do it, because they'll say it's a better way.

Of course, but if the major camera manufacturers support DNG like they now support JPG, the pressure will be on, and manufacturers refusing to provide DNG will lose sales over it.

Going forward, for sure. Going back to what I said earlier, we have two arguments here.

1. whether converting existing documents to DNG is worthwhile.

2. whether DNG worthwhile as a future standard format.

That's very last millenium. We can do a lot better, but it takes competition to reach there.

There's always a battle between competition, and someone trying to monopolise, that goes back as far as business has been a concept.

And when the competition stagnates because there are no credible threats to the market dominators around something has to change to lower the barrier of entry.

That's a concept that's been going on for decades in any kind of business. However, history does show us that new technologies have often been held back because of existing 'standards'.

While that is correct, what new technology would be held back by a universal raw format?

Whatever the next new technology is, who knows, but history has shown this to happen in the past.

They do now, yes. At first they didn't, they wanted you to buy their proprietary software - most of them charged you extra for them when they first came out (at least for an upgraded version if you wanted more than basic editing tools).

Once camera manufacturers realize they're manufacturing cameras and not selling proprietary raw converters they will have incentive to change to a unified raw format. I hope they do, even if it ends up not being DNG. All the same advantages apply to any open, strictly standardized raw format.

Some have, Olympus already switched their software to free. I don't know about others though.

Indeed, the theory is perfect, but the real world does not always follow - indeed, invariably it won't as long as someone thinks they can make more profit.

And when someone thinks they can make more profit by following, we will have started a major shift in power towards the consumer without causing harm to the camera manufacturers bottom line. That's a worthy goal in my world.

Yes, but it's all speculation. We can hope, but I don't trust any of them to play by the rules.

 

[theswede #100093]

Well, being as this is a speculative subject, I think we'll leave it at that.

That Photoshop and Apertura are stagnated is not hypothetical. FOSS software will clean their clock if they don't get their act together soon.

We don't really know what would happen, we can only hope for the best. History does not leave me optimistic though.

Query; what is the world's most installed OS across all computing devices at this point? Hint; it was created out of open standards and is as open as they can become.

Open standards always win in the end. It may take a while, and digital computers are very young, but the writing is on the wall.

Buggy, or simply lack of resource to develop?

Lack of resources means lack of commitment means lack of alternatives. Exactly what I expect will change with a lower barrier of entry.

But why the option? is it really necessary?

Yes, linear is necessary. It's to work around the issue with incompatible sensor arrays.

Ok, that to me suggests that DNG is not capable of being the standardised format we need it to be. A 'workaround' for some sensor arrays is not compatibility if it's not able to access all the data that those arrays have available (which is shown on the Apple bulletin I provided the link to).

The reason for this is that the less than half a dozen cameras affected are - less than half a dozen. If another sensor layout becomes viable and gains traction that's when the spec needs to be extended to be able to handle it.

And even so, we're talking a grand total of four sensor arrays in commercial existence today, two of which need special treatment (and are used in less that four cameras each). It's not something which hurts the point of the standard.

Of course we are distracting from the original point here, which was whether it's worth converting existing Raw files to DNG. I think that it is probably isn't, as there's currently no guarantee that all the information available will be usable.

There is no guarantee that all the information available in a raw file will be usable no matter what. The difference is very slight. Either way, standardizing on DNG will endure all the information available will be usable.It's worth fighting for.

The whole point of a standard is to eliminate problems opening files, if there are problems opening files, then the standard has failed. Of course I agree that Aperture should be keeping up, but again, if it can open one 'version' of a 'standard' file, and not another, then that is not a 'standardized' file.

Again, how can a spec protect against bugs, sloppiness and bad programming? Apple are not following the spec; how is that the fault of the spec?

Because the spec does not allow for all the sensors available properly. That surely is the whole point of it, no?

No, that's not the point of the spec, the point of the spec is to standardize file formats on cameras, meaning it should work for 99-% of cameras (and it does) and be ready to extend to the ones that make sense to extend it to (and it is) and handle the odd cases well (which it does with linear mode), and either way, incorrectly implementing a spec is not the fault of the spec. Especially when others with ostensibly fewer resources manage to get it right.

Hmm, we would possibly get more choices - if anyone though it adding any more to the ones already available.

We might, but the most important thing is that lowering barrier to entry puts some fire to the couches the present market leaders are relaxing in.

My own recent searches has already found there are more choices for Raw conversion than I thought there were. I've been trying out darktable and RawTherapee, just to see if they could really be replacements for the commercial options. They do a good job for sure, but IMHO, no, they're not ready for that, but for someone starting out, wanting a cheap way to edit and organise Raw images, yeah, they are a good start.

And the main reason they aren't more than that is the high barrier of entry.

But, they do show that having more choices does not mean better choices.

It means the market leaders feel the heat from competition, and are forced to improve. I'm sure the LR and Aperture marketing people are following Darktable and RawTherapee closely.

It might if it's allowed to mature into a proper standard, and one that isn't controlled by a large corporate.

This is the issue. The sole advantage of DNG is that it exists and works now. There is no need to start over. A new standard would have to start from scratch and make all the mistakes DNG has done all over again.

FWIW, I do agree that DNG as a concept is very much a good thing, but only for new cameras that support it natively, and we can trust that all the data is universally available in all software. Personally I'm not as optimistic as you seem to be.

Currently, I can't see any point in converting if you already have software that already processes your existing Raw files. Why bother adding workload?

At this point in time, the reason is pretty much none. If DNG becomes a standard, there is definitely a reason to convert, as it opens the door to using any conversion software one likes.

Exactly my point.

Actually, your point is that one standard has flawed implementations. Your examples show a lack of a single standard, and multiple splintered standards with multiple implementations causing problems.

Which is exactly how it is with raw today, and exactly how it would not be with a standard raw format.

Thus, your point is the exact opposite of what would be expected would DNG gain traction as a mainstream common format.

fantastically.

Even PDF has odd anomalies that sometimes cause apps to trip up though.

Which is still a lot better than pretty much anything around except JPG, and that is pretty darn good work for a standard encompassing as much as PDF does.

And yeah, no program is bug free, so sometimes PDF's will cause problems. I'd question reality if this was not the case, and no software ecosystem will ever be perfect -not the raw conversion we have now, and not a DNG based ecosystem - but some are better than others, and it's easy to be better than the raw mess we have on our hands now.

That depends. I do the same with Microsoft too, and I'm not suffering too badly for it

As do I, to the extent I am able due to work. But throwing away an open standard (which Microsoft have 0 of) just because you'd rather play into the hands of those pushing proprietary formats and pay extra for the effort and development time spent working around imposed limitations is not helping either you, the camera manufacturer or the raw conversion software developers.

Adobe have screwed us for too long, particularly here in the UK where their pricing policies have shown they are profiteering in an unreasonable manner. They do upgrades that cost more than full versions, and their software is about as buggy as anyone elses. How do we trust them with DNG?

The same way we trust them with PDF. We ensure it's an open standard.

ImageMagick, XnView, Xv, x2y, Pixillion, etc., etc. ... there's tons.

Name one with a standard Mac OS GUI!

That is an incredibly narrow view. The point isn't the application or GUI, that is trivial to add (and there are several native OSX wrappers around conversion libraries). The point is having the functionality available for the front end applications to use.

Many websites, for example, use ImageMagick to serve up images. If they can easily throw up color corrected raw images from any camera, present or future, how is that not a win for every web surfer out there?

If any web developer currently locked into PNG and JPG can just add not only a DNG, but a DNG with embedded lossless correction to any place in a web site, how is that not a tremendous step forward, allowing high end photography to take a leap forward?

With the advent of proper color spaces in browsers and the rise of 10 bit color in screens and cards, this would mean that images posted today could be rendered for today's screens now, and for larger gamut as it becomes available, all without any intervention or software updates.

It would mean that you could put up a gallery site today, pump raw images to it and have them displayed to the viewer at the best available gamut - and when you get a new camera, you just keep uploading, never worrying about software versions.

It's a different paradigm. A completely different way of thinking. Thinking "oh, it's like today, only the files are named DNG" is completely missing the implications. This has the possibility to be huge.

Going forward, for sure. Going back to what I said earlier, we have two arguments here.

1. whether converting existing documents to DNG is worthwhile.

If DNG becomes a future standard it will be.

2. whether DNG worthwhile as a future standard format.

It is to such an extent it is almost impossible to convey the implications.

Whatever the next new technology is, who knows, but history has shown this to happen in the past.

And when new technology arises it is because it can stand on the shoulders of giants and leverage existing technology. The present raw mire does nothing to help innovation and works hard to hinder it.

Some have, Olympus already switched their software to free. I don't know about others though.

Unfortunately Olympus are a small player. It would take Canikon to make waves on this matter.

Yes, but it's all speculation. We can hope, but I don't trust any of them to play by the rules.

That is the beauty of open standards. What do you think would happen if a camera maker would start churning out non-standard JPG which only a few applications can handle? They're get torn apart in the press, and lose sales like mad.

If DNG was standard, the same would apply to that.

There is no better weapon for consumer protection than open standards, and no worse enemy than proprietary standards.

Jesper

 

[Andy Hewitt]

theswede wrote:

Well, being as this is a speculative subject, I think we'll leave it at that.

That Photoshop and Apertura are stagnated is not hypothetical. FOSS software will clean their clock if they don't get their act together soon.

You're misunderstanding. What *will* happen in the future is speculative.

We don't really know what would happen, we can only hope for the best. History does not leave me optimistic though.

Query; what is the world's most installed OS across all computing devices at this point? Hint; it was created out of open standards and is as open as they can become.

Depending on what data you look at, for Desktops Windows has 85-91%, Mac OS 7-13%, Linux 1.2%, and others have the rest. Of all OS, iOS only has 4%.

Open standards always win in the end. It may take a while, and digital computers are very young, but the writing is on the wall.

Hardly.

The reason for this is that the less than half a dozen cameras affected are - less than half a dozen. If another sensor layout becomes viable and gains traction that's when the spec needs to be extended to be able to handle it.
And even so, we're talking a grand total of four sensor arrays in commercial existence today, two of which need special treatment (and are used in less that four cameras each). It's not something which hurts the point of the standard.

That's certainly not true. The entire microFourThirds line up is effected (covering models from Olympus, Panasonic and Leica (which at this time is not a small market share), as well as a few other Raw formats. And none of the Foveon sensors are supported at all.

Of course we are distracting from the original point here, which was whether it's worth converting existing Raw files to DNG. I think that it is probably isn't, as there's currently no guarantee that all the information available will be usable.

There is no guarantee that all the information available in a raw file will be usable no matter what. The difference is very slight. Either way, standardizing on DNG will endure all the information available will be usable.It's worth fighting for.

Look, I'm not the one coming up with this. From Apple's own documentation:

"However, because the decoding offered with directly supported cameras is tailored for each camera's RAW format, the basic decode for natively supported images may be better than with DNG images."

That of course does not mean that the Raw is always better, but it does cast a shadow of doubt.

"Auto Noise Compensation requires specific knowledge of the camera's sensor characteristics in combination with the ISO value of the image. DNG images do not provide the needed sensor information."

And that does suggest, as I've been saying all along, that DNG may not be guaranteed to supply all the data from the sensor as a proprietary Raw file does.

Please note that I accept that in the future, that could change as manufacturers build DNG support into their cameras, and include all the data that is needed for full processing. However, at this time, it is by no means certain that a converted Raw will.

Exactly my point.
Actually, your point is that one standard has flawed implementations. Your examples show a lack of a single standard, and multiple splintered standards with multiple implementations causing problems.

Which is exactly how it is with raw today, and exactly how it would not be with a standard raw format.

Thus, your point is the exact opposite of what would be expected would DNG gain traction as a mainstream common format.

No my point is that it's very common for standards to be broken to push boundaries, and achieve results that cannot be achieved within the standards.

ImageMagick, XnView, Xv, x2y, Pixillion, etc., etc. ... there's tons.

Name one with a standard Mac OS GUI!

That is an incredibly narrow view. The point isn't the application or GUI, that is trivial to add (and there are several native OSX wrappers around conversion libraries). The point is having the functionality available for the front end applications to use.

Not at all, the point here is that these need to be available to the average user needing to have an efficient and accurate workflow. Rooting around in command line, or waiting for someone to build a GUI front end is not that.

Going forward, for sure. Going back to what I said earlier, we have two arguments here.

1. whether converting existing documents to DNG is worthwhile.

If DNG becomes a future standard it will be.

IF the data can be converted fully and accurately - at this time it cannot.

2. whether DNG worthwhile as a future standard format.

It is to such an extent it is almost impossible to convey the implications.

Maybe we'll see.

Yes, but it's all speculation. We can hope, but I don't trust any of them to play by the rules.

That is the beauty of open standards. What do you think would happen if a camera maker would start churning out non-standard JPG which only a few applications can handle? They're get torn apart in the press, and lose sales like mad.

If DNG was standard, the same would apply to that.

There is no better weapon for consumer protection than open standards, and no worse enemy than proprietary standards.

For sure. But, as said, I'm not arguing against DNG at all, just my pessimism for what will develop.