What is Future of DNG File Format due to Shifts at Adobe

[theswede #100093]

Any image manipulation application that can open DNG files (and actually uses the raw data and not the jpeg preview) (a) needs to be a raw converter and (b) should have profiles for the camera model in question. At best, cameras using DNG files saves you a roundtrip via the DNG converter, but apart from this there is zero advantage if a camera uses the DNG format for its raw files.

Except you're future proof, as your software and your cameras will always work together. You're not tied into buying the latest Photoshop just to get the latest ACR.

The question is whether most people really want to use raw files from an un-profiled camera.

The profile can be in the DNG. This is in fact one of the main advantages of DNG, and will future-proof your software. Of course you can override the profile if you have one you prefer, or when advances in conversion are made, but most people are likely to be happy with the default profile. It's what they use now.

Jesper

 

[theswede #100093]

Hmm, that's not what I found when experimenting.

It's what controlled experiments have found.

JPG2000 images were noticeably better quality, and smaller. Even if they're not significantly smaller, surely an improvement in image quality is always worth having.

Not at the expense of computing power. Cameras have too little of that already. At best we're talking 20% smaller storage requirements, which is insignificant with modern storage media sizes and costs.

So why bother then?

Because you can keep the profile in the DNG as well. Thus you will have no change compared to how it is today even with image editors which do not know anything about your camera. You're not dependent on the software upgrade cycle, but can use any DNG aware software to edit any DNG camera photo.

Ok, so you're still relying on the software being able to optimise processing based on the camera model tag, if that needs an update to achieve, what's the benefit?

Only if the camera provides no profile or you want to override the one provided.

You might be able to open the image, and perhaps make some kind of adjustments on basic assumed settings, but it wouldn't be optimised. You'd not be much better off than you are now, as you'd still have to wait for a software update to be able to make use of optimised settings.

Could you please provide your source for that the software updates actually optimize image generation and do not only provide updated basic conversion functionality? It's the first time I hear that there is actually optimization going on in the image conversion. I have no idea what would be optimized and how, so your source for this would be appreciated.

Jesper

 

[Andy Hewitt]

theswede wrote:

Not at the expense of computing power. Cameras have too little of that already. At best we're talking 20% smaller storage requirements, which is insignificant with modern storage media sizes and costs.

Hmm, for sure, but computing power has also progressed an awful lot since the format was introduced, I'd suggest that all factors here are insignificant. Anyway, where not here to argue about JPG2000.

So why bother then?

Because you can keep the profile in the DNG as well. Thus you will have no change compared to how it is today even with image editors which do not know anything about your camera. You're not dependent on the software upgrade cycle, but can use any DNG aware software to edit any DNG camera photo.

Ok, so you're still relying on the software being able to optimise processing based on the camera model tag, if that needs an update to achieve, what's the benefit?

Only if the camera provides no profile or you want to override the one provided.

You might be able to open the image, and perhaps make some kind of adjustments on basic assumed settings, but it wouldn't be optimised. You'd not be much better off than you are now, as you'd still have to wait for a software update to be able to make use of optimised settings.

Could you please provide your source for that the software updates actually optimize image generation and do not only provide updated basic conversion functionality? It's the first time I hear that there is actually optimization going on in the image conversion. I have no idea what would be optimized and how, so your source for this would be appreciated.

Just have a look at the Raw Fine Tuning Settings in Aperture, you'll see that they change automatically for each camera model - it's even noticeable between very similar Olympus models that I use.

Have a look here too:

http://support.apple.com/kb/HT2709

A quick bit of searching finds that yes, DNG does enable you to open 'most' DNG files in software that supports it. However, as I said, it may not optimise the image processing for a given model unless the processing software has a profile for that camera model.

To me, that still means software needs to be updated to account for new models of cameras, and negates much of the point of using DNG.

It also seems that DNG isn't even entirely 'standard', as there are a few variations that may or may not work on some software. Pretty much like a normal Raw file then.

I don't think there's any reason to believe that using DNG future proofs you at all. It's just as likely that software developers would drop this format as much as they would proprietary Raw files.

 

[noirdesir]

theswede wrote:

Any image manipulation application that can open DNG files (and actually uses the raw data and not the jpeg preview) (a) needs to be a raw converter and (b) should have profiles for the camera model in question. At best, cameras using DNG files saves you a roundtrip via the DNG converter, but apart from this there is zero advantage if a camera uses the DNG format for its raw files.

Except you're future proof, as your software and your cameras will always work together. You're not tied into buying the latest Photoshop just to get the latest ACR.

As I said below, the camera and the software will only work well together if the camera has been profiled by the raw converter maker. And for that, I will always have to buy new versions of my raw converter, DNG or not.

The question is whether most people really want to use raw files from an un-profiled camera.

The profile can be in the DNG. This is in fact one of the main advantages of DNG, and will future-proof your software. Of course you can override the profile if you have one you prefer, or when advances in conversion are made, but most people are likely to be happy with the default profile. It's what they use now.

No, they don't. Why do you think Apple adds support for raw files from cameras that save their raw data natively in DNG files? Obviously because they tested/profiled the camera and adjusted their raw conversion parameters to optimize the result. Thus, people don't use the default profile (which might or might not be saved by the camera in the DNG file), they use Apple's profile. And the same applies for Adobe, they explicitly add support for new camera models that save their raw data as DNG files. Thus, again, people won't use the default profile, they will use the profile created by Adobe.

 

[noirdesir]

graybalanced wrote

The catch with that sentence is that the "ease and ubiquity" of most software being able to open most raw files is a huge illusion. The illusion is maintained by every single raw processor software company having a team of people dedicated to decoding proprietary raw files as new cameras roll out. How sustainable is this effort? So far all the important players seem dedicated to this, but as the number of cameras grows you kind of have to wonder.

The amount of personnel necessary to profile cameras is proportional to the number of cameras released per year. This number has grown but it won't grow indefinitely as the number of cameras the market can sustain is limited.

The real value is if cameras could save straight to it, but until that happens everybody is going to continue to think there is no need, because of the silent development tax we're paying for all the raw format decoding that has to go on.

Why then do we have to wait for Apple or Adobe to support, eg, new Leica cameras (which save their raw data as DNG files)? If there is no 'development' needed, why do we still have to wait?

That's why I support the idea of DNG and I really want cameras to save straight to it, but at the same time I don't bother converting my existing files. The value is hidden and in the future, and when those two things are true, few people care about making it happen. This is true in tech, in politics, in social and environmental issues, etc. As long as everything seems fine on the surface, we prefer to think everything is going to continue to be fine, no matter what is really going on under the surface. It's been pointed out that there are already Kodak raw formats that Kodak no longer supports.

Do you really think extracting the raw data from a raw file is the complicated part? In particular since no camera maker will change the formatting constantly? What takes time is to profile the camera and that is needed regardless of the file format.

 

[theswede #100093]

Just have a look at the Raw Fine Tuning Settings in Aperture, you'll see that they change automatically for each camera model - it's even noticeable between very similar Olympus models that I use.

Interesting. But again, this can be embedded in the DNG.

To me, that still means software needs to be updated to account for new models of cameras, and negates much of the point of using DNG.

Again, not if the DNG contains a good baseline set of parameters. Since the camera manufacturers ship raw converter software with the cameras they have that, and could easily do this.

It also seems that DNG isn't even entirely 'standard', as there are a few variations that may or may not work on some software.

DNG is entirely standard. Variations are deviations from spec and thus bugs.

I don't think there's any reason to believe that using DNG future proofs you at all. It's just as likely that software developers would drop this format as much as they would proprietary Raw files.

Of course software makers may drop any format. They may drop jpeg too if they decide to. But if DNG camera output with embedded profiles were to catch on it'd make all our lives easier, and open up the door for a lot of software to support DNG without the need for constant updates.

Unfortunately that hasn't happened, and isn't likely to. But there are definite advantages for us consumers if it were to happen.

Jesper

 

[Andy Hewitt]

theswede wrote:

Just have a look at the Raw Fine Tuning Settings in Aperture, you'll see that they change automatically for each camera model - it's even noticeable between very similar Olympus models that I use.

Interesting. But again, this can be embedded in the DNG.

Yes, I understand that a camera profile could be, and probably already is, embedded into the file.

To me, that still means software needs to be updated to account for new models of cameras, and negates much of the point of using DNG.

Again, not if the DNG contains a good baseline set of parameters. Since the camera manufacturers ship raw converter software with the cameras they have that, and could easily do this.

What would you call a 'good' baseline of parameters, how would you write a profile to optimise the file for every possible combination of camera and sensor that's available. It's not possible.

It also seems that DNG isn't even entirely 'standard', as there are a few variations that may or may not work on some software.

DNG is entirely standard. Variations are deviations from spec and thus bugs.

It's about as standard as JPG or TIFF is - just see how many variations of those there are too, with different compression methods and levels. They do have a common standard, but it can be deviated from, which can cause issues with some software.

I don't think there's any reason to believe that using DNG future proofs you at all. It's just as likely that software developers would drop this format as much as they would proprietary Raw files.

Of course software makers may drop any format. They may drop jpeg too if they decide to. But if DNG camera output with embedded profiles were to catch on it'd make all our lives easier, and open up the door for a lot of software to support DNG without the need for constant updates.

Unfortunately that hasn't happened, and isn't likely to. But there are definite advantages for us consumers if it were to happen.

But you're missing my point entirely. Using DNG merely enables a user to be able to open and edit a Raw file using any software, without having to update that software, using a basic set of parameters. However, it does not enable you to make use of features available with a newly designed sensor, or parameters embedded in the Raw file that might take advantage of a camera's new features. Even though a Raw file contains the data directly from the sensor, it's also known that other data is stored along with that to enable software to process the file better. This will by necessity be different for each sensor and camera model, depending on its specifications.

The whole point of using Raw image files in the first place is to be able to access image quality and editing functions that would otherwise not be available with other formats. Even if a DNG could provide this, the software would still have to recognise this information, and deal with it appropriately, which would mean it'd have to contain the profile for that camera.

All you're really gaining is an ability to make some use of Raw images from a brand new camera until such time the profile has been updated in the software you use.

 

[theswede #100093]

Yes, I understand that a camera profile could be, and probably already is, embedded into the file.

Do you really? You're arguing as if you do not understand this.

What would you call a 'good' baseline of parameters, how would you write a profile to optimise the file for every possible combination of camera and sensor that's available. It's not possible.

It's embedded in the DNG. It's not one profile for all cameras. It's one profile per camera - indeed, one profile per DNG, allowing the camera to provide fine tuning for the specific photo as required.

It's about as standard as JPG or TIFF is - just see how many variations of those there are too, with different compression methods and levels.

JPG has precisely one compression method. Varying the levels makes no difference in algorithm.

Loading…

en.wikipedia.org

TIFF is an ISO standard format. All implementations follow one of the three ISO standards encompassing TIFF and will report clearly if a TIFF file you attempt to open falls outside of the implementation used in the software.

Loading…

en.wikipedia.org

DNG has an open specification, and is completely backwards compatible with earlier versions of the specification. There are only feature addons in later versions, not basic changes as DNG is based on open standards.

Loading…

en.wikipedia.org

In other words, JPG, TIFF and DNG are all very well defined, well specified standards which software supporting them adheres very well to. There are few, if any, compatibility problems with JPG or DNG. TIFF is a bit different since the standard is incredibly broad and some (especially older) software does not implement all possible compression types. But you shouldn't use TIFF anyway.

They do have a common standard, but it can be deviated from, which can cause issues with some software.

No, the standard can not be deviated from. If that happens it is a bug.

But you're missing my point entirely. Using DNG merely enables a user to be able to open and edit a Raw file using any software, without having to update that software, using a basic set of parameters.

That is a non-point, since the DNG file can (and if DNG was popular would) contain an optimized set of parameters.

However, it does not enable you to make use of features available with a newly designed sensor, or parameters embedded in the Raw file that might take advantage of a camera's new features.

If there are new parameters which are not documented then they can not be used; this does not change whether the format is DNG or another raw format. If they are documented nothing will stop you from using them in any software.

Even though a Raw file contains the data directly from the sensor, it's also known that other data is stored along with that to enable software to process the file better.

That is called "metadata" and DNG has excellent support for this.

This will by necessity be different for each sensor and camera model, depending on its specifications.

There is no such necessity. Most metadata is identical between cameras.

The whole point of using Raw image files in the first place is to be able to access image quality and editing functions that would otherwise not be available with other formats. Even if a DNG could provide this, the software would still have to recognise this information, and deal with it appropriately, which would mean it'd have to contain the profile for that camera.

Which is exactly what it does.

All you're really gaining is an ability to make some use of Raw images from a brand new camera until such time the profile has been updated in the software you use.

You also gain the ability to use raw images from any camera with any software which supports DNG, allowing for much more competition in raw converters and raw capable software. At present a few raw converters completely dominate the market due to the esoteric nature of the raw formats and the constant struggle to keep up with ever changing raw formats.

This would allow a whole new ecosystem of applications which we can't even dream of today with our limited supply of options. It would be a marvelous time to be a digital darkroom artist.

Sadly I don't think that day will come, since most people seem stuck in the mindset of todays limited raw converters.

Jesper

 

[Andy Hewitt]

theswede wrote:

Yes, I understand that a camera profile could be, and probably already is, embedded into the file.

Do you really? You're arguing as if you do not understand this.

What would you call a 'good' baseline of parameters, how would you write a profile to optimise the file for every possible combination of camera and sensor that's available. It's not possible.

It's embedded in the DNG. It's not one profile for all cameras. It's one profile per camera - indeed, one profile per DNG, allowing the camera to provide fine tuning for the specific photo as required.

It's about as standard as JPG or TIFF is - just see how many variations of those there are too, with different compression methods and levels.

JPG has precisely one compression method. Varying the levels makes no difference in algorithm.

Loading…

en.wikipedia.org

TIFF is an ISO standard format. All implementations follow one of the three ISO standards encompassing TIFF and will report clearly if a TIFF file you attempt to open falls outside of the implementation used in the software.

Loading…

en.wikipedia.org

DNG has an open specification, and is completely backwards compatible with earlier versions of the specification. There are only feature addons in later versions, not basic changes as DNG is based on open standards.

Loading…

en.wikipedia.org

In other words, JPG, TIFF and DNG are all very well defined, well specified standards which software supporting them adheres very well to. There are few, if any, compatibility problems with JPG or DNG. TIFF is a bit different since the standard is incredibly broad and some (especially older) software does not implement all possible compression types. But you shouldn't use TIFF anyway.

That's certainly not the case though if you really read those pages:

On the DNG page:

"DNG is not (yet) a standard format, but is based on several open formats and/or standards"

The JPG page details a myriad of possible optional ways to create a JPG based image.

The TIFF page lists about 30 possible variations of the 'standard'.

It's certainly not definite that a .tif .dng or .jpg file will be of a known standard format. Of course most cameras have adopted a baseline format for each of them that they use, but that's not to say that variations of them don't exist.

Even though a Raw file contains the data directly from the sensor, it's also known that other data is stored along with that to enable software to process the file better.

That is called "metadata" and DNG has excellent support for this.

No, there are some proprietary tags and information in some Raw files that can only be accessed using that manufacturers software (unless someone has reverse engineered of course).

This will by necessity be different for each sensor and camera model, depending on its specifications.

There is no such necessity. Most metadata is identical between cameras.

The whole point of using Raw image files in the first place is to be able to access image quality and editing functions that would otherwise not be available with other formats. Even if a DNG could provide this, the software would still have to recognise this information, and deal with it appropriately, which would mean it'd have to contain the profile for that camera.

Which is exactly what it does.

The DNG does, but the software reading the file may well not.

All you're really gaining is an ability to make some use of Raw images from a brand new camera until such time the profile has been updated in the software you use.

You also gain the ability to use raw images from any camera with any software which supports DNG, allowing for much more competition in raw converters and raw capable software. At present a few raw converters completely dominate the market due to the esoteric nature of the raw formats and the constant struggle to keep up with ever changing raw formats.

This would allow a whole new ecosystem of applications which we can't even dream of today with our limited supply of options. It would be a marvelous time to be a digital darkroom artist.

Sadly I don't think that day will come, since most people seem stuck in the mindset of todays limited raw converters.

There's nothing limiting about Raw per se, after all, they allow you access to all of the data available from the sensor. However I do agree with you, a standardised format would be better without doubt, but it's only worthwhile if you can tap into the full data set for your camera.

The problem we have is that the camera manufacturers are reluctant to licence the information they use in their Raw formats.

 

[theswede #100093]

That's certainly not the case though if you really read those pages:

I've more than "really read" the pages. I've implemented image libraries.

On the DNG page:

"DNG is not (yet) a standard format, but is based on several open formats and/or standards"

Do you know what that means? It means DNG is not an ISO or ANSI standard. That doesn't mean itallows variants. It's still one specification not allowing deviations. Just because it is not a "standard format" (a formality) does not mean it is not strictly specified. It is, the specification is very clear, and it's open and unencumbered.

The JPG page details a myriad of possible optional ways to create a JPG based image.

The image can be created in precisely one way. JPG only has one compression method. No deviation allowed. The image can be packed in a JFIF or EXIF container, but if there is a JFIF header it doesn't matter which format is used, any old image manipulation program can read it since the actual image is packed the same way. Only metadata differs. And the only reason older software have problems with EXIF only files is because they don't recognize the metdata, not because the image differs. It doesn't.

The TIFF page lists about 30 possible variations of the 'standard'.

And it lists the exact method to tell the application which compression and file format is used. If your file tells the application it is one specific type of TIFF but you decided to vary the format a bit from that specification, your file is garbage. No deviation allowed. But you shouldn't use TIFF anyway.

It's certainly not definite that a .tif .dng or .jpg file will be of a known standard format.

It's most certainly definite. If a JPG has a JFIF header any software can open it. If you're daft enough to save a JPG with only an EXIF header it can possibly only be opened by more recent software, but the software will know from looking at the header exactly what the format of the JPG is, and if the header lies the file is so much garbage. Either way, all JPGs have the image data stored the same way. Only metadata differs.

And a DNG is even stricter. It has a version in the header, and if the file format does not match that version, it's broken. End of story. The spec allows for no deviation at all.

Of course most cameras have adopted a baseline format for each of them that they use, but that's not to say that variations of them don't exist.

None of the formats have any variations allowed outside of specification. None at all. If your camera generates an image which falls outside of spec there are no guarantees anything at all can read it. JPG has no variations at all outside metadata. DNG has versions, but they're backwards compatible so that matters little. TIFF is a special case since it tries to encompass so much ground, but you're still bound to the spec. Deviate and your image is garbage. And if you're storing photographs the number of sane TIFF choices fall down drastically. Most cameras storing TIFF just use uncompressed TIFF to avoid implementation issues and keep processing demands down (processing costs battery and causes heat, and is avoided by cameras as much as possible, which is one major reason JPG2000 has not caught on). But you shouldn't use TIFF anyway.

No, there are some proprietary tags and information in some Raw files that can only be accessed using that manufacturers software (unless someone has reverse engineered of course).

Manufacturer raw files are entirely proprietary, and the have their own names for tags containing standard information. That is precisely why DNG would be better.

The tags are named and scaled differently, but they still contain the same information. That's how the same raw software using the same sliders and buttons can open multiple formats of raw files. It maps the tags internally to the same functions. There are no "extra tags" containing mysterious sensor information which needs to be specifically implemented per camera. It's all well understood, and maps to DNG nicely.

The DNG does, but the software reading the file may well not.

So? The DNG contains it, then what does it matter whether the software has it or not?

That is what you consistently seem to not understand. If the profile is in the DNG, the software can just read it out, and the conversion will be optimized. No need to keep upgrading your raw software just because you have a new camera, or because someone sends you a DNG they took. Just open it, an it's optimized.

The revolution this would be is nothing short of astounding.

There's nothing limiting about Raw per se, after all, they allow you access to all of the data available from the sensor. However I do agree with you, a standardised format would be better without doubt, but it's only worthwhile if you can tap into the full data set for your camera.

And you can with DNG. That is precisely the point of DNG, and no argument you have provided (or can provide) has shown otherwise.

The problem we have is that the camera manufacturers are reluctant to licence the information they use in their Raw formats.

D'oh.

Jesper

 

[theswede #100093]

As I said below, the camera and the software will only work well together if the camera has been profiled by the raw converter maker. And for that, I will always have to buy new versions of my raw converter, DNG or not.

This only holds up if the raw converter you're talking about is always superior to the raw converter provided by the camera maker, which uses the same profile which would be embedded in the DNG. In practice the difference tends to be small, and usually down to algorithm and not parameters. And that difference will carry over if the profile is supplied by the camera manufacturer instead of reverse engineered.

No, they don't. Why do you think Apple adds support for raw files from cameras that save their raw data natively in DNG files?

Because marketing. If the camera is not listed as "supported" Apple will get bad writeup for that.

Obviously because they tested/profiled the camera and adjusted their raw conversion parameters to optimize the result.

That's not in the least obvious. It's expensive and unrewarding to spend money on that when the camera manufacturer already has done it. What is obvious is that Apple needs to list every camera in their "supported" section or people will not buy their software.

Thus, people don't use the default profile (which might or might not be saved by the camera in the DNG file), they use Apple's profile.

This does not at all follow. If the profile is embedded it was most likely extracted and instated in Apple's software as the default profile, or the embedded profile is actually used every time. There is no reason for Apple to do any more work than this, since the best they can do is duplicate the results they would have if they do this.

And the same applies for Adobe, they explicitly add support for new camera models that save their raw data as DNG files. Thus, again, people won't use the default profile, they will use the profile created by Adobe.

And the same applies for Adobe. If they do not list the camera as "supported", they will get slammed in media and people will avoid their software. Therefore they list the camera as "supported" no matter how much or little work (down to no work at all) they had to do to make that happen.

Jesper

 

[Andy Hewitt]

theswede wrote:

None of the formats have any variations allowed outside of specification. None at all.

Yes, I get that, but they do allow for variations within the specification, which is what I was talking about, I never mentioned anything about modifying the specification outside of the standard - that just creates a new file format.

But you shouldn't use TIFF anyway.

Why not?

Manufacturer raw files are entirely proprietary, and the have their own names for tags containing standard information. That is precisely why DNG would be better.

Although they do also add some non-standard information, and some even pre-process the images (some Nikons, and I suspect Canons too, have been known to apply noise reduction prior to saving the file).

The revolution this would be is nothing short of astounding.

Once you get the big manufacturers to accept they can no longer encrypt the data.

There's nothing limiting about Raw per se, after all, they allow you access to all of the data available from the sensor. However I do agree with you, a standardised format would be better without doubt, but it's only worthwhile if you can tap into the full data set for your camera.

And you can with DNG. That is precisely the point of DNG, and no argument you have provided (or can provide) has shown otherwise.

DNG sourced directly from the camera, yes. Converted DNGs maybe not - at least not until all the Raw image data files are unencrypted, or we get 100% accurate reverse engineering (which probably breaks the licensing agreements).

Either way, the worst thing for current Raw image users is if DNG did become a standard format. It would then for sure kill off support for proprietary Raw files. While everybody continues to create different Raw files, then software will have to continue supporting it.

 

[theswede #100093]

Yes, I get that, but they do allow for variations within the specification, which is what I was talking about, I never mentioned anything about modifying the specification outside of the standard - that just creates a new file format.

JPG allows no variation at all of the image. None. There is one compression method allowed, and one format to save it in. The only variation is the metadata container, and of the two variations allowed both can be present in the same file at the same time negating any problems.

DNG allows no variation at all except versions, which are backwards compatible by design so that won't matter. The spec is rigid.

TIFF is the only exception, but it has a robust mechanism for handling the variety. It's been superseded for pretty much all uses, and is really showing its age, but it's incredibly well engineered.

But you shouldn't use TIFF anyway.

Why not?

Because there are better image formats for everything you'd want to use TIFF for. The sole exception is if you want a 16 bit per color image. Then TIFF is the most portable format.

Although they do also add some non-standard information, and some even pre-process the images (some Nikons, and I suspect Canons too, have been known to apply noise reduction prior to saving the file).

No non-standard information of any value. And the pre-processing is outside of file format considerations.

The revolution this would be is nothing short of astounding.

Once you get the big manufacturers to accept they can no longer encrypt the data.

They don't now, so that would be no change at all.

DNG sourced directly from the camera, yes. Converted DNGs maybe not - at least not until all the Raw image data files are unencrypted, or we get 100% accurate reverse engineering (which probably breaks the licensing agreements).

Reverse engineering is fully legal and breaches no legal licensing agreement. And raw images are not encrypted. That would be rather pointless, as the whole purpose of taking photos is to view them.

Either way, the worst thing for current Raw image users is if DNG did become a standard format. It would then for sure kill off support for proprietary Raw files. While everybody continues to create different Raw files, then software will have to continue supporting it.

On the contrary. This would be the best thing possible for current raw image users, since all present raw images (with a few rare exceptions) can easily be converted to DNG, allowing them to be used with the vast proliferation of tools this would create.

The usefulness of raw images would become much greater, and that would benefit every raw image ever taken which can be converted to DNG - which is almost all of them.

I can think of nothing that would benefit digital photography more than the universal adoption of a single, standardized raw format with embedded camera profiles. Nothing at all. It would allow for so much technical progress it's impossible to even imagine the scope of it.

Jesper

 

[Andy Hewitt]

theswede wrote:

Yes, I get that, but they do allow for variations within the specification, which is what I was talking about, I never mentioned anything about modifying the specification outside of the standard - that just creates a new file format.

JPG allows no variation at all of the image. None. There is one compression method allowed, and one format to save it in. The only variation is the metadata container, and of the two variations allowed both can be present in the same file at the same time negating any problems.

So, there's no choice of 'progressive' or 'standard' compression then?

DNG allows no variation at all except versions, which are backwards compatible by design so that won't matter. The spec is rigid.

TIFF is the only exception, but it has a robust mechanism for handling the variety. It's been superseded for pretty much all uses, and is really showing its age, but it's incredibly well engineered.

But you shouldn't use TIFF anyway.

Why not?

Because there are better image formats for everything you'd want to use TIFF for. The sole exception is if you want a 16 bit per color image. Then TIFF is the most portable format.

Which is what most people use if they want highest quality images. Or of course there's JPG2000 .

The revolution this would be is nothing short of astounding.

Once you get the big manufacturers to accept they can no longer encrypt the data.

They don't now, so that would be no change at all.

Hmm apart from this quote from the Wiki page on Raw files:

"Several major camera manufacturers, including Nikon, Canon and Sony, encrypt portions of the file in an attempt to prevent third-party tools from accessing them"

DNG sourced directly from the camera, yes. Converted DNGs maybe not - at least not until all the Raw image data files are unencrypted, or we get 100% accurate reverse engineering (which probably breaks the licensing agreements).

Reverse engineering is fully legal and breaches no legal licensing agreement. And raw images are not encrypted. That would be rather pointless, as the whole purpose of taking photos is to view them.

See above.

Either way, the worst thing for current Raw image users is if DNG did become a standard format. It would then for sure kill off support for proprietary Raw files. While everybody continues to create different Raw files, then software will have to continue supporting it.

On the contrary. This would be the best thing possible for current raw image users, since all present raw images (with a few rare exceptions) can easily be converted to DNG, allowing them to be used with the vast proliferation of tools this would create.

Providing the conversion is actually clean, and works properly in all software - if you read the link I posted about Aperture and DNGs, you'll see that this is not always the case.

The usefulness of raw images would become much greater, and that would benefit every raw image ever taken which can be converted to DNG - which is almost all of them.

No, it isn't. For example, I could not convert my Olympus images to DNG and continue using them in Aperture, as it does not support the format that they're converted to.

I can think of nothing that would benefit digital photography more than the universal adoption of a single, standardized raw format with embedded camera profiles. Nothing at all. It would allow for so much technical progress it's impossible to even imagine the scope of it.

Strange as it seems, I do agree with you that a single image format would be much better, but I disagree that it wouldn't affect existing Raw users.

 

[theswede #100093]

So, there's no choice of 'progressive' or 'standard' compression then?

No. Only between storage order in the file. The compression method is the same. The same applies to using Huffman optimization. It's a reordering of data, and does not affect the way the image is stored (and thus read).

Which is what most people use if they want highest quality images. Or of course there's JPG2000 .

It's not what people should use if they want highest quality images. It's what people should use if they want maximum portability. You should not use TIFF, except to get images into whatever you're going to edit with. If you're just after high quality save it as PNG, or if you're editing as PSD or XCF depending on your preferred poison.

JPG2000 suffers from splintered standard (it's a lot like TIFF in this regard), high processing demands, suspected submarine patents and small gains over well established and much less complex standards. It'll never catch on.

Hmm apart from this quote from the Wiki page on Raw files:

"Several major camera manufacturers, including Nikon, Canon and Sony, encrypt portions of the file in an attempt to prevent third-party tools from accessing them"

A fancy way of saying they're scaling the parameters oddly. It's no big deal to figure out

Providing the conversion is actually clean, and works properly in all software - if you read the link I posted about Aperture and DNGs, you'll see that this is not always the case.

Considering that there's be thousands (at least) applications which can handle raw files if they were a single standard it doesn't much matter if a few of them are broken. They'll get lost in the dust of the best few hundred soon enough.

You're still thinking in artificial scarcity created by ever changing raw formats creating a limited playfield. Standardized raw would make mincemeat of Aperture, or force it to get its act together.

The usefulness of raw images would become much greater, and that would benefit every raw image ever taken which can be converted to DNG - which is almost all of them.

No, it isn't. For example, I could not convert my Olympus images to DNG and continue using them in Aperture, as it does not support the format that they're converted to.

This is a problem with Aperture. Not with DNG. And definitely not if DNG was the standard for all cameras and software going forward.

Strange as it seems, I do agree with you that a single image format would be much better, but I disagree that it wouldn't affect existing Raw users.

Oh, it would affect them all right! It would make their lives so much easier and better in the digital dark room it's almost impossible to imagine!

Apple would hate it though. They'd be forced to fix Aperture.

Jesper

 

[noirdesir]

theswede wrote:

As I said below, the camera and the software will only work well together if the camera has been profiled by the raw converter maker. And for that, I will always have to buy new versions of my raw converter, DNG or not.

This only holds up if the raw converter you're talking about is always superior to the raw converter provided by the camera maker, which uses the same profile which would be embedded in the DNG. In practice the difference tends to be small, and usually down to algorithm and not parameters. And that difference will carry over if the profile is supplied by the camera manufacturer instead of reverse engineered.

Since almost all camera maker supplied raw converters suck (exception Capture One and maybe Phocus, which I know little about) most people are not served by metadata that only the camera brand raw converter can make use of. And our discussion here by definition is about third-party raw converters because the camera-branded raw converters can get any sensor specifications directly from the manufacturer. They don't need metadata in the raw files.

No, they don't. Why do you think Apple adds support for raw files from cameras that save their raw data natively in DNG files?

Because marketing. If the camera is not listed as "supported" Apple will get bad writeup for that.

Why then does Adobe do the same? And why does Apple support only some of the cameras with DNG raws? And why do both companies sometimes start to officially support DNG cameras only in their second (or third) raw support update counting from the day the camera hit the shelfs?

Obviously because they tested/profiled the camera and adjusted their raw conversion parameters to optimize the result.

That's not in the least obvious. It's expensive and unrewarding to spend money on that when the camera manufacturer already has done it. What is obvious is that Apple needs to list every camera in their "supported" section or people will not buy their software.

There is a significant number of DNG cameras that are not on Apple's list (and even much more non-DNG cameras). Why would that be?

And the same applies for Adobe, they explicitly add support for new camera models that save their raw data as DNG files. Thus, again, people won't use the default profile, they will use the profile created by Adobe.

And the same applies for Adobe. If they do not list the camera as "supported", they will get slammed in media and people will avoid their software. Therefore they list the camera as "supported" no matter how much or little work (down to no work at all) they had to do to make that happen.

Yeah, and they even pretend to some work by waiting several weeks, sometimes months before they add support.

 

[theswede #100093]

Why then does Adobe do the same?

Because marketing.

And why does Apple support only some of the cameras with DNG raws?

You're asking why does Apple lag behind on features and appear to have stopped supporting Aperture. I have no idea, but no matter the reason I hope they stop doing it soon.

And why do both companies sometimes start to officially support DNG cameras only in their second (or third) raw support update counting from the day the camera hit the shelfs?

Because that's when they get down to it. Have you ever worked for a large corporation? Do you know what is involved in getting as much as a one keyword change pushed from development through the testing path out to production?

There is a significant number of DNG cameras that are not on Apple's list (and even much more non-DNG cameras). Why would that be?

I have no idea why Apple appear to have left Aperture behind. It's not doing them any favors, whatever the reason.

Yeah, and they even pretend to some work by waiting several weeks, sometimes months before they add support.

That's actually pretty close to what it feels like when you're trying to push a change through at a major company; "pretend to some work". The red tape is just insane.

Jesper

 

[noirdesir]

theswede wrote:

Why then does Adobe do the same?

Because marketing.

And why does Apple support only some of the cameras with DNG raws?

You're asking why does Apple lag behind on features and appear to have stopped supporting Aperture. I have no idea, but no matter the reason I hope they stop doing it soon.

And why do both companies sometimes start to officially support DNG cameras only in their second (or third) raw support update counting from the day the camera hit the shelfs?

Because that's when they get down to it. Have you ever worked for a large corporation? Do you know what is involved in getting as much as a one keyword change pushed from development through the testing path out to production?

There is a significant number of DNG cameras that are not on Apple's list (and even much more non-DNG cameras). Why would that be?

I have no idea why Apple appear to have left Aperture behind. It's not doing them any favors, whatever the reason.

Yeah, and they even pretend to some work by waiting several weeks, sometimes months before they add support.

That's actually pretty close to what it feels like when you're trying to push a change through at a major company; "pretend to some work". The red tape is just insane.

You have become a person fixated with the belief that a DNG file contains all information necessary for any raw converter to convert the raw data to the raw converter's satisfaction. You base this only on the fact (I'll believe you for the moment that it is a fact) that a DNG file contains some information about the colour response.

You have no proof that this is actually the case and you discredit any circumstantial evidence to the contrary using far-fetched theories. I'm afraid you have become immune to any rational arguments which makes any further discussion pointless.

 

[theswede #100093]

You have become a person fixated with the belief that a DNG file contains all information necessary for any raw converter to convert the raw data to the raw converter's satisfaction.

Thanks for the psych eval, but I pay better qualified people for that.

No, a DNG does not necessarily contain all information necessary for the raw converter's satisfaction - but then, neither does any other raw files or raw converters would not require any user adjustable values.

However, there are no limitations stopping a DNG from containing a full camera profile and sensible default values leading to as good results as the default settings in a typical raw converter which contains a dedicated profile for the camera used.

You base this only on the fact (I'll believe you for the moment that it is a fact) that a DNG file contains some information about the colour response.

I wonder where that came from. A DNG file can contain that, the standard allows it, but even if it doesn't you're only marginally worse off with a DNG you have no profile in than with a supported raw file. Once you've set one conversion up as you like it you can use that as your default for that camera model in the future. And that's the worst case.

You have no proof that this is actually the case and you discredit any circumstantial evidence to the contrary using far-fetched theories. I'm afraid you have become immune to any rational arguments which makes any further discussion pointless.

If you have any rational arguments, feel free to test this hypothesis.

Jesper

 

[noirdesir]

theswede wrote:

You have become a person fixated with the belief that a DNG file contains all information necessary for any raw converter to convert the raw data to the raw converter's satisfaction.

No, a DNG does not necessarily contain all information necessary for the raw converter's satisfaction - but then, neither does any other raw files or raw converters would not require any user adjustable values.

However, there are no limitations stopping a DNG from containing a full camera profile and sensible default values leading to as good results as the default settings in a typical raw converter which contains a dedicated profile for the camera used.

There are also no limitations stopping, eg, a NEF file, from containing a full camera profile and sensible default values (and documentation by Nikon how to read them) leading to as good results as the default settings in a typical raw converter which contains a dedicated profile for the camera used.

Thus, if my grandmother had wheels she'd be a wagon. You seem to have a problem distinguishing between what is possible in principle and what is reality.

You base this only on the fact (I'll believe you for the moment that it is a fact) that a DNG file contains some information about the colour response.

I wonder where that came from. A DNG file can contain that, the standard allows it, but even if it doesn't you're only marginally worse off with a DNG you have no profile in than with a supported raw file. Once you've set one conversion up as you like it you can use that as your default for that camera model in the future. And that's the worst case.

Except for Adobe where you users are given a tool to match colours automatically in order to create a full conversion profile with relative ease, this is normally next to impossible because trying to match all colours on a test chart manually with only the usual controls like WB (temp and tint), tone response curve, saturation and individual colour adjustments is just not possible. You'd have to match potentially dozens of colours with dozens of sliders. That is an optimisation problem that is simply impossible to solve by hand.

You have no proof that this is actually the case and you discredit any circumstantial evidence to the contrary using far-fetched theories. I'm afraid you have become immune to any rational arguments which makes any further discussion pointless.

If you have any rational arguments, feel free to test this hypothesis.

Rational includes reasonable, which in opposition to my arguments yours have not been.