DNG converter, older LR workflow help

[dmokn]

I've shot in RAW and converted to DNG in Lightroom for many years. In interest of reducing unnecessary bulk in my RAW file sizes, I'm looking for some ideas working through how to incorporate Adobe DNG converter into my workflow with Lightroom 6 (the last one pre-subscription). I have two questions related to dealing with unexpected duplicate files:

1. I just tried importing a couple photos directly from my SD card with the following settings:


Note that I was experimenting with compressing the RAW down to 10MP.

I was able to import the photos into LR fine. The problem I'm seeing is that if I look in my Windows folder, there are duplicates of the new DNG files, each with a ...-2 extension. When I "delete from disk" in LR (the option I've always used), the duplicate file seems to stay on my hard drive, even as the preview of the original file disappears as expected in LR. Any ideas?

2. Based on my quick experimentation, I'm not seeing much file degradation in going down to 10MP, especially for my less important shots. So I'm interested in compressing the majority of my older catalog. When I tried doing this in one folder, it created duplicates of all my photos, which I think would mean I'd have to go through every folder in LR, delete all original photos, and then import all the new compressed ones. Given years of folders and thousands of photos, this seems super onerous. Any ideas for a better workflow on this?

Thanks in advance.

 

[dj_paige]

You are getting the -2 because you either ran the DNG converter twice, or you imported the DNG using COPY as DNG.

You shouldn't see any degradation in image quality when converting to DNG. So your statement "not seeing much file degradation" implies you are seeing some. That doesn't make sense to me. I assume you "file degradation" you mean "image quality" degradation.

Please don't ever think about re-importing photos, that is a terrible mistake, and unnecessary. I don't have Lr 6 any more, but in the Library Module, you should be able to select the desired photo(s) and then use the menu commend Library->Convert Photo to DNG.

So I'm interested in compressing the majority of my older catalog. When I tried doing this in one folder, it created duplicates of all my photos

I don't know what you did here.

 

[dmokn]

Thanks for the quick reply. I figured out my first problem, which was almost what you said. By habit I had changed "Copy as DNG" to "Copy" when what I needed to do was change it to "Add," which worked.

Yes, by "file degradation" I did mean "degrading the image quality." What I was trying to get at was I have thousands of old previously-converted DNG files that I want to use the DNG converter to compress down to 10MP, which in my little experimentation so far, doesn't seem to degrade the image quality too much. I just haven't figured out a way to do a bulk conversion that replaces the DNGs I have versus create thousands of new versions.

 

[ishwanu]

You shouldn't see any degradation in image quality when converting to DNG. So your statement "not seeing much file degradation" implies you are seeing some. That doesn't make sense to me. I assume you "file degradation" you mean "image quality" degradation.

The degradation comment is a little vague. It is pretty impressive that the lossy compression algorithm can maintain something very close to the original pixel data in a file that's dramatically smaller in size, but it does introduce some degradation.

When he also reduces the pixel count, that's another layer of 'degradation'.

 

[dj_paige]

What I was trying to get at was I have thousands of old previously-converted DNG files that I want to use the DNG converter to compress down to 10MP, which in my little experimentation so far, doesn't seem to degrade the image quality too much. I just haven't figured out a way to do a bulk conversion that replaces the DNGs I have versus create thousands of new versions.

I am not aware that you can change the megapixels of a DNG via any method.

 

[ishwanu]

I just haven't figured out a way to do a bulk conversion that replaces the DNGs I have versus create thousands of new versions.

That would be pretty easy if you didn't need to maintain the consistency of a catalog: Tell the converter to place the new files in a temporary folder somewhere else, then overwrite the originals with them and delete the temporary folder. But the DAM that I use would see that they've all changed and would want me to rescan them, and I assume the Lightroom catalog would have even deeper issues.

 

[ishwanu]

What I was trying to get at was I have thousands of old previously-converted DNG files that I want to use the DNG converter to compress down to 10MP, which in my little experimentation so far, doesn't seem to degrade the image quality too much. I just haven't figured out a way to do a bulk conversion that replaces the DNGs I have versus create thousands of new versions.

I am not aware that you can change the megapixels of a DNG via any method.

 

[dj_paige]

What I was trying to get at was I have thousands of old previously-converted DNG files that I want to use the DNG converter to compress down to 10MP, which in my little experimentation so far, doesn't seem to degrade the image quality too much. I just haven't figured out a way to do a bulk conversion that replaces the DNGs I have versus create thousands of new versions.

I am not aware that you can change the megapixels of a DNG via any method.

Thanks, now I know.

To the original poster: I think the whole idea is just a recipe for mistakes to screw up your catalog, and one that takes a lot of work. I wouldn't do it. I also don't know if the edits you performed could be invalidated by replacing a DNG with a new DNG that has fewer pixels, especially local adjustments.

--
Paige Miller

 

[graybalanced]

Given years of folders and thousands of photos, this seems super onerous. Any ideas for a better workflow on this?

This problem was solved by the better workflow Adobe designed for Lightroom Classic that was described by dj_paige:

I don't have Lr 6 any more, but in the Library Module, you should be able to select the desired photo(s) and then use the menu commend Library->Convert Photo to DNG.

I also can’t remember if Lightroom 6 had the Convert Photo to DNG command, but if it does, that is an infinitely better way to do this. Because, the Convert Photo to DNG command does it this way:

1. You select any number of photos in your catalog. 1, 2, or a few thousand.
2. You do the Convert Photo to DNG command.
3. Lightroom does these for you:

• Offers same options as DNG Converter
• Converts a copy to DNG
• In the catalog database, it swaps out the original with the new DNG copy
• Deletes original (optional)
• Transfers all History, Metadata, etc. from original to DNG copy

It does all that housekeeping for you. No muss, no fuss.

The beauty of this of course, is that by auto-replacing originals with DNG copies in place, all your existing Develop edits, and all your original folder organization, and all your original collection organization, and all your keywords, captions, map location assignments etc are preserved and transferred to the DNG! You lose no work!

If you use DNG Converter independently instead and then import those, you are like starting over from scratch. History steps lost, collections lost, saved slide shows lost...

If you can do the mass DNG conversion in Lightroom itself you will save untold amounts of time and data.

 

[dmokn]

Given years of folders and thousands of photos, this seems super onerous. Any ideas for a better workflow on this?

This problem was solved by the better workflow Adobe designed for Lightroom Classic that was described by dj_paige:

I don't have Lr 6 any more, but in the Library Module, you should be able to select the desired photo(s) and then use the menu commend Library->Convert Photo to DNG.

I also can’t remember if Lightroom 6 had the Convert Photo to DNG command, but if it does, that is an infinitely better way to do this. Because, the Convert Photo to DNG command does it this way:

1. You select any number of photos in your catalog. 1, 2, or a few thousand.
2. You do the Convert Photo to DNG command.
3. Lightroom does these for you:

• Offers same options as DNG Converter
• Converts a copy to DNG
• In the catalog database, it swaps out the original with the new DNG copy
• Deletes original (optional)
• Transfers all History, Metadata, etc. from original to DNG copy

It does all that housekeeping for you. No muss, no fuss.

The beauty of this of course, is that by auto-replacing originals with DNG copies in place, all your existing Develop edits, and all your original folder organization, and all your original collection organization, and all your keywords, captions, map location assignments etc are preserved and transferred to the DNG! You lose no work!

If you use DNG Converter independently instead and then import those, you are like starting over from scratch. History steps lost, collections lost, saved slide shows lost...

If you can do the mass DNG conversion in Lightroom itself you will save untold amounts of time and data.

Thanks for the ideas, though things do look different on my end. Note that I have been using this method generally for many years, whether importing directly as DNG or importing in the native RAW format and then converting within LR later. So LR 6 does have the "Convert Photo to DNG" option, and it does keep any previous edits, but it doesn't have those compression options from DNG converter if that's what you're saying the new LR has. Here's how mine looks:



It looks like you guys are right that if I compress my current DNGs, it creates new copies without all my edits. The annoying workaround is importing them into the same folder as the originals and then syncing the edits one by one. I was hoping there would be a better way!

 

[graybalanced]

Thanks for the ideas, though things do look different on my end. Note that I have been using this method generally for many years, whether importing directly as DNG or importing in the native RAW format and then converting within LR later. So LR 6 does have the "Convert Photo to DNG" option, and it does keep any previous edits, but it doesn't have those compression options from DNG converter if that's what you're saying the new LR has.

OK, that screen shot helps fill in the blanks. Yes, support for lossy DNG in that feature was added in a later version of Lightroom Classic.

 

[ishwanu]

Thanks for the ideas, though things do look different on my end. Note that I have been using this method generally for many years, whether importing directly as DNG or importing in the native RAW format and then converting within LR later. So LR 6 does have the "Convert Photo to DNG" option, and it does keep any previous edits, but it doesn't have those compression options from DNG converter if that's what you're saying the new LR has.

OK, that screen shot helps fill in the blanks. Yes, support for lossy DNG in that feature was added in a later version of Lightroom Classic.

The lossy compression option is there. Apparently the option to reduce resolution is missing.

Using lossy compression alone can reduce the file size dramatically, so maybe that's enough.

 

[dmokn]

The lossy compression option is there. Apparently the option to reduce resolution is missing.

Using lossy compression alone can reduce the file size dramatically, so maybe that's enough.

That's a good call. I tried it and while it didn't let me finetune it at all, the compression reduced the file size almost as much as when I was reducing it to a 10MP in DNG converter. I think this is the move, thank you!

 

[ishwanu]

The lossy compression option is there. Apparently the option to reduce resolution is missing.

Using lossy compression alone can reduce the file size dramatically, so maybe that's enough.

That's a good call. I tried it and while it didn't let me finetune it at all, the compression reduced the file size almost as much as when I was reducing it to a 10MP in DNG converter. I think this is the move, thank you!

Glad that works for you. What was the uncompressed size?

 

[charlyw64]

The lossy compression option is there. Apparently the option to reduce resolution is missing.

Using lossy compression alone can reduce the file size dramatically, so maybe that's enough.

That's a good call. I tried it and while it didn't let me finetune it at all, the compression reduced the file size almost as much as when I was reducing it to a 10MP in DNG converter. I think this is the move, thank you!

What you are doing is a surefire way to botch your whole image collection and lock you into the Adobe cloud deeper than using the original RAW and a current LrC subscription would ever do!

Compressed DNG are a late addition to the standard and the change can not be undone. You lose the ability to use many of the other DNG converters and when (not if) LR 6 bites the dust due to OS incompatibilities you will be forced to go with the LrC subscription - but with images that have had their quality significantly reduced by two independent detrimental processes, 1. old demosaicing process, 2. lossy compression. If I were to advise you I would strongly suggest that you keep the original RAW files (before conversion to DNG) if you do not intend to subscribe to Adobe LrC ever.

 

[dmokn]

Glad that works for you. What was the uncompressed size?

My original DNGs were usually 13-16 MB and the compression took them down to 5-7 MB. I ended up doing a long process of de-selecting all my "best of" shots and then running the lossy conversion on the rest. I saved probably 100 GB on my hard drive so thanks again!

 

[dmokn]

What you are doing is a surefire way to botch your whole image collection and lock you into the Adobe cloud deeper than using the original RAW and a current LrC subscription would ever do!

Compressed DNG are a late addition to the standard and the change can not be undone. You lose the ability to use many of the other DNG converters and when (not if) LR 6 bites the dust due to OS incompatibilities you will be forced to go with the LrC subscription - but with images that have had their quality significantly reduced by two independent detrimental processes, 1. old demosaicing process, 2. lossy compression. If I were to advise you I would strongly suggest that you keep the original RAW files (before conversion to DNG) if you do not intend to subscribe to Adobe LrC ever.

Unfortunately I ran the process before seeing this, but thanks for the info. I did keep the original size DNGs for all my most important shots, and all of those have been exported as full-size JPEGs to the cloud as well. If I upgrade my camera in the future, I might move to shooting in RAW+JPG and just keep the RAWS for more important shots.

Curious why you recommend keeping the original RAWs versus DNGs in terms of future-proofing?

 

[charlyw64]

What you are doing is a surefire way to botch your whole image collection and lock you into the Adobe cloud deeper than using the original RAW and a current LrC subscription would ever do!

Compressed DNG are a late addition to the standard and the change can not be undone. You lose the ability to use many of the other DNG converters and when (not if) LR 6 bites the dust due to OS incompatibilities you will be forced to go with the LrC subscription - but with images that have had their quality significantly reduced by two independent detrimental processes, 1. old demosaicing process, 2. lossy compression. If I were to advise you I would strongly suggest that you keep the original RAW files (before conversion to DNG) if you do not intend to subscribe to Adobe LrC ever.

Unfortunately I ran the process before seeing this, but thanks for the info. I did keep the original size DNGs for all my most important shots, and all of those have been exported as full-size JPEGs to the cloud as well. If I upgrade my camera in the future, I might move to shooting in RAW+JPG and just keep the RAWS for more important shots.

Curious why you recommend keeping the original RAWs versus DNGs in terms of future-proofing?

Because DNG are anything but future proof - IMHO they only are temporary files within the Adobe eco system and they are not long term viable storage. They do lock you into the Adobe eco system more effectively than any other measure you could have made. Since you have no interest of updating to the subscription this is the completely wrong way of preserving your photos. You effectively have already destroyed your way of ever using the original manufacturers tools or about 80-90% of third party tools, especially with the lossy compression (which few 3rd party tools can even open properly). You should also consider that LR rewrites the DNG every time you make edits (if you have it set to write the changes not only to the catalog but also record the changes in the file system) and thus you need to backup the file completely, this will increase your backup costs (in time and space) considerably - and backup you must because every time the DNG is rewritten risks losing the file to corruption, thus it is imperative to keep several backup generations and regularly check these backups for viability.

LR6 was far from error free, I wouldn't touch old software with a long poking stick because it never gets checked by the manufacturer that it will still operate fully on newer operating system versions, I am a software developer myself and I wouldn't have used LR6 for more than a year beyond the availability of the successor because of the constantly changing environment!

 

[dmokn]

Curious why you recommend keeping the original RAWs versus DNGs in terms of future-proofing?

Because DNG are anything but future proof - IMHO they only are temporary files within the Adobe eco system and they are not long term viable storage. They do lock you into the Adobe eco system more effectively than any other measure you could have made. Since you have no interest of updating to the subscription this is the completely wrong way of preserving your photos. You effectively have already destroyed your way of ever using the original manufacturers tools or about 80-90% of third party tools, especially with the lossy compression (which few 3rd party tools can even open properly). You should also consider that LR rewrites the DNG every time you make edits (if you have it set to write the changes not only to the catalog but also record the changes in the file system) and thus you need to backup the file completely, this will increase your backup costs (in time and space) considerably - and backup you must because every time the DNG is rewritten risks losing the file to corruption, thus it is imperative to keep several backup generations and regularly check these backups for viability.

LR6 was far from error free, I wouldn't touch old software with a long poking stick because it never gets checked by the manufacturer that it will still operate fully on newer operating system versions, I am a software developer myself and I wouldn't have used LR6 for more than a year beyond the availability of the successor because of the constantly changing environment!

Good info, thanks. I guess I figured -- maybe incorrectly -- that the DNG format seems better poised to last long term than Olympus or Panasonic RAW file formats. But I suppose I might succumb to an Adobe subscription someday if I must. Do you have a favorite alternative that feels most universal and future-proof?

 

[charlyw64]

Curious why you recommend keeping the original RAWs versus DNGs in terms of future-proofing?

Because DNG are anything but future proof - IMHO they only are temporary files within the Adobe eco system and they are not long term viable storage. They do lock you into the Adobe eco system more effectively than any other measure you could have made. Since you have no interest of updating to the subscription this is the completely wrong way of preserving your photos. You effectively have already destroyed your way of ever using the original manufacturers tools or about 80-90% of third party tools, especially with the lossy compression (which few 3rd party tools can even open properly). You should also consider that LR rewrites the DNG every time you make edits (if you have it set to write the changes not only to the catalog but also record the changes in the file system) and thus you need to backup the file completely, this will increase your backup costs (in time and space) considerably - and backup you must because every time the DNG is rewritten risks losing the file to corruption, thus it is imperative to keep several backup generations and regularly check these backups for viability.

LR6 was far from error free, I wouldn't touch old software with a long poking stick because it never gets checked by the manufacturer that it will still operate fully on newer operating system versions, I am a software developer myself and I wouldn't have used LR6 for more than a year beyond the availability of the successor because of the constantly changing environment!

Good info, thanks. I guess I figured -- maybe incorrectly -- that the DNG format seems better poised to last long term than Olympus or Panasonic RAW file formats. But I suppose I might succumb to an Adobe subscription someday if I must. Do you have a favorite alternative that feels most universal and future-proof?

The only long term viable file type is basically the manufacturer RAW file. The assumption for creating DNG and pushing them (assuming altruistic behavior by Adobe here) was that an open, well specified RAW container format would be preferable to manufacturer specific RAW files. That has led to a completely bloated DNG file specification that only ever will be fully implemented by Adobe - all the rest of the manufacturers pick and mix the features they can easily support. And some things could not have been foreseen when the specification was developed (like masked sensor rows, the way sensor calibration must be performed, non-standard color filter array like RGGB or x-trans) and thus were latched onto the ever growing complexity of the DNG file format.