darktable 3.4.0 released

Started Dec 24, 2020 | Discussions thread
OP Henry Richardson Forum Pro • Posts: 20,575
Great info on undocumented stuff
3

Henry Richardson wrote:

I will write about my experience playing around with the darktable import function that imports Lightroom XMP files along with images and tries to translate some of the data for use by darktable. First, take a look at the following 2 links and then below you will see things I learned that are undocumented and I have never seen written about anywhere despite having searched, watching a couple of videos, etc.

Here is a 2013 article written by the developer about importing Lightroom XMP files along with image files into darktable:

https://www.darktable.org/2013/02/importing-lightroom-development/

You can also read the 3.4 manual info about importing Lightroom XMP files here:

https://www.darktable.org/usermanual/en/overview/sidecar-files/sidecar-import/

There are 2 different types of data to import:

  1. metadata (keywords (tags), star ratings, color labels, etc.)
  2. editing instructions

I experimented with it this year by importing a few of my files along with their Lightroom XMP sidecar files to see what darktable would do. I was using darktable 3.2.1 which was the release before this new 3.4.0.

I have 112k photos in my Lightroom catalog: 110k out-of-camera originals + 2k scanned film files. They all have keywords (tags), most have star ratings, many have color labels, many are in collections, etc. About 50k have been edited. I often go back to earlier edited photos and tweak them a bit. On rare occasions I undo all the edits and start over, but mostly just a few changes to an already edited photo. Some people suggest saving all the edited files to 16-bit TIFF files with the Lightroom editing baked in (sRGB or Adobe RGB or some other color space) which turns my 16.5mb Olympus PEN-F 20mp raw file into an 86mb compressed 16-bit TIFF file. And it turns my 3.85mb Olympus E-M10II 16mp JPEG file into an 83.5mb compressed 16-bit TIFF file. Now multiply that by all my files. And besides the explosion of storage space you are still stuck with baked in editing, baked in color space, etc. And you have thrown away all the advantages of non-destructive editing. So, for some people this is a reasonable solution, but it is not for me. On1, C1, ACDSee, and probably others also try to import the editing data from Lightroom, but none are even close to perfect. This is a very difficult thing to do and I no longer have any expectation that it is worthwhile to even do it to my photos.

My feeling these days is if I switch away from Lightroom then I most care about getting my keywords (tags), star ratings, color labels, collections, etc. since the editing translation to a different program is so far from perfect. With darktable you can get keywords (tags), star ratings, and color labels, but not collections. On1, C1, and ACDSee use the Lightroom catalog for the migration which has the collection data too whereas darktable uses the optional XMP files for each individual image file which is generated by Lightroom (you can get them by selecting your images and then hitting Ctrl-S to get them written). It is understandable why the free darktable uses the XMP files rather than the catalog. Adobe pretty regularly (every 2-3 years) changes the format of the catalog so if someone decodes it they have to keep decoding the new formats and updating the source code for the importer. The XMP files are text files and (presumably) are much easier to work with and they don't change.

Here are some important undocumented things I discovered:

  1. When you create the XMP files in Lightroom it only writes XMP files for raw files. If you have edited TIFFs and JPEGs also then there will be no XMP file for them so the editing info is lost and cannot be imported by darktable. Turns out this is very good thing for the reason I will mention below. The keywords (tags), star ratings, color labels, etc. are written in the XMP file for raw files and are written directly into the TIFF and JPEG files.
  2. When you import with darktable it picks up the keywords (tags), star ratings, color labels, etc. from the XMP file and also from the embedded data in TIFFs and JPEGs.
  3. You will see from the links above that darktable can only translate a small subset of the Lightroom editing instructions so right off the bat you can see this is not all that useful.
  4. The documentation says nothing about base curve (display-preferred), filmic (scene-preferred), or none. The darktable default is to use base curve (display-preferred), but it can be changed in preferences to either filmic (scene-preferred) or none. I spent a lot of time trying to figure this out with regards to importing Lightroom XMP files. After some research I found that the importer should not use either base curve or filmic when importing these, but it does. You have to set your preferences to none and then import. Even now I am not 100% sure about this, but from asking it seems like most people think that is what needs to be done and if you think about it then it probably makes sense.
  5. The darktable translation of even the subset of editing instructions is poor. I imported a few test photos with Lightroom XMP and the results were not good. Better off to not even do it since you have to undo it all and then manually turn on the base curve or filmic (depending on whether you want to use the old display-referred editing or the new scene-referred editing) before starting your editing. If you had any dust spot removal in Lightroom then darktable tries to duplicate it, but in my test it often did it in the wrong location so worse than not doing anything.
  6. Very unfortunately darktable gives you absolutely no control over how or even if it imports stuff from a Lightroom XMP file. Extremely bizarre because the darktable developers are constantly making everything even more complex with a huge number of often almost redundant modules, many knobs and levers to control every cryptic aspect of the module, etc. But in this case if there is an XMP file darktable automatically imports the data and does imperfect, incorrect edits on the subset of Lightroom instructions that it recognizes. You can't turn that off and you have zero control.
  7. One might think that you can use the darktable discard history command to get rid of the incorrect steps that the import function put in there. Wrong. This is not documented and almost certainly a bug, but discard history will not discard the steps that got added. It will only discard steps you did yourself. If it worked properly then you could select all your imported photos and discard the history to get rid of the mess the importer created. To get rid of the mess you will have to open each one up individually and manually remove the bad stuff from the history stack. Over and over for tens of thousands of photos.
  8. The thumbnails in the lighttable view are generated from the raw file after the editing steps have been applied. That means all the garbage images created by the importer will have garbage thumbnails too.

As it is now it is worse than useless, IMO. I most definitely want the metadata (keywords (tags), star ratings, color labels, etc.), but the editing stuff also gets automatically imported, translated incorrectly, and applied with no control over it. At the very least I would like to be able to tell it to import only the metadata, only the editing instructions, or both. Or a version of darktable that only imports the metadata since it seems to do that part correctly. It would probably only be about one line of code to disable the import of the editing stuff too or just use a conditional compilation instruction in the source code (#ifdef, etc. in C) when darktable is built.

I have provided great info about stuff that is undocumented and gotchas for anyone who wants to migrate from Lightroom to darktable.  Most of us don't want to lose all the metadata (keywording/tagging, star ratings, color labels, etc.), but also don't want to get all the useless editing translation that is mandatory and that cannot be removed except by manually deleting it from tens of thousands of files.  This is great info for everyone.  But, as usual, a few people seem to be annoyed that I have documented stuff that wasn't documented.  The question is why they are annoyed?

-- hide signature --

Henry Richardson
http://www.bakubo.com

Post (hide subjects) Posted by
jfw
Keyboard shortcuts:
FForum PPrevious NNext WNext unread UUpvote SSubscribe RReply QQuote BBookmark MMy threads
Color scheme? Blue / Yellow