darktable 3.4.0 released

Started Dec 24, 2020 | Discussions
OP Henry Richardson Forum Pro • Posts: 21,306
Re: Lightroom to darktable migration

His Dudeness wrote:

Seems like you know at least a little bit about coding C. Did you try to look into the code?

I just took a quick look into 'src/develop/lightroom.c' and yes, it seems pretty easy to focus on meta data.

I have not looked at the source code.  Yes, I suspect it would be easy to disable it.  Even if I disabled it myself in the code (couldn't build or test though since I have no development environment) it would have to be done by one of the darktable developers to be included in darktable releases.  Better than disabling would be having an option to select metadata, editing, or both.  But just disabling the garbage editing would be fine with me.

-- hide signature --

Henry Richardson
http://www.bakubo.com

His Dudeness
His Dudeness Contributing Member • Posts: 502
Re: Lightroom to darktable migration
1

Henry Richardson wrote:

His Dudeness wrote:

Seems like you know at least a little bit about coding C. Did you try to look into the code?

I just took a quick look into 'src/develop/lightroom.c' and yes, it seems pretty easy to focus on meta data.

I have not looked at the source code. Yes, I suspect it would be easy to disable it. Even if I disabled it myself in the code (couldn't build or test though since I have no development environment)

Setting up a development environment on Linux is quite simple (cannot tell for Windows)

it would have to be done by one of the darktable developers to be included in darktable releases.

No, you could submit a pull request on github.

Better than disabling would be having an option to select metadata, editing, or both.

Sure, but this would be the next step, once you know it works.

 His Dudeness's gear list:His Dudeness's gear list
Fujifilm X-E2 Nikon D750 Nikon D700 Nikon D300 Fujifilm X-T3 +12 more
sluggy_warrior Senior Member • Posts: 2,882
Re: Lightroom to darktable migration
1

Henry Richardson wrote:

All I want it the metadata, but darktable automatically imports and does garbage application of a subset of edits. How did you discard history? See my post. That doesn't work.

You can select the photos and click "discard history" under "history stack" module in lighttable. I often use it whenever I messes up the editing and wanted to start clean again, especially when copying the history/editing from one image to the others (e.g. focus stacking, panorama, ...)

OP Henry Richardson Forum Pro • Posts: 21,306
#7 discard history
1

sluggy_warrior wrote:

Henry Richardson wrote:

All I want it the metadata, but darktable automatically imports and does garbage application of a subset of edits. How did you discard history? See my post. That doesn't work.

You can select the photos and click "discard history" under "history stack" module in lighttable. I often use it whenever I messes up the editing and wanted to start clean again, especially when copying the history/editing from one image to the others (e.g. focus stacking, panorama, ...)

Please read again my point #7.

-- hide signature --

Henry Richardson
http://www.bakubo.com

OP Henry Richardson Forum Pro • Posts: 21,306
discard history
1

Henry Richardson wrote:

His Dudeness wrote:

Seems like you know at least a little bit about coding C. Did you try to look into the code?

I just took a quick look into 'src/develop/lightroom.c' and yes, it seems pretty easy to focus on meta data.

I have not looked at the source code. Yes, I suspect it would be easy to disable it. Even if I disabled it myself in the code (couldn't build or test though since I have no development environment) it would have to be done by one of the darktable developers to be included in darktable releases. Better than disabling would be having an option to select metadata, editing, or both. But just disabling the garbage editing would be fine with me.

Or just having discard history delete all the automatically added garbage editing would be good.  I think that is a bug.

-- hide signature --

Henry Richardson
http://www.bakubo.com

OP Henry Richardson Forum Pro • Posts: 21,306
Re: Lightroom to darktable migration

His Dudeness wrote:

Henry Richardson wrote:

His Dudeness wrote:

Seems like you know at least a little bit about coding C. Did you try to look into the code?

I just took a quick look into 'src/develop/lightroom.c' and yes, it seems pretty easy to focus on meta data.

I have not looked at the source code. Yes, I suspect it would be easy to disable it. Even if I disabled it myself in the code (couldn't build or test though since I have no development environment)

Setting up a development environment on Linux is quite simple (cannot tell for Windows)

it would have to be done by one of the darktable developers to be included in darktable releases.

No, you could submit a pull request on github.

Better than disabling would be having an option to select metadata, editing, or both.

Sure, but this would be the next step, once you know it works.

I have made these things known to the Lightroom import darktable developer.  I have also told them about the discard history bug.  A few months ago.

-- hide signature --

Henry Richardson
http://www.bakubo.com

sluggy_warrior Senior Member • Posts: 2,882
Re: #7 discard history

Henry Richardson wrote:

Please read again my point #7.

ah, sorry, I've never dealt with imported photos so discarding the history has always been working for me.

One work around that you can try on a small subset first: strip out everything in between the "<darktable:history>" tag (or whatever the equivalence of LR) in the XMP files. This can be done quickly with a command or script in GNU/Linux (a combination of find and grep should do). I don't have any XMP from LR so I can't tell the details, but they're simply plain text XML files, should be easy to manipulate.

Otherwise, use a SQLite DB browser, load up the library.db file, and delete all the rows in the history and history_hash tables related to the imported image (or empty that whole table if you want to delete history of all images in darktable).

Again, if you're comfortable with SQL and relational DB, it might be easy to examine the SQLite DB from LR, extract the data out and insert into darktable.

relatively simple data structure

the history steps of images identified by "imgid"

His Dudeness
His Dudeness Contributing Member • Posts: 502
Re: Lightroom to darktable migration

Henry Richardson wrote:

His Dudeness wrote:

Henry Richardson wrote:

His Dudeness wrote:

Seems like you know at least a little bit about coding C. Did you try to look into the code?

I just took a quick look into 'src/develop/lightroom.c' and yes, it seems pretty easy to focus on meta data.

I have not looked at the source code. Yes, I suspect it would be easy to disable it. Even if I disabled it myself in the code (couldn't build or test though since I have no development environment)

Setting up a development environment on Linux is quite simple (cannot tell for Windows)

it would have to be done by one of the darktable developers to be included in darktable releases.

No, you could submit a pull request on github.

Better than disabling would be having an option to select metadata, editing, or both.

Sure, but this would be the next step, once you know it works.

I have made these things known to the Lightroom import darktable developer. I have also told them about the discard history bug. A few months ago.

Probably this thread? https://discuss.pixls.us/t/importing-lightroom-xmp-sidecars/20668

What do you expect them to do? Did you file a bug report on github and provide meaningful test data? Did you create a feature request? It feels a little bit like you expect those people to do the work for you and even keep track of everything that has somewhere been mentioned in a forum.

Nevertheless, it seems like you are capable of writing at least some basic lines of C code. This is probably the only file to modify (at least for solving your problem locally): https://github.com/darktable-org/darktable/blob/master/src/develop/lightroom.c

Why don't you setup a development environment? Probably harder if you are on Windows, but all you need are the compiler environment, dependencies and a text editor. Instead of writing long complaints in forums, you could at least fix it for you locally and get your import done. This is perfectly okay. Creating a solution for all is another step and this is really a nice opportunity to contribute and give something back. Think about it.

 His Dudeness's gear list:His Dudeness's gear list
Fujifilm X-E2 Nikon D750 Nikon D700 Nikon D300 Fujifilm X-T3 +12 more
Mika Y.
Mika Y. Senior Member • Posts: 1,935
Re: darktable 3.4.0 released

sluggy_warrior wrote:

CAcreeks wrote:

No progress on Canon CR3, though.

yeah, unless Canon explicitly states that it's ok, there's no way around the current potential legal issue with the patent in CR3 format. darktable has the technical implementation ready for release, but can't. Not holding my breath.

https://github.com/darktable-org/rawspeed/issues/121

The later comments on this another Darktable issue related to the topic seem to be actually somewhat hopeful that the support might be released:

https://github.com/darktable-org/darktable/issues/2170

It also refers to this Exiv2 issue:

https://github.com/Exiv2/exiv2/issues/1229#issuecomment-727190658

This is just a result of cursory check for the situation though, I haven't been following it actively, so I might be missing something.

I guess the fact that Canon hasn't yet got mad at RawTherapee and ART developers releasing builds with rudimentary CR3 support is somewhat promising; maybe they're OK with this, even if they do not have interest in actively supporting open source projects.

 Mika Y.'s gear list:Mika Y.'s gear list
Canon EOS 5D Mark III Canon EOS Rebel T4i Canon EOS 5D Mark IV Canon EF-S 10-22mm F3.5-4.5 USM Canon EF 17-40mm f/4.0L USM +13 more
Mike Engles Senior Member • Posts: 1,742
Re: darktable 3.4.0 released

sluggy_warrior wrote:

CAcreeks wrote:

No progress on Canon CR3, though.

yeah, unless Canon explicitly states that it's ok, there's no way around the current potential legal issue with the patent in CR3 format. darktable has the technical implementation ready for release, but can't. Not holding my breath.

https://github.com/darktable-org/rawspeed/issues/121

Can you not use a CR3 to DNG converter, like Adobe's

It is free, so perhaps a honorary open source?

Now I imagine I will have to wash my mouth out!

MDE

CAcreeks
CAcreeks Forum Pro • Posts: 18,004
Re: darktable 3.4.0 released

Mika Y. wrote:

The later comments on this another Darktable issue related to the topic seem to be actually somewhat hopeful that the support might be released:

https://github.com/darktable-org/darktable/issues/2170

It also refers to this Exiv2 issue:

https://github.com/Exiv2/exiv2/issues/1229#issuecomment-727190658

Aha, the issue is ISOBMFF, also including HEIF, apparently not CR3 per se.

The exif2 dependency reminds me of this: https://xkcd.com/2347/

OP Henry Richardson Forum Pro • Posts: 21,306
lightroom.c

His Dudeness wrote:

I just took a quick look into 'src/develop/lightroom.c' and yes, it seems pretty easy to focus on meta data.

Where did you find that lightroom.c file?  I looked here and can't find any source code:

https://github.com/darktable-org/darktable

I guess that is the wrong place.  Can you point me to the right place?  Thanks.

-- hide signature --

Henry Richardson
http://www.bakubo.com

His Dudeness
His Dudeness Contributing Member • Posts: 502
Re: lightroom.c

Henry Richardson wrote:

His Dudeness wrote:

I just took a quick look into 'src/develop/lightroom.c' and yes, it seems pretty easy to focus on meta data.

Where did you find that lightroom.c file? I looked here and can't find any source code:

https://github.com/darktable-org/darktable

I guess that is the wrong place. Can you point me to the right place? Thanks.

It is the right place. Just go into the folder "src" then "develop" and there is lightroom.c.

This is the deep link:

https://github.com/darktable-org/darktable/blob/master/src/develop/lightroom.c

 His Dudeness's gear list:His Dudeness's gear list
Fujifilm X-E2 Nikon D750 Nikon D700 Nikon D300 Fujifilm X-T3 +12 more
OP Henry Richardson Forum Pro • Posts: 21,306
Re: Lightroom to darktable migration

His Dudeness wrote:

This is probably the only file to modify (at least for solving your problem locally): https://github.com/darktable-org/darktable/blob/master/src/develop/lightroom.c

Ahh, I was trying to find that file.  Thanks.

-- hide signature --

Henry Richardson
http://www.bakubo.com

OP Henry Richardson Forum Pro • Posts: 21,306
Great info on undocumented stuff
3

Henry Richardson wrote:

I will write about my experience playing around with the darktable import function that imports Lightroom XMP files along with images and tries to translate some of the data for use by darktable. First, take a look at the following 2 links and then below you will see things I learned that are undocumented and I have never seen written about anywhere despite having searched, watching a couple of videos, etc.

Here is a 2013 article written by the developer about importing Lightroom XMP files along with image files into darktable:

https://www.darktable.org/2013/02/importing-lightroom-development/

You can also read the 3.4 manual info about importing Lightroom XMP files here:

https://www.darktable.org/usermanual/en/overview/sidecar-files/sidecar-import/

There are 2 different types of data to import:

  1. metadata (keywords (tags), star ratings, color labels, etc.)
  2. editing instructions

I experimented with it this year by importing a few of my files along with their Lightroom XMP sidecar files to see what darktable would do. I was using darktable 3.2.1 which was the release before this new 3.4.0.

I have 112k photos in my Lightroom catalog: 110k out-of-camera originals + 2k scanned film files. They all have keywords (tags), most have star ratings, many have color labels, many are in collections, etc. About 50k have been edited. I often go back to earlier edited photos and tweak them a bit. On rare occasions I undo all the edits and start over, but mostly just a few changes to an already edited photo. Some people suggest saving all the edited files to 16-bit TIFF files with the Lightroom editing baked in (sRGB or Adobe RGB or some other color space) which turns my 16.5mb Olympus PEN-F 20mp raw file into an 86mb compressed 16-bit TIFF file. And it turns my 3.85mb Olympus E-M10II 16mp JPEG file into an 83.5mb compressed 16-bit TIFF file. Now multiply that by all my files. And besides the explosion of storage space you are still stuck with baked in editing, baked in color space, etc. And you have thrown away all the advantages of non-destructive editing. So, for some people this is a reasonable solution, but it is not for me. On1, C1, ACDSee, and probably others also try to import the editing data from Lightroom, but none are even close to perfect. This is a very difficult thing to do and I no longer have any expectation that it is worthwhile to even do it to my photos.

My feeling these days is if I switch away from Lightroom then I most care about getting my keywords (tags), star ratings, color labels, collections, etc. since the editing translation to a different program is so far from perfect. With darktable you can get keywords (tags), star ratings, and color labels, but not collections. On1, C1, and ACDSee use the Lightroom catalog for the migration which has the collection data too whereas darktable uses the optional XMP files for each individual image file which is generated by Lightroom (you can get them by selecting your images and then hitting Ctrl-S to get them written). It is understandable why the free darktable uses the XMP files rather than the catalog. Adobe pretty regularly (every 2-3 years) changes the format of the catalog so if someone decodes it they have to keep decoding the new formats and updating the source code for the importer. The XMP files are text files and (presumably) are much easier to work with and they don't change.

Here are some important undocumented things I discovered:

  1. When you create the XMP files in Lightroom it only writes XMP files for raw files. If you have edited TIFFs and JPEGs also then there will be no XMP file for them so the editing info is lost and cannot be imported by darktable. Turns out this is very good thing for the reason I will mention below. The keywords (tags), star ratings, color labels, etc. are written in the XMP file for raw files and are written directly into the TIFF and JPEG files.
  2. When you import with darktable it picks up the keywords (tags), star ratings, color labels, etc. from the XMP file and also from the embedded data in TIFFs and JPEGs.
  3. You will see from the links above that darktable can only translate a small subset of the Lightroom editing instructions so right off the bat you can see this is not all that useful.
  4. The documentation says nothing about base curve (display-preferred), filmic (scene-preferred), or none. The darktable default is to use base curve (display-preferred), but it can be changed in preferences to either filmic (scene-preferred) or none. I spent a lot of time trying to figure this out with regards to importing Lightroom XMP files. After some research I found that the importer should not use either base curve or filmic when importing these, but it does. You have to set your preferences to none and then import. Even now I am not 100% sure about this, but from asking it seems like most people think that is what needs to be done and if you think about it then it probably makes sense.
  5. The darktable translation of even the subset of editing instructions is poor. I imported a few test photos with Lightroom XMP and the results were not good. Better off to not even do it since you have to undo it all and then manually turn on the base curve or filmic (depending on whether you want to use the old display-referred editing or the new scene-referred editing) before starting your editing. If you had any dust spot removal in Lightroom then darktable tries to duplicate it, but in my test it often did it in the wrong location so worse than not doing anything.
  6. Very unfortunately darktable gives you absolutely no control over how or even if it imports stuff from a Lightroom XMP file. Extremely bizarre because the darktable developers are constantly making everything even more complex with a huge number of often almost redundant modules, many knobs and levers to control every cryptic aspect of the module, etc. But in this case if there is an XMP file darktable automatically imports the data and does imperfect, incorrect edits on the subset of Lightroom instructions that it recognizes. You can't turn that off and you have zero control.
  7. One might think that you can use the darktable discard history command to get rid of the incorrect steps that the import function put in there. Wrong. This is not documented and almost certainly a bug, but discard history will not discard the steps that got added. It will only discard steps you did yourself. If it worked properly then you could select all your imported photos and discard the history to get rid of the mess the importer created. To get rid of the mess you will have to open each one up individually and manually remove the bad stuff from the history stack. Over and over for tens of thousands of photos.
  8. The thumbnails in the lighttable view are generated from the raw file after the editing steps have been applied. That means all the garbage images created by the importer will have garbage thumbnails too.

As it is now it is worse than useless, IMO. I most definitely want the metadata (keywords (tags), star ratings, color labels, etc.), but the editing stuff also gets automatically imported, translated incorrectly, and applied with no control over it. At the very least I would like to be able to tell it to import only the metadata, only the editing instructions, or both. Or a version of darktable that only imports the metadata since it seems to do that part correctly. It would probably only be about one line of code to disable the import of the editing stuff too or just use a conditional compilation instruction in the source code (#ifdef, etc. in C) when darktable is built.

I have provided great info about stuff that is undocumented and gotchas for anyone who wants to migrate from Lightroom to darktable.  Most of us don't want to lose all the metadata (keywording/tagging, star ratings, color labels, etc.), but also don't want to get all the useless editing translation that is mandatory and that cannot be removed except by manually deleting it from tens of thousands of files.  This is great info for everyone.  But, as usual, a few people seem to be annoyed that I have documented stuff that wasn't documented.  The question is why they are annoyed?

-- hide signature --

Henry Richardson
http://www.bakubo.com

mstMuc
mstMuc Forum Member • Posts: 66
Re: darktable 3.4.0 released

Mike Engles wrote:

Can you not use a CR3 to DNG converter, like Adobe's

It is free, so perhaps a honorary open source?

if Adobe ports it for Linux it could be a solution

bastibe Contributing Member • Posts: 913
Re: Great info on undocumented stuff
1

A quick note on importing edits: none of Darktable's sliders do the exact same thing as Lightroom's. Thus even if you could import, say, exposure and saturation and contrast, the result would not be the same.

Strangely, however, a very few editing settings are imported, such as the input color space. I wish it wouldn't import any editing commands at all from Lightroom. It's just not useful in any way.

Regarding your comment about not being able to clear the history, I have a feeling you might be confusing imported editing settings with the default settings Darktable applies to every newly imported picture. There are a number of "edits" that Darktable applies to every picture just in order to be able to view and edit it at all. In fact all raw developers do, but usually hide it from the user. You will never have an empty history in Darktable, or an empty active module list.

 bastibe's gear list:bastibe's gear list
Ricoh GR III Fujifilm X-T3 Fujifilm X-Pro2 Fujifilm XF 60mm F2.4 R Macro Fujifilm XF 14mm F2.8 R +3 more
OP Henry Richardson Forum Pro • Posts: 21,306
Re: Great info on undocumented stuff
4

bastibe wrote:

A quick note on importing edits: none of Darktable's sliders do the exact same thing as Lightroom's. Thus even if you could import, say, exposure and saturation and contrast, the result would not be the same.

Strangely, however, a very few editing settings are imported, such as the input color space. I wish it wouldn't import any editing commands at all from Lightroom. It's just not useful in any way.

Yes, me too. Since it can't do good editing translations then I would prefer it not to do them at all. An option to say only pick up the metadata, but not the editing instructions would make more sense, I think.

Regarding your comment about not being able to clear the history, I have a feeling you might be confusing imported editing settings with the default settings Darktable applies to every newly imported picture. There are a number of "edits" that Darktable applies to every picture just in order to be able to view and edit it at all. In fact all raw developers do, but usually hide it from the user. You will never have an empty history in Darktable, or an empty active module list.

No, I am not confused. I know about the mandatory default settings that cannot be deleted. I am not talking about those. I am talking about the translated Lightroom settings that got added by the importer. The discard history command will not delete them, but if you open the image you can manually delete them from the history stack.

-- hide signature --

Henry Richardson
http://www.bakubo.com

OP Henry Richardson Forum Pro • Posts: 21,306
3.4.1 coming soon
5

I read a couple of days ago that a 3.4.1 bug fix release is coming soon. I got this list from a thread on another forum:

  • The new color calibration module (aka: “color mixer rgb”) has been fixed with mixing R,G,B values
  • Color calibration’s colorfulness has been updated to be more what a user expects
  • Several cameras had noise profile updates
  • Fixes for stars and labels
  • Combobox scrolling updates
  • Lightroom importing updates (to only import if no other XMP already exists, making it easier to reset to darktable defaults for processing)
  • Showing thumbnails when importing happens more frequently
  • Module rename fix
  • macOS specific fixes (scrolling fix, histogram fix, and misc. other Mac-related issues)
  • Various fixes for threading & memory free errors (stopping crashes and hangs)
  • And a bunch more
-- hide signature --

Henry Richardson
http://www.bakubo.com

OP Henry Richardson Forum Pro • Posts: 21,306
darktable 3.4.1 released
4

A bug fix release has been made:

https://github.com/darktable-org/darktable/releases/tag/release-3.4.1

-- hide signature --

Henry Richardson
http://www.bakubo.com

Keyboard shortcuts:
FForum MMy threads