Experience sharing - Setting up a photo management system with Linux Mint

[Rightsaidfred]

Dear all

I've been a darktable for Windows user for about one and a half years. My camera system is Fuji-X. I've always shared my Windows laptop with my family. Now I was lucky to obtain a PC from a friend with Linux Mint Ulyana Cinnamon, so I have my own system which is really cool.

Here I want to report about my experience in setting up a photo management system on Linux Mint. It took me quite a while. I was a Linux user until 10 years ago but forgot too many things After about two weeks (of course I could only work on it in my spare time) I had a working system, and it is fun to use! It is incredible what you can do without a big investment. Better invest into camera and lenses.

I want to share my experience. Experienced Linux users may laugh at me. But I am sure there are others who are happy to see a description. Comments welcome.

(1) I added a 2nd internal HDD

I bought a 4 TB internal HDD (Seagate BarraCuda) to add space for my photos. I managed to format it as ext4 with the graphical interface but permanent mounting did not work - don't know what I did wrong. So I decided to edit /etc/fstab.

How to do this can be found in the web. It took me an hour or two until I managed to understand what to do and how to do it but is actually no rocket science. You just need to add one single line to fstab. I added:

UUID=[xxx] /media/[name of disc] ext4 auto,rw,users 0 0

(The UUID of the drive can be figured out with $ blkid)

I added "users" because all users should be able to read and write ("rw") on the 2nd HDD. Of course I created an own user without superuser rights for all my photo work.

The editor mostly used to edit fstab, "vi" is a really weird tool, kind of a dinosaur. Once you understand the logic behind you will succeed. See: A Beginner’s Guide to Editing Text Files With Vi.

(2) darktable

darktable is my raw editor of choice. See here + the link therein for a description how I work with darktable. Basically, I work with the great LUTs from Stuart Sowerby to emulate the famous Fuji film simulations.

In the repository there was only v3.0.2. Of course you can work with this version but the current v3.2.1 comes with quite some improvements, so I wanted to have this one.

The latest version is available from the snapstore. I installed it but I didn't like those "loops" created by snapd. Maybe others can report. I decided to deleted it.

I eventually downloaded the .deb file from Opensuse and installed it.

$ sudo dpkg -i darktable_3.2.1-1.1_amd64.deb

I got an error message. So I did

$ sudo apt-get install -f

$ sudo dpkg -i darktable_3.2.1-1.1_amd64.deb

Many thanks to my friend who helped me to accomplish this.

(3) Lensfun database

darktable relies on the Lensfun database for lens correction profiles.

Lensfun is a great project. The Lensfun database contains a lot of lens correction profiles. Unfortunately, the project lacks active skilled members since about the beginning of 2019; lots of lens calibration images handed in by volunteers are waiting for being analyzed and processed to create lens profiles! But this is another story.

As a matter of fact, the Lensfun database that came with my version of darktable (maybe still from the repository's version) was completely outdated.

What you need to do to upgrade is described here. Let me describe in my own words how I did it:

• Download http://wilson.bronger.org/db/version_0.tar.bz2
• Unzip. You will get a folder "version_0". It contains a number of xml files - the profiles.
• Find Lensfun. In my case, it is in /usr/share/lensfun. Here you will find a folder called "version_1".
• Copy "version_0" to the same level, i.e., next to "version_1".
• Delete "version_1"
• Rename "version_0" as "version_1"
• Done

(4) Rapid Photo Downloader

This is a great tool to modify the file titles of raw and jpeg upon download from the camera, e.g., YYYYMMDD-Event-xxxx. It also can create folders and sub folders. I haven't actually found a way to use it without creating any new folder at all but that's something I can live with.

The version from the Linux Mint repository (0.9.22) was not the latest one and did not work. It just didn't start at all! Something was not OK with Python. So I got the latest version (0.9.24) from the website and followed the instruction. Success!

(5) XNView MP

My standard jpeg viewer with Linux is XViewer. It is fast but I can't do much more than displaying images.

With Windows, I use IrfanView to add text to jpegs, to resize jpegs, and for batch operations (e.g., to resize a large numbers of jpegs). Unfortunately, IrfanView is not available for Linux.

A good Linux alternative is XNView Multi Platform (MP). Again, in the repository was an outdated version. It could not detect my second HDD in the directory. So I downloaded the latest version from the website. Success!

I downloaded the .tgz and unzipped into /opt/. XNView MP can be started as a shell with xnview.sh and it works very well.

In the meantime I know the more professional way would have been to take the .deb 64 bit instead of the .tgz. You live and learn

(6) GIMP

Of course a must have with a Linux system. With Windows, I use it as a portable application.

In this case, there was no problem with the version from the repository.

But, as I said, for simply adding text to a jpeg or changing size, I use the XNView MP.

(7) ExifTool

A tool to edit Exif data. I need it to add information to those images I took with my manual lenses. I always enter the fl with the camera but that's all you can do when using a manual lens.

I got the tool as described here: How To Install and Use exiftool In Linux, Windows, Kali, Ubuntu, Mint with Examples.

With Windows, I used it with the ExifToolGUI, a graphical interface. To the best of my knowledge, the GUI is not available for Linux. Never mind, it can be easily run from the command line. Here are some examples: ExifTool Command-Line Examples.

(8) rsync

With Windows, I use FreeFileSync to mirror my HDD to external USB drives. This is my way to backup my data.

FreeFileSync is also available for Linux. But you actually don't need it. rysnc does it all.

I won't describe this command here, there's plenty in the web. Beginners should use the dry run function first, particularly if --delete is applied.

(9) Zerene Stacker

I've had a license for my Windows laptop, and luckily, Zerene is also available for Linux.

Installation is very easy. Download and unzip. I unzipped to /opt/.

This is my current system. C&C welcome!

You see quite a lot of issues resulted because the repository hosts outdated versions of the software.

BR,

Martin

--
https://500px.com/bachrocks

 

[CAcreeks]

Thank you Martin for the excellent essay! I've used IRIX then Linux for all my website needs, and switched to GIMP on Linux for photography when I found it superior to Photoshop. I have one question for you.

(4) Rapid Photo Downloader

This is a great tool to modify the file titles of raw and jpeg upon download from the camera, e.g., YYYYMMDD-Event-xxxx. It also can create folders and sub folders. I haven't actually found a way to use it without creating any new folder at all but that's something I can live with.

The version from the Linux Mint repository (0.9.22) was not the latest one and did not work. It just didn't start at all! Something was not OK with Python. So I got the latest version (0.9.24) from the website and followed the instruction. Success!

Is Rapid Photo Downloader color managed? If not, does it matter?

I have never tried it because was not in the Ubuntu repositories when I looked (it is now), thus I thought it might be difficult to manage. Thanks for installation tip.

Zerene stacker sound intriguing. I hadn't heard of it.

 

[Rightsaidfred]

Thank you Martin for the excellent essay!

Thank you. You're welcome.

Rightsaidfred, post: 64423438, member: 700823"]
(4) Rapid Photo Downloader

This is a great tool to modify the file titles of raw and jpeg upon download from the camera, e.g., YYYYMMDD-Event-xxxx. It also can create folders and sub folders. I haven't actually found a way to use it without creating any new folder at all but that's something I can live with.

The version from the Linux Mint repository (0.9.22) was not the latest one and did not work. It just didn't start at all! Something was not OK with Python. So I got the latest version (0.9.24) from the website and followed the instruction. Success!

Is Rapid Photo Downloader color managed? If not, does it matter?
[/QUOTE]
It just changes file titles and puts files into folders upon download from the camera, (hopefully) nothing else. I create file titles like YYYYMMDD-birthday-5732.jpeg.

I have never tried it because was not in the Ubuntu repositories when I looked (it is now), thus I thought it might be difficult to manage. Thanks for installation tip.

Zerene stacker sound intriguing. I hadn't heard of it.

Zerene and Helicon are the 'top dogs' of stacking software. I do a lot of macro photography. For stacking, I started with the free Picolay and one day went for Zerene that digests perspective changes during focusing better IMO. The English Wikipedia contains a good list of stacking software.

BR,

Martin

 

[PHXAZCRAIG]

I'm comfortable with Linux, though generally as a server running typical server apps (DNS, Apache, SQL, DHCP, firewalls, etc.

On windows I went from using a bunch of different software from each camera manufacturer (Dusplay Memeries for Sony, DPP for Canon, CaptureNX for Nikon) with different ingest and web publishing software to ... Lightroom for most of it.

I'm comfortable with Lightroom, and would be comfortable with other software once I put my mind to learn it. But I do need something to keep track of all my images the way Lightroom does. I tried IDImager some years back, but it was too slow.

I didn't see anything in your list that seemed a Digital Asset Manager. Does GIMP do that now?

 

[CAcreeks]

I didn't see anything in your list that seemed a Digital Asset Manager. Does GIMP do that now?

Definitely not.

darktable has some organization tools. I wouldn't go so far as to call them a DAM.

digiKam probably comes closes to a true DAM on Linux.

Various other programs such as gThumb, Shotwell, and Pix are not color managed.

Does anybody know about GNOME Photos? It is in the Mint Software Manager, but I had never heard of it until it appeared in a search today.

 

[Rightsaidfred]

I'm comfortable with Linux, though generally as a server running typical server apps (DNS, Apache, SQL, DHCP, firewalls, etc.

On windows I went from using a bunch of different software from each camera manufacturer (Dusplay Memeries for Sony, DPP for Canon, CaptureNX for Nikon) with different ingest and web publishing software to ... Lightroom for most of it.

I'm comfortable with Lightroom, and would be comfortable with other software once I put my mind to learn it. But I do need something to keep track of all my images the way Lightroom does. I tried IDImager some years back, but it was too slow.

I didn't see anything in your list that seemed a Digital Asset Manager. Does GIMP do that now?

So as CAcreeks pointed out, darktable has some DAM capabilities. The photos are organized in so-called film rolls. You can assign tags, star ratings, and the like.

Maybe this experience helps: Leaving Adobe Lightroom; or how to migrate from Lightroom to open source alternatives

Personally, I cannot contribute much more to your question. I have a classical folder system starting from 2004 when I started with digital photogtaphy because I never wanted to be too dependent on a certain software or license solution, and I now have my darktable tags in addition. That still does it for me. I am not a pro, I currently have only about 1.5 TB of images starting from 2004.

However, yes, I may need to improve in the future. I've shot raw for only 2 years since I acquired my Fujifilm system, so my storage space needs are heavily on the rise because the number of my images per time unit has strongly increased, and because I save raw and jpeg of every image

BR,

Martin

 

[PHXAZCRAIG]

I have about 4 terabytes of still images now. A couple of years ago I invested in a decent NAS to use a s a backup with plenty of capacity. I gave myself 27 terabytes to grow into. Gives me some room for video.

I think I have over 250,000 images now. I also have a reasonable folder structure and a naming scheme that is both sortable and guarantees unique names.

 

[I2K4]

Your Linux Mint use is similar to mine, but I find XnViewMP useful for batch filenaming (I'm too lazy for tagging and categorizing functions) using a DIY filing system, along with Catfish File Search (or Copernic in Windows). I'm also leery of over-investing time or money in particular products (humming Paul Simon's "50 ways to leave your software".) I recently started using LUTs in Darktable, including Sowerby's, but will throw in other G'MIC simulations with their helpful preview page. BTW the Ubuntu Handbook PPA for Darktable has been reliably updating within days of release, since last December, to current.version 3.2.x:

sudo add-apt-repository ppa:ubuntuhandbook1/darktable

sudo apt update

sudo apt install darktable

or use Mint Software Manager or Synaptic PM after adding the PPA.

 

[CAcreeks]

However, yes, I may need to improve in the future. I've shot raw for only 2 years since I acquired my Fujifilm system, so my storage space needs are heavily on the rise because the number of my images per time unit has strongly increased, and because I save raw and jpeg of every image

Some NEF raws that I developed lately came with very large previews inside, perhaps even full size. This might be a camera settable option. So with Nikon it might be possible not to save JPEG, if you trust exiv2 source code going forward.

Whereas RAF previews are significantly downsampled, so it's a good idea to save JPEG. Does darktable have an option to coordinate Raw and JPEG images?

 

[jtstanco]

Just a note on Darktable with respect to Canon CR3 files.

I just re-tested the latest version of Darktable and it appears that Canon CR3 files are still not supported. I get a notification that there are no supported images to import when I try to import my CR3 samples folder.

These same files are supported by both XnViewMP and Rawtherapee so it looks like Darktable is using different raw conversion packages than XnViewMP and/or Rawtherapee.

 

[CAcreeks]

Just a note on Darktable with respect to Canon CR3 files.

I just re-tested the latest version of Darktable and it appears that Canon CR3 files are still not supported. I get a notification that there are no supported images to import when I try to import my CR3 samples folder.

These same files are supported by both XnViewMP and Rawtherapee so it looks like Darktable is using different raw conversion packages than XnViewMP and/or Rawtherapee.

Yes, rawspeed instead of dcraw.

I'm not sure what the pros and cons are. Maybe speed? Nor do I understand the license issues. Fortunately I do not own, nor do I intend to buy, a Canon camera.

https://discuss.pixls.us/t/darktable-and-the-cr3/17498

 

[sluggy_warrior]

(3) Lensfun database

darktable relies on the Lensfun database for lens correction profiles.

Lensfun is a great project. The Lensfun database contains a lot of lens correction profiles. Unfortunately, the project lacks active skilled members since about the beginning of 2019; lots of lens calibration images handed in by volunteers are waiting for being analyzed and processed to create lens profiles! But this is another story.

As a matter of fact, the Lensfun database that came with my version of darktable (maybe still from the repository's version) was completely outdated.

What you need to do to upgrade is described here. Let me describe in my own words how I did it:

• Download http://wilson.bronger.org/db/version_0.tar.bz2
• Unzip. You will get a folder "version_0". It contains a number of xml files - the profiles.
• Find Lensfun. In my case, it is in /usr/share/lensfun. Here you will find a folder called "version_1".
• Copy "version_0" to the same level, i.e., next to "version_1".
• Delete "version_1"
• Rename "version_0" as "version_1"
• Done

Just want to point out that, on Ubuntu, you can install the "liblensfun-bin" package, then run the command "lensfun-update-data" (as often as you want), everything will be taken care of.

 

[Rightsaidfred]

(3) Lensfun database

darktable relies on the Lensfun database for lens correction profiles.

Lensfun is a great project. The Lensfun database contains a lot of lens correction profiles. Unfortunately, the project lacks active skilled members since about the beginning of 2019; lots of lens calibration images handed in by volunteers are waiting for being analyzed and processed to create lens profiles! But this is another story.

As a matter of fact, the Lensfun database that came with my version of darktable (maybe still from the repository's version) was completely outdated.

What you need to do to upgrade is described here. Let me describe in my own words how I did it:

• Download http://wilson.bronger.org/db/version_0.tar.bz2
• Unzip. You will get a folder "version_0". It contains a number of xml files - the profiles.
• Find Lensfun. In my case, it is in /usr/share/lensfun. Here you will find a folder called "version_1".
• Copy "version_0" to the same level, i.e., next to "version_1".
• Delete "version_1"
• Rename "version_0" as "version_1"
• Done

Just want to point out that, on Ubuntu, you can install the "liblensfun-bin" package, then run the command "lensfun-update-data" (as often as you want), everything will be taken care of.

Cool many thanks for this!

BR,

Martin

 

[TacoTruck]

digiKam probably comes closes to a true DAM on Linux.

What is missing from XnViewMP for your DAM needs?

 

[CAcreeks]

digiKam probably comes closes to a true DAM on Linux.

What is missing from XnViewMP for your DAM needs?

That's a good question.

I have XnViewMP installed on a Macbook, but it's a work machine owned by my employer, so XnView gets very little use. I'll take a look and get back to you.

On Linux, XvViewMP was not in the PPA, so I never installed it until I wanted a simple viewer for OS X. (Mojave) Photos is over-complicated and import oriented.

 

[Rightsaidfred]

Windows users are used to assign file and folder names with blanks and also some special characters. The Linux world doesn't like this.

My recommendation is not use blanks and special characters in Windows in order to stay flexible. It costs you nothing to avoid it, and you never know

I renamed now nearly all my photo files and folders that contained blanks and/or special characters like the German "ß", "ä", "ö", "ü".

The easiest option IMO is the detox command.

Installation of detox:

$ sudo apt update

$ sudo apt install detox

How to use detox:

$ detox -nrv -s utf_8 [path]

-n = Dry run, just remove after successful dry run

-r = do the job recursively (if wanted)

-v = verbose, show what you did

Can also be done without the utf_8 sequence.

The command is very fast.

The following article describes two alternatives: How to Replace Spaces in Filenames with Underscores on the Linux Shell. I tried both. They worked but not recursively.

Photo user specific practical problems

I had no problems with detox except from the files created by Zerene stacker. These often start with an "-" which would get removed with detox. I didn't want this because I did not want to damage my saved Zerene projects, so, in those sections that contain Zerene projects, I used detox just on a higher hierarchical level without the "-r" in order not to touch the Zerene files. The "replace_spaces.sh" script from the cited article would probably let the "-" untouched but it does not work recursively (at least I don't know how to accomplish this). Due to my folder structure, doing it all folder by folder, level by level with this script would be too much work. So I'll leave all those sections that contain Zerene projects untouched for the moment. Well, nothing is perfect.

No probs encountered with darktable after the renaming event. Of course I had to remove film rolls and re-import them if file or folder names had changed. But no prob whatsoever thanks to the xmp files.

I did the same with the first one of my backup drives.

Don't expect problems but I'll wait a bit until I rename files and folders of my further backup drives.

Think I'll take the chance and tidy up my collection and remove a number of 2nd class photos

BR,

Martin

--
https://500px.com/bachrocks

 

[Rightsaidfred]

...

(8) rsync

rysnc does it all.

...

Maybe a bit off-topic in a photo forum. OTOH, doing regular backups is extremely important for us photo people. And probably not all photogs are Linux top-notchers. So I thought I'll just share it here:

I found an Rsync Command Generator.

My way for mirroring is currently

$ rsync -rnhuv --delete path_to_source/ path_to_target

A dry run every single time is strongly recommended. Any mistake with the --delete is fatal. Just delete the "n" in case the dry run is successful.

I have two HDDs, and I keep one of them in my office in the company. In addition, the family has an SSD with the jpegs, and the more recent jpegs are additionally on a Fritz!NAS. Quite some work to manage all this

I described recently my folder structure and how I delete second class images in a two-step approach.

BR,

Martin

--
https://500px.com/bachrocks

 

[sluggy_warrior]

Maybe a bit off-topic in a photo forum. OTOH, doing regular backups is extremely important for us photo people. And probably not all photogs are Linux top-notchers. So I thought I'll just share it here:

I found an Rsync Command Generator.

My way for mirroring is currently

$ rsync -rnhuv --delete path_to_source/ path_to_target

A dry run every single time is strongly recommended. Any mistake with the --delete is fatal. Just delete the "n" in case the dry run is successful.

if you're in a timezone with daylight saving like mine, I had to allow 1-hour off, or it'll resync the whole drive twice a year

"--modify-window=3601"

 

[Rightsaidfred]

...

(2) darktable

darktable is my raw editor of choice. See here + the link therein for a description how I work with darktable. Basically, I work with the great LUTs from Stuart Sowerby to emulate the famous Fuji film simulations.

In the repository there was only v3.0.2. Of course you can work with this version but the current v3.2.1 comes with quite some improvements, so I wanted to have this one.

...

I eventually downloaded the .deb file from Opensuse and installed it.

$ sudo dpkg -i darktable_3.2.1-1.1_amd64.deb

I got an error message. So I did

$ sudo apt-get install -f

$ sudo dpkg -i darktable_3.2.1-1.1_amd64.deb

Many thanks to my friend who helped me to accomplish this.

...

This is how I just upgraded to the latest version (I have Linux Mint Ulyana).

Logon as su

$ wget https://download.opensuse.org/repos...ntu_20.04/amd64/darktable_3.4.0-1.1_amd64.deb

$ sudo dpkg -i darktable_3.4.0-1.1_amd64.deb

No error messages were shown. No problems after restart of darktable (as always).

BR,

Martin

--
https://500px.com/bachrocks
https://100asa.com/photographer/martin

 

[Rightsaidfred]

The professional denoising tool, Neat Image should be good as a standalone tool under Linux. It should be possible to apply it with TIFFs exported from darktable.

See here.

This is just an option I have come across and I wanted to share. I have not tested this yet since personally, my need for highly sophisticated denoising is not the greatest. Experience sharing welcome.

Kind regards,

Martin