De-duplication utilities and work flow

[wklee]

I'm currently cleaning out my hard disks of about 1TB worth of duplicates. Which de-duplication utility do you use? What's your work flow with regards to file management? I am a LR 3 user.

Thanks
--
Never buy version 1.0 of anything.

 

[BG454]

To find duplicate files I find this Auslogics Duplicate File Finder program (free) very useful: http://www.auslogics.com/en/software/duplicate-file-finder/

I wouldn't say I had a "workflow" when it comes to file management. After I download my image files from my memory card sorted by date, I add my copyright message, title and keywords etc. Having done that I then make two backup copies to external USB hard drives (I use Microsoft's "SyncToy" for this). Once all that's done I feel safe to delete them from the memory card.
I'm using IDImager for image management, so your requirements may be different.
--

Photographers feel guilty that all they do for a living is press a button. - Andy Warhol

 

[wklee]

I have files with different modification dates that are otherwise identical. I can't be sure if it happened from OS X or Windows (or how) since I was using both platforms.

It's really confusing and I think the duplicate file finders fail here.

--
Never buy version 1.0 of anything.

 

[Carey Brown]

Check out free duplicate file finders:
http://inasphere.com/programs.html

See: 'DelDups2' and 'FindDups2'

Utilities find ANY files that are byte-for-byte identical regardless of time stamp or file name.

They are a bit nerdy and require that Java be installed on your machine (usually is), but hey, it's FREE. Comes with a Word doc with documentation.

If your looking for a utility that finds image files that "look" similar, I've had good luck with ThumbsPlus (not free).

--
A pixel is a terrible thing to waste.

 

[BG454]

I have files with different modification dates that are otherwise identical. I can't be sure if it happened from OS X or Windows (or how) since I was using both platforms.

It's really confusing and I think the duplicate file finders fail here.

One option in the Auslogics duplicate finder is to find Image files only. I believe it ignores things like save date. Why not give it a try, nothing to lose, it's free.
--

Photographers feel guilty that all they do for a living is press a button. - Andy Warhol

 

[wklee]

I've also downloaded WinDirStat, a free utility. It gives a "tree map" view of every folder in the hard disk(s). I can tell at a glance the contents of my biggest folders.

Thanks
--
Never buy version 1.0 of anything.

 

[starfire990]

I use two great utilities for managing my photos.
1: Easy duplicate finder http://www.easyduplicatefinder.com

This helps me find exact duplicates. If I had copied from my camera to more than one folder without clearing out the old photos. I cant begin to tell you how much space this has saved me, when it comes to cleaning out my camera cards I can be quite forgetful.
2: Duplicate photo finder http://www.duplicatephotofinder.com

This is the best one, it finds similar photos. I know most of you are probably like me, taking several photos in a row, just trying to get just the right effect. well what do you do with all the not quite right ones? When it comes time to clean those out, this DPF software is great. I get a side by side comparison that I can check, and delete what I don't want to keep.

 

[richard01]

To delete out all the duplicate files from your system, you can make use of duplicate finder mac that is competent enough to detect and remove all the redundant duplicate files present on the system completely without wasting much of your time.

so, make a try to use duplicate finder mac software which will meet all your needs regarding removal of duplicate files.

http://www.duplicatefilefinderformac.com

 

[wklee]

I'm going to just keep the photos I want to print and delete the rest. There's too much junk.

--
Never buy version 1.0 of anything.

 

[BG454]

I'm going to just keep the photos I want to print and delete the rest. There's too much junk.

I recently helped someone who was researching for a book. We worked through a lot of old photos (35mm film) and in several cases found information that was valuable, not regarding the main subject, but often background details that were invaluable to our research. Often the act of recording a scene, particularly a street or city view, can provide a historic archive for future reference. I've made it a point to sometimes record mundane scenes of everyday life, knowing that in years to come these will become historic reference points.

Storage is cheap these days, think carefully before you discard anything. You can never be certain how useful what you have may become.

--

Photographers feel guilty that all they do for a living is press a button. - Andy Warhol

 

[wklee]

I'll keep that in mind.

Thanks.

I recently helped someone who was researching for a book. We worked through a lot of old photos (35mm film) and in several cases found information that was valuable, not regarding the main subject, but often background details that were invaluable to our research. Often the act of recording a scene, particularly a street or city view, can provide a historic archive for future reference. I've made it a point to sometimes record mundane scenes of everyday life, knowing that in years to come these will become historic reference points.

Storage is cheap these days, think carefully before you discard anything. You can never be certain how useful what you have may become.

--

Photographers feel guilty that all they do for a living is press a button. - Andy Warhol

--
Never buy version 1.0 of anything.

 

[wklee]

I read a discussion on this subject in Macintouch.com indicates that CrashPlan allows de-duplication before backup (and other solutions that may be of interest to others).

http://www.macintouch.com/readerreports/backup/index.html#d17mar2011
--
Never buy version 1.0 of anything.

 

[wklee]

I also realize now that when using LR2, I unchecked the "Don't Import Suspected Duplicates" checkbox. Leaving me with lots of dupes. I must have been experimenting with some settings which I forgot about.

Is there any reason to leave this setting turned off when it is on by default?

--
Never buy version 1.0 of anything.

 

[rm2]

Do you have RAW as well as JPEG files? If so, you can save quite a bit of room compressing the RAW files.

 

[wklee]

And thanks for the link to PCLinuxOS in another thread. I've played around with an earlier version in addition to Ubuntu 8 or 9 at the time.I had issues with ATI or Nvidia drivers back then. I hope it works better now.

--
Never buy version 1.0 of anything.

 

[rm2]

There have been quite a few updates to those drivers since then. Although if you have a very old card, who knows.

As far as compressing RAW images, I keep all my RAW images on a separate directory tree from the processed files. That way I can use a little program I made to easily compress and decompress the whole tree or only a particular branch. You can read about it here:

http://pclosmag.com/html/Issues/200801/page07.html

A windows version can be downloaded from here:

http://sites.google.com/site/rmcorrespond/7Squeeze_1_0b1_exe.zip

Let me know what you think.

 

[CSMR]

"Doublekiller" is a good free program.

 

[wklee]

I've tried both approaches they have pros and drawbacks. For example, I probably have folders approaching tens of thousands. Using the folder (or a group) approach canl be very tedious. However, using a disk based approach would increase the chance of me accidentally deleting the wrong files.

My current approach is to create a "Culling" folder and have subfolders by year and be sure that I have at least one copy of whatever I'm deleting. I then sweep the whole disk of the junk matching what's in that folder.

I've installed a program called CloneMaster which was mentioned in an earlier thread. It's very fast searching for duplicates. The trail's limitation is that I'll have to pay for using mass deletions.

--
Never buy version 1.0 of anything.

 

[wklee]

I had forgotten how much disk space the LR catalogs and previews took up (backups were set to once weekly). I'm clearing those as well.

--
Never buy version 1.0 of anything.