Need help find unique image files when merging portable drives

Hi

I'm in the process of merging 10+ years' worth of images residing on several portable drives - these are 80% in sync from a folder structure point of view, but there are some differences, and especially some files might exist on multiple drives, but not necessarily in the same folder. My problem is how to find unique files that are only on one specific drive, irrespective of the folder that it might be in, so that I consolidate it into my final master drive?

I've tried several Duplicate file finders (I like KDiff3) and folder comparison tools (Beyond Compare meet my needs) and managed to remove most duplicates and can easily review folder structures, but I'm having trouble completely synching both drives without creating more duplicates or leaving empty folders. There are potentially more than 15000 folders and 100 000+files...

The duplicate finder tools can show all duplicates across all drives and then allow me to remove the duplicates (in the process then sometimes leaving empty folders when all files were duplicates and then removed), and the folder compare tools compare 2 (or 3) folders and show which files they contain - and whether a file is found in both folder, or only in one (either left or right). BUT folder compare tools doesn't allow me to see if a file that is in only one folder is unique to a folder structure on a drive - it might be simply in a different folder. How do I identify these - "IF the file is unique to compared folders AND unique to the drive THEN keep it"?

Any help would be appreciated - if you know of a tool (preferably freeware) or process that can do this please help - I searched DPreview forums (and Googled the web profusely) but can't seem to find a workable solution - but somebody else might have done this already!

Thanks
Heinrich

Sounds like you need a unique file finder rather than duplicate finder - I never used one but am sure google can help.

Another strategy to try would be to migrate everything to a new drive as is (if capacity allows for that) and then eliminate duplicates. This has the benefit of having the backup copies on old drives, should you make a mistake in the process.
--
Peter

Peter Szymiczek said:

Sounds like you need a unique file finder rather than duplicate finder - I never used one but am sure google can help.

Another strategy to try would be to migrate everything to a new drive as is (if capacity allows for that) and then eliminate duplicates. This has the benefit of having the backup copies on old drives, should you make a mistake in the process.
--
Peter

Click to expand...

After this migration, you could use Total Commander's "Compare by content" option as well. This option can be set to ignore file dates.

Also see this thread.
http://forums.dpreview.com/forums/read.asp?forum=1004&message=37748844

Greets.
Fotonut.
--
Snap snap - click click.

FujiLinks - http://fujilinks.110mb.com/index.html

I'm the OP referenced in the link. I'm waiting for Beyond Compare to release its 3-way Folder Merge/Compare feature. The developers have said that they are testing this feature with no indication as to when it will be available.

Microsoft (now Phase One's) Expression Media has a "Find Similar" feature which you may find useful.

I'm still struggling with it so it is going to take me a while to clear my hard disks.

--
Never buy version 1.0 of anything.

Thanks for all the replies - even thought it didn't solve my problem exactly, I think I have a process that might work. I'll report back once I get my process tested.

@wklee: I really like Beyond Compare and would definitely use a 3 way compare/merge option once available, but for now the best I could find (freeware) is KDiff3 - it does 3 way comparisons and merging of files and folders, but it's file compare/merging is for text files only and not applicable to images. And it does not provide a "display" option to see the actual image in it's native/system viewer like Byond Compare. But it's usefull to get a quick overview of 3 way matching between folders...

Thanks
Heinrich (aka ddd_fish)

@ Fotonut: This is exactly my problem - there doesn't seem to be a real "unique file finder" available anywhere - everything I found (WinMerge, KDiff3, Beyond Compare, etc) compares complete structures and can then list "orphans" in one structure - but separate de-duping is still a pre-requirement.

I'm having great success using Auslogics Duplicate File Finder to find and clear duplicates (within my master or backup structures individually, or between both) - but the unique files in structure requirement is still elusive..

I'm using Beyond compare to find "orphans" in folders (comparing my master and backup structures and finding files that exist on only one side) - and this clears quite a few items, but to determine if the "orphan" found in a folder in the backup is really the only file and not just hiding somewhere else in the structure on the master side is time consuming - having to first do the deduping, then find orphans, then move them and finally removing the empty folders.

all of this is time consuming between a master and backup structure - but I have several backup structures to compare to my one master which makes this an extraordinarily time consuming process...

I knew my data safety paranoia was going to catch up with me: having a backup on-site, same town off-site, different continent off-site and in the cloud - is safe, but difficult to manage if you don't follow the same procedure each time!

Hiya wklee,

Member said:

I'm the OP referenced in the link. I'm waiting for Beyond Compare to release its 3-way Folder Merge/Compare feature. The developers have said that they are testing this feature with no indication as to when it will be available.

Click to expand...

Is this not the item you were referring to ?
http://www.scootersoftware.com/moreinfo.php?zz=moreinfo_merge

If not and the way i read it, you could do 2 runs. Once for the first 2 disks/folders into one. Then do a second run to merge a 3rth disk/folders with the first run.

Or am i missing something.

Greets.
Fotonut.
--
Snap snap - click click.

FujiLinks - http://fujilinks.110mb.com/index.html

Fotonut said:

Fotonut said:

I'm the OP referenced in the link. I'm waiting for Beyond Compare to release its 3-way Folder Merge/Compare feature. The developers have said that they are testing this feature with no indication as to when it will be available.

Click to expand...

Is this not the item you were referring to ?
http://www.scootersoftware.com/moreinfo.php?zz=moreinfo_merge

Click to expand...

Yes it is.

Fotonut said:

If not and the way i read it, you could do 2 runs. Once for the first 2 disks/folders into one. Then do a second run to merge a 3rth disk/folders with the first run.

Or am i missing something.

Click to expand...

I was hoping for a simpler way of doing things. Thanks for the suggestion.

Fotonut said:

Greets.
Fotonut.
--
Snap snap - click click.

FujiLinks - http://fujilinks.110mb.com/index.html

Click to expand...

--
Never buy version 1.0 of anything.

I should have read the link first!

Currently, Beyond Compare's 3 way merge only works with text only not folders.

--
Never buy version 1.0 of anything.

I am thinking of this file synchronization. comparison, replication and backup software.
http://www.tgrmn.com/

--
Never buy version 1.0 of anything.

I haven't tried this yet, but it looks very similar to functionality in Beyond Compare ( http://www.scootersoftware.com ) or Alway Sync ( http://allwaysync.com/ ) that I'm already using. Beyond Compare and ViceVersa both provide synching, backup and file/folder compare functionality - but not finding and removing duplicates. My ultimate program would be a compbination of Beyond Compare, Auslogics Duplicate File Finder, and a (not yet found) "unique file finder" that will allow me to compare folder structures, identify differences visually and then remove duplicates and consolidate them into selected folders...

My current process is a two step process with manual corrections:
1. Using Auslogics Duplicate Finder:
1.a. Identify duplicate files

1.b. Remove duplicates from folders where not needed (manual selection per folder - very time consuming since the tool cannot accept "select all duplicates from this folder and below" and only uses "select all duplicates from this folder".
2. Using Beyond Compare:

2.a. Compare two structures visually - require some manual root folder selections and folder renaming to match if the structures have been modified (i.e. folder slightly renamed or moved)

2.b. Beyond Compare will only compare files per folder - so any orphans and changed files must be moved and managed between folders manually.

Hope this helps!

Regards
Heinrich (akak ddd_fish)

PS: I have no connections to any of the software packages mentioned - I found them online and I'm a (mostly) satisfied user - not paid by or affiliated with them in any other way.

I just discovered that CrashPlan has a de-duplication feature but it is not free. I looked at the CrashPlan Free client and some of the advanced features are grayed out. Apparently, Dropbox will also detect duplicates and will give the option to de-dup as well.

http://support.crashplan.com/doku.php/reference_guide/settings#advanced_backup_settings
--
Never buy version 1.0 of anything.

Hiya ddd_fish,

ddd_fish said:

@ Fotonut: This is exactly my problem - there doesn't seem to be a real "unique file finder" available anywhere - everything I found (WinMerge, KDiff3, Beyond Compare, etc) compares complete structures and can then list "orphans" in one structure - but separate de-duping is still a pre-requirement.

Click to expand...

Snip.

I have been using Total Commander (prev Windows Commander and before that Norton Commander) for a long time now.

I find the synchronizing option to be very powerful. You can do a simple sync, but you can also do a compare by content to delete doubles. I think you can also set it to ignore file dates when doing a compare by content or synchronization.

As with the other options mentioned it is worth taking a look at. Work out which app works best for your needs.

Greets.
Fotonut.
--
Snap snap - click click.

FujiLinks - http://fujilinks.110mb.com/index.html

I checked with CrashPlan's developers and found out that the de-dup feature works in all their clients. Free means the de-dup option cannot be changed but it still works automatically.

wklee said:

I just discovered that CrashPlan has a de-duplication feature but it is not free. I looked at the CrashPlan Free client and some of the advanced features are grayed out. Apparently, Dropbox will also detect duplicates and will give the option to de-dup as well.

http://support.crashplan.com/doku.php/reference_guide/settings#advanced_backup_settings
--
Never buy version 1.0 of anything.

Click to expand...

--
Never buy version 1.0 of anything.

Hi - we are finishing a product to do exactly what you asked.

Will be readiy within 10 days

Would you like to review it for us?