Duplicate & Near-Same Photos

Started 2 months ago | Discussions
garywc
garywc Senior Member • Posts: 1,012
Duplicate & Near-Same Photos

I have photos spread over 147 folders on my PC.

Is there any way to compare the contents of the photos so I can delete the duplicates or near-duplicate photos?

 garywc's gear list:garywc's gear list
Canon PowerShot G16
darg Regular Member • Posts: 495
Re: Duplicate & Near-Same Photos
6

A bottle of good wine and some evenings in front of your PC to finally organize your collection

makes also a good New Years resolution.

-- hide signature --

When in doubt, C4
KJ6EPH

 darg's gear list:darg's gear list
Canon PowerShot A80 Olympus E-500 Olympus PEN E-P3 Panasonic Lumix DMC-GM1 Sony a7 II +12 more
garywc
OP garywc Senior Member • Posts: 1,012
Re: Duplicate & Near-Same Photos

The same photo may exist in multiple folders and the file-names may be different.

 garywc's gear list:garywc's gear list
Canon PowerShot G16
rurikw Senior Member • Posts: 2,818
Re: Duplicate & Near-Same Photos
3

garywc wrote:

The same photo may exist in multiple folders and the file-names may be different.

how did they end up that way?

 rurikw's gear list:rurikw's gear list
Nikon Coolpix 5000 Sony Cyber-shot DSC-R1 Sigma DP1 Merrill Sigma DP2 Merrill Sigma DP3 Merrill +28 more
TacticDesigns
TacticDesigns Veteran Member • Posts: 6,787
Re: Duplicate & Near-Same Photos

garywc wrote:

The same photo may exist in multiple folders and the file-names may be different.

If you haven't changed the EXIF data inside the photo, and the date+time that the photo was taken is still intact, you might be able to use a program to rename the file to have the date+time put at the beginning of the filename.

This can help to make each picture have a unique filename.

But also, it would let you copy all the photos to fewer folders and have all the photos put in chronological order to when the picture was taken.

Then you can browse through . . . and all the "same pictures" should be grouped together.

To rename JPG files this way I use a program like NAMEXIF.

To rename RAW files this way I use Adobe Lightroom Classic.

Take care & Happy Shooting!

-- hide signature --
 TacticDesigns's gear list:TacticDesigns's gear list
Fujifilm XP80 Nikon D5100 Pentax Q Pentax *ist DS Pentax K100D +9 more
Photoby Regular Member • Posts: 152
Re: Duplicate & Near-Same Photos

Doing this by hand would be an exhausting and error-prone task, taking a very long time.

If you can extract the EXIF data, and you (or someone you know or can hire) has some programming skills, you can:

  • extract all that data from each file
  • sort it all so that duplicates fall together

and then go through that list to find duplicates (even if they have different names).

That is not a difficult thing to do at all - and it could be automated to a lesser or greater degree.

It won't catch the near-dups, but in all likelihood, they fall nearby their 'cousins', which should help some.

More sophisticated (and maybe not possible on a PC) ...

there is now software that can do this based on the images themselves (think face recognition software, but for whole pictures).  Getty Images, and perhaps others, crawl the web looking for unlicensed copies of their images - and they find them in the most obscure places.

These work by turning the image into a digital fingerprint of some kind, and then comparing those.

garywc
OP garywc Senior Member • Posts: 1,012
Re: Duplicate & Near-Same Photos

Years ago, I uploaded many of the photos to various "albums" in Photobucket.

A few weeks ago, I downloaded the photos from Photobucket back to my PC.  The files from each Photobucket "album" were put in separate folders on my PC.

That's how the same photos ended up in multiple folders.

 garywc's gear list:garywc's gear list
Canon PowerShot G16
Mark Scott Abeln
Mark Scott Abeln Forum Pro • Posts: 14,350
Lightroom

Lightroom can detect duplicates.

 Mark Scott Abeln's gear list:Mark Scott Abeln's gear list
Nikon D200 Nikon D7000 Nikon D750 Nikon AF-S DX Nikkor 35mm F1.8G Nikon AF Nikkor 50mm f/1.8D +2 more
Marek M Senior Member • Posts: 1,553
By date

garywc wrote:

The same photo may exist in multiple folders and the file-names may be different.

I would copy them to one folder and organized by date. That way selection could be easier.

mamallama
mamallama Forum Pro • Posts: 56,712
Re: Duplicate & Near-Same Photos

garywc wrote:

I have photos spread over 147 folders on my PC.

Is there any way to compare the contents of the photos so I can delete the duplicates or near-duplicate photos?

What would be your purpose in doing that? What would it accomplish? Wouldn't just selecting the one of the near duplicates as Grade 1 for use and just let the others stay where they are be OK? Hard disk storage is cheap.

 mamallama's gear list:mamallama's gear list
Panasonic Lumix DMC-GX85 +1 more
garywc
OP garywc Senior Member • Posts: 1,012
Re: Duplicate & Near-Same Photos

The photos do not have all of these EXIF dates:

  • DateTimeOriginal
  • CreateDate
  • FileCreateDate
  • DateTaken
  • FileModifyDate
  • FileAccessDate
  • ModifyDate

so I don't know what date I should use to put the photos into chronological order.

Also, some of the duplicate photos have different names (somehow they got changed):

instead of the file-name being IMG_2551.JPG, IMG_2552.JPG, IMG_2553.JPG, etc

the file-names are $R3O8CVV.JPG, $RA2MEZZ.JPG, $RAQ2V7U.JPG, $RBMNWZ0.JPG, etc

 garywc's gear list:garywc's gear list
Canon PowerShot G16
Photoby Regular Member • Posts: 152
Re: Duplicate & Near-Same Photos

garywc wrote:

Years ago, I uploaded many of the photos to various "albums" in Photobucket.

A few weeks ago, I downloaded the photos from Photobucket back to my PC. The files from each Photobucket "album" were put in separate folders on my PC.

That's how the same photos ended up in multiple folders.

That's also probably how the names got changed from IMG_... to those other names.

mamallama
mamallama Forum Pro • Posts: 56,712
Re: Duplicate & Near-Same Photos

garywc wrote:

The photos do not have all of these EXIF dates:

  • DateTimeOriginal
  • CreateDate
  • FileCreateDate
  • DateTaken
  • FileModifyDate
  • FileAccessDate
  • ModifyDate

so I don't know what date I should use to put the photos into chronological order.

Also, some of the duplicate photos have different names (somehow they got changed):

instead of the file-name being IMG_2551.JPG, IMG_2552.JPG, IMG_2553.JPG, etc

the file-names are $R3O8CVV.JPG, $RA2MEZZ.JPG, $RAQ2V7U.JPG, $RBMNWZ0.JPG, etc

All the more reason you should label the primary ones with a good DAM program and leave the duplicate and near duplicate ones alone. They are doing no harm taking up storage space on your hard drive which is so cheap. Use the DAM to search the primary ones for use.

If you have thousands of images so mixed up, trying to eliminate and delete the duplicate and near duplicate ones can be a thankless and useless task.

 mamallama's gear list:mamallama's gear list
Panasonic Lumix DMC-GX85 +1 more
Keyboard shortcuts:
FForum MMy threads