Application for checking integrity of photos/backups?

[Trk]

Do you use any application for checking that photos and or backups have not been changed or corrupted? Can you recommend something?

 

[Zone8]

Trk wrote: Do you use any application for checking that photos and or backups have not been changed or corrupted? Can you recommend something?

I use FBackup - now up to edition 5.4. I make two backups every day at set time of 10:00 (it will also kick in should I start the computer later in the day). I have one MIRROR and other FULL - both set to be without any compression. It is also set to auto-confirm validity of its backups. Initially I did check many image files and all were OK - perfect file backups every time. Hope that proves useful. I have come to rely on this and it provides a secure backup that - so I have found - works perfectly. Cannot wish for anything more really! I did try a number of backup programmes before finding this - none of the others was anywhere near as good. That it autochecks itself is very reassuring and as I indicated - no problems with any files.

http://www.fbackup.com/

--
Zone8: Although I am a handsome genius, when I stand in front of a mirror, I vaguely recognise the ugly idjit standing on the other side!
LINK: For B+W with Epson 1400 (and other models) using black ink only PLUS other useful tips:
http://www.photosnowdonia.co.uk/ZPS/epson1400-B&W.htm
Cleaning DSLR Sensors, including Kodak DSLR Factory Cleaning method:
http://www.photosnowdonia.co.uk/ZPS/KodakDCS-sensorcleaning.htm (Includes links to "bassotto's" images)
Solving back/front focus problems on Sigma and most other DSLRs
http://www.photosnowdonia.co.uk/ZPS/backfocus.htm
PDF format list of lenses you can print or download - covers Italian Flag YES/NO for DCS 14n but applies to others. http://www.photosnowdonia.co.uk/ZPS/ItiFlagLensList.pdf

 

[Trk]

Thanks, I tried it and it looks ok, i.e. it is possible to check integrity of backups.

However it does not solve issue where source file is corrupted and in next full backup you backup corrupted file which means you have lost your data.

I wanted application that would also check files on various locations not only backups to detect changes to avoid situation where you are backing up corrupted files.

 

[Zone8]

Trk wrote: However it does not solve issue where source file is corrupted and in next full backup you backup corrupted file which means you have lost your data.

Sorry but I don't quite follow your line of thought. If the source file is corrupted - it's corrupted. Backing that up will not restore the original which, as you state, is already corrupted. Backups can only backup what is already existent as the original.

Perhaps I could cover something extra for you as an idea. When I copy my RAW files using a USB3 card reader (simple addition of USB3 PCIe card in desktop computer) or of course, slightly slower, USB2 card reader, I copy into two separate folders - on internal drive and external USB hard drive. I have folders limiting RAW files to 100 in any folder, for example 1-100, 101-200. 201-300 and so forth. That ensures no folder is crammed full of RAW files, so the software works quickly and efficiently and as a side benefit, no corruption takes place - never has due limiting the number of files being accessed at any one time. I thus have a backup of all RAW files in two places. The FBackup software does its work only on the folders on the internal hard drive, thus the original copies on the external drive are never affected. That sort of procedure should solve your potential problems. Should mention the files on the external drive are not in folders - just the lot in one folder as if required - can simply copy over the one file using a file manager. Incidentally, FBackup enables access to individual files that can be opened in the backed up section.

 

[Sean Nelson]

Trk wrote: However it does not solve issue where source file is corrupted and in next full backup you backup corrupted file which means you have lost your data.

Sorry but I don't quite follow your line of thought. If the source file is corrupted - it's corrupted. Backing that up will not restore the original which, as you state, is already corrupted. Backups can only backup what is already existent as the original.

Yes, but if you discover a corrupted file on your primary disk then you can go to your backups to see if you've got a copy that was made before the corruption occurred. That's the whole point of having backups, no?

The way to do this is to use a checksumming utility. You run the utility to get checksums of all your files and save them somewhere, then you run the utility again to verify that the files' checksums haven't changed. If you Google "checksum utility" you'll find a lot of choices.

Checksumming a file doesn't verify that the files are good to start with, but it does tell you whether or not they get changed in any way, shape or form.

 

[Trk]

Yes, but if you discover a corrupted file on your primary disk then you can go to your backups to see if you've got a copy that was made before the corruption occurred. That's the whole point of having backups, no?

Issue is that one does not discover that files are corrupted and she is backing up corrupted files overwriting good backups and if she discovers that files are corrupted they are already corrupted in backups. I want to avoid this because I have backups on external hard drives and I need not only know that backups are ok, but I need to know when I do backup that files I backup are also ok.

Thanks I will use checksumming utility.

 

[skyglider]

Yes, but if you discover a corrupted file on your primary disk then you can go to your backups to see if you've got a copy that was made before the corruption occurred. That's the whole point of having backups, no?

Issue is that one does not discover that files are corrupted and she is backing up corrupted files overwriting good backups and if she discovers that files are corrupted they are already corrupted in backups. I want to avoid this because I have backups on external hard drives and I need not only know that backups are ok, but I need to know when I do backup that files I backup are also ok.

Thanks I will use checksumming utility.

Is your friend running Windows 8? If so, does she modify any partitions outside of Win8 after a shutdown? Or has any partitions been modified by her or you using a partition manager? Or if running a dual boot, are files being modified in any partition that's visible to Win8 in the other partitions?

Win8 can result in file corruption if partitions are modified after a shutdown due to its Fast Startup feature which is enabled by default. There is a fix to prevent that.

Sky

 

[Kerry Pierce]

I don't use or know of a specific utility to constantly check files for corruption, so can't help you there.

Many years ago, I had a problem with file corruption and I lost a bunch of photos because my backups were also of the corrupted files. That is why I changed methods for my photos or other important files.

But, FWIW, I don't use a traditional backup program for my important files or photos. Instead, I use a file copy utility (called Bart, which can be found here ) that copies the original files to several removable SATA HDD's.

My photos are in unique, usually date stamped, folders where new files are only added when I edit the files in the folder. Only those edits are then copied to the backup folders/drives.

There are a number of utilities like Bart and I suspect that you might be able to find one that would do checksums. Bart can detect when a file has changed, but I don't know how that is accomplished.

The main thing to consider, IMO, is that you can reduce your risk by using a backup scheme like I use, rather than using a backup program that would happily do backups of corrupt files. Of course, this kind of thing probably takes a little more user input, but I don't trust my photos to a normal backup program, precisely for the reasons you state. I normally use Macrium Reflect Pro for my normal system backups.

Hope this makes sense to you.

Kerry

 

[Trk]

If so, does she modify any partitions outside of Win8 after a shutdown? Or has any partitions been modified by her or you using a partition manager? Or if running a dual boot, are files being modified in any partition that's visible to Win8 in the other partitions?
Win8 can result in file corruption if partitions are modified after a shutdown due to its Fast Startup feature which is enabled by default. There is a fix to prevent that.

I write hypothetically, not about issue when you modify partitions outside Windows with dual boot. I want to do backup of files, mainly photos and check integrity of backup and also files that are being backed up.

 

[Trk]

But, FWIW, I don't use a traditional backup program for my important files or photos. Instead, I use a file copy utility (called Bart, which can be found here ) that copies the original files to several removable SATA HDD's.

I am using open source Areca backup that copies new files also into different folders according to its scheme so there is available history of file versions, deleted files are available, etc., but Areca has advantages because it is backup software - e.g. you can merge incremental backups and release storage space, compress files, etc. But all backup programs trigger backup of files with modified date so issue occurs in case of full backups where history is lost.

I have now utility for continuously checking integrity of files with checksums so it should be ok, because Areca backup verifies files after backup and with additional integrity check I should be able to have uncorrupted files.

I wanted something like new resilient file system from Microsoft, but simpler - just copy/backup files onto the external harddrive and check integrity, I have files on two hard drives so in case of corruption I will be able to recover file. In case of resilient file system I would need to have probably 3 hard drives and documentation for it is not sufficient yet.

 

[Kerry Pierce]

But, FWIW, I don't use a traditional backup program for my important files or photos. Instead, I use a file copy utility (called Bart, which can be found here ) that copies the original files to several removable SATA HDD's.

I am using open source Areca backup that copies new files also into different folders according to its scheme so there is available history of file versions, deleted files are available, etc., but Areca has advantages because it is backup software - e.g. you can merge incremental backups and release storage space, compress files, etc. But all backup programs trigger backup of files with modified date so issue occurs in case of full backups where history is lost.

Yes, that's where the problem lies.

I have now utility for continuously checking integrity of files with checksums so it should be ok, because Areca backup verifies files after backup and with additional integrity check I should be able to have uncorrupted files.

That's great. What checksum utility did you end up with?

I wanted something like new resilient file system from Microsoft, but simpler - just copy/backup files onto the external harddrive and check integrity, I have files on two hard drives so in case of corruption I will be able to recover file. In case of resilient file system I would need to have probably 3 hard drives and documentation for it is not sufficient yet.

Yes, this is what I want to do as well. Traditional backups are great, excepting the circumstances you described.

Regardless, I'm glad that you figured out something that will work for you.

Kerry

 

[Trk]

That's great. What checksum utility did you end up with?

I started coding my own utility, because I want to have everything automatic, e.g. when drive is attached to PC it starts checking automatically when necessary.

 

[Kerry Pierce]

That's great. What checksum utility did you end up with?

I started coding my own utility, because I want to have everything automatic, e.g. when drive is attached to PC it starts checking automatically when necessary.

Ah, that's great that you have the skills to do that. You know that it will do exactly what you want done, when you've finished coding.

Kerry

 

[Kerry Pierce]

That's great. What checksum utility did you end up with?

I started coding my own utility, because I want to have everything automatic, e.g. when drive is attached to PC it starts checking automatically when necessary.

Just wanted to thank you for this thread. It got me up off my seat and looking for a solution that is for people like me, who don't have the skills to code it ourselves. :-D

After looking around for a while, I ended up on this blog which talks about backing up images. His blog made a lot of sense to me and he is using a commercially available product, called ViceVersaPro, which can be found here .

Here is a page from their FAQ, that says that it uses a CRC calculation for comparison of the files.

There are 2 versions, a ViceVersaPlus and ViceVersaPro. Both will do CRC comparisons, according to this page , which compares both versions on their site. The Plus is $35 and the Pro is $60.

Looks good to me! I don't know which one I'll buy, but there are 30 day trials of both, so I'll get it figured out.

Thanks again. This has been very beneficial to me.

Kerry

--
When is it "Okay" to be mean, petty or unethical?
-
my gallery of so-so photos
http://www.pbase.com/kerrypierce/root

 

[Sean Nelson]

That's great. What checksum utility did you end up with?

I started coding my own utility, because I want to have everything automatic, e.g. when drive is attached to PC it starts checking automatically when necessary.

After looking around for a while, I ended up on this blog which talks about backing up images. His blog made a lot of sense to me and he is using a commercially available product, called ViceVersaPro, which can be found here .
Here is a page from their FAQ, that says that it uses a CRC calculation for comparison of the files.

I just want to point out that there is a difference between checksumming the backup and checksumming the files being backed up.

Decent backup utilities provide a checksum option, but it applies to the backup file. The utility calculates a checksum as it writes the backup file to disk, and when it's finished writing the entire file it turns around and reads it back again, recalculating the checksum and comparing it to the original to ensure that the disk correctly stored every bit without error. But this does not check the original file that was backed up - if that original has been corrupted then the backup will have a perfect copy of the corruption.

What Trk is doing (I believe) is taking checksums of the original files and saving them. A year from now he can re-run the checksums and compare them against those he saved to ensure that the files haven't somehow been changed during that period of time. That doesn't have anything directly to do with a backup at all, except that if he finds that a file has mysteriously changed he'd hopefully be able to go to one of his backups and find one that has a copy of the file before it got corrupted.

 

[Alice Phoebe]

Do you use any application for checking that photos and or backups have not been changed or corrupted? Can you recommend something?

Recommend you EaseUS Todo Backup, it is a free backup software to backup files and folders. Backup files to secure your precious photos, office documents and videos.

Launch EaseUS Todo Backup, click File backup.

Select files to backup via User defined or File type.

If you select files via File type, You can choose backup based on file types - Documents Files, E-mail, Financial Files, Music Files, Picture Files, and Video Files, etc. Each type supports multi file formats. Tick the types you want to backup. Choose Picture Files.

You can Set backup schedule for your backups. Set schedule to backup files later, daily, weekly, monthly or upon event.

 

[Trk]

I just want to point out that there is a difference between checksumming the backup and checksumming the files being backed up.
Decent backup utilities provide a checksum option, but it applies to the backup file. ...

Yes, exactly.

I have initial version of utility, which continuously and automatically scans configured drives/folders (periodically, when drive is inserted in computer, etc.) and detects latent corruption, it is similar to resilient file system, but it does not automatically solve corruption, in case of corruption it shows repeatedly notifications and data must be repaired manually. I use this with open source Areca backup configured to check files after backup. This solution is good for me because I need half of hard drives required by RAID or similar.







For professionals I would recommend Microsoft resilient file system each volume at least two hard drives, e.g. volumes in PC at least two drives each protected with resilient filesystem to avoid corruption of source data and at least two harddrives for backup volumes.

 

[Kerry Pierce]

That's great. What checksum utility did you end up with?

I started coding my own utility, because I want to have everything automatic, e.g. when drive is attached to PC it starts checking automatically when necessary.

After looking around for a while, I ended up on this blog which talks about backing up images. His blog made a lot of sense to me and he is using a commercially available product, called ViceVersaPro, which can be found here .

Here is a page from their FAQ, that says that it uses a CRC calculation for comparison of the files.

I just want to point out that there is a difference between checksumming the backup and checksumming the files being backed up.

Decent backup utilities provide a checksum option, but it applies to the backup file. The utility calculates a checksum as it writes the backup file to disk, and when it's finished writing the entire file it turns around and reads it back again, recalculating the checksum and comparing it to the original to ensure that the disk correctly stored every bit without error. But this does not check the original file that was backed up - if that original has been corrupted then the backup will have a perfect copy of the corruption.

I don't know how any checksum utility would know whether or not the original file is already corrupt. At the first backup initiation, any program is going to assume that you, the operator, know the status of the files that you want backed up and that you wouldn't tell it to back up corrupt files.

Once the first backup is done, then the software can use its historical comparisons to ensure that a file has or hasn't changed.

What Trk is doing (I believe) is taking checksums of the original files and saving them. A year from now he can re-run the checksums and compare them against those he saved to ensure that the files haven't somehow been changed during that period of time. That doesn't have anything directly to do with a backup at all, except that if he finds that a file has mysteriously changed he'd hopefully be able to go to one of his backups and find one that has a copy of the file before it got corrupted.

I am sure that Trk is doing cool things with his coding. But that doesn't help those of us that haven't the skills to do that.

ViceVersa will take CRC checksums of the original file, if you specify such, according to the FAQ page that I linked to earlier.

Here's a quote from that page, "You can verify that files in a specified location have not changed by creating a snapshot of one or more folders at a given time and then re-comparing the folder content with the original snapshot at later time."

So, if this quote is to be believed, it seems to do what you think Trk is doing, by creating and keeping a snapshot of the original folder. Even so, if the user doesn't personally check each and every file to be backed up, there is a possibility that 1 or more files will be corrupt at the very beginning and will be faithfully backed up, even by the method you cite.

The old computer acronym GIGO still applies.

Having said all of that, if you know of a backup scheme that will do this level of file comparison or better, it might be helpful to others if you'd specify it.

Kerry

 

[Trk]

I don't know how any checksum utility would know whether or not the original file is already corrupt. At the first backup initiation, any program is going to assume that you, the operator, know the status of the files that you want backed up and that you wouldn't tell it to back up corrupt files.

Utility I am coding is storing information in database it will recognize that file is corrupt (=changed) and it works fully automatically.

You can additionally use file copy program that supports file verification after you copy your images from sdcard to harddrive. Then backup program that does file verification and then checksumming utility that will monitor all files on all drives to detect when file has been corrupted.

I will indeed offer my checksumming utility it for free as soon as it is finished, tested and I create some basic manual for it. I also use sha-1 hashes, not crc, to have increased reliability and possibility to extend functions.

 

[Kerry Pierce]

I don't know how any checksum utility would know whether or not the original file is already corrupt. At the first backup initiation, any program is going to assume that you, the operator, know the status of the files that you want backed up and that you wouldn't tell it to back up corrupt files.

Utility I am coding is storing information in database it will recognize that file is corrupt (=changed) and it works fully automatically.

I don't want to be argumentative, but I don't see how any program would know that a file was corrupted, before the program was used.

Assuming that all files are fine the first time that you run your program, then of course it will see the change.

You can additionally use file copy program that supports file verification after you copy your images from sdcard to harddrive. Then backup program that does file verification and then checksumming utility that will monitor all files on all drives to detect when file has been corrupted.

That's very cool. I haven't yet found a program that will do that monitoring routine.

I will indeed offer my checksumming utility it for free as soon as it is finished, tested and I create some basic manual for it. I also use sha-1 hashes, not crc, to have increased reliability and possibility to extend functions.

Wow, that is very generous of you sir! Even so, I'd be very happy to pay for such a utility, should you decide to at least set up a donation scheme for your hard work.

Kerry