External HDD question (Windows OS)

[Satyaa]

Background:

I use two 4TB external HDDs (probably 5 years old by now) on my Windows desktop (also 5+ years old) to back up my photos (among other things).

I use a free s/w called FreeFileSync to back up photos from my PC to HDD 1 every week. Then I use the same s/w to sync up HDD 1 to HDD 2 once a month.

The HDDs are formatted as NTFS drives. They're about 75% filled up at this point.

Question:

When I am sync'ing a large volume of files, usually over several GB, the target/destination HDD gets stuck at some point. I can't do much with it at that point. I have to stop the file sync program, log out, log back in, run error check and drive optimization (a.k.a. defrag) and then it moves along normally until next time.

This is always the case during monthly backups because the total files are usually over 20 GB. And the problem has existed for at least a couple of years, if not more.

Is that behavior normal with external drives? Is it because of their age and symptom of impending death? Is that because they're 75% filled up and beyond their optimum capacity? Or something else?

I have ruled out the s/w as a cause for this because I get the same problem even if I am copying the entire folder using Windows' normal copy/paste feature and the files are over several GB.

Thanks.

--
See my profile (About me) for gear and my posting policy.

 

[Karl_Guttag]

Background:

I use two 4TB external HDDs (probably 5 years old by now) on my Windows desktop (also 5+ years old) to back up my photos (among other things).

I use a free s/w called FreeFileSync to back up photos from my PC to HDD 1 every week. Then I use the same s/w to sync up HDD 1 to HDD 2 once a month.

The HDDs are formatted as NTFS drives. They're about 75% filled up at this point.

Question:

When I am sync'ing a large volume of files, usually over several GB, the target/destination HDD gets stuck at some point. I can't do much with it at that point. I have to stop the file sync program, log out, log back in, run error check and drive optimization (a.k.a. defrag) and then it moves along normally until next time.

This is always the case during monthly backups because the total files are usually over 20 GB. And the problem has existed for at least a couple of years, if not more.

Is that behavior normal with external drives? Is it because of their age and symptom of impending death? Is that because they're 75% filled up and beyond their optimum capacity? Or something else?

I have ruled out the s/w as a cause for this because I get the same problem even if I am copying the entire folder using Windows' normal copy/paste feature and the files are over several GB.

Thanks.

Not normal at all. You either have a Drive problem, a Software Problem, or a problem with your PC when doing certain tasks. The simplest way to figure out what is happening is to swap out the various elements and isolate the problem.

The defrag may or may not have anything to do with it. Unless the disk is nearly full, one defrag should last a long time.

I would start by swapping out the FreeFyleSync as there are many programs you can try (some may have a free trial period). You might even try and do some simple File Explorer move of huge sets of files to bypass the software. If you have or can use a different PC, try the drives with it. Unfortunately, the place with the most possible points-of-failure is the PC itself, including the CPU, memories, buses on the motherboard, and the internal drives. A good rule in debugging is to start with the easiest to swap out.

 

[BobKnDP]

Apparently, it's a common problem. It probably isn't due to failing hardware.

A little Googling gives the suggestion of enabling write caching for the external drives. (It's disabled by default.)

I'll have to try that myself. I haven't had a copy quit altogether, but I've seen transfer rates drop to low levels.

The downside is that data that is in the cache and not yet written to the disk can be lost if power fails or the drive is unplugged with using the "safely remove" feature.

If that helps, please let us know.

 

[BobKnDP]

Apparently, it's a common problem. It probably isn't due to failing hardware.

A little Googling gives the suggestion of enabling write caching for the external drives. (It's disabled by default.)

I'll have to try that myself. I haven't had a copy quit altogether, but I've seen transfer rates drop to low levels.

The downside is that data that is in the cache and not yet written to the disk can be lost if power fails or the drive is unplugged with using the "safely remove" feature.

If that helps, please let us know.

I just experimented with turning on write caching for an external USB drive. (An enclosure for a 3.5" disk. It has a high speed type B connector.)





The data transfer rates using Windows Explorer still went up and down, in the familiar manner. Pfui.

 

[maltmoose]

Its not normal. I also use freefilesync and seems solid to me with large mirrors.

I used to have an issue with an external SDD and it would always crash randomly with copies. I finally figured it out though. The SSD was fine but the enclosure was the issue. I bought an enclosure with a different chipset and its 100% solid in transfers now.

 

[Billiam29]

Background:

I use two 4TB external HDDs (probably 5 years old by now) on my Windows desktop (also 5+ years old) to back up my photos (among other things).

I use a free s/w called FreeFileSync to back up photos from my PC to HDD 1 every week. Then I use the same s/w to sync up HDD 1 to HDD 2 once a month.

Could you elaborate on why you’re not just syncing PC-to-HDD 2 once per month? With a file sync-based regimen, I’m having trouble envisioning why you’re using HDD 1 as the backup source for your monthly backup to HDD 2.

This of course does not address what is going on with your current situation. Having said that, if you take HDD 1 out of the picture as being the source for HDD 2 and everything works, then maybe that’s all you need.

 

[Craig Gillette]

I added a second external drive to do backups to recently. The process failed the first time for reasons I don't really recall (and may not understand), but IIRC, it had to do with available "copying(?)" space for the backup program to work, not on the receiving drive but on my C drive. I'd been downloading my photos to the C drive so it was perhaps more full than it should have been. I moved enough files to my D drive to bring C down to about half full and about 1/3 full on D using Explorer and tried again and it did fine, backing up from the installed drives.

So, with having been using the drives for so long, perhaps you've crept into a space problem on one of the drives?

 

[Carey Brown]

Background:

I use two 4TB external HDDs (probably 5 years old by now) on my Windows desktop (also 5+ years old) to back up my photos (among other things).

I use a free s/w called FreeFileSync to back up photos from my PC to HDD 1 every week. Then I use the same s/w to sync up HDD 1 to HDD 2 once a month.

Could you elaborate on why you’re not just syncing PC-to-HDD 2 once per month? With a file sync-based regimen, I’m having trouble envisioning why you’re using HDD 1 as the backup source for your monthly backup to HDD 2.

This of course does not address what is going on with your current situation. Having said that, if you take HDD 1 out of the picture as being the source for HDD 2 and everything works, then maybe that’s all you need.

Your backup scheme doesn't make sense to me either. You should have at least two backup drives that you swap out in rotation so that if one fails you haven't lost every thing. In your scheme HDD-2 is your only real backup. Your HDD-1 is just a temporary backup. Assuming that you have enough room on your internal drives for all of your original files then I would just rotate between HDD-1 and HDD-2 and forget the 1-->2 copying.

It has been suggested that you swap things out one at a time to narrow down the culprit. You may want to get a third external drive and see if that works. Either way you'd end up with 3 drives in your backup rotation which would be better still.

There are memory test programs, have you tried one of those?

 

[LynniePad]

Background:

I use two 4TB external HDDs (probably 5 years old by now) on my Windows desktop (also 5+ years old) to back up my photos (among other things).

I use a free s/w called FreeFileSync to back up photos from my PC to HDD 1 every week. Then I use the same s/w to sync up HDD 1 to HDD 2 once a month.

The HDDs are formatted as NTFS drives. They're about 75% filled up at this point.

Some form of incremental backup, where you only add new or changed files to the backup drive, might solve your problem and save a lot of time.

Unless you are frequently changing most of your data, you can also set aside large chunks of it for long-term storage, only refreshing it every now and then.

I’d also question the logic of copying HDD1 to HDD2. Better to cycle them.

Question:

When I am sync'ing a large volume of files, usually over several GB, the target/destination HDD gets stuck at some point. I can't do much with it at that point. I have to stop the file sync program, log out, log back in, run error check and drive optimization (a.k.a. defrag) and then it moves along normally until next time.

Defrag is unlikely to be the actual cure to your problem.

This is always the case during monthly backups because the total files are usually over 20 GB. And the problem has existed for at least a couple of years, if not more.

Sometimes a rogue file can cause this sort of error. Look for a file that has an overly long file name, or unusual characters in the file name.

 

[Satyaa]

Background:

I use two 4TB external HDDs (probably 5 years old by now) on my Windows desktop (also 5+ years old) to back up my photos (among other things).

I use a free s/w called FreeFileSync to back up photos from my PC to HDD 1 every week. Then I use the same s/w to sync up HDD 1 to HDD 2 once a month.

The HDDs are formatted as NTFS drives. They're about 75% filled up at this point.

Question:

When I am sync'ing a large volume of files, usually over several GB, the target/destination HDD gets stuck at some point. I can't do much with it at that point. I have to stop the file sync program, log out, log back in, run error check and drive optimization (a.k.a. defrag) and then it moves along normally until next time.

This is always the case during monthly backups because the total files are usually over 20 GB. And the problem has existed for at least a couple of years, if not more.

Is that behavior normal with external drives? Is it because of their age and symptom of impending death? Is that because they're 75% filled up and beyond their optimum capacity? Or something else?

I have ruled out the s/w as a cause for this because I get the same problem even if I am copying the entire folder using Windows' normal copy/paste feature and the files are over several GB.

Thanks.

Not normal at all. You either have a Drive problem, a Software Problem, or a problem with your PC when doing certain tasks. The simplest way to figure out what is happening is to swap out the various elements and isolate the problem.

The defrag may or may not have anything to do with it. Unless the disk is nearly full, one defrag should last a long time.

I would start by swapping out the FreeFyleSync as there are many programs you can try (some may have a free trial period). You might even try and do some simple File Explorer move of huge sets of files to bypass the software.

Please see the part highlighted in my last paragraph above.

If you have or can use a different PC, try the drives with it.

This is something I can try on on of my laptops

Unfortunately, the place with the most possible points-of-failure is the PC itself, including the CPU, memories, buses on the motherboard, and the internal drives. A good rule in debugging is to start with the easiest to swap out.

Thanks

 

[Satyaa]

Thanks for the tips. I'll try that

 

[Satyaa]

OK. Thanks for confirming your experience.

 

[Satyaa]

Background:

I use two 4TB external HDDs (probably 5 years old by now) on my Windows desktop (also 5+ years old) to back up my photos (among other things).

I use a free s/w called FreeFileSync to back up photos from my PC to HDD 1 every week. Then I use the same s/w to sync up HDD 1 to HDD 2 once a month.

Could you elaborate on why you’re not just syncing PC-to-HDD 2 once per month? With a file sync-based regimen, I’m having trouble envisioning why you’re using HDD 1 as the backup source for your monthly backup to HDD 2.

Good question. It started long time ago when I was also re-organizing, deleting files, etc., on the primary HDD and wanted those changes to be sync'ed up.

This of course does not address what is going on with your current situation. Having said that, if you take HDD 1 out of the picture as being the source for HDD 2 and everything works, then maybe that’s all you need.

That's a good point. I'll trying to sync HDD 2 with the desktop to see if it improves anything.

Thanks

 

[Satyaa]

Background:

I use two 4TB external HDDs (probably 5 years old by now) on my Windows desktop (also 5+ years old) to back up my photos (among other things).

I use a free s/w called FreeFileSync to back up photos from my PC to HDD 1 every week. Then I use the same s/w to sync up HDD 1 to HDD 2 once a month.

Could you elaborate on why you’re not just syncing PC-to-HDD 2 once per month? With a file sync-based regimen, I’m having trouble envisioning why you’re using HDD 1 as the backup source for your monthly backup to HDD 2.

This of course does not address what is going on with your current situation. Having said that, if you take HDD 1 out of the picture as being the source for HDD 2 and everything works, then maybe that’s all you need.

Your backup scheme doesn't make sense to me either. You should have at least two backup drives that you swap out in rotation so that if one fails you haven't lost every thing. In your scheme HDD-2 is your only real backup. Your HDD-1 is just a temporary backup.

I have taken care of that. Between the camera cards, desktop and two HDDs, there are always two copies of my files, sometimes three. I delete the files on the card once the processed files on desktop have been backed up. When I sync one HDD to the second, there are three copies probably for a year. I delete photos older than a year from my desktop and each HDD has a copy at that point.

Assuming that you have enough room on your internal drives for all of your original files then I would just rotate between HDD-1 and HDD-2 and forget the 1-->2 copying.

It has been suggested that you swap things out one at a time to narrow down the culprit. You may want to get a third external drive and see if that works. Either way you'd end up with 3 drives in your backup rotation which would be better still.

There are memory test programs, have you tried one of those?

I have tried HDD test programs both from the manufacturer and third-party. Found no issues.

Thanks

 

[Satyaa]

I added a second external drive to do backups to recently. The process failed the first time for reasons I don't really recall (and may not understand), but IIRC, it had to do with available "copying(?)" space for the backup program to work, not on the receiving drive but on my C drive. I'd been downloading my photos to the C drive so it was perhaps more full than it should have been. I moved enough files to my D drive to bring C down to about half full and about 1/3 full on D using Explorer and tried again and it did fine, backing up from the installed drives.

So, with having been using the drives for so long, perhaps you've crept into a space problem on one of the drives?

That's a good point.

I don't have any of these photos/videos on C drive. Only Windows and some documents. It is a 1TB SSD and more than half is free.

The D drive is a 2TB SSD, which is where my photos and videos are. It has more free space than my external HDDs have now.

Thanks.

 

[Satyaa]

Background:

I use two 4TB external HDDs (probably 5 years old by now) on my Windows desktop (also 5+ years old) to back up my photos (among other things).

I use a free s/w called FreeFileSync to back up photos from my PC to HDD 1 every week. Then I use the same s/w to sync up HDD 1 to HDD 2 once a month.

The HDDs are formatted as NTFS drives. They're about 75% filled up at this point.

Some form of incremental backup, where you only add new or changed files to the backup drive, might solve your problem and save a lot of time.

I don't do a backup the way Windows does. I sync up files to mimic the folder structure, because when I need photos from more than a year ago, I get them from the HDD.

Unless you are frequently changing most of your data, you can also set aside large chunks of it for long-term storage, only refreshing it every now and then.

I’d also question the logic of copying HDD1 to HDD2. Better to cycle them.

Between the two HDDs, I do swap between the primary and secondary once in a while. They are same model/brand, so I identify them with a nylon band around them. The one with blue band is primary for few months, then one with gold band is the primary for few months.

Question:

When I am sync'ing a large volume of files, usually over several GB, the target/destination HDD gets stuck at some point. I can't do much with it at that point. I have to stop the file sync program, log out, log back in, run error check and drive optimization (a.k.a. defrag) and then it moves along normally until next time.

Defrag is unlikely to be the actual cure to your problem.

This is always the case during monthly backups because the total files are usually over 20 GB. And the problem has existed for at least a couple of years, if not more.

Sometimes a rogue file can cause this sort of error. Look for a file that has an overly long file name, or unusual characters in the file name.

That's something I have to look for.

Thanks

 

[LynniePad]

I use two 4TB external HDDs (probably 5 years old by now) on my Windows desktop (also 5+ years old) to back up my photos (among other things).

I use a free s/w called FreeFileSync to back up photos from my PC to HDD 1 every week. Then I use the same s/w to sync up HDD 1 to HDD 2 once a month.

The HDDs are formatted as NTFS drives. They're about 75% filled up at this point.

Some form of incremental backup, where you only add new or changed files to the backup drive, might solve your problem and save a lot of time.

I don't do a backup the way Windows does. I sync up files to mimic the folder structure, because when I need photos from more than a year ago, I get them from the HDD.

I actually use Robocopy (Windows built-in program). With the correct options, it will do a full backup the first time you run it, while subsequent runs will look after new or changed files. Very quick if you opt for multi-threaded execution.

Sometimes a rogue file can cause this sort of error. Look for a file that has an overly long file name, or unusual characters in the file name.

That's something I have to look for.

MS-Word has a nasty habit of using the first line of a document as the default file name. We had a cluster of such files (submitted by students) that caused copy problems. That’s one place to look.

 

[Satyaa]

I use two 4TB external HDDs (probably 5 years old by now) on my Windows desktop (also 5+ years old) to back up my photos (among other things).

I use a free s/w called FreeFileSync to back up photos from my PC to HDD 1 every week. Then I use the same s/w to sync up HDD 1 to HDD 2 once a month.

The HDDs are formatted as NTFS drives. They're about 75% filled up at this point.

Some form of incremental backup, where you only add new or changed files to the backup drive, might solve your problem and save a lot of time.

I don't do a backup the way Windows does. I sync up files to mimic the folder structure, because when I need photos from more than a year ago, I get them from the HDD.

I actually use Robocopy (Windows built-in program). With the correct options, it will do a full backup the first time you run it, while subsequent runs will look after new or changed files. Very quick if you opt for multi-threaded execution.

I heard of Robocopy tool (may have used it in the past) but was not aware of it being built-in with Windows. I will check it out.

Sometimes a rogue file can cause this sort of error. Look for a file that has an overly long file name, or unusual characters in the file name.

That's something I have to look for.

MS-Word has a nasty habit of using the first line of a document as the default file name. We had a cluster of such files (submitted by students) that caused copy problems. That’s one place to look.

Thanks for that tip

 

[Carey Brown]

Background:

I use two 4TB external HDDs (probably 5 years old by now) on my Windows desktop (also 5+ years old) to back up my photos (among other things).

I use a free s/w called FreeFileSync to back up photos from my PC to HDD 1 every week. Then I use the same s/w to sync up HDD 1 to HDD 2 once a month.

Could you elaborate on why you’re not just syncing PC-to-HDD 2 once per month? With a file sync-based regimen, I’m having trouble envisioning why you’re using HDD 1 as the backup source for your monthly backup to HDD 2.

This of course does not address what is going on with your current situation. Having said that, if you take HDD 1 out of the picture as being the source for HDD 2 and everything works, then maybe that’s all you need.

Your backup scheme doesn't make sense to me either. You should have at least two backup drives that you swap out in rotation so that if one fails you haven't lost every thing. In your scheme HDD-2 is your only real backup. Your HDD-1 is just a temporary backup.

I have taken care of that. Between the camera cards, desktop and two HDDs, there are always two copies of my files, sometimes three. I delete the files on the card once the processed files on desktop have been backed up. When I sync one HDD to the second, there are three copies probably for a year. I delete photos older than a year from my desktop and each HDD has a copy at that point.

Assuming that you have enough room on your internal drives for all of your original files then I would just rotate between HDD-1 and HDD-2 and forget the 1-->2 copying.

It has been suggested that you swap things out one at a time to narrow down the culprit. You may want to get a third external drive and see if that works. Either way you'd end up with 3 drives in your backup rotation which would be better still.

There are memory test programs, have you tried one of those?

I have tried HDD test programs both from the manufacturer and third-party. Found no issues.

Thanks

I meant *memory* as in RAM.

 

[skyglider]

The HDDs are formatted as NTFS drives. They're about 75% filled up at this point.

I have ruled out the s/w as a cause for this because I get the same problem even if I am copying the entire folder using Windows' normal copy/paste feature and the files are over several GB.

Although you said the HDDs are formatted as NTFS, your problem really sounds like a FAT format problem. FAT has a max file size of 4GB.

Are you able to copy files of less than 4GB OK?