Software for mass copy to external SSD

[Sean Nelson]

In theory, explorer is fine. In practice, stuff happens. I prefer to trust, but verify.

You are worrying about a zero-risk problem in practice, and you can't verify 95% of the file movements that are happening all the time and that you are not aware of.

I agree that problems of this sort are few and far between, but they do happen. I started having verification problems on my backups that I initially blamed on the disk, but when they kept happening with different drives I was forced to troubleshoot further and eventually discovered that I had an intermittent memory problem. The only thing that caught it in normal use was the end-to-end verification of my backups. The POST memory test at startup missed it, it wasn't until I resorted to running lengthy memory diagnostics that I was able to pin it down.

Read-after-write is useless in this kind of situation because the error occurs before the data is sent to the disk. Alas, memory is the single point of failure where zero error checking is done in most consumer systems. And I run everything to spec, those folks who play around with overclocking are more at risk.

As a result of this experience I now only buy systems with ECC memory. That will hopefully prevent this kind of problem from going undetected again. but Murphy lurks under the covers and for the irreplaceable data I've collected over a lifetime a policy of "trust but verify" keeps me happy. It's not as if I have to do it by hand, I have this thing called a "computer" that does it for me while I do other work.

Each person can decide for themselves what precautions to take based on how much they value their data, but they should base those decisions on an understanding of what kinds of things can go wrong.

 

[Chris Noble]

In theory, explorer is fine. In practice, stuff happens. I prefer to trust, but verify.

You are worrying about a zero-risk problem in practice, and you can't verify 95% of the file movements that are happening all the time and that you are not aware of.

I agree that problems of this sort are few and far between, but they do happen. I started having verification problems on my backups that I initially blamed on the disk, but when they kept happening with different drives I was forced to troubleshoot further and eventually discovered that I had an intermittent memory problem. The only thing that caught it in normal use was the end-to-end verification of my backups. The POST memory test at startup missed it, it wasn't until I resorted to running lengthy memory diagnostics that I was able to pin it down.

Read-after-write is useless in this kind of situation because the error occurs before the data is sent to the disk. Alas, memory is the single point of failure where zero error checking is done in most consumer systems. And I run everything to spec, those folks who play around with overclocking are more at risk.

As a result of this experience I now only buy systems with ECC memory. That will hopefully prevent this kind of problem from going undetected again. but Murphy lurks under the covers and for the irreplaceable data I've collected over a lifetime a policy of "trust but verify" keeps me happy. It's not as if I have to do it by hand, I have this thing called a "computer" that does it for me while I do other work.

Each person can decide for themselves what precautions to take based on how much they value their data, but they should base those decisions on an understanding of what kinds of things can go wrong.

Periodic RAM problems can occur. I sympathize with you on that issue as it can be nasty. Unfortunately, ECC memory does not catch all of them.

You should be running a systems check on your computer periodically. For my Dells, it's called SupportAssist and runs by default once a month, It includes RAM tests and other system-board tests. The RAM tests are quite robust against periodic failure modes, but no test is absolutely fool-proof.

Doing forensic analysis on backup files will uncover some systems problems, but it's an indirect and convoluted and unreliable substitute. The best approach is a direct systems test, combined with a backup system that meets the three backup requirements: redundant, regular, and automated. My backup runs automated every night, and the backup system runs its own automated file-integrity tests monthly. The human error element is almost always the #1 failure mode.

And waiting until you observe problems before you run diagnostic tests is too late. They should be conducted on a regular basis, and scheduled by automated routines, to avoid the almost-inevitable "I forgot" scenario.

 

[Henry Richardson]

I had a bad experience with file corruption during a File Explorer ~450gb copy from one external HDD to another external HDD. That is one of the reasons I started using Syncback with the verify option.

Years ago I bought a new 1tb external HDD so I setup my Windows laptop to copy all the data from a 500gb external HDD using File Explorer. I discovered about 10 of my camera raw files were corrupted on the 1tb drive. They were not consecutive photos, they were taken at various times and in various folders. They were not corrupted on the source drive though. I never knew what happened during the all night copy, but if I had used Syncback with the verify option then possibly it would have caught it. Since then I use Syncback with verify. Maybe just lucky, but I have not had any more file corruption problems since doing it this way. At least that I have discovered.

 

[Carey Brown]

I had a bad experience with file corruption during a File Explorer ~450gb copy from one external HDD to another external HDD. That is one of the reasons I started using Syncback with the verify option.

Years ago I bought a new 1tb external HDD so I setup my Windows laptop to copy all the data from a 500gb external HDD using File Explorer. I discovered about 10 of my camera raw files were corrupted on the 1tb drive. They were not consecutive photos, they were taken at various times and in various folders. They were not corrupted on the source drive though. I never knew what happened during the all night copy, but if I had used Syncback with the verify option then possibly it would have caught it. Since then I use Syncback with verify. Maybe just lucky, but I have not had any more file corruption problems since doing it this way. At least that I have discovered.

So, did Syncback not fill the bill of your original post?

 

[Colin Creevey]

I had a bad experience with file corruption during a File Explorer ~450gb copy from one external HDD to another external HDD. That is one of the reasons I started using Syncback with the verify option.

Years ago I bought a new 1tb external HDD so I setup my Windows laptop to copy all the data from a 500gb external HDD using File Explorer. I discovered about 10 of my camera raw files were corrupted on the 1tb drive. They were not consecutive photos, they were taken at various times and in various folders. They were not corrupted on the source drive though. I never knew what happened during the all night copy, but if I had used Syncback with the verify option then possibly it would have caught it. Since then I use Syncback with verify. Maybe just lucky, but I have not had any more file corruption problems since doing it this way. At least that I have discovered.

I've had the same issue. Terracopy (free) also has the option to turn on verify. It hashes the source file and hashes the output then compares the hashes.

 

[Sean Nelson]

I had a bad experience with file corruption during a File Explorer ~450gb copy from one external HDD to another external HDD. ... I never knew what happened during the all night copy...

I wrote a utility for myself that monitors the event log and lets me know if there are any errors that get reported, which includes errors involving reading or writing to drives. But I've never come across any USB drives that actually report errors. I don't know if it's a shortcoming of the protocol or what, but because of the lack of error notification I always verify anything I write to USB drives that I consider important.

And since my backups go to USB drives, my backup utility does a backup pass and then a verify pass.

 

[TatumUK]

I always generated a SHA-256 hash for any critical data I move. I then check the checksums on the copied data to verify the data's integrity.

 

[Henry Richardson]

I had a bad experience with file corruption during a File Explorer ~450gb copy from one external HDD to another external HDD. That is one of the reasons I started using Syncback with the verify option.

Years ago I bought a new 1tb external HDD so I setup my Windows laptop to copy all the data from a 500gb external HDD using File Explorer. I discovered about 10 of my camera raw files were corrupted on the 1tb drive. They were not consecutive photos, they were taken at various times and in various folders. They were not corrupted on the source drive though. I never knew what happened during the all night copy, but if I had used Syncback with the verify option then possibly it would have caught it. Since then I use Syncback with verify. Maybe just lucky, but I have not had any more file corruption problems since doing it this way. At least that I have discovered.

I've had the same issue. Terracopy (free) also has the option to turn on verify. It hashes the source file and hashes the output then compares the hashes.

Yes, I have used teracopy for years and always use the verify option. I usually use it when I am copying something pretty big such as my LrC catalog file, but not for mass copy to a new external drive.

 

[Michael Firstlight]

The fastest mass file copying program for Windows can vary based on your specific needs and the nature of your file transfer tasks. Several tools are optimized for speed and reliability. Here are some of the most popular and widely recognized ones:

1. TeraCopy: This is one of the most popular file copy utilities for Windows. It's designed to copy and move files at the maximum possible speed, offers robust error recovery, and integrates seamlessly with Windows Explorer.
2. FastCopy: This is another tool that claims to be the fastest copy/delete software on Windows. It supports different modes of file copying, like differential copy (copying only different or non-existing files). It's highly configurable and can achieve impressive speeds.
3. Robocopy (Robust File Copy): This is a command-line tool that's been built into Windows for several years. It's highly robust, offering a slew of features and options. It's particularly powerful for network transfers and offers multi-threaded copying since Windows 7.
4. UltraCopier: This is an open-source tool designed to enable faster and more reliable file copying. It provides a user-friendly interface, error management, and customizable skins.
5. RichCopy: Originally developed by a Microsoft engineer, this tool offers a GUI and supports multi-threaded file copying, which can significantly speed up file transfers.
6. XXCOPY: It's a command-line tool that extends the capabilities of XCOPY, a built-in Windows tool. It's versatile and powerful but has a steeper learning curve due to its command-line nature.
7. Copy Handler: This tool offers high customization, pause/resume functionality, and integrates with the Windows shell. It's also optimized for speed.

When determining the "fastest" tool, consider the following factors:

• Nature of the Transfer: The speed can vary based on whether you're transferring many small files or a few large ones, or if you're copying data to/from SSDs, HDDs, or over a network.
• User Interface vs. Command-Line: GUI tools might be more user-friendly, but command-line tools can be more powerful and flexible, especially for batch operations or scripting.
• Additional Features: Some tools provide extra features like error recovery, verification, or the ability to pause and resume transfers.
• System and Software Compatibility: Ensure the tool is compatible with your version of Windows and doesn't conflict with other software.

I personally use TerraCopy for pure speed with bulk file transfers:

• Optimizations:
  • TeraCopy: It's designed to copy and move files at the maximum possible speed. It uses dynamically adjusted buffers to reduce the seek times. Also, asynchronous copy speeds up file transfer between two physical hard drives.
  • Windows Explorer: The built-in file copy feature in Windows has improved over various versions of the OS, but it might not be as optimized for speed as TeraCopy in certain scenarios.
• Error Handling:
  • TeraCopy: One of TeraCopy's main features is its robust error recovery. If a file transfer fails, it will often retry multiple times. If it's unable to read a file after several attempts, it will skip that file and continue with the rest. At the end of the copy job, it provides a report detailing any problematic files.
  • Windows Explorer: In older versions of Windows, if Explorer encountered an error during a copy job (e.g., unreadable file), it might stop the entire process. Newer versions have improved this behavior, but TeraCopy's error handling is generally more detailed and user-friendly.
• Features:
  • TeraCopy: It provides additional features, like pause and resume file transfers, verify files after copying, and integrate with Windows Explorer's context menu for seamless usage.
  • Windows Explorer: While it provides a more straightforward and integrated experience, it might lack advanced features like those provided by TeraCopy.

 

[David Lal]

Thanks Michael. A useful summary. My main problem right now is remembering what I've backed up and where!

 

[Carey Brown]

The fastest mass file copying program for Windows can vary based on your specific needs and the nature of your file transfer tasks. Several tools are optimized for speed and reliability. Here are some of the most popular and widely recognized ones:

1. TeraCopy: This is one of the most popular file copy utilities for Windows. It's designed to copy and move files at the maximum possible speed, offers robust error recovery, and integrates seamlessly with Windows Explorer.
2. FastCopy: This is another tool that claims to be the fastest copy/delete software on Windows. It supports different modes of file copying, like differential copy (copying only different or non-existing files). It's highly configurable and can achieve impressive speeds.
3. Robocopy (Robust File Copy): This is a command-line tool that's been built into Windows for several years. It's highly robust, offering a slew of features and options. It's particularly powerful for network transfers and offers multi-threaded copying since Windows 7.
4. UltraCopier: This is an open-source tool designed to enable faster and more reliable file copying. It provides a user-friendly interface, error management, and customizable skins.
5. RichCopy: Originally developed by a Microsoft engineer, this tool offers a GUI and supports multi-threaded file copying, which can significantly speed up file transfers.
6. XXCOPY: It's a command-line tool that extends the capabilities of XCOPY, a built-in Windows tool. It's versatile and powerful but has a steeper learning curve due to its command-line nature.
7. Copy Handler: This tool offers high customization, pause/resume functionality, and integrates with the Windows shell. It's also optimized for speed.

When determining the "fastest" tool, consider the following factors:

• Nature of the Transfer: The speed can vary based on whether you're transferring many small files or a few large ones, or if you're copying data to/from SSDs, HDDs, or over a network.
• User Interface vs. Command-Line: GUI tools might be more user-friendly, but command-line tools can be more powerful and flexible, especially for batch operations or scripting.
• Additional Features: Some tools provide extra features like error recovery, verification, or the ability to pause and resume transfers.
• System and Software Compatibility: Ensure the tool is compatible with your version of Windows and doesn't conflict with other software.

I personally use TerraCopy for pure speed with bulk file transfers:

• Optimizations:
  • TeraCopy: It's designed to copy and move files at the maximum possible speed. It uses dynamically adjusted buffers to reduce the seek times. Also, asynchronous copy speeds up file transfer between two physical hard drives.
  • Windows Explorer: The built-in file copy feature in Windows has improved over various versions of the OS, but it might not be as optimized for speed as TeraCopy in certain scenarios.
• Error Handling:
  • TeraCopy: One of TeraCopy's main features is its robust error recovery. If a file transfer fails, it will often retry multiple times. If it's unable to read a file after several attempts, it will skip that file and continue with the rest. At the end of the copy job, it provides a report detailing any problematic files.
  • Windows Explorer: In older versions of Windows, if Explorer encountered an error during a copy job (e.g., unreadable file), it might stop the entire process. Newer versions have improved this behavior, but TeraCopy's error handling is generally more detailed and user-friendly.
• Features:
  • TeraCopy: It provides additional features, like pause and resume file transfers, verify files after copying, and integrate with Windows Explorer's context menu for seamless usage.
  • Windows Explorer: While it provides a more straightforward and integrated experience, it might lack advanced features like those provided by TeraCopy.

One flavor of optimization that you don't mention is copying only files that have changed. That can be a huge saving in time right there. There are typically two ways that this can be implemented: by comparing source and destination files sizes and modification times, or, by looking to see if the Windows "modified" bit is set for the file. I don't know if things have improved over time but there was a point where I caught the modified bit not getting set for files I just modified, so I no longer trust it. I gather Robocopy uses this approach.

You also don't mention that the verification step, if necessary, can add significant time.

Some programs like FreeFileSync can copy files that are in use IF you run the program as Administrator. This is often the reason other programs will skip over files.

Thank you for the nice list all in all.

 

[Chris Noble]

What software do you use to do a mass copy of 1.81tb from an external 2tb SSD to an external 4tb SSD? Performance and data integrity are both important.

Drag and drop in Windows Explorer. I just moved about 250GB of files in a few minutes.

Faster even than posting a new discussion thread on DPReview and reading all the complicated responses.

 

[Henry Richardson]

The fastest mass file copying program for Windows can vary based on your specific needs and the nature of your file transfer tasks. Several tools are optimized for speed and reliability. Here are some of the most popular and widely recognized ones:

1. TeraCopy: This is one of the most popular file copy utilities for Windows. It's designed to copy and move files at the maximum possible speed, offers robust error recovery, and integrates seamlessly with Windows Explorer.
2. FastCopy: This is another tool that claims to be the fastest copy/delete software on Windows. It supports different modes of file copying, like differential copy (copying only different or non-existing files). It's highly configurable and can achieve impressive speeds.
3. Robocopy (Robust File Copy): This is a command-line tool that's been built into Windows for several years. It's highly robust, offering a slew of features and options. It's particularly powerful for network transfers and offers multi-threaded copying since Windows 7.
4. UltraCopier: This is an open-source tool designed to enable faster and more reliable file copying. It provides a user-friendly interface, error management, and customizable skins.
5. RichCopy: Originally developed by a Microsoft engineer, this tool offers a GUI and supports multi-threaded file copying, which can significantly speed up file transfers.
6. XXCOPY: It's a command-line tool that extends the capabilities of XCOPY, a built-in Windows tool. It's versatile and powerful but has a steeper learning curve due to its command-line nature.
7. Copy Handler: This tool offers high customization, pause/resume functionality, and integrates with the Windows shell. It's also optimized for speed.

When determining the "fastest" tool, consider the following factors:

• Nature of the Transfer: The speed can vary based on whether you're transferring many small files or a few large ones, or if you're copying data to/from SSDs, HDDs, or over a network.
• User Interface vs. Command-Line: GUI tools might be more user-friendly, but command-line tools can be more powerful and flexible, especially for batch operations or scripting.
• Additional Features: Some tools provide extra features like error recovery, verification, or the ability to pause and resume transfers.
• System and Software Compatibility: Ensure the tool is compatible with your version of Windows and doesn't conflict with other software.

I personally use TerraCopy for pure speed with bulk file transfers:

• Optimizations:
  • TeraCopy: It's designed to copy and move files at the maximum possible speed. It uses dynamically adjusted buffers to reduce the seek times. Also, asynchronous copy speeds up file transfer between two physical hard drives.
  • Windows Explorer: The built-in file copy feature in Windows has improved over various versions of the OS, but it might not be as optimized for speed as TeraCopy in certain scenarios.
• Error Handling:
  • TeraCopy: One of TeraCopy's main features is its robust error recovery. If a file transfer fails, it will often retry multiple times. If it's unable to read a file after several attempts, it will skip that file and continue with the rest. At the end of the copy job, it provides a report detailing any problematic files.
  • Windows Explorer: In older versions of Windows, if Explorer encountered an error during a copy job (e.g., unreadable file), it might stop the entire process. Newer versions have improved this behavior, but TeraCopy's error handling is generally more detailed and user-friendly.
• Features:
  • TeraCopy: It provides additional features, like pause and resume file transfers, verify files after copying, and integrate with Windows Explorer's context menu for seamless usage.
  • Windows Explorer: While it provides a more straightforward and integrated experience, it might lack advanced features like those provided by TeraCopy.

Yes, I use teracopy and mentioned it in my post above:

https://www.dpreview.com/forums/post/67267849

 

[Henry Richardson]

Drag and drop in Windows Explorer. I just moved about 250GB of files in a few minutes.

Faster even than posting a new discussion thread on DPReview and reading all the complicated responses.

I have had a bad experience doing a mass copy using it. You missed a lot by posting without reading. Why the nastiness and snark? Especially when you are oh so wrong? See these:

https://www.dpreview.com/forums/post/67263814

https://www.dpreview.com/forums/post/67264133

Of course, if you are happy with it then all is good.

 

[Henry Richardson]

One flavor of optimization that you don't mention is copying only files that have changed. That can be a huge saving in time right there. There are typically two ways that this can be implemented: by comparing source and destination files sizes and modification times, or, by looking to see if the Windows "modified" bit is set for the file. I don't know if things have improved over time but there was a point where I caught the modified bit not getting set for files I just modified, so I no longer trust it. I gather Robocopy uses this approach.

For the initial mass copy I do not care about that, but afterwards I use Syncback also to copy any new/modified files over too.

You also don't mention that the verification step, if necessary, can add significant time.

I use verification, but it does make it a bit slower.

Some programs like FreeFileSync can copy files that are in use IF you run the program as Administrator. This is often the reason other programs will skip over files.

On my PCs I use Syncback, but on my Mac I use FreeFileSync.

Thank you for the nice list all in all.

 

[New Wrycuda]

One flavor of optimization that you don't mention is copying only files that have changed. That can be a huge saving in time right there. There are typically two ways that this can be implemented: by comparing source and destination files sizes and modification times, or, by looking to see if the Windows "modified" bit is set for the file. I don't know if things have improved over time but there was a point where I caught the modified bit not getting set for files I just modified, so I no longer trust it.

I gather Robocopy uses this approach.

It certainly does, and I have been advocating the use of Robocopy for “Incremental/Cumulative” backup for some years now. One caveat is that it’s necessary to include a command-line switch to handle timing issues…

Robocopy “e:\shared\current files” “f:\data\current files” /e /dcopy:t /mt:8

(/mt:8 is actually the default; I believe that there’s no point trying to use more than the available physical cores).

I have a couple of servers, each with a desktop shortcut that is labelled “Quick Backup”, and this adds new/changed files to external SSDs in a matter of seconds. The last time I set up a new server, I used the Robocopy version of the data, and this passed every test.

To placate the “Backup Obsessed”, I should add that I also backup by more traditional methods.

 

[Chris Noble]

Drag and drop in Windows Explorer. I just moved about 250GB of files in a few minutes.

Faster even than posting a new discussion thread on DPReview and reading all the complicated responses.

I have had a bad experience doing a mass copy using it. You missed a lot by posting without reading. Why the nastiness and snark? Especially when you are oh so wrong?

I did read the whole complicated thread; especially the one that suggested to you that a single bad experience many years ago may not be a reliable guide; especially when millions of people use Windows Explorer to do millions of file copies every day. Suggests that maybe something else was going on for you; and I posted in this thread and then forgot all about it.

But I happened to do a big transfer today and remembered this old thread! Which I noticed is still getting complicated suggestions to address this very simple task...

And just my commenting that you got many complicated responses (which is true) is not "nasty and snarky". I think you are projecting your own feelings of frustration.

Good luck with the file copying! It's not complicated. Try Windows Explorer. Or don't.

 

[Henry Richardson]

Drag and drop in Windows Explorer. I just moved about 250GB of files in a few minutes.

Faster even than posting a new discussion thread on DPReview and reading all the complicated responses.

I have had a bad experience doing a mass copy using it. You missed a lot by posting without reading. Why the nastiness and snark? Especially when you are oh so wrong?

I did read the whole complicated thread; especially the one that suggested to you that a single bad experience many years ago may not be a reliable guide; especially when millions of people use Windows Explorer to do millions of file copies every day. Suggests that maybe something else was going on for you; and I posted in this thread and then forgot all about it.

But I happened to do a big transfer today and remembered this old thread! Which I noticed is still getting complicated suggestions to address this very simple task...

And just my commenting that you got many complicated responses (which is true) is not "nasty and snarky". I think you are projecting your own feelings of frustration.

Good luck with the file copying! It's not complicated. Try Windows Explorer. Or don't.

Well, you obviously did not read these:

https://www.dpreview.com/forums/post/67257643

https://www.dpreview.com/forums/post/67258011

There was one reason only for the very slow copy and Robert hit on it.

And you may have missed this post for you from earlier:

https://www.dpreview.com/forums/post/67287229

 

[Michael Firstlight]

Thanks Michael. A useful summary. My main problem right now is remembering what I've backed up and where!

Thank you! Yes! You are correct. I only addressed bulk file needs. I personally also use FreeFileSync aside from Terracopy - they make a great pair.

Of course neither fully satisfy a disaster recovery strategy. Along with these I still use Acronis and create both incremental and full back-ups and keep the full back-ups off-site. Those backup's have saved my bacon in the past.

Regards,
Michael

 

[Chris Dubea]

Thanks all for a great discussion!

Learned about some products I wasn't aware of.

My first response to the OP would have been to use robocopy. I've used it for years.

Now I have other options!

Awesome