Software for mass copy to external SSD

Carey Brown said:

I'm guessing that a transfer from one external drive to another involves two steps: transfer from 'A' to memory, then transfer from memory to 'B'. This, in theory , would result in half the anticipated transfer rate.

Click to expand...

A program can copy at full speed (i.e, the speed of the slower of the two drives) even though a transfer requires reading from one drive into memory and then writing to the other. It does this by using double buffering. Here's how it works:

• Read chunk 1 from drive 1
• Write chunk 1 to drive 2 while reading chunk 2 from drive 1
• Write chunk 2 to drive 2 while reading chunk 3 from drive 1
• Write chunk 3 to drive 2 while reading chunk 4 from drive 1

...and so on.

So you're keeping both drives busy at the same time.

TomFid said:

Chris Noble said:

Henry Richardson said:

What software do you use to do a mass copy of 1.81tb from an external 2tb SSD to an external 4tb SSD? Performance and data integrity are both important.

Click to expand...

Keep it simple.

Click to expand...

This certainly works.

For extra security afterwards, you can use a file sync tool like FreeFileSync or a diff tool like Kdiff to verify that everything made it.

Click to expand...

Every file copy function, including the basic WIndows Explorer, does that.

All those other packages that posters on this thread are suggesting do additional functions, that the OP is not looking for. Some of them also change the structure of the target drive, which he is also not looking to do.

Windows 11

SC489 said:

I would just copy using the standard Explorer

Click to expand...

Years ago when I had smaller external HDDs (120gb, 160gb, 500gb, etc.) I would do that to copy everything to a new, larger one. Then one time after tens of thousands of files, about half the total copy in GB, there was some sort of error glitch and it stopped. I could find no way to restart it so that it would continue where it left off. When I ran it again it tried to copy all the earlier files again. This was long ago, maybe 15 or so years ago. My recollection is fuzzy, but I think it got errors because all the earlier copied files were already there. I think what I finally ended up doing was deleting everything on the new drive (many GB of previously copied files) by doing a quick format and then using File Explorer again to copy everything. The second time, fortunately, did not get an error glitch and it finished.

Are you saying it can now restart where it left off so that using File Explorer to copy 1.81tb taking many, many hours is now safe?

This problem is why I started using Syncback to do this many years ago. It easily continues where it left off whenever it stops for any reason, including me stopping it. Also, has several performance and verification options that one can use.

TorsteinH said:

Windows 11

Click to expand...

.. sorts of other things won't work.

;-)

David Lal said:

gscotten said:

Macrium Free lets you set the target partition size when cloning.

Click to expand...

Does it George? That's good news. I am sure the last time I did a clone with Macrium I was not able to do that. It's either new (and welcome) functionality or something I missed.

Click to expand...

Here is the documentation. Cloning a disk - Knowledgebase 8.0 - Macrium Reflect Knowledgebase

Check out steps 4 and 5. "Fill Space" or click the wrench to set your size. Or right click the partition and choose "Partition Size and Layout".

It has been that way for as long as I can remember.

New Wrycuda said:

Dare I suggest Robocopy?

Click to expand...

Absolutely Robocopy, and you already have it in Windows. I use it often at work.

Or Free File Sync if you want to download something with a GUI and use it for other things too. I use it at home for data backups.

Eric Carlson said:

New Wrycuda said:

Dare I suggest Robocopy?

Click to expand...

Absolutely Robocopy, and you already have it in Windows. I use it often at work.

Click to expand...

It seems that many users are unaware of the following feature of Robocopy as previously mentioned…

I’m not sure if the “multi-threaded” option would work for simple copying, but it might be worth trying (/mt:4 in your case). It certainly accelerates the backup process that I use to add new or changed files to the target. (If you run a Robocopy script on subsequent occasions, it skips existing files and only copies new stuff).

I have to admit that the first time I tried this, the process was over so soon that I thought something had gone wrong.

I’ve been using it this way for several years. Clicking a Desktop shortcut saves recent work in a matter of seconds. Obviously, I also use conventional backup, but Robocopy is a good safety net for vital work. A handy safety feature is that any files that may have been deleted from the working folders are retained by Robocopy.

gscotten said:

David Lal said:

gscotten said:

Macrium Free lets you set the target partition size when cloning.

Click to expand...

Does it George? That's good news. I am sure the last time I did a clone with Macrium I was not able to do that. It's either new (and welcome) functionality or something I missed.

Click to expand...

Here is the documentation. Cloning a disk - Knowledgebase 8.0 - Macrium Reflect Knowledgebase

Check out steps 4 and 5. "Fill Space" or click the wrench to set your size. Or right click the partition and choose "Partition Size and Layout".

It has been that way for as long as I can remember.

Click to expand...

In that case, it is just something I missed :-(

The copy finished during the night so all seems well now. I am using the new 4tb SSD now.

I went back to using the free Syncback once the suggestion in this thread to go into the Device Manager and enable write caching (which is turned off by default). I saw that it was finally running at a very good speed. Syncback is just as fast as robocopy and File Explorer and it has a nice GUI with many options. I have been using it for many years to backup files to external drives.

I usually use the verify option in Syncback, but I had turned it off temporarily to try to speed things up earlier when the copy was so slow. After I saw it was now running well with the Device Manager adjustment I turned it back on. The verify option did not slow it down much. With verify it reads the destination file it just wrote and compares it to the source file. Of course, there is no guarantee because it is possible that the read data is actually data that has been cached in OS memory. But, it is better than nothing.

So, this is what I did to try to get a good speed:

Windows:

set sleep to never so it can run for hours
Windows Defender: Real-time protection turned off
disable internet so no activity (downloads and install of Windows updates, etc.)

4tb SSD:

Device Manager: drive properties, policies, better performance (enable write caching)

Syncback:

set option in copy/delete, advanced, turn off make safe copies which gives better performance

Henry Richardson said:

SC489 said:

I would just copy using the standard Explorer

Click to expand...

Years ago when I had smaller external HDDs (120gb, 160gb, 500gb, etc.) I would do that to copy everything to a new, larger one. Then one time after tens of thousands of files, about half the total copy in GB, there was some sort of error glitch and it stopped. I could find no way to restart it so that it would continue where it left off. When I ran it again it tried to copy all the earlier files again. This was long ago, maybe 15 or so years ago. My recollection is fuzzy, but I think it got errors because all the earlier copied files were already there. I think what I finally ended up doing was deleting everything on the new drive (many GB of previously copied files) by doing a quick format and then using File Explorer again to copy everything. The second time, fortunately, did not get an error glitch and it finished.

Are you saying it can now restart where it left off so that using File Explorer to copy 1.81tb taking many, many hours is now safe?

This problem is why I started using Syncback to do this many years ago. It easily continues where it left off whenever it stops for any reason, including me stopping it. Also, has several performance and verification options that one can use.

Click to expand...

Years ago is over. Any issues during a copy using explorer will result in an appropriate set of choices. You will be able to complete the copy and will know what is going on in the unlikely event of an issue.

In the time that you have spent with this thread you could have done the copy hundreds of times. Now this is bad use of your time so pore efficiency.

Morris

Henry Richardson said:

Robert Zanatta said:

Henry Richardson said:

TomFid said:

Henry Richardson said:

How long would you expect the full copy of 1.81tb to take? I will use this laptop with nothing else running during the copy, internet disabled, and Windows Defender Real-time protection turned off:

Dell laptop, Windows 10 22H2
i7-8565U, 4 cores
32gb RAM

Of course, I know you cannot say exactly. But do you think a single digit number of hours? Many hours? Days?

Click to expand...

I second the recommendation for robocopy, particularly with the /v flag for verification. The verify step roughly doubles time though. It can take a little head-scratching to get a well-tuned command line though. There's a GUI for it, but I remember bouncing off that for some forgotten reason.

Click to expand...

I started robocopy last night using this command line:

robocopy g: h: /e /zb

Click to expand...

For the new drive, go into it's properties and enable write caching. Also disable 8.3 file name creation of you don't need short file names.

Click to expand...

That is helping a lot. Thank you. Here is the Task Manager doing the copy now after doing that. Before doing it the graph at the bottom the label at the top right was just at 100Kb/s to 500Kb/s. In just 6 minutes it has copied 10gb. Many times more than hours and hours of copying before had managed.

Click to expand...

The Device Manager adjustment was the essential thing that changed the speed. Using robocopy, File Explorer, or Syncback were all the same speed and irrelevant.

This morning I found some old notes from when I did a copy to another SSD about 2 years ago and I found that I had written in the notes to be sure to make the change in Device Manager, but I had forgotten about it. Thank you to Robert for reminding me! Apparently, you and I are the only people here who know about this and its importance. As soon as I saw your suggestion I knew that was the problem.

New Wrycuda said:

Eric Carlson said:

New Wrycuda said:

Dare I suggest Robocopy?

Click to expand...

Absolutely Robocopy, and you already have it in Windows. I use it often at work.

Click to expand...

It seems that many users are unaware of the following feature of Robocopy as previously mentioned…

I’m not sure if the “multi-threaded” option would work for simple copying, but it might be worth trying (/mt:4 in your case). It certainly accelerates the backup process that I use to add new or changed files to the target. (If you run a Robocopy script on subsequent occasions, it skips existing files and only copies new stuff).

I have to admit that the first time I tried this, the process was over so soon that I thought something had gone wrong.

I’ve been using it this way for several years. Clicking a Desktop shortcut saves recent work in a matter of seconds. Obviously, I also use conventional backup, but Robocopy is a good safety net for vital work. A handy safety feature is that any files that may have been deleted from the working folders are retained by Robocopy.

Click to expand...

I've used much higher thread counts at work. The default is 8 if you use /MT with no number. I've used something like /MT:32 or 64 (don't remember exactly). 128 is the max.

--
- Eric, http://www.invisiblerobot.com/

Eric Carlson said:

New Wrycuda said:

Eric Carlson said:

New Wrycuda said:

Dare I suggest Robocopy?

Click to expand...

Absolutely Robocopy, and you already have it in Windows. I use it often at work.

Click to expand...

It seems that many users are unaware of the following feature of Robocopy as previously mentioned…

I’m not sure if the “multi-threaded” option would work for simple copying, but it might be worth trying (/mt:4 in your case). It certainly accelerates the backup process that I use to add new or changed files to the target. (If you run a Robocopy script on subsequent occasions, it skips existing files and only copies new stuff).

I have to admit that the first time I tried this, the process was over so soon that I thought something had gone wrong.

I’ve been using it this way for several years. Clicking a Desktop shortcut saves recent work in a matter of seconds. Obviously, I also use conventional backup, but Robocopy is a good safety net for vital work. A handy safety feature is that any files that may have been deleted from the working folders are retained by Robocopy.

Click to expand...

I've used much higher thread counts at work. The default is 8 if you use /MT with no number. I've used something like /MT:32 or 64 (don't remember exactly). 128 is the max.

Click to expand...

My suggested thread count was based on the OP hardware…

Eric Carlson said:

Eric Carlson said:

Dell laptop, Windows 10 22H2
i7-8565U, 4 cores
32gb RAM

Click to expand...

Click to expand...

With my 8 physical cores, I use /mt:8, (Yes, I know that’s the default), which I think is a reasonable choice; it’s certainly fast enough at that setting, with a “fast backup” taking just a couple of seconds to deal with a day’s work. My Office data is about 40Gb, while the 400Gb Photo Archive on a similar computer doesn’t take much longer to update with a day’s shooting.

Mentioning the often misunderstood subject of “File Attributes” usually clears up doubts about backup behaviour.

Morris0 said:

Henry Richardson said:

SC489 said:

I would just copy using the standard Explorer

Click to expand...

Years ago when I had smaller external HDDs (120gb, 160gb, 500gb, etc.) I would do that to copy everything to a new, larger one. Then one time after tens of thousands of files, about half the total copy in GB, there was some sort of error glitch and it stopped. I could find no way to restart it so that it would continue where it left off. When I ran it again it tried to copy all the earlier files again. This was long ago, maybe 15 or so years ago. My recollection is fuzzy, but I think it got errors because all the earlier copied files were already there. I think what I finally ended up doing was deleting everything on the new drive (many GB of previously copied files) by doing a quick format and then using File Explorer again to copy everything. The second time, fortunately, did not get an error glitch and it finished.

Are you saying it can now restart where it left off so that using File Explorer to copy 1.81tb taking many, many hours is now safe?

This problem is why I started using Syncback to do this many years ago. It easily continues where it left off whenever it stops for any reason, including me stopping it. Also, has several performance and verification options that one can use.

Click to expand...

Years ago is over. Any issues during a copy using explorer will result in an appropriate set of choices. You will be able to complete the copy and will know what is going on in the unlikely event of an issue.

In the time that you have spent with this thread you could have done the copy hundreds of times. Now this is bad use of your time so pore efficiency.

Morris

Click to expand...

Indeed. Very common on this forum to gravitate to debates about convoluted and complex suggestions to fix simple problems. I just moved about 200GB from one drive to another yesterday evening, in one operation using Explorer; in the background of course, while I was doing other things.

In all this confusion, I have lost track. I am wondering, How long did your copy take and how many TBs were copied? Thanks and glad it has worked.

Chris Noble said:

Very common on this forum to gravitate to debates about convoluted and complex suggestions to fix simple problems. I just moved about 200GB from one drive to another yesterday evening, in one operation using Explorer; in the background of course, while I was doing other things.

Click to expand...

The biggest drawback to using Explorer, IMHO, is that the new copies of the files that it creates get the current date/time as their "creation" date. By default Robocopy preserves the original creation and modification dates on the new copies of the files.

That may or may not matter to someone, but I think it's something to be aware of when deciding which tool to use.

Sean Nelson said:

Chris Noble said:

Very common on this forum to gravitate to debates about convoluted and complex suggestions to fix simple problems. I just moved about 200GB from one drive to another yesterday evening, in one operation using Explorer; in the background of course, while I was doing other things.

Click to expand...

The biggest drawback to using Explorer, IMHO, is that the new copies of the files that it creates get the current date/time as their "creation" date.

Click to expand...

Actually, that's one of the benefits of Explorer, and done for a reason. See below.

Sean Nelson said:

By default Robocopy preserves the original creation and modification dates on the new copies of the files.

That may or may not matter to someone, but I think it's something to be aware of when deciding which tool to use.

Click to expand...

True enough, but often dangerous to rely on the creation date, since that applies to the file wrapper and not the contents. The most relevant and useful date is the Modified Date, which applies to the contents. Explorer does not change that date, which is why the Modified Date is often earlier than the Created Date in systems where files are moved around a lot. The Modified Date will tell you the last time the file contents were changed, and is the one that programs look at.

Updating the Creation Date is actually recommended, as it tells you the last time the file was moved.

Chris Noble said:

TomFid said:

Chris Noble said:

Henry Richardson said:

What software do you use to do a mass copy of 1.81tb from an external 2tb SSD to an external 4tb SSD? Performance and data integrity are both important.

Click to expand...

Keep it simple.

Click to expand...

This certainly works.

For extra security afterwards, you can use a file sync tool like FreeFileSync or a diff tool like Kdiff to verify that everything made it.

Click to expand...

Every file copy function, including the basic WIndows Explorer, does that.

Click to expand...

Win Explorer verifies that it finished the for-loop over the files, but it doesn't verify file content (neither does robocopy for that matter, unless /v is used).

In theory, explorer is fine. In practice, stuff happens. I prefer to trust, but verify.

TomFid said:

Chris Noble said:

TomFid said:

Chris Noble said:

Henry Richardson said:

What software do you use to do a mass copy of 1.81tb from an external 2tb SSD to an external 4tb SSD? Performance and data integrity are both important.

Click to expand...

Keep it simple.

Click to expand...

This certainly works.

For extra security afterwards, you can use a file sync tool like FreeFileSync or a diff tool like Kdiff to verify that everything made it.

Click to expand...

Every file copy function, including the basic WIndows Explorer, does that.

Click to expand...

Win Explorer verifies that it finished the for-loop over the files, but it doesn't verify file content (neither does robocopy for that matter, unless /v is used).

Click to expand...

Neither do the myriad of programs that run on your computer and that are constantly copying and moving files -- your browser cache for example.

TomFid said:

In theory, explorer is fine. In practice, stuff happens. I prefer to trust, but verify.

Click to expand...

You are worrying about a zero-risk problem in practice, and you can't verify 95% of the file movements that are happening all the time and that you are not aware of.

And, by the way, Write file integrity is executed by a Read-after-Write function on each block transfer.

But go ahead, run those additional verification programs. Can't hurt, and makes the time go by for you.