Reading mode: Light Dark

Longterm storage of backup HDD : Waterproof container?

Go to last post
S

Sebastian Cohen

Forum Enthusiast
Messages
255
Reaction score
141
So, I am doing an archival backup of EVERYTHING. This will be the "Doomsday Vault Backup" and it won't get backed up further or connected. It is going on a 3.5" HDD, no case. FYI, I will still be doing my regular backup routine with my other drives as normal.

Lately I have been thinking about how I store my backup drives, in general. I know they are airtight so air moisture variations shouldn't really be an issue. Problem is, they usually end up all over the place. Drawers, bottom of, cardboard boxes, usually bottom of.

So I've been thinking of sticking them into one of those plastic IKEA airtight refrigerator boxes. The type has this rubber seal which would also make them waterproof. It is an added protection AND it will be easier to find. I can also put in a note with what/where etc so I don't have to connect it if I don't remember. Sounds like I have a ton of discs, but you would be surprised how confused you can get with just a couple when you find one 4 years later.

SO, long question short. I will be creating a permanent "environment" in that box and I am wondering if or how that can be negative for the drive when the outside fluxtuates?

I am planning on throwing in one or two of those silica gel pouches as well, should negate any negative variations?

This will prob just lie there for 3-4 years.

I am aware of that the lubrication might dry out in the bearings, but this has not been an issue so far.
 
Chris Noble said:
OK, I give up, the chkdsk documentation clearly states what it does and which types of errors it corrects (not just "reports"). You seem to want to argue for argument's sake rather than reading the documentation on this useful utility.
Click to expand...
oh, the irony.

you're wrong in about 7 different ways here, but if you think an antiquated tool from the DOS era will protect your data, it's your funeral.
 
kelpdiver said:
Chris Noble said:
OK, I give up, the chkdsk documentation clearly states what it does and which types of errors it corrects (not just "reports"). You seem to want to argue for argument's sake rather than reading the documentation on this useful utility.
Click to expand...
oh, the irony.

you're wrong in about 7 different ways here, but if you think an antiquated tool from the DOS era will protect your data, it's your funeral.
Click to expand...
Kelp, you get the prize.
 
Chris Noble said:
Sean Nelson said:
Chkdsk checks for the consistency of file system metadata, but it doesn't care about what's inside data files, apart from the fact that if you use "/r" then it makes sure the disk reports that it can read them.
Click to expand...
/r means repair, not report.

OK, I give up, the chkdsk documentation clearly states what it does and which types of errors it corrects (not just "reports"). You seem to want to argue for argument's sake rather than reading the documentation on this useful utility.
Click to expand...
From the Chkdsk page at Microsoft.com (bolding is mine):
Chris Noble said:
/f Fixes errors on the disk. The disk must be locked. If chkdsk cannot lock the drive, a message appears that asks you if you want to check the drive the next time you restart the computer.

/r Locates bad sectors and recovers readable information. The disk must be locked. /r includes the functionality of /f, with the additional analysis of physical disk errors.
Click to expand...
The basic purpose of ChkDsk is to analyze the file system to ensure its consistency. For example, it checks that every cluster on the disk is either allocated to a file, to file metadata, or appears in the list of available clusters.

/f will "fix" any such error it finds. For example if it finds clusters that aren't allocated to files and don't appear on the list of available clusters, it will create dummy files that contain those clusters so that you can access the data and do something with it. I assume this is what you're referring to when you say "repair".

/r reads all the sectors on the drive in order to see if the drive reports that any of them are unreadable. It also includes the "/f" option, so in that sense it does do "repairs", but it's essential purpose over and above /f is to check all the blocks on the drive.

Here's the simple version of why I use checksumming:

Checksums will tell me if the data inside any files has changed.

Chkdsk will not do that.
 
Sean Nelson said:
From the Chkdsk page at Microsoft.com (bolding is mine):

/f Fixes errors on the disk.

/r Locates bad sectors and recovers readable information. /r includes the functionality of /f.

Checksums will tell me if the data inside any files has changed.
Click to expand...
Checksum gives you no information as to the source of the error, nor does it fix the error (if the error is fixable). Your only recourse is to recopy the file, which may not fix what caused the problem. And the original file may no longer be available (if you time-stamp your backups, which I hope you do). And how do you run this check on all the files in your backup system in an automated way?

To verify the integrity of a backup system, you need to be able to distinguish between HDD sector-level errors (what Chkdsk does), file system errors (what Scan Drive does), and file content errors (which depend on how each program stores data in its own file type). Each of the three error types is possible. Checksum does none of that.

I'll continue using Chkdsk and Extended SMART to periodically verify the integrity of my backup system, and verify-after-write on my backup software to verify the file data itself. You continue relying on checksum.
 
Last edited:
Chris Noble said:
Sean Nelson said:
From the Chkdsk page at Microsoft.com (bolding is mine):

/f Fixes errors on the disk.

/r Locates bad sectors and recovers readable information. /r includes the functionality of /f.

Checksums will tell me if the data inside any files has changed.
Click to expand...
Checksum gives you no information as to the source of the error, nor does it fix the error (if the error is fixable).
Click to expand...
Yes, as I noted above.

The purpose of checksum is to regularly confirm that my backups are still intact. No matter what the cause. It's not to fix the problem, it's to discover that a problem exists.

I don't know about you, but if I discover that my backups are no longer intact then I will replace the drive before it comes to the point where I actually need to restore from those backups.
 
Last edited:
Sean Nelson said:
The purpose of checksum is to regularly confirm that my backups are still intact. No matter what the cause. It's not to fix the problem, it's to discover that a problem exists.
Click to expand...
How and how often do you do that? At any one time I probably have a dozen backup files in the backup set, with new incremental ones being created every night and older ones being consolidated or flushed. How do you update your checksum-checking system across all the backup files? Automation and background activity are important to me.
Sean Nelson said:
I don't know about you, but if I discover that my backups are no longer intact then I will replace the drive before it comes to the point where I actually need to restore from those backups.
Click to expand...
I don't know about you, but I don't want to wait until I have a checksum error before taking action. That's a pretty serious problem. Disks are often already degrading before checksum errors show up. I prefer to avoid problems rather than having the problem occur, not knowing exactly what was the cause, and hoping that a disk swap and rebuild has solved it.
 
Last edited:
Chris Noble said:
Kelp, you get the prize.
Click to expand...
of course I do. Site reliability is a big part of how I make a living.

You have a pet rock solution that hasn't had to face an actual test. I have millions of customers. (Don't house them on ntfs windows systems etiher, FTR)
 
Chris Noble said:
Sean Nelson said:
The purpose of checksum is to regularly confirm that my backups are still intact. No matter what the cause. It's not to fix the problem, it's to discover that a problem exists.
Click to expand...
How and how often do you do that? At any one time I probably have a dozen backup files in the backup set, with new incremental ones being created every night and older ones being consolidated or flushed. How do you update your checksum-checking system across all the backup files? Automation and background activity are important to me.
Click to expand...
I only do this for my full backups which are done weekly and monthly. Daily incremental backups are done on alternating USB flash drives. If I loose one of these I don't consider it to be catastrophic.

The backup utility I wrote to do my file-based backups generates a checksum as it writes the backup set and writes it to a separate ".MD5" file when the backup completes, no separate pass is needed. So there's no real overhead to creating the initial checksums.

My backup script automatically deletes the oldest backups (including the saveset file, log file and associated ".MD5" file) as needed to make space for the new one, runs the backup, and then runs the checksum utility to verify all the ".MD5" files on the drive to make sure that both the newest backup and all the prior backups are still correct.

So the work involved is inserting a disk into my USB drive dock once a week, clicking the script link, and then later when the backup is finished checking the output window to see that no errors were found.
Chris Noble said:
I don't know about you, but I don't want to wait until I have a checksum error before taking action. That's a pretty serious problem.
Click to expand...
The strategy is: use every available means to detect problems. Checksum verification is another method that has the potential to detect problems that other strategies miss. It's essentially testing that your backup can be restored without actually restoring it.
 
Last edited:
Sean Nelson said:
Chris Noble said:
Sean Nelson said:
The purpose of checksum is to regularly confirm that my backups are still intact. No matter what the cause. It's not to fix the problem, it's to discover that a problem exists.
Click to expand...
How and how often do you do that? At any one time I probably have a dozen backup files in the backup set, with new incremental ones being created every night and older ones being consolidated or flushed. How do you update your checksum-checking system across all the backup files? Automation and background activity are important to me.
Click to expand...
I only do this for my full backups which are done weekly and monthly. Daily incremental backups are done on alternating USB flash drives. If I loose one of these I don't consider it to be catastrophic.
Click to expand...
I'd rather not lose any of my data.
Sean Nelson said:
The backup utility I wrote to do my file-based backups...
Click to expand...
I'd rather use a commercial solution.
Sean Nelson said:
generates a checksum as it writes the backup set and writes it to a separate ".MD5" file when the backup completes, no separate pass is needed. So there's no real overhead to creating the initial checksums.

My backup script automatically deletes the oldest backups (including the saveset file, log file and associated ".MD5" file) as needed to make space for the new one, runs the backup, and then runs the checksum utility to verify all the ".MD5" files on the drive to make sure that both the newest backup and all the prior backups are still correct.

So the work involved is inserting a disk into my USB drive dock once a week, clicking the script link, and then later when the backup is finished checking the output window to see that no errors were found.
Click to expand...
Way too complicated and unreliable for me. To each their own.
Sean Nelson said:
Sean Nelson said:
I don't know about you, but I don't want to wait until I have a checksum error before taking action. That's a pretty serious problem.
Click to expand...
The strategy is: use every available means to detect problems.
Click to expand...
"Every available means" is actually not a good strategy. Just enough complementary means, with redundancy, that detect degradation before failure and don't require restores every time there is a fault is a better strategy.
Sean Nelson said:
Checksum verification is another method that has the potential to detect problems that other strategies miss. It's essentially testing that your backup can be restored without actually restoring it.
Click to expand...
I'd rather fix problems earlier rather than rely solely on restoring backups after the problem (without a diagnosis) has declared itself. Your system seems to satisfy you, mine works well for me, we can leave it at that.
 
Last edited:
Sean Nelson said:
CAcreeks wrote:
Many backup and sync programs automatically verify checksums.
Macrium does as well, I believe by default.
Click to expand...
You have to be careful, because sometimes what seems like a verify actually isn't.

I run my image backups using Macrium and I always use the verify option - but I have 64GB of RAM and with the backup image files usually running in the 50+GB range I've found that Macrium doesn't bypass the RAM cache when it re-reads the backup to verify it.

My backup typically takes about 12 or 13 minutes to run, and the subsequent verify should take the same ballpark time, but it often completes in just a couple of minutes without any disk activity going on. Macrium is happily reporting that the file has been successfully verified ...
Click to expand...
Oh oh!

I wonder what happens with 16GB, which is what we have on all our Windows machines.

Thanks for the warning. I will check duration of Macrium verify and get back to this thread, if it doesn't fill up beforehand.

(Waterproof container, huh?)
 
Last edited:
CAcreeks said:
Sean Nelson said:
CAcreeks said:
Many backup and sync programs automatically verify checksums.

Macrium does as well, I believe by default.
Click to expand...
You have to be careful, because sometimes what seems like a verify actually isn't.

I run my image backups using Macrium and I always use the verify option - but I have 64GB of RAM and with the backup image files usually running in the 50+GB range I've found that Macrium doesn't bypass the RAM cache when it re-reads the backup to verify it.

My backup typically takes about 12 or 13 minutes to run, and the subsequent verify should take the same ballpark time, but it often completes in just a couple of minutes without any disk activity going on. Macrium is happily reporting that the file has been successfully verified ...
Click to expand...
Oh oh!

I wonder what happens with 16GB, which is what we have on all our Windows machines.

Thanks for the warning. I will check duration of Macrium verify and get back to this thread, if it doesn't fill up beforehand.
Click to expand...
I thought that the backup file was closed at the end of the backup operation, and re-opened immediately to do the Auto Verify. That would flush the RAM cache, wouldn't it? Also, I thought that RAM cache was for multiple reads, not read-after-write. But I'm not an expert on this.
 
Chris Noble said:
I thought that the backup file was closed at the end of the backup operation, and re-opened immediately to do the Auto Verify. That would flush the RAM cache, wouldn't it? Also, I thought that RAM cache was for multiple reads, not read-after-write. But I'm not an expert on this.
Click to expand...
Windows caches both file reads and file writes. Since it's not uncommon for programs to read data back that has recently been written, data written to the disk is retained in cache so that a program which then tries to read it doesn't have to wait for the disk.

Programs can use the options in the Windows File I/O API to bypass file caching, but the fact that Macrium shows a verify progress bar for me that merrily zaps through the entire backup set in way too short a time, coupled with the fact that there's no disk activity going on while this is happening, tells me that they're not using those options.

The fact that Windows caches disk writes (i.e., it doesn't do them immediately but waits for an opportune time) means that there's a window of vulnerability where a memory error could corrupt data before its written to the disk. These kinds of errors are very hard to detect without ECC memory because from the drive's point of view everything is just fine. This is one of the reasons why (a) I use ECC memory, and (b) why I always verify my backups.

If you've only got 16GB of memory it's likely that your backup images will be a fair bit larger than that, so the cached data written at the start of the image will have to be flushed before you get to the end of the backup.
 
Last edited:
Chris Noble said:
Sean Nelson said:
Robert Zanatta said:
By default, Windows enabled write caching on internal drives and disables it for removable drives.
Click to expand...
I believe that write caching is enabled by default for external drives that are formatted with NTFS.
Click to expand...
But presumably, writes to a NAS (however it's formatted) would not be cached?
Click to expand...
That I cannot say.

Try copying a large file to the NAS while nothing else is going on and monitoring disk activity - if the disk activity continues after the copy completes then it's being write cached.

Or alternatively, try copying a large file from the NAS, and then try copying it a second time. If there's no disk activity the second time, then it's being read cached.
 
Sean Nelson said:
Chris Noble said:
Sean Nelson said:
Robert Zanatta said:
By default, Windows enabled write caching on internal drives and disables it for removable drives.
Click to expand...
I believe that write caching is enabled by default for external drives that are formatted with NTFS.
Click to expand...
But presumably, writes to a NAS (however it's formatted) would not be cached?
Click to expand...
That I cannot say.

Try copying a large file to the NAS while nothing else is going on and monitoring disk activity - if the disk activity continues after the copy completes then it's being write cached.

Or alternatively, try copying a large file from the NAS, and then try copying it a second time. If there's no disk activity the second time, then it's being read cached.
Click to expand...
That's a bit unreliable because the cache flushing priority depends on how many other processes are active at the same time and on how much demand there is for RAM from other processes.
 
Chris Noble said:
Sean Nelson said:
Try copying a large file to the NAS while nothing else is going on...
Click to expand...
That's a bit unreliable because the cache flushing priority depends on how many other processes are active at the same time and on how much demand there is for RAM from other processes.
Click to expand...
Yeah, if you can't do it on a quiesced system then it's harder to tell, but unless the system is quite busy I'd expect that you could suss it out with a few attempts.

But perhaps the better method is to go to Device Manager, navigate to the "Disk Drives" node, right click on the specific drive in that node and select "properties". Then click the "Policies" tab of the resulting dialogue box to see whether write caching is enabled.
Robert Zanatta said:
I just did an installation of Windows and the transfer to the external USB drive was slow. The policy for the drive was set to quick removal. I changed it to better performance and data transfer to the drive improved significantly.
Click to expand...
The worst performance difference comes if you're copying over lots of small files because "quick removal" forces the system to update the file system metadata immediately to ensure it's as consistent as possible even if you yank the drive at some random time.

It's worse on NTFS drives because the file system is journalled and every change has to update not just the metadata itself but also the transaction log. I believe this is why USB drives with NTFS file systems get mounted with caching enabled by default.
 
As I said, my drive was mounted with caching disabled, which is what the quick removal setting does. Microsoft changed this in Windows 10.

docs.microsoft.com

Windows default media removal policy

Manage default media removal policy in Windows.
docs.microsoft.com
 
You must log in or register to reply here.

Similar threads

R
  • User review User review
From Synology to UGREEN: My Journey to a Smarter, Faster NAS with OMV7
Replies
3
Views
10K
New Wrycuda
N
JRET
Storage need for speed
Replies
13
Views
3K
joger
joger
Satyaa
External HDD recommendation
2
Replies
24
Views
536
Satyaa
Satyaa
M
DIY I made a dry cabinet |Minimalist design wooden
Replies
0
Views
2K
MrHNH
M
R
  • Question Question
Solution for alternating backup disks?
Replies
12
Views
13
Richard Pugh
R

Keyboard shortcuts

f
Forum
m
My threads
Mobile site
About
Editorial content
Cameras & Lenses
Community
All content, design, and layout are Copyright © 1998–2025 Digital Photography Review All Rights Reserved.
Reproduction in whole or part in any form or medium without specific written permission is prohibited.
When you use DPReview links to buy products, the site may earn a commission.
©GPS Media - Guides, Products, Services.
Back
Top