Recommendations for long term storage: RAID or ??

Started 3 months ago | Discussions
gilliano Regular Member • Posts: 106
Recommendations for long term storage: RAID or ??

So I have reached well over 1TB of photos/ video. Although I have a fairly robust multi-copy backup system in place I do not want to use a dead end like an external HD for any kind of long term storage (long time field researcher and data manager, seen way too many people do that and lose everything). My internet connection is slow and not unlimited and I am aware of the ways cloud systems can go away/ go out of business so cloud back up is not my preference. I'm looking at RAIDs but not really sure what is practical / affordable for a small operation like mine. Any specific recommendations? Anything to consider? Alternatives? Assume I know nothing about them, which is basically where I'm at.

Also atm I am limited to USB 2.0 on my current computer but in the next year or so will upgrade to a new computer with 3.0 or faster file share capability. Not sure if I should wait or if most systems can handle both types of connection?

 gilliano's gear list:gilliano's gear list
Canon EOS M5
Billiam29 Senior Member • Posts: 1,726
Re: Recommendations for long term storage: RAID or ??

Perhaps you could clarify your concerns a bit further. You mention that you “…do not want to use a dead end like and external HD for any kind of long term storage” but your inquiry is about RAID systems which are composed of exactly that, external hard drives.

OP gilliano Regular Member • Posts: 106
Re: Recommendations for long term storage: RAID or ??

Billiam29 wrote:

Perhaps you could clarify your concerns a bit further. You mention that you “…do not want to use a dead end like and external HD for any kind of long term storage” but your inquiry is about RAID systems which are composed of exactly that, external hard drives.

I'm concerned about a single external hard drive failing, particularly with no warning. As we all should be! It does happen fairly often.

A RAID is an array of hard drives with automatic multiple backups and the ones I'm familiar with have built in checks and warnings of impending failure for each individual drive in the array. It's a much more robust storage and backup system. However I know nothing about smaller affordable at-home RAIDs that I've found online, only big commercial types and so I'd like some input from those who are on good buys, personal experience, practical tips etc.

 gilliano's gear list:gilliano's gear list
Canon EOS M5
Billiam29 Senior Member • Posts: 1,726
Re: Recommendations for long term storage: RAID or ??
2

Sorry to drag this out, but when you used the phrase “long term storage” I thought you were referring to some sort of archival purpose where the storage could be set on a shelf somewhere (literally or figuratively). Are you just referring to storage that you’re not actively working or editing from of but is still attached to your PC and online?

gilliano wrote:

A RAID is an array of hard drives with automatic multiple backups and the ones I'm familiar with have built in checks and warnings of impending failure for each individual drive in the array. It's a much more robust storage and backup system. However I know nothing about smaller affordable at-home RAIDs that I've found online, only big commercial types and so I'd like some input from those who are on good buys, personal experience, practical tips etc.

No RAID configurations offer backups. Full stop. RAID 1 configurations offer instantaneous copies of your data on two hard drives so that you can keep working if one of them fails. Any accidents on your part, file corruptions, malware infections, etc… are copied perfectly between the two hard drives. No RAID configuration of any kind offers multiple automatic backups. There may be some external disk systems that do things analogous to this but that’s a function of software running on those systems, not a RAID configuration they are using.

Sean Nelson
Sean Nelson Forum Pro • Posts: 13,802
Re: Recommendations for long term storage: RAID or ??
1

gilliano wrote:

...I'm looking at RAIDs but not really sure what is practical / affordable for a small operation like mine. Any specific recommendations? Anything to consider? Alternatives? Assume I know nothing about them, which is basically where I'm at.

It seems to me that you're confusing a need for bulk storage to hold all your data with the need to back it up to make sure it doesn't get lost. Those are two different requirements that need two different solutions.

As far as storage goes, 1TB is trivial. My recommendation would be to simply get a bigger hard drive - they're cheap, fast enough for most bulk storage needs, and a lot simpler to manage than a RAID configuration. To be safe RAID requires that you test and document recovery procedures and deal with the vagaries of the software, firmware or hardware you choose for your RAID implementation.

As far as backup goes, Billiam29 is exactly right when he says that RAID is NOT a backup. The only sure backup is a copy you make and then remove from your system so that it's not vulnerable to accidental deletion, corruption, virus attack, or theft or damage that happens to the rest of the system. And IMHO the simplest backup is to just copy your files (using whatever method you choose, including backup software) to a USB drive that you then disconnect from the system and store elsewhere. For better protection you'd have two or more of these backup drives that you'd alternate in cycles, and the gold standard would be to store the latest copy to a different location.

The real purpose of a RAID system organized with redundancy is to eliminate the time needed to restore your files if a disk failure occurs. It's insurance against downtime, not against data loss. If the availability of your system is critical in some business or personal sense then that's the reason to use RAID. But even if you do, you'll still need a backup strategy that keeps a copy of your data offline, because there are a lot more ways to loose your data than just disk failure.

PHXAZCRAIG
PHXAZCRAIG Forum Pro • Posts: 16,626
my suggestions
2

With only 1TB of data, you have several options, and don't necessarily need RAID.   You do need backup.

Option 1:   Two big hard drives in the same PC, using mirroring to automatically (and quickly) have any data written or erased on one drive duplicated on a second.    This isn't a good backup, but in a sense is the minimum you should do to protect yourself against a hard drive failure.

Problem: A lot of bad things can happen to both drives in a single PC.    You really need to have a backup copy on a separate device.

Good thing:  if either drive fails, you should still have a working pc.

Option 2:  Two big hard drives in two different devices (PC's or other) with some sort of automatic copy routine to copy your important data across a network connection to a second hard drive.   Either one fails, you have at least most of your data on one device or the other.

RAID is involved in option 1 in the form of RAID0 or mirroring.

For my part, I've not tried to prevent my PC's drives from failing (mirroring), but instead put a lot of eggs in a big backup basket.    I have multiple systems backing up to a single device, and in that device I use RAID5 to protect against a drive failure without costing me as many drives as mirroring would.   I used to have a FreeBSD box (homemade) using software RAID5 to give me the capacity of 2 drives from a total of 3.   Mirroring would have cost me two drives from a total of four, so RAID5 is cheaper on disks.    After several years that box failed and I replaced it with a QNAP 451+ with 4 drives in RAID5 (gives capacity of 3 drives, which in my case is 27TB).

My late wife was a big fan of external hard drives for backup.  Cheaper than internal drives, which always made me suspicious.   And those drives seemed to reliably fail soon after the warranty expired.  Still, way better than nothing for backup.

If you have two pc's and an ethernet network connecting them, your easiest answer is to use MS networking to map a drive from one to the other, then some scheduled batch program to copy the appropriate data from one PC to the other.  I use Robocopy,

There is a problem using backup software - you have to choose between one-way sync (copy newer files) or true synchronization, meaning that if you delete a file on the source, that file gets deleted on the destination too.    This is a problem for accidental deletions.

The alternative problem, which I face, is if you copy over data only one-way, you eventually end up with stuff on the backup that you deleted on the source.   In my case that is a lot of files, because I import all my images to a temporary directory for rating, culling and post-processing.   Once I've done all my edits, I use Lightroom to move the data to a permanent directory structure.   Which means I end up with two copies of a lot of stuff.   I periodically delete those directories manually.

-- hide signature --

Phoenix Arizona Craig
www.cjcphoto.net
"In theory, practice and theory are the same. In practice, they're not."

 PHXAZCRAIG's gear list:PHXAZCRAIG's gear list
Nikon D80 Nikon D200 Nikon D300 Nikon D700 Nikon 1 V1 +37 more
Sean Nelson
Sean Nelson Forum Pro • Posts: 13,802
Re: my suggestions
2

PHXAZCRAIG wrote:

My late wife was a big fan of external hard drives for backup. Cheaper than internal drives, which always made me suspicious. And those drives seemed to reliably fail soon after the warranty expired. Still, way better than nothing for backup.

For backups I swear by "bare" hard drives connected to my PC via a USB 3.0 hard drive dock.  The nice thing about bare drives is that they're cheaper, you know for sure exactly what drive you're getting (which isn't the case when you buy drives in enclosures), and they're compact enough to fit into a bank safety deposit box even when wrapped with bubble wrap.

calson Forum Pro • Posts: 10,379
Re: Recommendations for long term storage: RAID or ??
1

There are NAS boxes that support 1GB Ethernet and that is the fastest data connection with your current computer, assuming that it is a laptop. With a desktop you could add a USB 3.1 port with a PCIe board.

The least expensive approach with NAS is to buy an enclosure (QNAP or Symbology) that takes 2 hard drives and use RAID1 that copies all data to both hard drives so each mirrors the other one. When one drive fails a replacement is put into the enclosure and the NAS will then copy data from the good drive to the new drive and at the same time you can access the data on the good drive.

With a NAS with Ethernet you or others can connect it to a wireless router and access it with a laptop from anywhere in the house. A wireless capable television can also access the files on the NAS and display them.

-- hide signature --

"It is horrifying that we have to fight our own government to save the environment." Ansel Adams

 calson's gear list:calson's gear list
Sony RX100 Nikon D850
WryCuda Forum Pro • Posts: 10,165
Re: Recommendations for long term storage: RAID or ??
1

gilliano wrote:

So I have reached well over 1TB of photos/ video. Although I have a fairly robust multi-copy backup system in place I do not want to use a dead end like an external HD for any kind of long term storage (long time field researcher and data manager, seen way too many people do that and lose everything). My internet connection is slow and not unlimited and I am aware of the ways cloud systems can go away/ go out of business so cloud back up is not my preference. I'm looking at RAIDs but not really sure what is practical / affordable for a small operation like mine. Any specific recommendations? Anything to consider? Alternatives? Assume I know nothing about them, which is basically where I'm at.

Also atm I am limited to USB 2.0 on my current computer but in the next year or so will upgrade to a new computer with 3.0 or faster file share capability. Not sure if I should wait or if most systems can handle both types of connection?

I have a similar amount of data, and file copying is all I’ve ever used for backup. I haven’t lost anything in 30+ years of wrangling data.

I divide data into “Archive” and “Current” so that I don’t have to copy everything on a regular basis. Every five years or so, I consign additional data to the Archive set, which is refreshed yearly.

I run manual backups about once a month, while vital projects are backup up daily.

I have data copied across the network to several places...

  • Main server
  • Standby computer
  • My photo computer (Current data + Photos)
  • Travel computer (Current data only)

In addition, each computer has its dedicated external USB 3.0 HDD.

In my experience, RAID is a bad idea for a small operation. NAS could be useful, provided that is backed up.

I’m not sure about your “both types of connection” query. USB 2 and USB 3 are compatible, apart from the speed. You may be able to add a USB 3 adapter to your current computer.

Do think about networking as an option. In an emergency, any of my computers could assume the role of “main computer”.

 WryCuda's gear list:WryCuda's gear list
Nikon D7100 Sony a6000 Sony a7
OP gilliano Regular Member • Posts: 106
Re: Recommendations for long term storage: RAID or ??

calson wrote:

There are NAS boxes that support 1GB Ethernet and that is the fastest data connection with your current computer, assuming that it is a laptop. With a desktop you could add a USB 3.1 port with a PCIe board.

The least expensive approach with NAS is to buy an enclosure (QNAP or Symbology) that takes 2 hard drives and use RAID1 that copies all data to both hard drives so each mirrors the other one. When one drive fails a replacement is put into the enclosure and the NAS will then copy data from the good drive to the new drive and at the same time you can access the data on the good drive.

With a NAS with Ethernet you or others can connect it to a wireless router and access it with a laptop from anywhere in the house. A wireless capable television can also access the files on the NAS and display them.

I think this is what I need: the combination of mirrored drives and the automatic backup/ copy/ restore sounds right. My main problem right now is just time spent manually backing up and  the occasional LR cataloging nightmare when I screw up. Automating would be huge. TThe speed would be a huge bonus: I am on a desktop. Thanks! I have never set anything like this up myself and was missing a few key terms and bits of the picture.

fwiw, I have a lot more than 1TB of data but I might have a TB in working status at any one time (video I have yet to review/ edit together mostly and sometimes a glut of the stupidly large files that the Canon software produces).

For true archiving I store tiff/jpg files on Blu-Ray discs and raw on the cloud. I'm not sure the cloud options are going to be around in 50 years but the discs should be!! It's cheap to do, I highly recommend it for long term data storage if you don't have access to some kind of professionally managed tape back up set-up.

 gilliano's gear list:gilliano's gear list
Canon EOS M5
Sean Nelson
Sean Nelson Forum Pro • Posts: 13,802
Re: Recommendations for long term storage: RAID or ??
1

gilliano wrote:

calson wrote:

The least expensive approach with NAS is to buy an enclosure (QNAP or Symbology) that takes 2 hard drives and use RAID1 that copies all data to both hard drives so each mirrors the other one. When one drive fails a replacement is put into the enclosure and the NAS will then copy data from the good drive to the new drive and at the same time you can access the data on the good drive.

I think this is what I need: the combination of mirrored drives and the automatic backup/ copy/ restore sounds right.

My main problem right now is just time spent manually backing up and the occasional LR cataloging nightmare when I screw up.

I want reiterate and to make sure you understand that RAID will not work as a backup for you.   If you screw up your LR catalogue a mirrored RAID system will make sure that the "copy" is exactly as screwed up as the original.

Backups to external USB drives can be scheduled to run automatically as long as you leave your computer on overnight and you remember to leave a drive plugged in.   In the morning you unplug the drive and squirrel it away somewhere, not exactly an onerous task.

WryCuda Forum Pro • Posts: 10,165
Re: Recommendations for long term storage: RAID or ??

Sean Nelson wrote:

gilliano wrote:

I think this is what I need: the combination of mirrored drives and the automatic backup/ copy/ restore sounds right.

Forget the "automatic restore" option. 

My main problem right now is just time spent manually backing up and the occasional LR cataloging nightmare when I screw up.

I want to reiterate and to make sure you understand that RAID will not work as a backup for you. If you screw up your LR catalog a mirrored RAID system will make sure that the "copy" is exactly as screwed up as the original.

My "recent data" is small enough to permit historical backups, so that I can restore previous data from up to 12 months previously. Important projects also exhibit "version control", so that editing errors can be easily undone.

I have actually known students who have a single file called "PhD thesis final version", or similar, with no other preliminary documents at all.

Such as...

  • Chapter1001
  • Chapter1002
  • Chapter1003
  • Chapter2001
  • etc.
 WryCuda's gear list:WryCuda's gear list
Nikon D7100 Sony a6000 Sony a7
OP gilliano Regular Member • Posts: 106
Re: Recommendations for long term storage: RAID or ??

Sean Nelson wrote:

gilliano wrote:

calson wrote:

The least expensive approach with NAS is to buy an enclosure (QNAP or Symbology) that takes 2 hard drives and use RAID1 that copies all data to both hard drives so each mirrors the other one. When one drive fails a replacement is put into the enclosure and the NAS will then copy data from the good drive to the new drive and at the same time you can access the data on the good drive.

I think this is what I need: the combination of mirrored drives and the automatic backup/ copy/ restore sounds right.

My main problem right now is just time spent manually backing up and the occasional LR cataloging nightmare when I screw up.

I want reiterate and to make sure you understand that RAID will not work as a backup for you. If you screw up your LR catalogue a mirrored RAID system will make sure that the "copy" is exactly as screwed up as the original.

Backups to external USB drives can be scheduled to run automatically as long as you leave your computer on overnight and you remember to leave a drive plugged in. In the morning you unplug the drive and squirrel it away somewhere, not exactly an onerous task.

Personally I do not consider backing up to a single external drive good enough insurance. Seen too many of them fail.

The LR catalog issue: I back up my internal hard drive to two separate external hard drives. I can point LR at any of the three drives as they are, in theory identical. Problem: creating two identical backups manually is time consuming and difficult to do perfectly over a long period of time, inevitably differences creep in. RAID is designed to automate this multiple-disk process and with only one backup action for each point in time there should be no further LR issues.

Using the terms calson has provided (NAS/DAS) I am finding what I need: consumer oriented arrays packaged with some various nifty software and hardware options and targeted at photographers so there is minimal maintenance. Some sync with popular cloud options natively, which would be cool if I had decent internet speeds.

So far I'm looking at Drobo and Synology but am open to other suggestions? Ideally whatever I got would be partitioned and back up to each partition on alternate days but as it's really just to back up my working hard drive I probably don't really need that.

 gilliano's gear list:gilliano's gear list
Canon EOS M5
OP gilliano Regular Member • Posts: 106
Re: Recommendations for long term storage: RAID or ??

WryCuda wrote:

Sean Nelson wrote:

gilliano wrote:

I think this is what I need: the combination of mirrored drives and the automatic backup/ copy/ restore sounds right.

Forget the "automatic restore" option.

My main problem right now is just time spent manually backing up and the occasional LR cataloging nightmare when I screw up.

I want to reiterate and to make sure you understand that RAID will not work as a backup for you. If you screw up your LR catalog a mirrored RAID system will make sure that the "copy" is exactly as screwed up as the original.

My "recent data" is small enough to permit historical backups, so that I can restore previous data from up to 12 months previously. Important projects also exhibit "version control", so that editing errors can be easily undone.

I have actually known students who have a single file called "PhD thesis final version", or similar, with no other preliminary documents at all.

Such as...

  • Chapter1001
  • Chapter1002
  • Chapter1003
  • Chapter2001
  • etc.

I would love to maintain a historical backup system but getting into video is making it all so unwieldy. I should probably split the photo and video for back up purposes.

As far as students go about once a year I hear about one losing their whole thesis when they lose their laptop which had the sole copy. I have no idea how anyone makes it to graduate school without learning about the magic of thumb drives.

As a field scientist I have seen some very creative ways to destroy digital storage: Write in the Rain paper and SD cards will survive almost anything. Hard drives, laptops and tablets: not so much.

 gilliano's gear list:gilliano's gear list
Canon EOS M5
Sean Nelson
Sean Nelson Forum Pro • Posts: 13,802
Re: Recommendations for long term storage: RAID or ??

gilliano wrote:

Sean Nelson wrote:

I want reiterate and to make sure you understand that RAID will not work as a backup for you. If you screw up your LR catalogue a mirrored RAID system will make sure that the "copy" is exactly as screwed up as the original.

Personally I do not consider backing up to a single external drive good enough insurance. Seen too many of them fail.

I totally agree.  That's why I have two external drives that I alternate in a weekly cycle and two more that I alternate in a monthly cycle.  My backup drives are larger than my online drive, allowing me to keep multiple backups on each drive going back several months.

The LR catalog issue: I back up my internal hard drive to two separate external hard drives. I can point LR at any of the three drives as they are, in theory identical. Problem: creating two identical backups manually is time consuming and difficult to do perfectly over a long period of time, inevitably differences creep in.

That seems like a lot of fuss to me.   I just stick a drive in my external USB drive dock once a week and let my backup copy everything.  I don't use Lightroom, but I do have Bridge, Thumbsplus, and a lot of other project-oriented software - all of the files and metadata catalogues on the entire disk just get backed up and the drive is removed when it's done.  If I loose the online drive or if it gets corrupted I can restore an individual file or the entire drive.

I also stick a USB flash drive in every evening and back up just the changed files, it takes a couple of minutes.  So in the worst case of disk or other failure I just have to reenter 1 days' worth of stuff, not usually a big deal for the kind of work I do.

I don't have to restore stuff very often, maybe once a year or so - and it's usually because I accidentally deleted something I didn't mean to or because I want to back out of some lengthy editing of program code that went down a rabbit hole.

SC489 Senior Member • Posts: 1,198
Re: Recommendations for long term storage: RAID or ??
1

I wouldn't bother with the complexities of RAID. I would make scheduled Macrium Reflect backups to an internal or external permanently connected external drive and manual backups to another external drive stored separately from the PC. That way you are protected against a single backup  drive failure or serious malware attack affecting all drives. Uses the Reflect Verify feature and viBoot to check you can boot the backup image.

 SC489's gear list:SC489's gear list
Sony Cyber-shot DSC-HX80 Panasonic Lumix DMC-GX7 Olympus M.Zuiko Digital ED 9-18mm F4.0-5.6 Panasonic Lumix G Vario 45-150mm F4-5.6 ASPH Mega OIS Panasonic Lumix G Vario 14-42mm F3.5-5.6 II ASPH Mega OIS +4 more
OP gilliano Regular Member • Posts: 106
Re: Recommendations for long term storage: RAID or ??

My understanding is that LR catalog is SQL based so the image folders need to stay in order. I have a 1TB internal HD and probably 3TB of data and so my catalog has to point to a minimum of two drives. Then I have to back those up etc etc Doable but you have to be careful and being 100% careful over time is statistically unlikely. And there is not much fun to be had backing up original images and losing all your edits.

 gilliano's gear list:gilliano's gear list
Canon EOS M5
Robert Zanatta Senior Member • Posts: 1,905
Re: Recommendations for long term storage: RAID or ??
2

I like using a USB 3 external dock.  I can swap drives on the fly as it supports ejecting media.

I have multiple drives.  My Seagate 3TB Barracuda is fast in both transfer rate and access time.  With a file synchronization app it only takes a few minutes to backup/sync 30GB of changes.  Bur my older WD Black 2TB drives are noticeably slower than the Seagate... 

If you don't have USB 3 port, an add-in card is inexpensive.

 Robert Zanatta's gear list:Robert Zanatta's gear list
Canon EOS 5D Mark IV
TomFid Veteran Member • Posts: 3,474
Re: Recommendations for long term storage: RAID or ??
1

gilliano wrote:

So far I'm looking at Drobo and Synology but am open to other suggestions? Ideally whatever I got would be partitioned and back up to each partition on alternate days but as it's really just to back up my working hard drive I probably don't really need that.

I have a Drobo that I like pretty well. Performance is reasonable, it's easy to interact with, and it recently survived a drive failure gracefully. However, that drive failure has me wondering if it runs hotter than the same drives would in a big computer case with more cooling.

Either way, DAS is not a full backup - if your house burns down it goes with everything else. So I think you still need off-site, either external HDs or cloud.

DerKeyser Contributing Member • Posts: 626
Re: Recommendations for long term storage: RAID or ??

gilliano wrote:

So I have reached well over 1TB of photos/ video. Although I have a fairly robust multi-copy backup system in place I do not want to use a dead end like an external HD for any kind of long term storage (long time field researcher and data manager, seen way too many people do that and lose everything). My internet connection is slow and not unlimited and I am aware of the ways cloud systems can go away/ go out of business so cloud back up is not my preference. I'm looking at RAIDs but not really sure what is practical / affordable for a small operation like mine. Any specific recommendations? Anything to consider? Alternatives? Assume I know nothing about them, which is basically where I'm at.

Also atm I am limited to USB 2.0 on my current computer but in the next year or so will upgrade to a new computer with 3.0 or faster file share capability. Not sure if I should wait or if most systems can handle both types of connection?

Actually there’s a lot of considerations to take depending on what result you are looking for. 
Let’s just get this one out of the way: RAID is not backup. It’s just a single disk with better physical hardwáre redundancy

I wrote the blog post below a few years back, and I have litteraly had more than a hundred people writing me and thanking me for making them aware of all the “other” things to keep in mind when designing a solution.

I’m not saying that my solution is the best - it’s just a very good low cost way of solving all the “problems” some/most of the offered solutions in this thread inherently contains (even though they are either denied by the author, or extremely unlikely).

Just read it through and identify the qualities you would like your solution to have. Then you can create that completely different from mine if that is your best solution. I don’t care - I only care about you getting a proper workflow/backup solution that will persist through human error, hardware errors and natural disasters.

Using a NAS to secure your workflow.

-- hide signature --

Happy Nikon Shooter
See my articles and galleries at: https://wolffmadsen.dk

Keyboard shortcuts:
FForum MMy threads