How do you backup?

I get that RAID is still necessary for high-volume pro shooters where you can't be down for an hour and there are hundreds of files added almost daily.

But for you shooters who don't shoot super-high volume every week and have less than 5 TB of total image files? No way RAID!
Come on Greg. RAID is good.

Btw where did you get the 5TB figure?

RAID isn't by any means a necessity, but it is something I choose to have, for many reasons.

01. 8 x 10TB drives in RAID6 gives me 54.5 TB in usable capacity. If I am to run single drives, I'd have eight different drives in my PC and be looking for files in differently volumes and will probably have to keep a list on what is where. Now I have all of that under one volume.

02. Speed: It significantly increases transfer rate. I am getting almost SSD like speed (not IOPS, but transfer speed), from a 54.5 TB volume. Imaging having a space that big and have the speed too using SSD, or even normal single drives.

fefac462a0354b64bc58c8e6468ade0d.jpg

It reaches first gen NVMe SSD speeds in most cases, and having that is amazing. I have no lag accessing catalogs or raw files. Even a 35GB PSB files opens pretty quick in PS, try that with a single drive and you will soon notice how slow it is.

And the best thing is, all my other drives in the system are NVMe SSDs (2 x 1TB, 1 x 2TB) and three SATA SSDs (3 x 2TB), so between all the drives whenever I copy or move data, I get SSD-like speeds even though my biggest drive is mechanical 😁

03. Hardware RAID volumes are protected at boot: no boot time virus can affect the volumes as it is completely independent of the host OS. Any drives connected to the PC can be accessed during boot as they are using the OS, but a hardware RAID controller firmware loads independently before boot and protects it.

04. My controller does monthly full surface check on all drives, and also a consistency check on the whole volume, once a month, scheduled by me about five years ago. Imagine doing this to all separate drives every month manually. It sure does makes maintenance easy.

05. Unexpected power loss during copying data: No problem. The Backup Battery Unit (BBU) will transfer the data in the cache when power is restored, so no file corruption on the file that were interrupted.

06. You can disconnect and connect another drive and resume work. I don't even have to do that. It just stays online, it will use one of the hot-swap drives to start rebuild straightaway without requiring the user to do anything.

RAID has more advantages than disadvantages. RAID is in no way, dead. Have you seen all the latest RAID controllers coming out with NVMe features? You can have NVMe SSDs as cache and what not. Expensive but amazing. RAID is not complex at all. Very easy to do if you have the right components, and once done it will serve you for years.

I stopped relying on normal hard drives a long time ago. All the drives I use are enterprise grade drives. Most people either run software RAID through enclosure / NAS, or hybrid RAIDs with normal desktop drives which cannot even maintain TLER, so it is expected those systems will run into problems.
RAID introduces more risk than it mitigates.
Not really, it has more advantages than disadvantages.
There are super reliable 8 TB HDDs now. Just work off of one and use GoodSync to sync to as many other internal or external drives as you want.

In the unlikely event that the 8TB HDD fails, just immediately switch to another synced drive. Good Sync is not like a backup program that creates coded gibberish on the backup disk where you can't see what is there and there is always a risk on the restore process. Good Sync creates a duplicate of whatever folders you back up. You can go right to it.

All you would have to do is reconnect the LR catalog to the new disk.

RAID is dead. It is obsolete. It is too complex. It has to much risk of not working correctly.
Not dead at all. In fact, a new controller just came out with 8GB NVMe SSD cache, and U2 NVMe supports and many others. It is very modern.

RAID only creates problems if sub-standard components are used: like standard hard drives.
a RAID of 5 disks has more chance of creating problems than using just one disk and backing it up.
Actually a RAID10 array with 5 drive (one hot swap) will be a really reliable array. Even if it is software RAID (but not windows RAID)
I probably shoot 15,000 files a year now max?

RAID? No thanks.
RAID isn't for everyone. Your synced drive system is good, I'm not denying it. For me RAID is an amazing technology that I wish I knew about it 10 more years ago :)
Most of you could live to be 200 years old and never fill an 8TB drive unless you have video.
I've only been doing photography since 2008, and started shooting RAW since 2011, and I now have about 13TB of photos, and I don't do video at all.

--
IG: https://www.instagram.com/manzurfahim/
website: https://www.manzurfahim.com
 
Last edited:
I hear you on the speed, but that will soon not be the case.

RAID is dead.
Not dead at all, Greg. Look at this. PCIe 4.0 RAID controller with NVMe and all the bells and whistles :D

I could easily run it right now. I have a stack of 3 8TB drives in my PC and could easily add two or three more for cheap. (I have a huge Be Quiet! Dark Base Pro 900 case with 8 HDD rails.)
You will run into problems, because I think most of your drives are not designed to be in a RAID array. Enterprise drives designed for RAIDs are expensive. I just bought 2 x 18TB drives, each cost me $600 USD. I'll slowly buy more as I cannot afford to buy them all at once. But I can replace them one or even two at a time without any downtime, and it will rebuild it in the background.
I think some people run Raid because its cool and part of the hobby. Not you Stan....

But again, RAID of course has its uses, especially for some of the high-end guys on this Board and pro high-volume photographers who can't go down for an hour in the sutdio.
 
I get that RAID is still necessary for high-volume pro shooters where you can't be down for an hour and there are hundreds of files added almost daily.

But for you shooters who don't shoot super-high volume every week and have less than 5 TB of total image files? No way RAID!
Come on Greg. RAID is good.

Btw where did you get the 5TB figure?

RAID isn't by any means a necessity, but it is something I choose to have, for many reasons.

01. 8 x 10TB drives in RAID6 gives me 54.5 TB in usable capacity. If I am to run single drives, I'd have eight different drives in my PC and be looking for files in differently volumes and will probably have to keep a list on what is where. Now I have all of that under one volume.

02. Speed: It significantly increases transfer rate. I am getting almost SSD like speed (not IOPS, but transfer speed), from a 54.5 TB volume. Imaging having a space that big and have the speed too using SSD, or even normal single drives.

fefac462a0354b64bc58c8e6468ade0d.jpg

It reaches first gen NVMe SSD speeds in most cases, and having that is amazing. I have no lag accessing catalogs or raw files. Even a 35GB PSB files opens pretty quick in PS, try that with a single drive and you will soon notice how slow it is.

And the best thing is, all my other drives in the system are NVMe SSDs (2 x 1TB, 1 x 2TB) and three SATA SSDs (3 x 2TB), so between all the drives whenever I copy or move data, I get SSD-like speeds even though my biggest drive is mechanical 😁

03. Hardware RAID volumes are protected at boot: no boot time virus can affect the volumes as it is completely independent of the host OS. Any drives connected to the PC can be accessed during boot as they are using the OS, but a hardware RAID controller firmware loads independently before boot and protects it.

04. My controller does monthly full surface check on all drives, and also a consistency check on the whole volume, once a month, scheduled by me about five years ago. Imagine doing this to all separate drives every month manually. It sure does makes maintenance easy.

05. Unexpected power loss during copying data: No problem. The Backup Battery Unit (BBU) will transfer the data in the cache when power is restored, so no file corruption on the file that were interrupted.

06. You can disconnect and connect another drive and resume work. I don't even have to do that. It just stays online, it will use one of the hot-swap drives to start rebuild straightaway without requiring the user to do anything.

RAID has more advantages than disadvantages. RAID is in no way, dead. Have you seen all the latest RAID controllers coming out with NVMe features? You can have NVMe SSDs as cache and what not. Expensive but amazing. RAID is not complex at all. Very easy to do if you have the right components, and once done it will serve you for years.

I stopped relying on normal hard drives a long time ago. All the drives I use are enterprise grade drives. Most people either run software RAID through enclosure / NAS, or hybrid RAIDs with normal desktop drives which cannot even maintain TLER, so it is expected those systems will run into problems.
RAID introduces more risk than it mitigates.
Not really, it has more advantages than disadvantages.
There are super reliable 8 TB HDDs now. Just work off of one and use GoodSync to sync to as many other internal or external drives as you want.

In the unlikely event that the 8TB HDD fails, just immediately switch to another synced drive. Good Sync is not like a backup program that creates coded gibberish on the backup disk where you can't see what is there and there is always a risk on the restore process. Good Sync creates a duplicate of whatever folders you back up. You can go right to it.

All you would have to do is reconnect the LR catalog to the new disk.

RAID is dead. It is obsolete. It is too complex. It has to much risk of not working correctly.
Not dead at all. In fact, a new controller just came out with 8GB NVMe SSD cache, and U2 NVMe supports and many others. It is very modern.

RAID only creates problems if sub-standard components are used: like standard hard drives.
a RAID of 5 disks has more chance of creating problems than using just one disk and backing it up.
Actually a RAID10 array with 5 drive (one hot swap) will be a really reliable array. Even if it is software RAID (but not windows RAID)
I probably shoot 15,000 files a year now max?

RAID? No thanks.
RAID isn't for everyone. Your synced drive system is good, I'm not denying it. For me RAID is an amazing technology that I wish I knew about it 10 more years ago :)
Most of you could live to be 200 years old and never fill an 8TB drive unless you have video.
I've only been doing photography since 2008, and started shooting RAW since 2011, and I now have about 13TB of photos, and I don't do video at all.
Manzur, that is an amazing post and RAID for you is great. I probably was too harsh on calling RAID dead, but it is old tech that was developed for spinning smaller volume drives.

You are very technical, have high-end everything and a full understanding of it all and how best to do it. I am amazed that you have 13 TB of pure raw images. That is a huge amount of image files for a single shooter that is not out doing school portraits all day every day with a team of assistants and other photographers.

I said 5 TB because 8TB HDDs are cheap and common and if you have around 4 TB of data an 8 TB HDD can work great for you until the 6 and 8 TB SSDs start becoming very common and much cheaper.

If I had 13 TB of data I would do what you are doing and start stacking hard drives and running raid the way you are.

But won't it be great when we have 8TB super fast (non sata) SSDs for say, 250 bucks? Wow. That would change everything. But you would still do what you are doing and probably just pile up 4 or 5 of those and run a raid....

--
Greg Johnson, San Antonio, Texas
 
Manzur, that is an amazing post and RAID for you is great. I probably was too harsh on calling RAID dead, but it is old tech that was developed for spinning smaller volume drives.
It is an old tech for sure, but now they have all upgraded. NVMe SSD support, trim support, and they can now throughput like 10-15GBps due to SSD RAID. RAID got a lot more exciting.
You are very technical, have high-end everything and a full understanding of it all and how best to do it. I am amazed that you have 13 TB of pure raw images. That is a huge amount of image files for a single shooter that is not out doing school portraits all day every day with a team of assistants and other photographers.
You are being too kind with your words Greg :) And I used to trek a lot, and used to take a lot of photos. Back in 2016, I walked the whole of Scotland from one end to another, and on that trip alone I took 500GB or so photos in 20 days. I've only started shooting portraits three years ago.
I said 5 TB because 8TB HDDs are cheap and common and if you have around 4 TB of data an 8 TB HDD can work great for you until the 6 and 8 TB SSDs start becoming very common and much cheaper.

If I had 13 TB of data I would do what you are doing and start stacking hard drives and running raid the way you are.

But won't it be great when we have 8TB super fast (non sata) SSDs for say, 250 bucks? Wow. That would change everything. But you would still do what you are doing and probably just pile up 4 or 5 of those and run a raid....
Ohh Greg, I cannot wait for 8TB NVMe / SATA SSDs to become cheap. Imagine the speed and reliability we will get from them. But unfortunately I think it will take a long time for them to become available at $250. And yes, I'll probably end up doing RAID with them again, you guessed it right Greg.

Stay safe and well.
 
I came up with this simple solution over the years:

1. Create a Capture One session on my USB 3 SanDisk 2TB SSD (my "working drive").

2. Import to this Capture One session.

3. Copy this session to Synology1 immediately after the files are imported.

4. Synology1 is automatically sync'd to Synology2 via Syncthing. Synology2 lives offsite in another geographic location.
 
I agree with you on old or new hardware raid solutions that lock you into their ecosystem are indeed useless. However using software based stuff like ZFS is miles better than a single hard disk, or manually syncing many disks. Sure you have to get a computer with ECC RAM but after that you are guaranteed to have no silent data-corruption, ever. ZFS will tell you which disk is about to fail, as soon as some blocks are written weirdly. You may have already had a corrupted jpeg somewhere, cannot happen on ZFS. It rebuilds really fast if there is a failure. And, the most important thing it does: snapshots. You can go back in time recovering accidental deletes for example. Snapshots are created in a second, hassle free. Also it makes backups incredibly easy by sending snapshots to another system/disk. No need to sync and check file dates or something else unreliable and slow.

Check FreeNAS for an easy to use storage system using this amazing filesystem. I built myself a completely silent NAS server using SSDs over 10gbit ethernet. I geht like 600 MBytes/s speed and secure storage from that, I'd call this far from dead (the machine is old and limited by the 6Gbit SATA bus, built today you can easily saturate the 10gbit ethernet)

Since a RAID is not a backup, just a failsafe disk, you still have to have a backup strategy. The sending of snapshots, another major feature of ZFS, makes it super efficient, even for cloud use. Check rsync.net they support receiving ZFS snapshots. Or you build yourself a second machine you put at a friends house, or your weekend home. Presto snapshot history and backup available at two locations, fully automated by FreeNAS, no manual interaction needed. If a fire or flood hits your house, you have a backup.

Another advantage, let's suppose your computer dies, you can bring the disks to someone with linux or BSD on it and it will be readable, no special hardware required, only the software driver to access ZFS is needed.

Manually struggling about with single disks is dead, and dangerous if you ask me ;)
 
Just to be clear, not all RAID Levels provide redundancy or fault tolerance, specifically RAID Level 0 is not fault tolerant.

Daily backups to magnetic tape is still used by cloud storage vendors and provides multiple copies of data as a function of time.


Jeff
 
Just to be clear, not all RAID Levels provide redundancy or fault tolerance, specifically RAID Level 0 is not fault tolerant.
RAID 0 — what we used to call striping — isn’t RAID, but merely AID.
Daily backups to magnetic tape is still used by cloud storage vendors and provides multiple copies of data as a function of time.
I used to use LTO, but cycling through the generations got to be really expensive, and LTO drives we slow to upgrade drivers. I just use more disks now, plus cloud backup.
 
After having number of issues with storage devices (dead HDDs and SSDs, bit flips on newer SMR drives, RAID controller sudden death) I just revamped data protection strategy both for the photo archives (hobby) and home business (need continuity). I will spare business data considerations.

My model makes assumption on the storage consumption, which are fairly timid, thus i am erring on the side of reliability. Currently I work with 4TB archive and 500GB annual ingest, probably doubling next year as I retire and spend more time shooting.

I am reasonably versed in business data protection, but chose not to invest into enterprise grade software suites due to the cost. I do backup business and development workstations (Separate from photo workstation) using Acronis suite, both daily and real time.

Real time backup I also evaluated, but for the large photo archive i am not satisfied with reliability and performance for either TimeMachine, FileHistory or Acronis Real Time backup. Staying away, tho it would be nice to have.

In the field : I am finishing the day with 3 verified copies if i am car camping, risking when backpacking (very rare). In camera mirrored cards stay filled till the end of the trip. At the end of the day I back them up to the laptop (1TB built in) and in the same time to the external SSD (1TB). After copying usually I run rsync checksum verification, seen enough in flight corruption on both USB and SD busses. SSD i keep on me when in the airport.

Home studio: My previous scheme was 3 tier storage (as soon as possible get to verified 3 copies) to protect against hardware failure, human errors and storage integrity failures. Workstation working storage are striped SSDs, no redundancy. ECC RAM. Mirrored 8TB USB3 drive set is attached. I have a home/business server, which houses 12 4TB HDDs (NAS grade, mixed batches and vendors to lower chances of concurrent RAID disk mortality) and 2 NVMe SSDs as a low latency tier. Just upgraded server in winter to Windows 2019, parity storage spaces (if you care about details), 64GB ECC RAM Older server with 8TB still runs, but using FreeNAS BSD software, as I am still pondering switch from Windows to Unix.

On a workstation twice a day running rsync as a cron job to replicate files from working storage to USB. On demand rerunning the job after a large editing session, which doesn't happen too frequently, yet. Copying parameters based on checksum, ignoring timestamps. Once a day rsync job to verify checksum integrity, sending email if any violation occurred.

Twice a week or on demand after the large ingest, identical job is running to the home server. After rsync copy job (copy based on checksum as well, assuming rsync daemon o on the server side). Server file system is ReFS, which by default is configured to run regular integrity verification, integrated with RAID5 logic , on violation I get an immediate warning. Even with that I run weekly rsync checksum verification job of workstation versus server. Older server runs now ZFS with L2ARC cache.

Added recently: I was missing protection against natural disasters, which in a seismic region can be real (I know pro folks who lost massive amounts of data to fires and earthquakes). After evaluating p2p replication to the house in another city I ruled against it. Instead I spent 2 weeks evaluating current cloud offerings and with my relatively timid data set settled on MSP360 with Azure Cold archive back end, configuring client side encryption with my key. If an archive grows too fast I will move to Backblaze (second choice) unlimited plan. I am only on second week of running 4th tier backup to the cloud, by the end of the year I think I will develop some opinions on the experience.

PS I could simplify all this with enterprise grade or custom configured distributed system with replication and integrity checking, but I don't want to go back to high tech chores, instead want to maximize time on creative endeavors. I admit the setup above is cludgy and techies would blame me for the lack of elegance and simplicity, but it works for the last 8 years. Cloud does look attractive these days, huge inroads been made in enterprise reliability and performance for archives going into 10s and even 100s of TBs. Of course one needs decent uplink, i am enjoying affordable access to a 500Mbps fiber link.
 
Two G Technology 10TB hard drives in two separate locations and cloud storage.
 
Jim K turned me onto GoodSync and I love it.

As I said above I work off of one internal 8 TB HDD in my PC. On that HDD I have only my images, organized into folders of my design.

I have not interest in backing up any OS, programs or anything other than image files, scanned documents and MS Office files.

My MS Office files are in the cloud, but my image files are not.

I hate backup programs that are coded gibberish on the backup. I want to look in the backup disk and see my folder and images exactly as they would appear in Windows File Explorer.

GoodSync creates that exact duplicate folder structure on the disk you back up to - as many as you want - you create a job for each disk that stays on a menu.

You can go to that disk and it is exactly like going to your main drive that is being backed up. Technically, you would not even have to restore if your primary disk went down. (I would like to know that for sure though).

Plus, I hate having multiple versions of files in a photo image folder on the backup. I just want the last change.

A backup program is only as good as its ability to restore....

Plus, Raid is not a backup. It is very complicated resiliency. I have the resiliency with my multiple synced disks from Goodsync, while actually working off just one big HDD.

Speaking of one big main disk - I think within 1 year that one main disk that I work off will be a big 8 TB Sata SSD! There are some good SATA 8 TB SSDs coming out now at around 800 bucks. If that gets to 500 I will get one and use it as my main data drive for my images.

Imagine if that were M.2 and operating on your Motherboard!
 
I use a similar synch utility called, “Synchredible.” It looks functionally identical. I use it to back up my Lightroom Catalog and preivews, and my Library of raw photos, etc. I do this manually after new sessions working. It goes on an external HDD that gets locked in my safe in a fireproof container. The safe itself has a good fire rating.

I also use the Windows back up to back up everything (system image included) once a week onto an external Glyph HDD. This just sits permanently attached to my computer.

Since I don’t consider any of my photography “all that important” for posterity (no one will care when I’m pushing up daisies), my back up system is “good enough” for my purposes.

Rand
 
Hi,

If you want to lose everything, just stick it onto the cloud. It'll all Go Away at some point in time. And then there is the security nightmare. No. Not for me.

Stan
I have zero interest at this point for sending any of my raw images up to the cloud as any kind of storage or backup. Storage is cheapo and getting cheaper. And it is 50,000 times faster and better than the cloud.
Cloud storage has a place in backup chains. It does not replace onsite backups. It does not replace RAIDs. It does provide geographical diversity, as do disks in safe deposit boxes. When I had to evacuate because of a fire a few days ago, I was glad that I had current cloud backups.

Jim
 
Hi,

If you want to lose everything, just stick it onto the cloud. It'll all Go Away at some point in time. And then there is the security nightmare. No. Not for me.

Stan
I have zero interest at this point for sending any of my raw images up to the cloud as any kind of storage or backup. Storage is cheapo and getting cheaper. And it is 50,000 times faster and better than the cloud.
Cloud storage has a place in backup chains. It does not replace onsite backups. It does not replace RAIDs. It does provide geographical diversity, as do disks in safe deposit boxes. When I had to evacuate because of a fire a few days ago, I was glad that I had current cloud backups.

Jim
 
Hi,

If you want to lose everything, just stick it onto the cloud. It'll all Go Away at some point in time. And then there is the security nightmare. No. Not for me.

Stan
I have zero interest at this point for sending any of my raw images up to the cloud as any kind of storage or backup. Storage is cheapo and getting cheaper. And it is 50,000 times faster and better than the cloud.
Cloud storage has a place in backup chains. It does not replace onsite backups. It does not replace RAIDs. It does provide geographical diversity, as do disks in safe deposit boxes. When I had to evacuate because of a fire a few days ago, I was glad that I had current cloud backups.

Jim
Well said. I agree. I might send my 4 TB to the cloud if it is feasible just as the ultimate safety along with the synced disks I described having.
GoodSync has a Backblaze mode.





97cd84d9b17d466a8bed2c487103eb00.jpg.png





--
 
I measured the time it takes to write and read a 1.44Gb PSD file from/to PS.

SSD: 2600MB/s read and write
External HDD: 130 Mb/s read and write

Write
SSD: 48sec
External HD: 58 sec

Read
SSD: 8.4 sec
External HD: 9.2 sec

I keep only catalogs, caches, and preview images on SSD. The images themselves reside on slow and silent external HDs.
 
I measured the time it takes to write and read a 1.44Gb PSD file from/to PS.

SSD: 2600MB/s read and write
External HDD: 130 Mb/s read and write

Write
SSD: 48sec
External HD: 58 sec

Read
SSD: 8.4 sec
External HD: 9.2 sec

I keep only catalogs, caches, and preview images on SSD. The images themselves reside on slow and silent external HDs.
I can't figure out what your reply has to do with the post to which you replied.
 
I measured the time it takes to write and read a 1.44Gb PSD file from/to PS.

SSD: 2600MB/s read and write
External HDD: 130 Mb/s read and write

Write
SSD: 48sec
External HD: 58 sec

Read
SSD: 8.4 sec
External HD: 9.2 sec

I keep only catalogs, caches, and preview images on SSD. The images themselves reside on slow and silent external HDs.
I can't figure out what your reply has to do with the post to which you replied.
Sorry, I hit Reply instead of Reply to Thread (or Reply to the appropriate branch of discussion). It was a contribution to the discussion of whether one needs fast storage for photos.

I typically read DPR in Flat view, and that also leads me to make mistakes like that.
 
I measured the time it takes to write and read a 1.44Gb PSD file from/to PS.

SSD: 2600MB/s read and write
External HDD: 130 Mb/s read and write

Write
SSD: 48sec
External HD: 58 sec

Read
SSD: 8.4 sec
External HD: 9.2 sec

I keep only catalogs, caches, and preview images on SSD. The images themselves reside on slow and silent external HDs.
I can't figure out what your reply has to do with the post to which you replied.
Sorry, I hit Reply instead of Reply to Thread (or Reply to the appropriate branch of discussion). It was a contribution to the discussion of whether one needs fast storage for photos.

I typically read DPR in Flat view, and that also leads me to make mistakes like that.
Now that I understand it was unrelated to my post, no problem.

Ji
 

Keyboard shortcuts

Back
Top