How do you backup?

[Manzur Fahim]

Hello everyone,

Hope you are well and safe.

There is a recent thread about photo storage and it got me interested about knowing what everyone else is doing in terms of their backup, whether they are photos or other files. I'm quite paranoid about data backup, ever since I lost about 3TB of data due to a PCI SATA card not being compatible with 3TB drives. Anyway, here is how I do backup and why I do it this way.

01. My first backup starts with the camera. I always use two cards in the camera, configured in backup mode. I use large capacity cards, and I don't format after every shoot. The files stays in the card until the cards are full, and then I use a different set of cards and so on. Once I have all cards used up, I started using the very first set after formatting them in camera, and this is how I keep all cards in rotation. So for a good amount of time, all my raw data have a copy in the cards.

02. After creating LR catalog, I copy the raw files from the card to my primary storage system. I use a LSI enterprise grade PCIe 3.0 8x hardware RAID controller which can take eight hard drives.

I am currently using 8 x 10TB HGST He10 enterprise grade hard drives, and have them configured in a RAID 6 array. These drives are designed to tolerate a workload of 540TB per year, and are built to a much higher standard than standard desktop or NAS drives.

This gives me 54.5 TB of usable capacity in Windows. The RAID controller has some useful features, I have scheduled a monthly Petrol read, which surface scans all the hard drive once a month, and a consistency check function that checks the virtual drive for any inconsistency once a month. The controller has a battery which protects any data in the cache in case of a power failure.

03. I manually copy all the data in a G-Drive USB 3.1 Gen1 RAID enclosure, which has 2 x 10TB (Seagate EXOS enterprise drives) drives in RAID 0, with a usable space of 18.1 TB. I can backup to this drive at a decent speed (around 320MB/s). I backup to this drive on the first day of the month.

I'm thinking of upgrading this to a USB 3.1 Gen2 model which would improve the backup speed, as these drives can go up to 240MB/s on their own. RAID 0 is getting bottle-necked with the enclosure I have.

04. I backup to a single 16TB drive (Seagate EXOS) on the 10th of every month. This is my second backup of the photos.

05. I backup to multiple SSDs (2 x 3.84TB Samsung PM863 enterprise SSDs, 3 x 2TB Samsung 850 EVO, 860 EVO drives). This is my third backup of the photos. I do this backup on the 20th of every month.

The multiple backups acts as backup redundancies, but mostly acts as different versions of the backup. For example, on the 22nd of any month I realized that I did a mistake and deleted something from my hard drive on 15th of that month. And the backup from 20th did not have that file. But luckily I still have the backup from 10th of the month which still has that file. This way I can go back and undo my work up to 30 days in the past.

What is your backup method? I'd very much like to hear about it.

Thank you and stay safe.

 

[Rob de Loe]

Pretty much the same as you... Lots of devices, lots of redundancy, some devices off-site, rotations.

 

[Michael Prince]

I am pretty lazy. I have a couple of disk arrays, and back them up to the cloud with BackBlaze, which works great. I currently have 12 terabytes up there!

 

[cortex95x]

I am pretty lazy. I have a couple of disk arrays, and back them up to the cloud with BackBlaze, which works great. I currently have 12 terabytes up there!

What happens when your Internet connection goes down and you need an image that is stored in 'Cloud'? I for one never use the 'Cloud' for anything, and prefer a redundant backup system and RAID array, in order to maintain complete control of my data at all times, and that does not only apply to images. So, in essence, I follow a routine that is similar to Manzur's, and often add portable SSD drives to the equation.

 

[sczin]

My Method

Dual cards in camera at all times

Before traveling:

I sink my desktop drive with an external drive A.

While traveling at the end of each day:

• Raw files copied to small external drive B
• Raw files imported into lightroom (catalog is stored on external drive A). During import of Raw files are also stored on external drive A
• I deep RAW files on cards until I get home (I have lots of high capacity cards)

After traveling and while at home:

Using Allsway Sync application, sync external drive A with Desktop internal Raid 1

Using EaseUS backup software, make two copies of internal Raid 1.

First copy gets stored in bank safe box.

Second copy is updated weekly at home, then rotated to the bank monthly.

So, I have redundancy in the desktop with the RAID, plus one onsite back up and one offsite backup.

Hope this helps

Steven Castro

website www.stevencastrophotography.com

 

[Theia]

Two cards in the camera, copied to laptop if I’m away for any length of time. Keep the files on the cards unless I need the space. At the office, copy the files to the Synology NAS with RAID (powered through a UPS). The NAS backs up to a local large external drive every night and to CrashPlan continuously for “offsite” storage. I do not do local physical offsite storage because I assume it will also be destroyed when the ‘quake hits. If the office goes up in flames and the NAS and back-up are destroyed, then I’ll resort to CrashPlan. So far, there have been no card failures, only insignificant 'quakes, no fires, and only one drive failure in the NAS.

 

[Barry Twycross]

My back up strategy depends on whether I'm at home, or out somewhere (vacation etc, pandemics permitting). I'm only doing this for fun, so I dan't have to be too rigorous.

I'm willing to risk up to a days shooting, but once I get back to base (home or hotel, etc), I'll make sure there's two copies of every photo (not necessarily every file).

Only my GFX has dual card slots, I've got those configured for raw+jpeg, that's good enough for two copies for me. All my other cameras only have one card slot, so I can potentially lose up to a day's shooting if I have a card failure. As I said, I'm willing to take that risk.

I have had some cards do dodgy things, but only ever lost 1 or 2 photos that way, and then that card is immediately retired.

The first thing I do when I get back to base, is to upload the jpegs to Photos on the Mac. There's now a copy in the library, and on the card. The cards don't get erased until there are two other copies of every file. I specifically bought 10x cards for my travel camera, so I can keep copies on them, and also have the copy on the hard drive.

I'm not quite so worried about most of the raw files. A few I want to work with may be copied to the hard drive, (and enter the regular backup if at home). The others are destined to be archived and "forgotten".

Now I have enough space I have an archive folder on my hard drive. Before I had enough space, I archived the files to my phone, or my wife's iPad while not at home. I got ridiculously large amount of storage for them, specifically so they were available as a backup.

Once I get back home (vacation over etc), then the hard drive gets it's usual backup. That's using "Time Machine", which is regular unattended incremental backup to a "Time Capsule" appliance.

Now Apple no longer makes Time Capsules, I'm wondering what to migrate the Time Machine host to.

The photos library is part of the Time Machine backup, so will get backed up in an hour or two.

The raw archive is excluded from the regular backup. I'll manually copy the archive to an external drive, to make sure there's at least two copies extant.

Once there are enough copies, I'll feel free to recycle the cards, but I don't make any effort to erase them until I need to use one. So there's another potential copy lying around.

The Time Capsule is supposed to be archived every month and the archive stored off site (in my desk draw at work). Due to the pandemic, I'm no longer working off site, so I need to work that bit out, and I've been having trouble with the archiving, probably another good reason to consider a replacement for the Time Capsule. I'll also store a copy of my raw archive offsite as well.

 

[Erik Kaffehr]

Hello everyone,

Hope you are well and safe.

There is a recent thread about photo storage and it got me interested about knowing what everyone else is doing in terms of their backup, whether they are photos or other files. I'm quite paranoid about data backup, ever since I lost about 3TB of data due to a PCI SATA card not being compatible with 3TB drives. Anyway, here is how I do backup and why I do it this way.

01. My first backup starts with the camera. I always use two cards in the camera, configured in backup mode. I use large capacity cards, and I don't format after every shoot. The files stays in the card until the cards are full, and then I use a different set of cards and so on. Once I have all cards used up, I started using the very first set after formatting them in camera, and this is how I keep all cards in rotation. So for a good amount of time, all my raw data have a copy in the cards.

02. After creating LR catalog, I copy the raw files from the card to my primary storage system. I use a LSI enterprise grade PCIe 3.0 8x hardware RAID controller which can take eight hard drives.

I am currently using 8 x 10TB HGST He10 enterprise grade hard drives, and have them configured in a RAID 6 array. These drives are designed to tolerate a workload of 540TB per year, and are built to a much higher standard than standard desktop or NAS drives.

This gives me 54.5 TB of usable capacity in Windows. The RAID controller has some useful features, I have scheduled a monthly Petrol read, which surface scans all the hard drive once a month, and a consistency check function that checks the virtual drive for any inconsistency once a month. The controller has a battery which protects any data in the cache in case of a power failure.

03. I manually copy all the data in a G-Drive USB 3.1 Gen1 RAID enclosure, which has 2 x 10TB (Seagate EXOS enterprise drives) drives in RAID 0, with a usable space of 18.1 TB. I can backup to this drive at a decent speed (around 320MB/s). I backup to this drive on the first day of the month.

I'm thinking of upgrading this to a USB 3.1 Gen2 model which would improve the backup speed, as these drives can go up to 240MB/s on their own. RAID 0 is getting bottle-necked with the enclosure I have.

04. I backup to a single 16TB drive (Seagate EXOS) on the 10th of every month. This is my second backup of the photos.

05. I backup to multiple SSDs (2 x 3.84TB Samsung PM863 enterprise SSDs, 3 x 2TB Samsung 850 EVO, 860 EVO drives). This is my third backup of the photos. I do this backup on the 20th of every month.

The multiple backups acts as backup redundancies, but mostly acts as different versions of the backup. For example, on the 22nd of any month I realized that I did a mistake and deleted something from my hard drive on 15th of that month. And the backup from 20th did not have that file. But luckily I still have the backup from 10th of the month which still has that file. This way I can go back and undo my work up to 30 days in the past.

What is your backup method? I'd very much like to hear about it.

Thank you and stay safe.

Hi,

My strategy is twofold:

On travel, I use moderate size USB cards (64 GB), I don't shoot a lot, like 100-200 images per day. Reason is that I hate sculling lots of near identical images.

On important travel, I would save images to my laptop and do a backup to an attached external drive, possibly with dual SSD disks, while keeping images on my SD-card.

At home, I use a 12 TB RAID 5 attached to USB-3. The setup here is limited by HDD performance. Keep in mind that HDD performance depends on access time, rotation speed and file fragmentation and disk caching strategy.

At home, I would also have an external 12TB RAID 5 for backup, using 'GoodSync'. In Macintosh times, I used to have 'TimeMachine' that was an excellent solution. With windows I feel we have a lot of 'crapware'. Reason I skipped Apple is pricing. I like the software but not the pricing on hardware.

At home I also have a remote backup, using 'Backblaze', it insures against single point failure, like the house burning down.

This backup strategy does not protect against file corruption. Bad files simply migrate to existing backups.

Apple's TimeMachine keeps versions of files, so it may help with file corruption to some extent.

Best regards

Erik

 

[Michael Prince]

I am pretty lazy. I have a couple of disk arrays, and back them up to the cloud with BackBlaze, which works great. I currently have 12 terabytes up there!

What happens when your Internet connection goes down and you need an image that is stored in 'Cloud'? I for one never use the 'Cloud' for anything, and prefer a redundant backup system and RAID array, in order to maintain complete control of my data at all times, and that does not only apply to images. So, in essence, I follow a routine that is similar to Manzur's, and often add portable SSD drives to the equation.

I can download with my iPhone if I have too, or my laptop using my iPhone as a hotspot. It''s great not having to travel with a bunch of drives.

 

[Greg7579]

I have about 4.5 TB of data - every picture I have taken in my live (TIFF scans of all the film and slides, jpegs and mostly raw) on one internal 8TB HDD.

I work off of that on my PC. One drive. No RAID!

I of course don't back up the OS or any programs - only my pictures folder which contains 90,000 images from a lifetime of shooting.

That 8 TB drive is synced to another identical 8 TB drive in my PC. I use GoodSync (thanks Jim). I do that again to a second identical internal 8TB HDD.

Then I do it a 3rd, 4th and 5th time to three external HDDs. One of those is kept off site.

So I operate off of one disk, no RAID, and back it up using Good Sync 5 times.

That simple. I don't care if my main drive crashes. Why? Because Goodsync creates an exact duplicate of that data in the same visible Windows folder structure. I literally could go right to it. I would not really even have to restore.

I will never run RAID! No way. Maybe I would if I had a business and shot everyday for money.

When there are 6 and 8 TB M.2 PCIe NVMe SSDs for less than 400 bucks, I will switch that all to SSD. Can't wait!

 

[RivkiLocker]

Nothing as sophisticated as your specific setup, but some common elements.

I store all my photos (raws) on an external hard drive and backup to a second hard drive (using time machine on my Mac) and to a cloud backup system. (Carbonite). I use a 3TB external drive, and it lasts me a little over a year based on my current photography patterns. I also backup jpegs of my best photos (4 and 5 star) to Apple Photos and Amazon photos. Works well. I lost some files one time (deleted them by mistake) and was able to retrieve them very easily.

 

[Thomas Guenther]

I do it the lazy, but efficient way:

1. Copy pictures from card to computer.

2. Rename pictures on OS (Mac)

3. Import pictures into COne catalogue, including copying the files to their target destination on an external ssd volume into an annual folder.

4. Copy catalogue to its backup destination, besides internal backup from COne.

5. Copy new pictures from ssd drive (working drive) to backup drive (6TB hd).

6. Delete folder ( see 1. ) from computer

7. Format card afterwards in camera.

8. If the working drive (ssd) is full, I go for new one.

Looks like this:







I now only work with a hd dock, where I can swap independent drives to my needs, e. g. the dock is switched off, if I do not do editing. The backup hd is only powered on and connected regarding its backing up purpose. No problems. with any kind of surge or whatever problems in my home grid.

--
some lenses - some bodies

 

[Greg7579]

I get that RAID is still necessary for high-volume pro shooters where you can't be down for an hour and there are hundreds of files added almost daily.

But for you shooters who don't shoot super-high volume every week and have less than 5 TB of total image files? No way RAID!

RAID introduces more risk than it mitigates.

There are super reliable 8 TB HDDs now. Just work off of one and use GoodSync to sync to as many other internal or external drives as you want.

In the unlikely event that the 8TB HDD fails, just immediately switch to another synced drive. Good Sync is not like a backup program that creates coded gibberish on the backup disk where you can't see what is there and there is always a risk on the restore process. Good Sync creates a duplicate of whatever folders you back up. You can go right to it.

All you would have to do is reconnect the LR catalog to the new disk.

RAID is dead. It is obsolete. It is too complex. It has to much risk of not working correctly.

a RAID of 5 disks has more chance of creating problems than using just one disk and backing it up.

I probably shoot 15,000 files a year now max?

RAID? No thanks.

Most of you could live to be 200 years old and never fill an 8TB drive unless you have video.

 

[Stan Disbrow]

Hi,

RAID gains speed via spindle sync. Turns platter drives into a reasonable facsimile of a drum storage unit. All that head switching between head steps. And, far cheaper to run than SSD drives at this point.

I've been running RAID since 1990. Began the first of the 340 MB 3.5" IBM SCSI drives. And I will continue to do so for data drives.

Backup these days is on USB drives. Lots of them. Multiple copies in multiple places.

Stan

 

[Greg7579]

Hi,

RAID gains speed via spindle sync. Turns platter drives into a reasonable facsimile of a drum storage unit. All that head switching between head steps. And, far cheaper to run than SSD drives at this point.

I've been running RAID since 1990. Began the first of the 340 MB 3.5" IBM SCSI drives. And I will continue to do so for data drives.

Backup these days is on USB drives. Lots of them. Multiple copies in multiple places.

Stan

I hear you on the speed, but that will soon not be the case.

RAID is dead.

I could easily run it right now. I have a stack of 3 8TB drives in my PC and could easily add two or three more for cheap. (I have a huge Be Quiet! Dark Base Pro 900 case with 8 HDD rails.)

I think some people run Raid because its cool and part of the hobby. Not you Stan....

But again, RAID of course has its uses, especially for some of the high-end guys on this Board and pro high-volume photographers who can't go down for an hour in the sutdio.

 

[Stan Disbrow]

Hi,

If you want to lose everything, just stick it onto the cloud. It'll all Go Away at some point in time. And then there is the security nightmare. No. Not for me.

Stan

 

[Snap Happy]

1. 2 cards in camera.
2. Ingest to SSD in laptop for editing.
3. Goodsync one-way sync to 4x4TB Synology NAS (running SHR 2 disk redundancy), I trigger this manually after ingest and after edit sessions.
4. Scheduled overnight backup from NAS to Amazon Glacier.

I periodically delete sessions off the laptop once I have finished post-processing. Provided I have synced to NAS, they'll still be available to browse there, and I can copy back if I want to re-edit, and Goodsync takes care of any changes.

 

[Greg7579]

Hi,

If you want to lose everything, just stick it onto the cloud. It'll all Go Away at some point in time. And then there is the security nightmare. No. Not for me.

Stan

I have zero interest at this point for sending any of my raw images up to the cloud as any kind of storage or backup. Storage is cheapo and getting cheaper. And it is 50,000 times faster and better than the cloud.

 

[mplkn]

I have three Network Attached Storage boxes, each with a pair of identical drives operating in RAID 1 configuration.

I have a pair of USB SSD enclosures that each hold two SSDs in a very small package. They are bus powered and configured each with a pair of SSDs in RAID 1 configuration. These function, at times, as the intermediate stopover spot for files from SD cards when I can't get to my NAS boxes.

I have a pair of Western Digital Wi-Fi equipped self powered (rechargable) hard drives each with an SD card reader on board. They can perform a copy based on anything new appearing in the SD card. These drives are usually the first stop for my files on their way from SD card to storage.

I can understand folks' squeamishness about RAID, well, especially configurations that in which a virtual volume is striped across multiple physical volumes.

But RAID 1 is simple mirroring. What goes on one drive goes on both. The drives are interchangable, and can be mounted, as is, as normal drives in a regular system. RAID 1 is simply the drive equivalent of running two SSDs in parallel in the camera. RAID 1 simply automates what smart folks do manually: make redundant copies on separate, independent devices. There's no special formatting. If a drive has to be replaced you just insert the new drive, and the system copies everything from the existing partner, making a new mirror. It's really simple.

By far the greatest long term threat to our images is our reliance on digital storage. When you create a digital image file that you intend to keep, maybe even pass down, you're committing to moving it from drive to drive whenever necessary, probably every few years at least, forever.

Or else it's gone.

I love film negatives and slides. I have negatives that I personally developed over half a century ago. They're chemically stable. With proper storage they will last a very long time. I digitize them to work with them (unless I'm making an analog, chemically based enlargement in my darkroom), but otherwise I leave them undisturbed.

When I have a digital image I want to give an extra measure of protection to, I print it, using pigment-based ink on archival-grade paper. This amounts to a "forward pass" in time. Properly stored, the print will probably last at least a century.

I've worked in university archive and library settings. The value of old images is simply impossible to calculate. Very few are actually worthless in historical terms, many are useful as artifacts, and some are immensely valuable, irreplaceable, really. They are primary source material for historians. We simply can't know, at the time, what an image we make today might mean to someone in the future, if it manages to survive.

All of the backup solutions discussed in this thread are vastly better than the ultimate loss of these images. Most folks' images will be lost.

By the way, the other missing piece here is historical context. Even a date, penciled on the back of an ancient snapshot, is vastly better than a snapshot with no identification. But modern metadata, date, time, exposure info, GPS coordinates, these are a Godsend to future users of the image. To my thinking (wearing my information science hat) the metadata, stored with the image, stored IN the file: this is the real revolution. Any archeologist will tell you historical context is everything. That's why they photograph a pottery shard, in place, before they pull it from the earth. That's why they record what was also found NEAR the shard. It's all for piecing together context. The shard by itself is a piece of pottery. In context it's suddenly from a place and time, a companion to other things from that place and time, and the people living there.

So our job is to help out those future archeologists and historians, to help explain, to folks who will never be here, just what a world we've been living in. The beauty. The madness. We're documenting it, folks, and with some care, and some luck, it'll help folks out a thousand years from now, who otherwise could never believe we actually lived this way.

 

[Steve Bingham]

Hello everyone,

Hope you are well and safe.

There is a recent thread about photo storage and it got me interested about knowing what everyone else is doing in terms of their backup, whether they are photos or other files. I'm quite paranoid about data backup, ever since I lost about 3TB of data due to a PCI SATA card not being compatible with 3TB drives. Anyway, here is how I do backup and why I do it this way.

01. My first backup starts with the camera. I always use two cards in the camera, configured in backup mode. I use large capacity cards, and I don't format after every shoot. The files stays in the card until the cards are full, and then I use a different set of cards and so on. Once I have all cards used up, I started using the very first set after formatting them in camera, and this is how I keep all cards in rotation. So for a good amount of time, all my raw data have a copy in the cards.

02. After creating LR catalog, I copy the raw files from the card to my primary storage system. I use a LSI enterprise grade PCIe 3.0 8x hardware RAID controller which can take eight hard drives.

I am currently using 8 x 10TB HGST He10 enterprise grade hard drives, and have them configured in a RAID 6 array. These drives are designed to tolerate a workload of 540TB per year, and are built to a much higher standard than standard desktop or NAS drives.

This gives me 54.5 TB of usable capacity in Windows. The RAID controller has some useful features, I have scheduled a monthly Petrol read, which surface scans all the hard drive once a month, and a consistency check function that checks the virtual drive for any inconsistency once a month. The controller has a battery which protects any data in the cache in case of a power failure.

03. I manually copy all the data in a G-Drive USB 3.1 Gen1 RAID enclosure, which has 2 x 10TB (Seagate EXOS enterprise drives) drives in RAID 0, with a usable space of 18.1 TB. I can backup to this drive at a decent speed (around 320MB/s). I backup to this drive on the first day of the month.

I'm thinking of upgrading this to a USB 3.1 Gen2 model which would improve the backup speed, as these drives can go up to 240MB/s on their own. RAID 0 is getting bottle-necked with the enclosure I have.

04. I backup to a single 16TB drive

This is what I use except I don't have MF.

(Seagate EXOS) on the 10th of every month. This is my second backup of the photos.

05. I backup to multiple SSDs (2 x 3.84TB Samsung PM863 enterprise SSDs, 3 x 2TB Samsung 850 EVO, 860 EVO drives). This is my third backup of the photos. I do this backup on the 20th of every month.

The multiple backups acts as backup redundancies, but mostly acts as different versions of the backup. For example, on the 22nd of any month I realized that I did a mistake and deleted something from my hard drive on 15th of that month. And the backup from 20th did not have that file. But luckily I still have the backup from 10th of the month which still has that file. This way I can go back and undo my work up to 30 days in the past.

What is your backup method? I'd very much like to hear about it.

Thank you and stay safe.