What are your archival & backup policies?

Started May 11, 2012 | Discussions
Lights
Veteran MemberPosts: 3,184Gear list
Like?
Re: What are your archival & backup policies?
In reply to szlevi, May 12, 2012

HD+DVD for Raw. JPG's to another HD just in case. Keeper jpgs to another DVD just in case, but usually only after getting enough to fill.

I do realize DVD's and CD's are corruptible - but haven't had much issue with either them or 10 year old CD's if careful of initial quality.
When my HD's fail load the files back

I even print 4x6's of my family or important memories, just in case...and it's really cheap.

Did lose a month's shots last year because of being too lazy to back up and a HD's MFT went out, but saved most with PC inspector.
--

http://www.pbase.com/madlights
http://barriolson.aminus3.com/

Reply   Reply with quote   Complain
Klarno
Senior MemberPosts: 2,908Gear list
Like?
Re: What are your archival & backup policies?
In reply to szlevi, May 12, 2012

All of my new photographs are stored simultaneously on my computer and on a time machine backup drive. Very easy, makes it so I don't have to worry about it.

When I need to make space on my hard drive, I move them over to an external drive, which is mirrored against another. I use a scheduled Carbon Copy Cloner task to keep everything synchronized, such as when I make any changes in my older photo libraries.

I should be able to easily scale this procedure up to incorporate more drives for more redundancy.

-- hide signature --
 Klarno's gear list:Klarno's gear list
Olympus PEN E-PL1 Panasonic Lumix DMC-GH2 Olympus E-M1 Olympus Zuiko Digital 14-54mm 1:2.8-3.5 II Olympus Zuiko Digital 11-22mm 1:2.8-3.5 +4 more
Reply   Reply with quote   Complain
Jim Holloman
Regular MemberPosts: 140
Like?
Re: Carbonite can also fail
In reply to Optical1, May 12, 2012
Reply   Reply with quote   Complain
Guy Parsons
Forum ProPosts: 17,602Gear list
Like?
Re: Long, not for attention span deficient readers.
In reply to pnieuwla, May 12, 2012

pnieuwla wrote:

Firstly, no reliance on any "cloud" or online storage, it's all here at home somewhere in the home network.

And what if, unfortunately, your home would be struck by a disaster (such as an alien attack) or 'just' burn down?

In the end nothing really matters, so why worry?

But as said elsewhere, the house is safe, no incidents (touch wood) in 42 years, plus we have numerous backups that do go with us or out of the house when it is vacant for more than a day.

I trust my methods about 100 times more than I would trust any "cloud" solution, the wind blows and the clouds evaporate!

If aliens attack then it might be worthwhile using their backups on Mars or wherever, just in case Earth blows up.

Regards............. Guy

 Guy Parsons's gear list:Guy Parsons's gear list
Olympus PEN E-PL5 Panasonic Lumix G Vario 45-150mm F4-5.6 ASPH Mega OIS Olympus 12-40mm F2.8 Panasonic Lumix DMC-LX3 Casio Exilim EX-ZR1000 +15 more
Reply   Reply with quote   Complain
Jim Holloman
Regular MemberPosts: 140
Like?
Re: External eSATA dock
In reply to szlevi, May 12, 2012

I use the Communicator eSATA docking station and 2 2TB hard disk drives. New images are copied to BOTH drives with directory names in format YYMMDD_Subject. I then delete the images from my computer. The station has eSATA and USB 2 ports.

It can also clone a disk without being attached to a computer -- but takes 12 hours to clone a 2TB drive (be careful which hole you plug each drive into -- it can also clone an empty drive to a drive full of data -- my clones front to rear). Purchased the 2 WD Green drives for $100 each. They are now up $120 each. The station works with PC (my choice) and Macs.

If you want an offsite copy, then buy three drives and rotate the offsite drive.

Thru Amazon ~$60:
Dual HDD Docking w/Sim/Flash Card Reader/USB Hub

http://www.amazon.com/Dual-Docking-Flash-Card-Reader/dp/B003L139KS/ref=sr_1_sc_3?s=electronics&ie=UTF8&qid=1336866468&sr=1-3-spell

http://www.camera2000.com/en/dual-2-5-3-5-sata-hdd-docking-station-usb-esata-backup.html

http://www.youtube.com/watch?v=JGqtGkVS6Sk

A similar docking station at Amazon:
http://www.amazon.com/StarTech-com-eSATA-Drive-Docking-Station/dp/B004I6OCRO

Reply   Reply with quote   Complain
Promit
Contributing MemberPosts: 890Gear list
Like?
Re: What are your archival & backup policies?
In reply to szlevi, May 13, 2012

Backup to a RAID 1 NAS box at home, and everything also goes to a large storage account I have with SpiderOak. I don't consider any of the other methods mentioned -- optical discs, prints, etc to be vaguely credible methods of storage. Google Picasa also has a substantial subset of images for sharing, as does Facebook, but neither of those are proper backup solutions.

 Promit's gear list:Promit's gear list
Sony SLT-A77 Olympus OM-D E-M5 Panasonic Lumix DMC-GH3 Panasonic Lumix G 20mm F1.7 ASPH Panasonic Lumix G Vario HD 14-140mm F4-5.8 OIS +11 more
Reply   Reply with quote   Complain
Mark B UK
Contributing MemberPosts: 905Gear list
Like?
Re: What are your archival & backup policies?
In reply to Promit, May 13, 2012

I shoot everything RAW, which I store in Aperture on a 2TB iMac that backs up to a similar-capacity Time Machine. For additional security the Mac is in my home office (a building in my garden) and the Time Machine is in my house, IMO the simplest and most secure backup and file management system available.

 Mark B UK's gear list:Mark B UK's gear list
Olympus OM-D E-M5 Olympus M.Zuiko Digital ED 12mm 1:2 Olympus M.Zuiko Digital ED 60mm 1:2.8 Macro Panasonic 12-35mm F2.8 Panasonic Lumix G X Vario 35-100mm F2.8 OIS
Reply   Reply with quote   Complain
Jeff
Veteran MemberPosts: 4,470
Like?
Get a copy of the DAM book, pronto ...
In reply to szlevi, May 13, 2012

Do yourself a favor and get a copy of the DAM book http://www.thedambook.com/ , and check out their web site while your at it. It was the best money I ever spent on photography.

Reply   Reply with quote   Complain
Jeff
Veteran MemberPosts: 4,470
Like?
About keywords ....
In reply to Guy Parsons, May 13, 2012

Lot's of good advice here, but respectfully I have to disagree about using keywords.

Like you, I use sequential folders for a base level organization of files on disk. But at best it's one dimensional by time. It's also redundant since time is embedded in the file's meta data.

Renaming with unique file names, then embedding keywords and other metadata in the image files is the key (imho) to efficiently maintaining a large set of images. My own strategy is ---

a. Be sure camera's clock is set accurately. If using two cams, be sure they're synced.

b. On uploading, rename images using date/time/serial number. For example, 20120513_080312. LR can use metadata for batch renaming. The idea is to be unique, and sortable. I also add copyright, etc. I happen to use LR, but there a lot of tools for batch uploading and renaming. Files are in folders organized by month.

c. I take a few minutes to flag files for deletion, add keywords -- especially people -- and some location info. That's usually enough, and only takes a few minutes. A while back I bought the controlled vocabulary keyword set, which makes the job easy and organized.

d. I use 0-5 stars for rating files, labels, collections, and smart collections in LR to manage the library. I've got 60K images, and this works quite well.

e. Store the metadata with the images. That's a simple insurance policy against catalog corruption. I've rebuilt the catalog from metadata at least twice over the last 5 years or so.

My 2 cents,
Jeff

Reply   Reply with quote   Complain
szlevi
Contributing MemberPosts: 932Gear list
Like?
Excellent posts...
In reply to szlevi, May 16, 2012

...thank you.

I did a test run with Adobe's DNG Converter and the size difference, even with all things embedded (full RAW, full JPEG etc), is apparently around 6-8% extra so I might just go down the DNG road.

As for cloud backup anyone dismissing it is making a huge mistake - having geographically dispersed backups is the key element for any valid DR scenario (part of my job is to design & deploy such systems.)

Also Google does not own your intellectual property, never will - it's just a misunderstanding created by some clueless journalists who didn't understand that in order to Google be able to move your data around the globe between their DCs they need to have certain legal rights to your data (eg to avoid stupid lawsuits.)

Anyone thinking Google would ever willingly abuse your data ie use your images without your consent is dreaming: that would immediately destroy Google's reputation thus their entire business model of relying on anonymous data mining while providing ads and services and they would earn nothing in return.

szlevi wrote:

Since my Canon age has come to an end I decided to tackle this issue while I'm waiting for my OM-D... I only have 8-10k shots so it won't take long but not sure about the format: DNG or DNG w/ embedded RAW or DNG w/ embedded RAW + JPEG preview?

Or I should just forget DNG and keep archiving my RAW shots the usual way (a ReadyNAS Pro 6 at home + an unlimited online backup account & full-size JPEGs on Picasa as final backup replicas in case of some nationwide disaster) and don't worry about future software support...?

How do you archive your shots?

-- hide signature --

http://instagr.am/p/JvSMWFBYyl
(Now you cannot say I don't have a single pic! :P)

-- hide signature --

http://instagr.am/p/JvSMWFBYyl
(Now you cannot say I don't have a single pic! :P)

 szlevi's gear list:szlevi's gear list
Olympus OM-D E-M5 Olympus M.Zuiko Digital ED 40-150mm 1:4-5.6 R Olympus M.Zuiko Digital 45mm 1:1.8 Olympus M.Zuiko Digital ED 12-50mm 1:3.5-6.3 EZ Adobe Photoshop Lightroom 4 +7 more
Reply   Reply with quote   Complain
Florida45
Regular MemberPosts: 179
Like?
Re: Backup: Cloudless to Portable HD in Safe Deposit Box
In reply to pnieuwla, May 16, 2012

I keep everything in folders labeled by date and subject on my desktop PC's hard drive. Then once a year or so I copy everything (including tax records, etc) to a portable hard drive that resides in a free Safe Deposit box at the bank.

Reply   Reply with quote   Complain
Jeff
Veteran MemberPosts: 4,470
Like?
Cloud ....
In reply to szlevi, May 16, 2012

szlevi wrote:

...thank you.

I did a test run with Adobe's DNG Converter and the size difference, even with all things embedded (full RAW, full JPEG etc), is apparently around 6-8% extra so I might just go down the DNG road.

As for cloud backup anyone dismissing it is making a huge mistake - having geographically dispersed backups is the key element for any valid DR scenario (part of my job is to design & deploy such systems.)

This is an excellent point, and I agree that cloud storage will continue to grow exponentially. The trouble is that at this point large collections -- say 500G+ of data -- take much time to sync, and prices are still high.

On the other hand, off-site backup a week or so out of date still provides good coverage. What I do is maintain a working drive, and then use rsync to maintain a pair of backups that occasionally rotate between home and work. This works pretty well. Hopefully cloud storage will continue to drop in price so that we get to a point where its a good choice for large collections.

Also Google does not own your intellectual property, never will - it's just a misunderstanding created by some clueless journalists who didn't understand that in order to Google be able to move your data around the globe between their DCs they need to have certain legal rights to your data (eg to avoid stupid lawsuits.)

Anyone thinking Google would ever willingly abuse your data ie use your images without your consent is dreaming: that would immediately destroy Google's reputation thus their entire business model of relying on anonymous data mining while providing ads and services and they would earn nothing in return.

szlevi wrote:

Since my Canon age has come to an end I decided to tackle this issue while I'm waiting for my OM-D... I only have 8-10k shots so it won't take long but not sure about the format: DNG or DNG w/ embedded RAW or DNG w/ embedded RAW + JPEG preview?

Or I should just forget DNG and keep archiving my RAW shots the usual way (a ReadyNAS Pro 6 at home + an unlimited online backup account & full-size JPEGs on Picasa as final backup replicas in case of some nationwide disaster) and don't worry about future software support...?

How do you archive your shots?

-- hide signature --

http://instagr.am/p/JvSMWFBYyl
(Now you cannot say I don't have a single pic! :P)

-- hide signature --

http://instagr.am/p/JvSMWFBYyl
(Now you cannot say I don't have a single pic! :P)

Reply   Reply with quote   Complain
szlevi
Contributing MemberPosts: 932Gear list
Like?
Re: Backup: Cloudless to Portable HD in Safe Deposit Box
In reply to Florida45, May 16, 2012

That's nice except if a storm or an earthquake wipes out your area - then even if the deposit survives and your drive is found it will be most likely dead from a mechanical standpoint due to the drops it suffered when the bank collapsed.

Trust me, there's no way around mirroring/transporting copies of your files across vast distances. At work I set up could-based backup 2x a day for our active projects and it's mirrored to the Midwest (we are in Manhattan) so we can have anything from a server room fire to an earthquake or a terrorist attack we will still only lose a day's work in worst case (we also run full weeklies overnight to tape.)

Florida45 wrote:

I keep everything in folders labeled by date and subject on my desktop PC's hard drive. Then once a year or so I copy everything (including tax records, etc) to a portable hard drive that resides in a free Safe Deposit box at the bank.

-- hide signature --

http://instagr.am/p/JvSMWFBYyl
(Now you cannot say I don't have a single pic! :P)

 szlevi's gear list:szlevi's gear list
Olympus OM-D E-M5 Olympus M.Zuiko Digital ED 40-150mm 1:4-5.6 R Olympus M.Zuiko Digital 45mm 1:1.8 Olympus M.Zuiko Digital ED 12-50mm 1:3.5-6.3 EZ Adobe Photoshop Lightroom 4 +7 more
Reply   Reply with quote   Complain
szlevi
Contributing MemberPosts: 932Gear list
Like?
Re: Cloud - another LOOOOOONG reply...
In reply to Jeff, May 17, 2012

Jeff,

here's your cheap cloud: http://www.backblaze.com/online-backup-performance.html

For $50/year or $95/2-y (they have monthly plans too) you get completely unlimited backup space , with unlimited file size as well. Yes, the initial backup will take weeks - so what? It runs in the background, you can throttle it, exclude files etc. Yes, they probably definitely do not have Google's geo-redundant backend but hey, then just get a 1TB Google Drive for $50/month (or team up with someone and split the space & cost under a common account) for your most valuable data.

FWIW I chose a different, more granular path - I call it a multilevel approach, both in terms of money and features...

1. Geo-redundant level : Basically it's a collection of cheap, redundant but limited services, relying on the world's biggest information service providers. Originals are not stored at this level solely due to costs.

a) For pictures I'm grandfathered into Google's previously available $20/year 20GB storage plan (they don't offer annual plans anymore) which allows me to store all my shots in original-sized JPEGs & YT-quality videos in Picasa plus some growth. This is my golden insurance in case everything else fails.

b) For my music I use Amazon Cloud Drive's 20GB for $20/year plan w/ unlimited MP3 storage & streaming (latter one is pretty awesome); I also store a second set of JPGs here so this will likely to go up to 50GB for $50/year after I got my OM-D. Another golden insurance policy.

2. Enterprise-grade redundancy w/ (likely) single-location level:   These services, while lacking Google's or Amazon's global reach, offer very affordable unlimited or quasi-unlimited services which allows me to upload and store all my originals here.

a) For my videos (up to 1080p) I use Vimeo now, it's $59 a year and the only limitation is 500GB/week upload which is more than fine for me especially that they also store the originals I upload. (Previously I used the truly unlimited Motionbox but it was sold to HP. :() It's also great for sharing videos in nice quality vs YT, comes with unlimited HD embedding etc.

b) For all files I deem somewhat important I use Backblaze, a truly unlimited online backup service. This also means it holds another copy of all my originals - details about their DC, how they run it etc: http://blog.backblaze.com/

3. Highly fault-tolerant but single-server, single/local level:   This is the primary location of all my files, at home.

By default everything sits on my ReadyNAS Pro 6 at home, currently running at 4TB in RAID6, holding all my original files, music in FLAC etc. It's connected to an IBM-branded APC Smart-UPS 1500VA USB (daily battery self-test runs at 4AM.) I have a spare drive in case I need to replace a failed one and, of course, I keep all images and videos on a HDD on my desktop as well.

Summary

In other words my full annual cost is $150 as of today (might be $180 by next year if I have to upgrade my Amazon Cloud Drive.)

In return I enjoy multiple layers of redundant copies including geographically redundant services for compressed but high quality (web-, even print-ready) formats.

To lose any of my data completely would require of not two but three or more independently stored copies to be destroyed simultaneously eg for my music it means fire at home, data loss at Backblaze and Amazon deletes my entire MP3 collection accidentally but for my pictures it would require fire at home, data loss at Backblaze (originals gone) plus both Amazon and Google suffering some unimaginable data loss where both loses all my JPEGs etc... highly unlikely if you ask me.

Jeff wrote:

szlevi wrote:

...thank you.

I did a test run with Adobe's DNG Converter and the size difference, even with all things embedded (full RAW, full JPEG etc), is apparently around 6-8% extra so I might just go down the DNG road.

As for cloud backup anyone dismissing it is making a huge mistake - having geographically dispersed backups is the key element for any valid DR scenario (part of my job is to design & deploy such systems.)

This is an excellent point, and I agree that cloud storage will continue to grow exponentially. The trouble is that at this point large collections -- say 500G+ of data -- take much time to sync, and prices are still high.

On the other hand, off-site backup a week or so out of date still provides good coverage. What I do is maintain a working drive, and then use rsync to maintain a pair of backups that occasionally rotate between home and work. This works pretty well. Hopefully cloud storage will continue to drop in price so that we get to a point where its a good choice for large collections.

Also Google does not own your intellectual property, never will - it's just a misunderstanding created by some clueless journalists who didn't understand that in order to Google be able to move your data around the globe between their DCs they need to have certain legal rights to your data (eg to avoid stupid lawsuits.)

Anyone thinking Google would ever willingly abuse your data ie use your images without your consent is dreaming: that would immediately destroy Google's reputation thus their entire business model of relying on anonymous data mining while providing ads and services and they would earn nothing in return.

-- hide signature --

http://instagr.am/p/JvSMWFBYyl
(Now you cannot say I don't have a single pic! :P)

-- hide signature --

http://instagr.am/p/JvSMWFBYyl
(Now you cannot say I don't have a single pic! :P)

 szlevi's gear list:szlevi's gear list
Olympus OM-D E-M5 Olympus M.Zuiko Digital ED 40-150mm 1:4-5.6 R Olympus M.Zuiko Digital 45mm 1:1.8 Olympus M.Zuiko Digital ED 12-50mm 1:3.5-6.3 EZ Adobe Photoshop Lightroom 4 +7 more
Reply   Reply with quote   Complain
Jeff
Veteran MemberPosts: 4,470
Like?
Looks intriguing ...
In reply to szlevi, May 18, 2012

Thanks for pointing out Backblaze. I didn't know about them, and they certainly are offering a great deal. Need to try them out.

Reply   Reply with quote   Complain
Keyboard shortcuts:
FForum MMy threads