Long-Term Image File Storage Choices

Chaplain Mark · Feb 3, 2019

Having read the above replies, I'm leaning toward rendundant external HDD's.

Thanks very much, all who replied...!!

--
Chaplain Mark
-----
'Tis better to have a camera and not need one than to need a camera and not have one.
--------------
In pursuit of photographic excellence.

gipper51 · Feb 3, 2019

Numerous options as listed here, but probably the simplest and cheapest method to prevent data loss for a typical home user is a basic 3 drive backup plan:

-Main storage drive with all files. Should not be a C: drive.

-On site backup of main drive (usually an external drive, unplugged when not in use in case of catastrophic failure like power surges or hacks).

-Off site backup of main drive. Stored off site in case of fire, natural disaster, theft, etc. at home.

This is easy to manage for a typical home user, say somebody with less than 14TB of data. If you have more than 14TB then yeah it gets complicated and expensive since it exceeds today's single HDD capacities. But in the age of 4TB HDDs costing under $150, and 8TB under $250 you can easily and cheaply store lots of data with minimal risk of losing it... assuming you are diligent with the backups.

Johann Kruger · Feb 3, 2019

We copy and verify from legacy file systems onto ZFS hard drives and removable tapes to avoid Bit Rot aka Silent Data Corruption. Thanks.

jfriend00 · Feb 3, 2019

gipper51 said:
Numerous options as listed here, but probably the simplest and cheapest method to prevent data loss for a typical home user is a basic 3 drive backup plan:

-Main storage drive with all files. Should not be a C: drive.

-On site backup of main drive (usually an external drive, unplugged when not in use in case of catastrophic failure like power surges or hacks).

-Off site backup of main drive. Stored off site in case of fire, natural disaster, theft, etc. at home.

This is easy to manage for a typical home user, say somebody with less than 14TB of data. If you have more than 14TB then yeah it gets complicated and expensive since it exceeds today's single HDD capacities. But in the age of 4TB HDDs costing under $150, and 8TB under $250 you can easily and cheaply store lots of data with minimal risk of losing it... assuming you are diligent with the backups.

This is a good plan. I would suggest that using an online backup service such as BackBlaze (or other similar services) is the simpler and best option for the off-site backup. The two things that make it best are 1) someone else manages the aging of hard drives an 2) it's never out-of-date because you procrastinated updating the off-site drive.

In all cases, you will also have to implement a drive replacement plan where you decide how old you let a drive get before you replace it. With seldom used backup drives, you can't "wait" until you see warning signs of deterioration. By then, it's likely too late. You have to be more proactive than that by designating a time period for replacement and sticking to that.

Chaplain Mark · Feb 3, 2019

gipper51 said:
Numerous options as listed here, but probably the simplest and cheapest method to prevent data loss for a typical home user is a basic 3 drive backup plan:

-Main storage drive with all files. Should not be a C: drive.

-On site backup of main drive (usually an external drive, unplugged when not in use in case of catastrophic failure like power surges or hacks).

-Off site backup of main drive. Stored off site in case of fire, natural disaster, theft, etc. at home.

This is easy to manage for a typical home user, say somebody with less than 14TB of data. If you have more than 14TB then yeah it gets complicated and expensive since it exceeds today's single HDD capacities. But in the age of 4TB HDDs costing under $150, and 8TB under $250 you can easily and cheaply store lots of data with minimal risk of losing it... assuming you are diligent with the backups.

Janoch · Feb 3, 2019

jfriend00 said:
Janoch said:

Chaplain Mark said:

Janoch said:

I use an USB3 HD dock and have a handful of drives for external storage.

For convenience also use an internal dedicated backup drive... is mirrored regularly to one of the externals.

Photos (raw and projects) are copied to 3 drives... represents lots of hours! :-D

Discs will not be kept for more than 3 to 5 years, then they'll be replaced.

And I have no confidence in any kind of flash drives btw....

Click to expand...

Does an SD card fall into the flash drive category?

Click to expand...

Yes. Worthless! :-D

Just remembered, I have a "small" disc with a bit of music on, a Seagate Barracuda 7200.7 160Gb... that thing is close to being 15 years old and runs perfectly fine without any problems.

Click to expand...

Just because it still runs after 15 years doesn't mean you're playing with the best odds and that's a smart choice for backup.

Of course not.

That's why it only holds a music collection of no value.

I only mentioned it because Mark was looking for a lifespan of 5-10 years... which is no problem at all for "good" HDDs...

tkbslc · Feb 3, 2019

Janoch said:
jfriend00 said:

Janoch said:

Chaplain Mark said:

Janoch said:

I use an USB3 HD dock and have a handful of drives for external storage.

For convenience also use an internal dedicated backup drive... is mirrored regularly to one of the externals.

Photos (raw and projects) are copied to 3 drives... represents lots of hours! :-D

Discs will not be kept for more than 3 to 5 years, then they'll be replaced.

And I have no confidence in any kind of flash drives btw....

Click to expand...

Does an SD card fall into the flash drive category?

Click to expand...

Yes. Worthless! :-D

Just remembered, I have a "small" disc with a bit of music on, a Seagate Barracuda 7200.7 160Gb... that thing is close to being 15 years old and runs perfectly fine without any problems.

Click to expand...

Just because it still runs after 15 years doesn't mean you're playing with the best odds and that's a smart choice for backup.

Click to expand...

Of course not.

That's why it only holds a music collection of no value.

I only mentioned it because Mark was looking for a lifespan of 5-10 years... which is no problem at all for "good" HDDs...

The problem is predictability. It's basically a coin toss whether a 5+ year old hard drive is going to power on again if it has been left sitting.

Janoch · Feb 3, 2019

Well... I've only been in the game for some +30 years, so what do I know? :-D

A docking station with "loose" discs is still the most reliable and payable solution available.

And they are not sitting still, but kept on a rotation scheme for updates. Any problems will most probably be detected at this point and you do not keep a single baclup only.

So far I've never had a disc failing.... only mounted ones due to lightning strikes. That event quickly ruled out NAS btw.... the odds for two discs failing is very, very small.

There are things I fear way more in this life! ;-)

gipper51 · Feb 3, 2019

jfriend00 said:
gipper51 said:

Numerous options as listed here, but probably the simplest and cheapest method to prevent data loss for a typical home user is a basic 3 drive backup plan:

-Main storage drive with all files. Should not be a C: drive.

-On site backup of main drive (usually an external drive, unplugged when not in use in case of catastrophic failure like power surges or hacks).

-Off site backup of main drive. Stored off site in case of fire, natural disaster, theft, etc. at home.

This is easy to manage for a typical home user, say somebody with less than 14TB of data. If you have more than 14TB then yeah it gets complicated and expensive since it exceeds today's single HDD capacities. But in the age of 4TB HDDs costing under $150, and 8TB under $250 you can easily and cheaply store lots of data with minimal risk of losing it... assuming you are diligent with the backups.

Click to expand...

This is a good plan. I would suggest that using an online backup service such as BackBlaze (or other similar services) is the simpler and best option for the off-site backup. The two things that make it best are 1) someone else manages the aging of hard drives an 2) it's never out-of-date because you procrastinated updating the off-site drive.

I've never considered an online service but maybe I should. Part of me still says I don't want my data on somebody else's servers, but I know that's old thinking.

jfriend00 said:
In all cases, you will also have to implement a drive replacement plan where you decide how old you let a drive get before you replace it. With seldom used backup drives, you can't "wait" until you see warning signs of deterioration. By then, it's likely too late. You have to be more proactive than that by designating a time period for replacement and sticking to that.

In my own use, HDD age has never been a concern because I tend to not go crazy on the storage size when I buy drives. I'd rather not pay a premium now for space I won't need until a few years down the road. And when I do need it, that same size will be much cheaper by then.

My 3.5" HDDs usually are in use 5-6 years. My main drive is always the largest capacity, and my external backups are a combination of my former main drive that gets demoted to an external enclosure as my on site backup, and a 2.5" HDD for my offsite.

I like to upgrade my main drive when it hits about 50% capacity, my backups are getting squeezed by then. I recently upgraded my main drive from 3TB to 5TB, and my backups are now 3TB (former main) and a 2TB USB 2.5". My total data backup is about 1.4TB. I'll upgrade the 2.5" drive to a 4TB model soon and be good for a few years for that drive. Once my data gets near 3TB, I'll upgrade the main drive again and shuffle the old one down. At my rate it will take a few years to get to 3TB of data.

I've had good luck with HDDs serving for 5+ years between main and backup duty. Not too concerned if a backup drive fails because of age since I'll have two more copies.

--
My site:
http://www.gipperich-photography.com

PHXAZCRAIG · Feb 3, 2019

I'd like to see a fast tape. Would be something if faster than the disk arrays from Barracuda for instance.

For the average joe, I just don't know. It's a new medium, and it hasn't had very many decades of refinement. I just think having all my pics on my QNAP will be as good as anything for my daughters to inherit. They will have to figure out how to pass it on, should anyone care. I do have some family photos from my grandfather going back to 1940. Seems worth keeping around, though I still have the slides they came from. If the slides haven't rotted away in the meantime.

Optical disks eventually corrode, except maybe those early gold cd's. (Remember $15 cd blanks?) Hard drives fail. Disk standards change and readers go extinct. It's a real mess.

tkbslc · Feb 3, 2019

PHXAZCRAIG said:
I'd like to see a fast tape. Would be something if faster than the disk arrays from Barracuda for instance.

Since we were talking large business, the newest revisions of LTO/Ultrium can do 300-360MB/s uncompressed. Given that you typically see these in robotic libraries with 4-8 drives, each with their own dedicated fibre channel or SAS link, that is 1.4-2.8TB/s. If you enable compression, you can go faster, but it's unpredictable. At those speeds, the tape is usually not even the bottleneck.

Not reasonable for home or small business use, obviously.

a_c_skinner · Feb 4, 2019

For a basic user the answer is actually very easy:

Two external HDDs, ideally three or more, all with the same files on them. Preferably buy different types of disk so if there is a product wide defect you will not lose all of the data.

Only have one disk connected for routine use, only connect others when needed and ideally keep one in another building.

kelpdiver · Feb 4, 2019

Janoch said:
Well... I've only been in the game for some +30 years, so what do I know? :-D

Well, tell us, what do you know? 5 years is the service life for an enterprise drive. Coin flip is overstating the failure p, but it's sufficiently non zero that you should be moving onward.

And after 30 years, surely you've seen the bad models, with failure rates well outside the normal couple percentage points per year.

if you're going to make an appeal to expertise, cite some data.

Janoch said:
So far I've never had a disc failing.... only mounted ones due to lightning strikes. That event quickly ruled out NAS btw.... the odds for two discs failing is very, very small.

The failure p is not a random event, as proven by unrecoverable R5 failures. If you bought the drives as same time (iow, same lot and one fails, the odds for another failure are markedly higher.

kiriak · Feb 4, 2019

gipper51 said:
Numerous options as listed here, but probably the simplest and cheapest method to prevent data loss for a typical home user is a basic 3 drive backup plan:

-Main storage drive with all files. Should not be a C: drive.

-On site backup of main drive (usually an external drive, unplugged when not in use in case of catastrophic failure like power surges or hacks).

-Off site backup of main drive. Stored off site in case of fire, natural disaster, theft, etc. at home.

This is easy to manage for a typical home user, say somebody with less than 14TB of data. If you have more than 14TB then yeah it gets complicated and expensive since it exceeds today's single HDD capacities. But in the age of 4TB HDDs costing under $150, and 8TB under $250 you can easily and cheaply store lots of data with minimal risk of losing it... assuming you are diligent with the backups.

I do more or less the same

1) critical data in my pc's SSD (encrypted in case it will be stolen)

2) the above data (backed up) and everything else on the NAS (one bay, i do not mess with video or movies) - personal data (bank accounts, passwords etc) again on encrypted shares

3) All data of the NAS backed up (every couple of weeks) on 2 external HDDs , that are rotated, so every time one HDD backup is outside home, in the paranoid case sth happens when I have the HDD in my home. I keep these in my work's office in another location in the city (encrypted of course). These backups have also file history, in case I delete sth by accident or get corrupted.

important: check data consistency of the backups every few months

When I'll become more familiar with networks and security, I'll consider having my old NAS, in my mother's home in another town and having the offline backup there (or in my sister's home on the other side of the continent) made automatically over the net, after making the initial heavy work locally.

Another thing I'm looking is in filesystems with better or more transparent integrity checks, this is what worries me mostly, silent file corruption that will spread to the backups.

Sean Nelson · Feb 4, 2019

Chaplain Mark said:
Which media would be more reliable for long-term storage of my photo, SD card, or external HDD?

If you value your photos, then you need to do two things regardless of which media you choose:

You need to store at least two copies on different media, and ideally keep those media in different physical locations.
You need to create checksums of all your files and then use them to verify the data from time to time so that if there are any problems you can recover your photos from another copy.

This is true whether you're storing your photos on hard drives, SD cards, optical discs, tape(!), etc. etc. Every storage media can degrade over time.

I personally wouldn't advise the use of flash media (SD cards, USB flash drives, or SSDs) for long-term storage because the static charges that hold the data in this type of memory degrade over time - unless it's always kept on-line so that the controller can detect and refresh degrading memory pages.

IMHO, because of the need to periodically verify your files, hard drives are the most sensible choice. They have high capacity, low cost per GB, and you can plug one in and let it grind away verifying files for as long as it takes without having to constantly swap out different media.

Fazal Majid · Feb 4, 2019

NAND flash memory is not archival, it only last for a couple years of cold storage. You need hard drives, optical or tape for archival purposes.

Johann Kruger · Feb 4, 2019

Optical and Magnetic media easily suffer from bit degradation, a.k.a bit rot, a.k.a. silent data corruption. Use ZFS files to guarantee your data remains exactly the same as you copied and verified. Otherwise some of your images could soon be short only showing a fraction of the total, others will have huge colorful streaks, big white blotches, sprinkles, and snow.

a_c_skinner · Feb 4, 2019

It is really difficult to get a good estimate but reading the Wiki article on silent data loss it seems to me that in normal (life time) storage the chances are somewhere between 1 in 10^6 and 1 in 10^9 files will be lost. That means if you have two independent copies the chances of losing a file are very small indeed (say 1 in 10^12).

I'd be really grateful if someone can point me to a better estimate or perhaps I should say one I can understand properly.

--
Andrew Skinner

kiriak · Feb 4, 2019

Johann Kruger said:
Optical and Magnetic media easily suffer from bit degradation, a.k.a bit rot, a.k.a. silent data corruption. Use ZFS files to guarantee your data remains exactly the same as you copied and verified. Otherwise some of your images could soon be short only showing a fraction of the total, others will have huge colorful streaks, big white blotches, sprinkles, and snow.

for my workflow,

in which part should I try to implement ZFS ?

1. in my pc (dual boot win10/ Fedora Linux)

2. NAS (synology)

3. external HDDs used as off-site backups

can this be done or is meaningful only in step 3?

what about other filesystems as Btrfs ?

William Carson · Feb 4, 2019

a_c_skinner said:
It is really difficult to get a good estimate but reading the Wiki article on silent data loss it seems to me that in normal (life time) storage the chances are somewhere between 1 in 10^6 and 1 in 10^9 files will be lost. That means if you have two independent copies the chances of losing a file are very small indeed (say 1 in 10^12).

I'd be really grateful if someone can point me to a better estimate or perhaps I should say one I can understand properly.

It is surprising the lack of support for the M disk storage option. All the other types of digital storage are acknowledged to last no more -at a stretch- of 20-30 years. ( I still have a archival CD burned in the 1980s that is readable) Yet testing of M disks will hold data for past 100 years. Yet, everyone discounts this option. It comes down to what is the purpose of storage? If it is truly 'archival' and expected to be accessible in 50-100 years of basement museum type storage, then HHD, SSD, Tape, is out. M disks will -at this stage of technology- be the only option. If the data is just expected to be needed to be there for 5-10 years, HD is better than SSD or flash. For really non critical quickly changing data that is continually used (refreshed) SSD is anyones best choice because of ease of use and speed.

Long-Term Image File Storage Choices

Veteran Member

Veteran Member

Senior Member

Forum Pro

Veteran Member

Senior Member

Forum Pro

Senior Member

Veteran Member

Forum Pro

Forum Pro

Forum Pro

Veteran Member

Active member

Forum Pro

Senior Member

Senior Member

Forum Pro

Active member

Veteran Member

Similar threads

Keyboard shortcuts