Bit rot

Started 2 months ago | Discussions
Barugon Veteran Member • Posts: 9,064
Bit rot
1

I was reading Fstoppers and came across this article about bit rot. The author has the notion that his images, once written to a digital medium, will degrade over time. He has some pretty big misconceptions here.

It's true that all digital storage can only be written to some (very large) number of times and that applies to the individual bits (however blocks of data are written at once), not the whole volume. Many newer technologies, like SSds, employ a rotation mechanism for writing data. That is, it will write to a specific location but it won't write to that location again (provided that location is marked as available) until it has gone through all the available storage locations. Once it has gone all the way around then it will write to that location again. This helps distribute the wear and greatly increases the longevity of storage devices.

Keep in mind that once you write a file to some storage medium then it stays put and that location is not written to again. You can read from an SSD, SD or other digital storage any number of times without degradation. All electronic devices can still ultimately fail so it's still a good idea to maintain more than one backup or consider using DVD or Blu-ray.

The article above suggests archiving your images on film. To me, this is an insane solution to the problem of potential data errors. I suggest using a data recovery format like par2 or rav when you write your images to a backup. The rav recovery file format is available via WinRAR but I personally use par2 as it's open source (I also use Linux). Here's how it works.

Create a recovery:

par2 c -n1 PS5_0087.RW2

The -n1 parameter tells it put all the recovery information in one file. The par recovery format was invented when people used the highly unreliable usenet for sharing files, so par2 will split the recovery data into multiple smaller files so that if one or two are missing it can still repair the original file.

To verify a file run:

par2 v PS5_0087.RW2

This will probably tell you "All files are correct, repair is not required." However, if you do need to repair the file then run:

par2 r PS5_0087.RW2

This, to me, is a much better way to ensure the integrity of your images.

RiccoT Forum Member • Posts: 91
Re: Bit rot

absolutely agree. film is lossy by design. I don't know what's wrong with this guy.

For Backup I highly recommend optical storage. I use 100GB M-Disk for that.

Maybe multiple cloud storages would also work. but that's not so cheap.

And yes, obviously we all should just print our favorite images

 RiccoT's gear list:RiccoT's gear list
Samsung NX1 Samsung NX 30mm F2 Pancake Samsung NX 50-200mm F4-5.6 OIS Samsung NX 85mm F1.4 ED SSA Samsung NX 45mm F1.8 +3 more
D Cox Forum Pro • Posts: 28,648
Re: Bit rot
1

RiccoT wrote:

absolutely agree. film is lossy by design. I don't know what's wrong with this guy.

Why do you say that film is "lossy by design" ? I see no reason why film can't be used to record digital data, similar to bar codes.

Silver based film has a very long life, and it can be sepia toned to give an even longer life. However, the need for chemical processing is inconvenient.

For Backup I highly recommend optical storage. I use 100GB M-Disk for that.

M-disks are good, but they take up a lot of space compared to a 10 Terabyte hard drive.

Maybe multiple cloud storages would also work. but that's not so cheap.

And yes, obviously we all should just print our favorite images

As for bit rot, it was a real problem in the days of floppy drives. Although modern hard drives are still magnetic recorders, they do seem to be quite reliable. Heavy users such as Google have statistics.

Clay tablets, once they are baked, have a long life, but their storage density is very low. And they can be broken -- there are many bits missing from the tale of Gilgamesh.

Don Cox

 D Cox's gear list:D Cox's gear list
Sigma fp
OP Barugon Veteran Member • Posts: 9,064
Re: Bit rot

D Cox wrote:

RiccoT wrote:

absolutely agree. film is lossy by design. I don't know what's wrong with this guy.

Why do you say that film is "lossy by design" ? I see no reason why film can't be used to record digital data, similar to bar codes.

Silver based film has a very long life, and it can be sepia toned to give an even longer life. However, the need for chemical processing is inconvenient.

For Backup I highly recommend optical storage. I use 100GB M-Disk for that.

M-disks are good, but they take up a lot of space compared to a 10 Terabyte hard drive.

Optical disks have the advantage that they contain no electronics that can fail over time due to heat fatigue.

Maybe multiple cloud storages would also work. but that's not so cheap.

And yes, obviously we all should just print our favorite images

As for bit rot, it was a real problem in the days of floppy drives. Although modern hard drives are still magnetic recorders, they do seem to be quite reliable. Heavy users such as Google have statistics.

I've had way more failures with hard drives than I have with SSDs. Come to think of it, I haven't had any SSD failures yet. Anyway, I personally don't use any magnetic storage now.

LuxShots
LuxShots Contributing Member • Posts: 702
Idiot
2

He is an absolute idiot.

Bit rot is a term coined by Software Engineering that depicts when code has been written so long ago, that it hasn't been refactored to keep up with changes in technology, be that speed improvements, code architecture improvements or new means and methods for how the code is supposed to operate in the latest realization of the software.

I speak this with the utmost knowledge, as I have been a Sr Software Engineer for quite awhile ;-).

I better way to deal with failures of the recording medium that are bound to happen at some point, is to reduce the statistical probability of it happening at all. If you have all of your data on just one hard drive (mechanical or SSD) you are playing with fire, as it can fail at anytime without warning. The term Sudden Infant Death describes how even a brand new electronic device can fail in the first 49-72 hours of use, while the one right next to it on the shelf will hold data for a decade!

The way I handle my data is using a RAID 5 NAS solution. If any one drive fails, every bit if my data is 100% in tact. albeit at a lower reading rate. I can then replace the bad drive in a hot swappable manner, and get back to full functionality after the system integrates the new drive.

My NAS is connected to my desktop computer using a 10GB SPF+ network connection, and to the rest of my network over the 1GB Ethernet connections, which makes my data available over the internet through a secure VPN connection.

I only run Seagate IronWolf NAS drives, and I have 5 4TB drives giving me 16TB of storage space, as RAID-5 uses one drive for parity. The drives run 24/7 in my NAS, and after they get to a predetermined amount of hours, I move them to the NAS backup pool, which is only powered on weekly for backups. At some point in time, these drives will be retired completely, and replaced with higher capacity drives with at least 10TB per drive.

I have 5TB of cloud drive space, that holds at least one full backup and the latest incremental backup, so if I suffer from a theft or fire, I still have my data, and my wallet just gets a tad lighter.

Everyone doing digital image creation that wants their imagery to last a few lifetimes should invest the money on a NAS based data storage solution, as well as instructions for where it is and how they go about getting to it for loved ones.

I'm still working on getting my wife up to speed, but the things on my NAS unit is as important to her as they are to me, so it's worth the effort for her to learn.

FYI, I use the QNAP TS-932X, and I added 16GB of RAM to it. This has 5 3.5" storage drive bays and 4 2.5" SSD cache drive bays. I use 4 256 GB SSD drives for read cache, as I edit 5.9K video directly off of my NAS drive, and this greatly increases performance. This model is currently in legacy status, but other models with more power and a similar price are available. A cheaper way to go would be to at least get a 4 drive NAS such as a QNAP TS-431K.

 LuxShots's gear list:LuxShots's gear list
Panasonic Lumix DC-S1H Panasonic S 24-105mm F4 Macro OIS Panasonic S1 Sigma 12-24mm F4.5-5.6 EX DG Aspherical HSM Canon EF 50mm F1.8 STM +9 more
Tom Schum
Tom Schum Forum Pro • Posts: 11,257
Re: Bit rot
1

I've also called it "data evaporation", when data stored as a pattern of electron charge or as a pattern of magnetic polarization basically evaporates when the charge bleeds off over a period of years.  This is such a touchy subject that you will have trouble finding much about it on the internet.  USB sticks nowadays are pretty good but not recommended for long term storage.  Magnetic rotating platter storage is still considered the best for long term.

I saw data evaporation in a laser that simply stopped working because the program, stored in 20-year old EPROM devices, got spontaneously corrupted.

In cloud-based storage you can be pretty sure your data will survive, unless you stop paying the fees!  This is because most cloud storage is RAID, I believe.  Of course the cloud might be hacked, in which case all your data will be lost forever.

In your own private storage, RAIDs can prevent losses but only so long as you keep them working.  This is usually done with software that is beyond your control, software that one day might stop being updated.  And, one power supply glitch can fry all your drives at once.

With photographic film, you might or might not lose image data, but if the chemical processing is not ideal, lifetime issues come into play.  And, of course, you need a way to view the images you have stored.

When all is said and done, it seems to me that the best advice, advice I've been hearing for years, is "save your images as archival prints".  Of course you have to store them in an archival way, and keep the environment from destroying them.  A little flood in the basement can end it all.  If not, in future centuries your descendants living in caves or orbiting Saturn can still view the prints directly.  That is, if they have eyes like ours...

I remember seeing data CDs you could buy that advertised 600 year stability.  Great, but can you read them in a CD reader at that point?  Highly doubtful CD readers will still exist.

This goes on and on.  Maybe the Egyptians had the best idea:  paint on a wall in an underground crypt in a dry country, then hide the crypt.

-- hide signature --

Tom Schum
Copper: Mankind's favorite electrical conductor

 Tom Schum's gear list:Tom Schum's gear list
Fujifilm X30 Sigma dp0 Quattro Panasonic ZS100 Fujifilm X-E1 Sigma sd Quattro +17 more
Keyboard shortcuts:
FForum MMy threads