bit Rot examples

So any data deterioration/rotting means some unwanted/unexpected change occurred in the medium, and that includes unwanted change in electric charge in the NAND cells, although there's many other reasons for 'deterioration'.
NAND, by the very design, "forget" the stored data. That is because even for read operations you have to address cells, means you put voltage on that to read the state, and that alone can modify the charge in the cell and also neighboring cells. As has been outlined to you already as well, the tighter you pack all that, the more likely this is to happen. So it is "unwanted", but certainly "expected". Write operations work with higher voltages, go figure what happens there. And then there is the general loss of charge, the fact that the cells detoriate with changing the charge (the smaller, the faster), etc.

Which is why you have larger amounts of the total NAND capacity reserved for error corrections, and I'm talking like 30-40% there, not 1% or whatever you hope it to be. They can correct for example up to 8 bits in a certain set of data, and if they go past a margin (lets say >5 bits have errors), this part of the NAND is getting copied to some other place and eventually marked free. Because, as Joep also noted, the cells are usually not really dead/broken, but need to be reset. Which means, yes, even reading flash media triggers such copy operations eventually. And there are a number of other consequences down the line. You may have heard of TRIM, for example ...

So, data isn't stored as 0 and 1, but it is charge, and depending on the circuitry it is then interpreted as 0 or 1. Harddisks for example don't store 0 and 1 which you see directly, they have some intricate encoding scheme for the actual 0 and 1. Or look at USB, which has some encoding scheme for 0 and 1 as well, so a "high" level there isn't an 1 ... and there are things like https://en.wikipedia.org/wiki/Emitter_coupled_logic (there is no clear "on" or "off" anymore) or for example https://en.wikipedia.org/wiki/Phase_Shift_Keying which stores way more than just 0 or 1 to obtain higher bandwidths over analog media ... which means there can be gradual loss of data, because of the underlying physicalities. And modern busses usually have a "link training" phase where the different participants on the bus optimize automagically what is called https://en.wikipedia.org/wiki/Eye_pattern, because analog world and all that ...

What you see as 0 and 1 from the media is a couple stages removed from the actual physical level already.
 
So any data deterioration/rotting means some unwanted/unexpected change occurred in the medium, and that includes unwanted change in electric charge in the NAND cells, although there's many other reasons for 'deterioration'.
NAND, by the very design, "forget" the stored data.
It's not 'designed'. It's a technical limitation.
That is because even for read operations you have to address cells, means you put voltage on that to read the state, and that alone can modify the charge in the cell and also neighboring cells. As has been outlined to you already as well, the tighter you pack all that, the more likely this is to happen. So it is "unwanted", but certainly "expected". Write operations work with higher voltages, go figure what happens there. And then there is the general loss of charge, the fact that the cells detoriate with changing the charge (the smaller, the faster), etc.

Which is why you have larger amounts of the total NAND capacity reserved for error corrections, and I'm talking like 30-40% there, not 1% or whatever you hope it to be.
Yeah I'd expect it to be around 1% when using Hamming codes, for example https://www.elnec.com/sw/samsung_ecc_algorithm_for_256b.pdf

3 bytes (22 bits in fact) of ECC per 256 bytes of data ~= 1.17%

or 3 bytes (full 24 bits) of ECC per 512 bytes = 0.58%

More advanced (and time-consuming) methods may be able to correct more than 1 flipped bit, as you say below, but they will have around 10% redundancy afaik.

40% doesn't make much sense - you might as well just duplicate the data like in RAID.

Anyway, the encoding rate, even if it's 40%, wasn't the point. On the contrary, the better the redundancy is, the less likely you'll experience the data corruption.
They can correct for example up to 8 bits in a certain set of data, and if they go past a margin (lets say >5 bits have errors), this part of the NAND is getting copied to some other place and eventually marked free. Because, as Joep also noted, the cells are usually not really dead/broken, but need to be reset.
Which means, yes, even reading flash media triggers such copy operations eventually. And there are a number of other consequences down the line. You may have heard of TRIM, for example ...

So, data isn't stored as 0 and 1,
You might want to check the thread, again I never claimed the data was stored as 0s and 1s.

The logical data being stored is 0s and 1s. But it is stored as analog charge. It's encoded as charge in the physical medium.
 
40% doesn't make much sense - you might as well just duplicate the data like in RAID.

...
Simply duplicating the data allows you to detect, but not correct errors. If the data on drive A is a zero, and the data on drive B is a one, then you know that one of them is wrong, but you don't know which one. Now if you had three copies, then you can detect and correct single bit errors.

Error detection strategies are more complex that simply keeping multiple copies. However, they are also more efficient at detecting and correct errors.
 
40% doesn't make much sense - you might as well just duplicate the data like in RAID.

...
Simply duplicating the data allows you to detect, but not correct errors.
So there's no RAIDs with two drives in the array? :) Normally you in RAIDs there's also data redundancy in each drive.
If the data on drive A is a zero, and the data on drive B is a one, then you know that one of them is wrong, but you don't know which one.
I will/may still have simple Hamming ECC with just 0.5% redundancy. So I will know exactly which one failed and will be able to correct any number of flipped bits, if they're all flipped on the same [faulty] drive.
Now if you had three copies, then you can detect and correct single bit errors.

Error detection strategies
Error detection is much easier to do that error correction, the above was about ECC (error correction codes).
are more complex that simply keeping multiple copies.
Yes, but above a was commenting on the case of ECC with 40% redundancy which is too much in my opinion.
 
So any data deterioration/rotting means some unwanted/unexpected change occurred in the medium, and that includes unwanted change in electric charge in the NAND cells, although there's many other reasons for 'deterioration'.
NAND, by the very design, "forget" the stored data. That is because even for read operations you have to address cells, means you put voltage on that to read the state, and that alone can modify the charge in the cell and also neighboring cells. As has been outlined to you already as well, the tighter you pack all that, the more likely this is to happen. So it is "unwanted", but certainly "expected". Write operations work with higher voltages, go figure what happens there. And then there is the general loss of charge, the fact that the cells detoriate with changing the charge (the smaller, the faster), etc.

Which is why you have larger amounts of the total NAND capacity reserved for error corrections, and I'm talking like 30-40% there, not 1% or whatever you hope it to be. They can correct for example up to 8 bits in a certain set of data, and if they go past a margin (lets say >5 bits have errors), this part of the NAND is getting copied to some other place and eventually marked free. Because, as Joep also noted, the cells are usually not really dead/broken, but need to be reset. Which means, yes, even reading flash media triggers such copy operations eventually. And there are a number of other consequences down the line. You may have heard of TRIM, for example ...

So, data isn't stored as 0 and 1, but it is charge, and depending on the circuitry it is then interpreted as 0 or 1. Harddisks for example don't store 0 and 1 which you see directly, they have some intricate encoding scheme for the actual 0 and 1. Or look at USB, which has some encoding scheme for 0 and 1 as well, so a "high" level there isn't an 1 ... and there are things like https://en.wikipedia.org/wiki/Emitter_coupled_logic (there is no clear "on" or "off" anymore) or for example https://en.wikipedia.org/wiki/Phase_Shift_Keying which stores way more than just 0 or 1 to obtain higher bandwidths over analog media ... which means there can be gradual loss of data, because of the underlying physicalities. And modern busses usually have a "link training" phase where the different participants on the bus optimize automagically what is called https://en.wikipedia.org/wiki/Eye_pattern, because analog world and all that ...

What you see as 0 and 1 from the media is a couple stages removed from the actual physical level already.
Well said. Quoted the text so others might have a change to read it if they missed the first post.
 
JPEGs that 'deteriorate' with repeated saving, reopening, and resaving cycles do so for a very different reason that has nothing to do with the passage of time. It happens because of the way the compression algorithm works when it's applied recursively.
Erm... I don't think anyone was talking about that (re-compressing images). I think they are talking about files simply going bad over time. (Which is really files going bad because of media going bad.)
Your take regarding what the OP is talking about with me isn't necessary, and your selective removal of the actual context of that discussion is in fact detrimental to the goal of clear communication. Not helping.

You could engage him yourself, though, if you have something to say to him. For example, you might try asking him what this statement of his means:
I am talking about photos that deteriorate over time faster than the recording medium decays.
I guess, in a sense, all decay of charge mixing up of locations is part of medium failure.

However. when I mean that the jpg decays before the medium does, I mean that you can use the medium safely for new jpgs - the core physical parts of it still work.

For example an SSD hard drive words fine for years but if it lacks power any data on it will fail. Power it up format from scratch and keep it powered new jpgs will be fine.

The lost jpgs are lost because the DESIGN of the medium will cause that to happen. Like any near water tight device even working perfectly fine will lose water given enough time.

Or a better example as it avoids factional entropy - a cooler for you food, will keep it cool but only for a while. If the cooler breaks or gets old this will get worse.
 
40% doesn't make much sense - you might as well just duplicate the data like in RAID.

...
Simply duplicating the data allows you to detect, but not correct errors.
So there's no RAIDs with two drives in the array? :) Normally you in RAIDs there's also data redundancy in each drive.
Yes, two drive RAIDs have more than 100% overhead. For each bit you store, there's a duplicate on the other drive, plus error correction and detection on each drive.
If the data on drive A is a zero, and the data on drive B is a one, then you know that one of them is wrong, but you don't know which one.
I will/may still have simple Hamming ECC with just 0.5% redundancy. So I will know exactly which one failed and will be able to correct any number of flipped bits, if they're all flipped on the same [faulty] drive.
Yes. In terms of protecting against common hardware level errors involving a handful of bits, error detection/correction is the way to go.
Now if you had three copies, then you can detect and correct single bit errors.

Error detection strategies
Error detection is much easier to do that error correction, the above was about ECC (error correction codes).
I think we both agree that good error detection and correction codes use far less than 100% overhead on many media types.

are more complex that simply keeping multiple copies.
Yes, but above a was commenting on the case of ECC with 40% redundancy which is too much in my opinion.
I would expect that the need for redundancy is strongly correlated to error characteristics of the underlying media. For instance, I believe that CD media has a 33% overhead for ECC (There's one byte of ECC for every 3 byte of data). Keep in mind that we still expect some uncorrectable errors on audio CDs, but this is generally acceptable as an error may come through as a momentary click, rather than rendering the entire audio track unusable. Data CDs require additional ECC data.

A figure of 40% overhead doesn't sound at all unreasonable for certain media.
 
Last edited:
So any data deterioration/rotting means some unwanted/unexpected change occurred in the medium, and that includes unwanted change in electric charge in the NAND cells, although there's many other reasons for 'deterioration'.
NAND, by the very design, "forget" the stored data.
It's not 'designed'. It's a technical limitation.
That is because even for read operations you have to address cells, means you put voltage on that to read the state, and that alone can modify the charge in the cell and also neighboring cells. As has been outlined to you already as well, the tighter you pack all that, the more likely this is to happen. So it is "unwanted", but certainly "expected". Write operations work with higher voltages, go figure what happens there. And then there is the general loss of charge, the fact that the cells detoriate with changing the charge (the smaller, the faster), etc.

Which is why you have larger amounts of the total NAND capacity reserved for error corrections, and I'm talking like 30-40% there, not 1% or whatever you hope it to be.
Yeah I'd expect it to be around 1% when using Hamming codes, for example https://www.elnec.com/sw/samsung_ecc_algorithm_for_256b.pdf

3 bytes (22 bits in fact) of ECC per 256 bytes of data ~= 1.17%

or 3 bytes (full 24 bits) of ECC per 512 bytes = 0.58%

More advanced (and time-consuming) methods may be able to correct more than 1 flipped bit, as you say below, but they will have around 10% redundancy afaik.

40% doesn't make much sense - you might as well just duplicate the data like in RAID.

Anyway, the encoding rate, even if it's 40%, wasn't the point. On the contrary, the better the redundancy is, the less likely you'll experience the data corruption.
Yes, ok, I remembered that part wrong. It is like 3-5% which are kept spare. Multibit + burst error detection and correction, management/meta data, and potentially even a 2nd stage error correction because all that is stored in the same NAND and may fall to pieces by bit rot like the actual data.
 
I would expect that the need for redundancy is strongly correlated to error characteristics of the underlying media.
Yes, plus the requirements for the data loss. As you said (below), audio CDs can be more tolerant to the data loss.

But the point is, the physical 'bit rot' as a random bit corruption is highly unlikely to affect your photos. You just won't see it because of the error correction.
For instance, I believe that CD media has a 33% overhead for ECC (There's one byte of ECC for every 3 byte of data). Keep in mind that we still expect some uncorrectable errors on audio CDs, but this is generally acceptable as an error may come through as a momentary click, rather than rendering the entire audio track unusable. Data CDs require additional ECC data.

A figure of 40% overhead doesn't sound at all unreasonable for certain media.
--
https://www.instagram.com/quarkcharmed/
https://500px.com/quarkcharmed
 
Last edited:
So any data deterioration/rotting means some unwanted/unexpected change occurred in the medium, and that includes unwanted change in electric charge in the NAND cells, although there's many other reasons for 'deterioration'.
NAND, by the very design, "forget" the stored data.
It's not 'designed'. It's a technical limitation.
That is because even for read operations you have to address cells, means you put voltage on that to read the state, and that alone can modify the charge in the cell and also neighboring cells. As has been outlined to you already as well, the tighter you pack all that, the more likely this is to happen. So it is "unwanted", but certainly "expected". Write operations work with higher voltages, go figure what happens there. And then there is the general loss of charge, the fact that the cells detoriate with changing the charge (the smaller, the faster), etc.

Which is why you have larger amounts of the total NAND capacity reserved for error corrections, and I'm talking like 30-40% there, not 1% or whatever you hope it to be.
Yeah I'd expect it to be around 1% when using Hamming codes, for example https://www.elnec.com/sw/samsung_ecc_algorithm_for_256b.pdf

3 bytes (22 bits in fact) of ECC per 256 bytes of data ~= 1.17%

or 3 bytes (full 24 bits) of ECC per 512 bytes = 0.58%

More advanced (and time-consuming) methods may be able to correct more than 1 flipped bit, as you say below, but they will have around 10% redundancy afaik.

40% doesn't make much sense - you might as well just duplicate the data like in RAID.

Anyway, the encoding rate, even if it's 40%, wasn't the point. On the contrary, the better the redundancy is, the less likely you'll experience the data corruption.
Yes, ok, I remembered that part wrong. It is like 3-5% which are kept spare. Multibit + burst error detection and correction, management/meta data, and potentially even a 2nd stage error correction because all that is stored in the same NAND and may fall to pieces by bit rot like the actual data.
Tandy PC had 12% error correction - a check bit in each set of 8. It was deemed to expensive by others to adopt -
 
JPEGs that 'deteriorate' with repeated saving, reopening, and resaving cycles do so for a very different reason that has nothing to do with the passage of time. It happens because of the way the compression algorithm works when it's applied recursively.
Erm... I don't think anyone was talking about that (re-compressing images). I think they are talking about files simply going bad over time. (Which is really files going bad because of media going bad.)
Your take regarding what the OP is talking about with me isn't necessary, and your selective removal of the actual context of that discussion is in fact detrimental to the goal of clear communication. Not helping.

You could engage him yourself, though, if you have something to say to him. For example, you might try asking him what this statement of his means:
I am talking about photos that deteriorate over time faster than the recording medium decays.
I guess, in a sense, all decay of charge mixing up of locations is part of medium failure.

However. when I mean that the jpg decays before the medium does,
Unfortunately you keep repeating the same misconceptions.

II mean that you can use the medium safely for new jpgs - the core physical parts of it still work.
No, it's highly unlikely to happen because of the error correction.
For example an SSD hard drive words fine for years but if it lacks power any data on it will fail.
Misconceptions are getting worse.

The primary reason for an SSD to fail is reaching the limit for program-erase cycles, not powering it off.
Power it up format from scratch and keep it powered new jpgs will be fine.
And that you're also very unlikely to experience. More likely the whole drive will become unreadable
The lost jpgs are lost because the DESIGN of the medium will cause that to happen.
No. What happens is a technical problem and design works around it by introducing the error correction mechanisms.
Like any near water tight device even working perfectly fine will lose water given enough time.

Or a better example as it avoids factional entropy - a cooler for you food, will keep it cool but only for a while. If the cooler breaks or gets old this will get worse.
 
Yes, ok, I remembered that part wrong. It is like 3-5% which are kept spare. Multibit + burst error detection and correction, management/meta data, and potentially even a 2nd stage error correction because all that is stored in the same NAND and may fall to pieces by bit rot like the actual data.
Tandy PC had 12% error correction - a check bit in each set of 8. It was deemed to expensive by others to adopt -
Sorry you simply misunderstood what it was about. It was about Hamming, BCH and other sophisticated types of error correction.

A 'check bit in each set of 8' is not an error correction at all.
 
JPEGs that 'deteriorate' with repeated saving, reopening, and resaving cycles do so for a very different reason that has nothing to do with the passage of time. It happens because of the way the compression algorithm works when it's applied recursively.
Erm... I don't think anyone was talking about that (re-compressing images). I think they are talking about files simply going bad over time. (Which is really files going bad because of media going bad.)
Your take regarding what the OP is talking about with me isn't necessary, and your selective removal of the actual context of that discussion is in fact detrimental to the goal of clear communication. Not helping.

You could engage him yourself, though, if you have something to say to him. For example, you might try asking him what this statement of his means:
I am talking about photos that deteriorate over time faster than the recording medium decays.
I guess, in a sense, all decay of charge mixing up of locations is part of medium failure.

However. when I mean that the jpg decays before the medium does,
Unfortunately you keep repeating the same misconceptions.
II mean that you can use the medium safely for new jpgs - the core physical parts of it still work.
No, it's highly unlikely to happen because of the error correction.
For example an SSD hard drive words fine for years but if it lacks power any data on it will fail.
Misconceptions are getting worse.

The primary reason for an SSD to fail is reaching the limit for program-erase cycles, not powering it off.
Power it up format from scratch and keep it powered new jpgs will be fine.
And that you're also very unlikely to experience. More likely the whole drive will become unreadable
The lost jpgs are lost because the DESIGN of the medium will cause that to happen.
No. What happens is a technical problem and design works around it by introducing the error correction mechanisms.
Like any near water tight device even working perfectly fine will lose water given enough time.

Or a better example as it avoids factional entropy - a cooler for you food, will keep it cool but only for a while. If the cooler breaks or gets old this will get worse.
Your not very forth coming. "Misconceptions"

Via SSD what are the misconceptions that I have expressed? What is the reality?

Ok here is a text please explain in layman terms the right interpretation

"Enterprise SSDs, however, have entirely different characteristics. An enterprise drive stored at 25C and operated at 40C has a retention rate of just 20 weeks. In worst-case scenarios or high storage temps, the data on an enterprise drive can start to fail within seven days. 3D NAND, which uses an older manufacturing process, might rate better in such metrics, but JEDEC doesn’t include that information.

These timelines aren’t likely to make much difference to consumers, who keep devices in daily use, but they do matter in the long-term and outside client usage scenarios. If you’re a serious computer enthusiast, you likely have a collection of drives that you removed from old systems when you built a new machine. Some people will carefully step through and ensure that each and every bit of data from the old installation was archived to local backup or cloud storage, but it’s much more common for people to copy the critical data and leave the rest of their information on the original disk. With a mechanical drive, that’s not a problem — you can take a drive out of rotation and typically assume it will spin back up two years later, provided you store it properly."

From what i Read SSD need power to maintain data integrity. This is wrong apparently so what does this actually say?

Since this is far more complex beyond a laymans understanding, lets keep to discussing the SSD misconception. Further, its the think I have almost no knowledge on - so the Dunning–Kruger_effect is at maximum so I have the most to learn here.

https://www.extremetech.com/computing/205382-ssds-can-lose-data-in-as-little-as-7-days-without-power
 
Last edited:
Via SSD what are the misconceptions that I have expressed? What is the reality?
"jpg decays before the medium does" - it's physically impossible and phrasing is very wrong indeed. The 'medium' is whatever stores your data, if the medium is intact, your data is intact (although there may be other issues during data transfer). The electrons in the NAND memory cells and the charge they produce are the physical 'medium'.
Ok here is a text please explain in layman terms the right interpretation

"Enterprise SSDs, however, have entirely different characteristics. An enterprise drive stored at 25C and operated at 40C has a retention rate of just 20 weeks. In worst-case scenarios or high storage temps, the data on an enterprise drive can start to fail within seven days. 3D NAND, which uses an older manufacturing process, might rate better in such metrics, but JEDEC doesn’t include that information.
Do you have enterprise SSDs?..
Since this is far more complex beyond a laymans understanding, lets keep to discussing the SSD misconception. Further, its the think I have almost no knowledge on - so the Dunning–Kruger_effect is at maximum so I have the most to learn here.

https://www.extremetech.com/computing/205382-ssds-can-lose-data-in-as-little-as-7-days-without-power
That article got it all wrong

https://www.pcworld.com/article/292...nt-lose-data-if-left-unplugged-after-all.html

--
https://www.instagram.com/quarkcharmed/
https://500px.com/quarkcharmed
 
Last edited:
Via SSD what are the misconceptions that I have expressed? What is the reality?
The reality is you are quoting material from 2015.
AND please send me info that updates or changes how SSD work. So what year o f SSD due I need to buy to be safe?

Currently I have man old SSD waht year should I junk and what years are changed?

you imply 2015 and less junk. So whens the year this changes?
 
AND please send me info that updates or changes how SSD work. So what year o f SSD due I need to buy to be safe?
I do not have the time, space, or patience to explain half of the reasons why buying a seven year old drive is a bad idea.
 
AND please send me info that updates or changes how SSD work. So what year o f SSD due I need to buy to be safe?
I do not have the time, space, or patience to explain half of the reasons why buying a seven year old drive is a bad idea.
Tahts a straw man.

I asked you when was the NEW SUPERIOR drives that you speak of/ This year?

2015 is 6 years by the way, less since the article was not write at the start.

Lets be lclear send me one link that updates and invalids the info of the link about powering of SDD drives. Pcmaga claims its still current info.

ONE LInk of one Trusted source. The rule that something is old does not work by itself, we sitlil after all still use 20+year jpgs with little change.
 

Keyboard shortcuts

Back
Top