Is flash memory unreliable due to exFAT?

[kelpdiver]

CDs/DVDs seem to be going out of fashion but I prefer them to flash sticks for distributing photos and so on to friends and family. Also they are a lot easier to file along with relevant paperwork.

DVDs are good for filed storage, but getting kind of lousy for sharing data with others. Laptops stopped having drives a decade ago, and a lot of people stopped buying desktops around then as well.

 

[Sean Nelson]

Is flash memory unreliable because of exFAT, or because it is flash memory?

It's because of flash memory.

Data in flash memory cells is stored using a static charge. This is the same kind of charge that you get if you rub a balloon on a wool sweater - because of the charge the balloon will stick to a wall. But the charge dissipates over time - the balloon will eventually fall off the wall and the data in a flash memory cell will eventually "evaporate" (the balloon falls much, much faster).

While I barely used thumb drives I don't think I have even had a SD/microSD or old Sony memory stick die or get corrupted.

The reason that flash drive capacity has been going up over the years is because the individual cells have been getting smaller and the number of bits of information stored in an individual cell have been going up. Both of these "advances" mean that the data will become unreadable more quickly. Drives try to mitigate this by using error correction codes which can recover the data even if some bits go missing, but you can only go so far with that. At some point, once enough of the static charges have evaporated, the data becomes unreadable.

So those old 8MB memory sticks with one bit per cell and cells the size of acreage could retain data for decades, while today's 512GB flash chips with two or three bits per cell and cells the size of angels on the head of a pin fare much much worse. It's not because modern drives are "cheap" in the sense of not being well made. It's because poorer data retention is an inevitable tradeoff for the improved capacity and lower cost per byte.

All of this is why it's a bad idea to use flash memory as archival storage. If you do plan to retain flash memory data over a long period of time, it's important to read all of the data periodically so that the chip's controller can discover correctable errors before they become permanent ones - when it reads erroneous data that can be corrected it rewrites the data, which puts new, fresh static charges on the cells to give them a new lease on life.

Checksumming your files and then verifying the checksums every so often is the best way to do this. Not only does it read all the data, it will tell you if any of the data was unreadable (because the checksums won't match). This is important because you don't get otherwise notified of read errors from SD cards or USB thumb drives. If you know that a file has errors then you can recover a good copy of it from one of the other copies you have (because we all make copies of our important files on redundant media, right?)

 

[David Lal]

CDs/DVDs seem to be going out of fashion but I prefer them to flash sticks for distributing photos and so on to friends and family. Also they are a lot easier to file along with relevant paperwork.

DVDs are good for filed storage, but getting kind of lousy for sharing data with others. Laptops stopped having drives a decade ago, and a lot of people stopped buying desktops around then as well.

Yes, indeed - witness the ascendency of the mobile telephone. Most of my family and many friends are Apple Mac users as well and that adds another complication since I am firmly in the 'proper' PC camp.

For small files it's not too much of an issue as I have web server space so can upload there then distribute a link. For large files (I wrote an e-book on Spanish grammar, create videos for example) measuring 6 or 7 gigabytes it is an issue. I know there are ways round this but it is an extra embuggerance I could do without. It was so much easier to pop a dual-layer DVD in an envelope and post it off in the mail.

 

[Chris Noble]

Personally, I only use thumb drives to move files from one computer to another. Never as a repository. I also keep one for emergency booting.

And with the growing number of cloud options, I have less and less need for them. I don't recall the last time I used one for anything.

I am curious about how people use thumb drives today. Seems like an obsolete technology.

 

[David Lal]

I am curious about how people use thumb drives today. Seems like an obsolete technology.

Somewhat like you Chris, I'd never use one for serious data storage but I do have a bunch of them to boot various different machines to different operating systems, recovery etc. In fact just now I created one to boot Linux/Ubuntu on an elderly Win7 laptop.

My other use which I have mentioned already is to send data, especially large data files such as photographs to people whose computers have no CD drive.

 

[Chris Noble]

I am curious about how people use thumb drives today. Seems like an obsolete technology.

Somewhat like you Chris, I'd never use one for serious data storage but I do have a bunch of them to boot various different machines to different operating systems, recovery etc. In fact just now I created one to boot Linux/Ubuntu on an elderly Win7 laptop.

My other use which I have mentioned already is to send data, especially large data files such as photographs to people whose computers have no CD drive.

Hi David,

I only keep a few, for backup booting/recovery. But even there I have a separate boot partition on my computers in case my main partition is corrupted beyond being bootable or I want to boot outside the installed OS.

For distributing photos and other files, I long ago switched, first to Dropbox and now to Google Photos & Docs. Much easier to send a link that the recipients can open on their computer and cellphone and download the images or docs if they want, on Apple or Android or Windows. The free Google One account is 15 GB, and you can get 100 GB for $2/month, 2 TB for $10/month.

 

[David Lal]

For distributing photos and other files, I long ago switched, first to Dropbox and now to Google Photos & Docs. Much easier to send a link that the recipients can open on their computer and cellphone and download the images or docs if they want, on Apple or Android or Windows. The free Google One account is 15 GB, and you can get 100 GB for $2/month, 2 TB for $10/month.

Thanks for the info Chris. I will have to investigate. Our ADSL infrastructure is pretty inferior to much of the rest of Europe and certainly to the USA so our up- and download speeds can be dire. especially in rural areas. An 8 gb video (one DL DVD's worth) could take for ever to upload at at 20 mb/s so that is another consideration.

However, it's certainly worth me looking.

 

[Soaptrail]

Is flash memory unreliable because of exFAT, or because it is flash memory?

It's because of flash memory.

Data in flash memory cells is stored using a static charge. This is the same kind of charge that you get if you rub a balloon on a wool sweater - because of the charge the balloon will stick to a wall. But the charge dissipates over time - the balloon will eventually fall off the wall and the data in a flash memory cell will eventually "evaporate" (the balloon falls much, much faster).

While I barely used thumb drives I don't think I have even had a SD/microSD or old Sony memory stick die or get corrupted.

The reason that flash drive capacity has been going up over the years is because the individual cells have been getting smaller and the number of bits of information stored in an individual cell have been going up. Both of these "advances" mean that the data will become unreadable more quickly. Drives try to mitigate this by using error correction codes which can recover the data even if some bits go missing, but you can only go so far with that. At some point, once enough of the static charges have evaporated, the data becomes unreadable.

So those old 8MB memory sticks with one bit per cell and cells the size of acreage could retain data for decades, while today's 512GB flash chips with two or three bits per cell and cells the size of angels on the head of a pin fare much much worse. It's not because modern drives are "cheap" in the sense of not being well made. It's because poorer data retention is an inevitable tradeoff for the improved capacity and lower cost per byte.

All of this is why it's a bad idea to use flash memory as archival storage. If you do plan to retain flash memory data over a long period of time, it's important to read all of the data periodically so that the chip's controller can discover correctable errors before they become permanent ones - when it reads erroneous data that can be corrected it rewrites the data, which puts new, fresh static charges on the cells to give them a new lease on life.

Checksumming your files and then verifying the checksums every so often is the best way to do this. Not only does it read all the data, it will tell you if any of the data was unreadable (because the checksums won't match). This is important because you don't get otherwise notified of read errors from SD cards or USB thumb drives. If you know that a file has errors then you can recover a good copy of it from one of the other copies you have (because we all make copies of our important files on redundant media, right?)

gotcha, so basicially the same as SSD "davances" SLC to TLC to MLC ... and each time the long term duration is reduced but as you said they have error checking and overprovisioning (i think that is the right word) to prevent issues.

 

[CAcreeks]

I'm surprised by the negative references to Verbatim in this thread. I thought it had a good name, certainly in terms of CDs/DVDs - I hope so as I have a pack of 50 Verbatim dual layer recordable DVDs arriving today :-(

They are not a prime vendor of USB flash sticks (thumb drives) but it appears they are well respected for optical media. From Wikipedia:

• In 1993, Verbatim started its successful foray into optical disc with CD-R media.
• In 1997, Verbatim released the world's first CD-RW format media.
• In 2001, Verbatim released the world's first DVD+R format media.
• Introduced the first 8.5 GB DVD+R DL in 2004, followed by DVD-R DL in 2005.

Never heard of the last one; we use BluRay.

 

[Chris Noble]

For distributing photos and other files, I long ago switched, first to Dropbox and now to Google Photos & Docs. Much easier to send a link that the recipients can open on their computer and cellphone and download the images or docs if they want, on Apple or Android or Windows. The free Google One account is 15 GB, and you can get 100 GB for $2/month, 2 TB for $10/month.

Thanks for the info Chris. I will have to investigate. Our ADSL infrastructure is pretty inferior to much of the rest of Europe and certainly to the USA so our up- and download speeds can be dire. especially in rural areas. An 8 gb video (one DL DVD's worth) could take for ever to upload at at 20 mb/s so that is another consideration.

However, it's certainly worth me looking.

2 hrs for ADSL at 8 Mbps... Start it before you go to bed .

 

[Soaptrail]

I'm surprised by the negative references to Verbatim in this thread. I thought it had a good name, certainly in terms of CDs/DVDs - I hope so as I have a pack of 50 Verbatim dual layer recordable DVDs arriving today :-(

They are not a prime vendor of USB flash sticks (thumb drives) but it appears they are well respected for optical media. From Wikipedia:

• In 1993, Verbatim started its successful foray into optical disc with CD-R media.
• In 1997, Verbatim released the world's first CD-RW format media.
• In 2001, Verbatim released the world's first DVD+R format media.
• Introduced the first 8.5 GB DVD+R DL in 2004, followed by DVD-R DL in 2005.

Never heard of the last one; we use BluRay.

After a while i only bought Verbatim optical media. But after buying a BD-R writer for my PC I backed up my RAW files on 34 BD-R verbatim discs. I validated they worked and appended a few of them. To save space i deleted those RAW from my computer. Then i went to pull a RAW file from the discs. I could not read data on 33 of the 34 discs. I even tried my home BD players.

Made me sad to get not one or two bad discs but all but one was horrendous.

 

[Chris Noble]

I'm surprised by the negative references to Verbatim in this thread. I thought it had a good name, certainly in terms of CDs/DVDs - I hope so as I have a pack of 50 Verbatim dual layer recordable DVDs arriving today :-(

They are not a prime vendor of USB flash sticks (thumb drives) but it appears they are well respected for optical media. From Wikipedia:

• In 1993, Verbatim started its successful foray into optical disc with CD-R media.
• In 1997, Verbatim released the world's first CD-RW format media.
• In 2001, Verbatim released the world's first DVD+R format media.
• Introduced the first 8.5 GB DVD+R DL in 2004, followed by DVD-R DL in 2005.

Never heard of the last one; we use BluRay.

Verbatim did make floppy disks in the 1970s in California. The company went out of business in 1985. The brand was acquired by Kodak, who then sold it to Mitsubishi in 1990, who sold it to CMC Magnetics (Taiwanese company) in 2019 for $32M (not a lot of money for a consumer electronics brand).

So a variety of companies have made products under the Verbatim brand. Although CMC owns the brand name now, that does not guarantee that they make those products. They are likely made by one or more Chinese OEMs.

 

[David Lal]

After a while i only bought Verbatim optical media. But after buying a BD-R writer for my PC I backed up my RAW files on 34 BD-R verbatim discs. I validated they worked and appended a few of them. To save space i deleted those RAW from my computer. Then i went to pull a RAW file from the discs. I could not read data on 33 of the 34 discs. I even tried my home BD players.

Made me sad to get not one or two bad discs but all but one was horrendous.

My word! That is indeed a very sad tale. Over here Blu-Ray never really took off, at least, not in my social circles: I don't know anybody who has a Blu-Ray drive/player.

 

[Chris Noble]

After a while i only bought Verbatim optical media. But after buying a BD-R writer for my PC I backed up my RAW files on 34 BD-R verbatim discs. I validated they worked and appended a few of them. To save space i deleted those RAW from my computer. Then i went to pull a RAW file from the discs. I could not read data on 33 of the 34 discs. I even tried my home BD players.

Made me sad to get not one or two bad discs but all but one was horrendous.

My word! That is indeed a very sad tale. Over here Blu-Ray never really took off, at least, not in my social circles: I don't know anybody who has a Blu-Ray drive/player.

The topic of DVDs for data storage has come up repeatedly. The DVD format was specifically developed for movies, which can handle high error rates because of the benign requirements of our eyesight. Applying them to data storage was an afterthought, based on intense error-correction protocols and the insistent push of marketing people. Some of us never went near a DVD-RW for data storage for that reason. But even now, there are still advocates, on this forum and elsewhere.

Soaptrail, did you approach specialized data-recovery outfits to see if they can help you recover your data?

 

[kelpdiver]

The topic of DVDs for data storage has come up repeatedly. The DVD format was specifically developed for movies, which can handle high error rates because of the benign requirements of our eyesight. Applying them to data storage was an afterthought, based on intense error-correction protocols and the insistent push of marketing people. Some of us never went near a DVD-RW for data storage for that reason. But even now, there are still advocates, on this forum and elsewhere.

Let's not conflate the issues of writeable media with the format. DVDs were used to deliver software without issue until the general death of optical drives as standard equipment. And the M-Disk has proven out as a reliable way to archive data.

But as people wanted and needed lot of blank media, a lot of unscrupulous types took advantage with low quality products at cheaper prices, and often pretended to be well regarded brands. Verbatim, among others, suffered from a rash of counterfeits and that may have been the case for the prior poster. There also seemed to be a lot of trouble for folks that burned at high speed on one drive, which could not be read on others. Recording at a slower speed could lower that problem to a degree.

 

[kelpdiver]

Yes, indeed - witness the ascendency of the mobile telephone. Most of my family and many friends are Apple Mac users as well and that adds another complication since I am firmly in the 'proper' PC camp.

One of my most useful thumb drives is a Sandisk wifi capable usb stick. I've been taking that on vacations for many years now as it can solve a number of transfer situations, though it does require an agent install. I now also have a drive with both usb and lightning connectors to allow faster transfers for IOS products as the wireless is pretty slow.

 

[CAcreeks]

Yes, indeed - witness the ascendency of the mobile telephone. Most of my family and many friends are Apple Mac users as well and that adds another complication since I am firmly in the 'proper' PC camp.

One of my most useful thumb drives is a Sandisk wifi capable usb stick. I've been taking that on vacations for many years now as it can solve a number of transfer situations, though it does require an agent install. I now also have a drive with both usb and lightning connectors to allow faster transfers for IOS products as the wireless is pretty slow.

Cool idea! New to me, thanks.

16GB Sandisk Connect

128GB Sandisk Connect

(on Amazon)

Transcend and Toshiba made (or still make) WiFi SD(XC) cards but they are way overpriced.

 

[Chris Noble]

The topic of DVDs for data storage has come up repeatedly. The DVD format was specifically developed for movies, which can handle high error rates because of the benign requirements of our eyesight. Applying them to data storage was an afterthought, based on intense error-correction protocols and the insistent push of marketing people. Some of us never went near a DVD-RW for data storage for that reason. But even now, there are still advocates, on this forum and elsewhere.

Let's not conflate the issues of writeable media with the format. DVDs were used to deliver software

No, CDs were. Much lower density. And sometimes DVD-Rs, never recordable DVDs.

without issue until the general death of optical drives as standard equipment. And the M-Disk has proven out as a reliable way to archive data.

But as people wanted and needed lot of blank media, a lot of unscrupulous types took advantage with low quality products at cheaper prices, and often pretended to be well regarded brands. Verbatim, among others, suffered from a rash of counterfeits and that may have been the case for the prior poster. There also seemed to be a lot of trouble for folks that burned at high speed on one drive, which could not be read on others. Recording at a slower speed could lower that problem to a degree.

... lipstick on a pig .

 

[Soaptrail]

After a while i only bought Verbatim optical media. But after buying a BD-R writer for my PC I backed up my RAW files on 34 BD-R verbatim discs. I validated they worked and appended a few of them. To save space i deleted those RAW from my computer. Then i went to pull a RAW file from the discs. I could not read data on 33 of the 34 discs. I even tried my home BD players.

Made me sad to get not one or two bad discs but all but one was horrendous.

My word! That is indeed a very sad tale. Over here Blu-Ray never really took off, at least, not in my social circles: I don't know anybody who has a Blu-Ray drive/player.

The topic of DVDs for data storage has come up repeatedly. The DVD format was specifically developed for movies, which can handle high error rates because of the benign requirements of our eyesight. Applying them to data storage was an afterthought, based on intense error-correction protocols and the insistent push of marketing people. Some of us never went near a DVD-RW for data storage for that reason. But even now, there are still advocates, on this forum and elsewhere.

Soaptrail, did you approach specialized data-recovery outfits to see if they can help you recover your data?

I did not attempt any recovery other than what i could try on my PC. But since I had the JPEG's it was not a true loss. That would have been better to put in my initial post.

I never had issues with writing to CD-R nor DVD+/-R but those BD-R's were frustrating. I bought them on Newegg so i presume they were not counterfeit but no idea. I did pull teeth to at least get Verbatim to give me a new spindle of BD-r which I still have since i will not trust them to put data on. In Fact i don't have a 5.25 drive anymore on my main PC so i only use internal drives and a NAS for backups which is not perfect but so far it suites me.

 

[David Lal]

Yes, indeed - witness the ascendency of the mobile telephone. Most of my family and many friends are Apple Mac users as well and that adds another complication since I am firmly in the 'proper' PC camp.

One of my most useful thumb drives is a Sandisk wifi capable usb stick. I've been taking that on vacations for many years now as it can solve a number of transfer situations, though it does require an agent install. I now also have a drive with both usb and lightning connectors to allow faster transfers for IOS products as the wireless is pretty slow.

Cool idea! New to me, thanks.

16GB Sandisk Connect

128GB Sandisk Connect

(on Amazon)

Transcend and Toshiba made (or still make) WiFi SD(XC) cards but they are way overpriced.

Yes, I have a couple of those smaller Sandisk WiFi-enabled SD cards also my D750 will communicate over WiFi but I have never bothered with all that. Popping the card into a card reader is fine for me.