Is flash memory unreliable due to exFAT?

[Tom_N]

The topic of DVDs for data storage has come up repeatedly. The DVD format was specifically developed for movies, which can handle high error rates because of the benign requirements of our eyesight. Applying them to data storage was an afterthought, based on intense error-correction protocols and the insistent push of marketing people. Some of us never went near a DVD-RW for data storage for that reason. But even now, there are still advocates, on this forum and elsewhere.

The CD-Audio format was specifically developed for music, with the idea that players could paper over some uncorrectable errors without listeners noticing. Once people started using CDs for data storage, CD-ROM/CD-R/CD-RW discs had to layer on more robust error correction than what you'd get on CD-Audio.

DVD-Video and Blu-Ray Video were designed to store audio and video in the form of computer files in the first place. There's some extra copy protection/DRM crud – like the decryption key area of DVDs that consumer drives can read, but not record – but file storage was not an afterthought, the way it was with CDs.

 

[Sean Nelson]

Let's not conflate the issues of writeable media with the format. DVDs were used to deliver software

No, CDs were. Much lower density. And sometimes DVD-Rs, never recordable DVDs.

He didn't say "recordable DVDs", he said "DVDs", which is true. Plenty of software was delivered on DVD discs. The DVD format was designed from the ground up with a file system format use to store data files, even for video - so it was ideally suited for software delivery in an era where CDs were no longer large enough (my Adobe CS6 master collection install kit comprises six DVDs) and high speed Internet connections were very uncommon.

 

[Chris Noble]

Let's not conflate the issues of writeable media with the format. DVDs were used to deliver software

No, CDs were. Much lower density. And sometimes DVD-Rs, never recordable DVDs.

He didn't say "recordable DVDs", he said "DVDs",

Seems like you did not read the original post that started this sub-thread.

 

[Chris Noble]

The topic of DVDs for data storage has come up repeatedly. The DVD format was specifically developed for movies, which can handle high error rates because of the benign requirements of our eyesight. Applying them to data storage was an afterthought, based on intense error-correction protocols and the insistent push of marketing people. Some of us never went near a DVD-RW for data storage for that reason. But even now, there are still advocates, on this forum and elsewhere.

The CD-Audio format was specifically developed for music, with the idea that players could paper over some uncorrectable errors without listeners noticing. Once people started using CDs for data storage, CD-ROM/CD-R/CD-RW discs had to layer on more robust error correction than what you'd get on CD-Audio.

DVD-Video and Blu-Ray Video were designed to store audio and video in the form of computer files in the first place. There's some extra copy protection/DRM crud – like the decryption key area of DVDs that consumer drives can read, but not record – but file storage was not an afterthought, the way it was with CDs.

You are confusing files used to store digital video vs. files used to store other types of data. The error-rate requirements are radically different.

 

[kelpdiver]

Let's not conflate the issues of writeable media with the format. DVDs were used to deliver software

No, CDs were. Much lower density. And sometimes DVD-Rs, never recordable DVDs.

He didn't say "recordable DVDs", he said "DVDs",

Seems like you did not read the original post that started this sub-thread.

nah...the subthread is about your nonsense about the DVD format not having data integrity. That's already been corrected.

For your statement to work, the movie industry would have intended to ship us product on writeable discs.

I have to wonder how you can claim to never had received software on a dvd before. Installing an OS or an Adobe product off a barrel of cds would be really painful.

 

[Chris Noble]

Let's not conflate the issues of writeable media with the format. DVDs were used to deliver software

No, CDs were. Much lower density. And sometimes DVDs, never recordable DVDs.

He didn't say "recordable DVDs", he said "DVDs",

Seems like you did not read the original post that started this sub-thread.

nah...the subthread is about your nonsense about the DVD format not having data integrity. That's already been corrected.

For your statement to work, the movie industry would have intended to ship us product on writeable discs.

I have to wonder how you can claim to never had received software on a dvd before. Installing an OS or an Adobe product off a barrel of cds would be really painful.

Go ahead; use DVD-Rs. It's your data.

 

[Sean Nelson]

DVD-Video and Blu-Ray Video were designed to store audio and video in the form of computer files in the first place. There's some extra copy protection/DRM crud – like the decryption key area of DVDs that consumer drives can read, but not record – but file storage was not an afterthought, the way it was with CDs.

You are confusing files used to store digital video vs. files used to store other types of data. The error-rate requirements are radically different.

There's only one file system, one physical data encoding method and one error detection and correction strategy for BluRay disks (and a single separate set of these for DVDs), and they are used both for disks that contain data files as well as for video disks. If you insert a BluRay or DVD video disk into a computer with an optical drive you can browse its file system using Windows File Explorer.

The only wrinkle is that the video files are encrypted and you can't make any sense of them without the decryption key that's stored in a special dedicated area that's not writable by consumer disc burners.

This is what the main video folder of my BluRay of "Gravity" looks like. The disk is in a drive that can burn BluRay discs but the disc itself is a BD-ROM that was pressed, aluminized and finished at a factory:



All DVD and BluRay disks, be they pressed at the factory or burned in a writable drive, are essentially disks containing data files as far as a computer is concerned.

 

[Chris Noble]

DVD-Video and Blu-Ray Video were designed to store audio and video in the form of computer files in the first place. There's some extra copy protection/DRM crud – like the decryption key area of DVDs that consumer drives can read, but not record – but file storage was not an afterthought, the way it was with CDs.

You are confusing files used to store digital video vs. files used to store other types of data. The error-rate requirements are radically different.

There's only one file system, one physical data encoding method and one error detection and correction strategy for BluRay disks (and a single separate set of these for DVDs), and they are used both for disks that contain data files as well as for video disks. If you insert a BluRay or DVD video disk into a computer with an optical drive you can browse its file system using Windows File Explorer.

The only wrinkle is that the video files are encrypted and you can't make any sense of them without the decryption key that's stored in a special dedicated area that's not writable by consumer disc burners.

This is what the main video folder of my BluRay of "Gravity" looks like. The disk is in a drive that can burn BluRay discs but the disc itself is a BD-ROM that was pressed, aluminized and finished at a factory:



All DVD and BluRay disks, be they pressed at the factory or burned in a writable drive, are essentially disks containing data files as far as a computer is concerned.

What you say is basically correct, but beside my point, which is that recordable optical disks can't be used to store data files reliably -- and that has nothing to do with the file system. But go ahead and use them for that purpose if you want. It's your data, and this is just a discussion forum.

 

[CAcreeks]

After a while i only bought Verbatim optical media. But after buying a BD-R writer for my PC I backed up my RAW files on 34 BD-R verbatim discs. I validated they worked and appended a few of them. To save space i deleted those RAW from my computer. Then i went to pull a RAW file from the discs. I could not read data on 33 of the 34 discs. I even tried my home BD players.

Maybe your mistake was "appending to" a few of them. Although standards allow appending to an already finalized disc image, I would not trust software to do it correctly.

I don't trust dual-layer BluRay either, but perhaps it works better than dual-layer DVD-R or DVD+R ever did for us. Single layer BluRay is limited to 25GB, which is not much.

Made me sad to get not one or two bad discs but all but one was horrendous.

Saving huge Raw files (or even larger DNG files) seems like the wrong approach now that we have lossless HEIC.

GIMP can save "nearly lossless HEIC" at 8, 10, or 12 bits per color channel. I don't know what nearly lossless means. A regular HEIC export is 1/10 the size of a NEF image I tried (from Nikon Z7). And half the size of the JPEG Q92 image that DxO PhotoLab produced. It's early days: exiftool does not work on HEIC, or GIMP failed to save EXIF.

 

[CAcreeks]

Go ahead; use DVD-Rs. It's your data.

It has been posted here that DVD+R was designed for data storage, whereas CDR was not, however my experience is that CDR discs have lasted for decades (so far) whereas DVD+R discs sometimes have not. I'm hoping BluRay is better than DVD.

Taiyo-Yuden CDR was supposed to be more durable than other manufactures, but of course the problem with CDR is very small capacity.

 

[Sean Nelson]

All DVD and BluRay disks, be they pressed at the factory or burned in a writable drive, are essentially disks containing data files as far as a computer is concerned.

What you say is basically correct, but beside my point, which is that recordable optical disks can't be used to store data files reliably -- and that has nothing to do with the file system. But go ahead and use them for that purpose if you want. It's your data, and this is just a discussion forum.

Hey, I never advocated for anything. Just pointing out that the only difference between factory-pressed disks and writable disks is in the physical storage format of the bits (pressed, aluminized pits and lands vs. dye spots, etc). Everything above that, including error detection and correction, the file system, etc. is the same.

 

[Chris Noble]

All DVD and BluRay disks, be they pressed at the factory or burned in a writable drive, are essentially disks containing data files as far as a computer is concerned.

What you say is basically correct, but beside my point, which is that recordable optical disks can't be used to store data files reliably -- and that has nothing to do with the file system. But go ahead and use them for that purpose if you want. It's your data, and this is just a discussion forum.

Hey, I never advocated for anything. Just pointing out that the only difference between factory-pressed disks and writable disks is in the physical storage format of the bits (pressed, aluminized pits and lands vs. dye spots, etc). Everything above that, including error detection and correction, the file system, etc. is the same.

The difference between the two that is relevant to this thread is that the writable disks are much less reliable and stable over time than the factory-pressed disks.

 

[CAcreeks]

Chris argued...

Hey, I never advocated for anything. Just pointing out that the only difference between factory-pressed disks and writable disks is in the physical storage format of the bits (pressed, aluminized pits and lands vs. dye spots, etc). Everything above that, including error detection and correction, the file system, etc. is the same.

Good point (as usual) Sean.

ISO 9660, the usual filesystem for optical disc, has in Rockridge extensions ownership and permission information.

Not exFAT. All files are global read, write, execute. They do seem to have ownership, but I'm not sure where that comes from.

 

[kelpdiver]

What you say is basically correct, but beside my point, which is that recordable optical disks can't be used to store data files reliably -- and that has nothing to do with the file system.

That's your replacement point, to be accurate. You've given up on the original argument that the DVD format was built around unreliability.

 

[kelpdiver]

Taiyo-Yuden CDR was supposed to be more durable than other manufactures, but of course the problem with CDR is very small capacity.

that brand got counterfeited more than any other.

 

[bakubo]

Have you experienced external SSD corruption? I have not, yet.

Last week I lost my 6th USB thumb drive, if I recall correctly. More than can be counted on one hand. On both Mac and PC, this ruined thumb drive claimed reformatting was required. Unmountable. Data not readable.

Is flash memory unreliable because of exFAT, or because it is flash memory?

I'm wondering if I should reformat my external SSD devices, mostly Samsung T7. They come with exFAT, which is convenient because of its near universality. Here's an informative writeup about exFAT reliability; save time by skipping to conclusion.

https://pawitp.medium.com/notes-on-exfat-and-reliability-d2f194d394c2

Amusing how your post about exFAT and SSDs was hijacked and morphed into a big fight about DVDs.

I am much more interested in your subject about exFAT and SSDs.

 

[Chris Noble]

What you say is basically correct, but beside my point, which is that recordable optical disks can't be used to store data files reliably -- and that has nothing to do with the file system.

That's your replacement point, to be accurate. You've given up on the original argument that the DVD format was built around unreliability.

I shouldn't waste time on this as you could simply read and try to understand the posts. The DVD format was built around video streaming, which has a much lower acceptable-error-rate requirement than other types of data. Nothing is "built around unreliability"... what a ridiculous concept.

 

[CAcreeks]

Have you experienced external SSD corruption? I have not, yet.

Amusing how your post about exFAT and SSDs was hijacked and morphed into a big fight about DVDs.

I am much more interested in your subject about exFAT and SSDs.

Yes, I conclude that the mostly deskside PC users in this forum have not been using external SSD (such as the Samsung T7) in large numbers. External SSD appeals mostly to laptop users.

On Reddit groups, I don't see evidence that exFAT on SSD is a terrible choice.

Linux to the rescue (or chkdsk):
https://www.phoronix.com/news/Linux-exFAT-Progs-1.2

 

[Sean Nelson]

The DVD format was built around video streaming, which has a much lower acceptable-error-rate requirement than other types of data.

Sorry, folks, I have to rebut this because it's just not true.

By the time the DVD format was developed CDs had proven their worth as a file storage medium. Unlike CDs, onto which file storage capabilities were retrofitted, DVDs were designed from the ground up as a file storage medium with full error detection and correction capabilities. The ISO-13346 file system, also known as "Universal Disk Format", was chosen as the format for data storage.

DVD Video is the standard that's layered on top of that file system to specify how video is stored. It describes the folder structure, video codecs, subtitles, programming interface that drives DVD menus, etc.

So the implication that DVDs have poorer error correction capabilities because video is more tolerant of errors is just plain false. Both data DVDs and Video DVDs are built on the same file system that was designed to use robust error detection and correction to avoid data loss.

Chris is saying that writable DVDs are a poor data storage mechanism, and I'm not arguing against that. But it's not because the generic DVD format is a poor data storage mechanism, it's because the specific media used with writable disks isn't as robust as factory pressed and aluminized disks. Errors that may occur when reading DVD discs (be they pressed or burned) are due to faults or degradation of one kind or another in the physical media that have become so bad as to exceed the capabilities of the error correcting codes, which are quite sophisticated.

 

[Chris Noble]

The DVD format was built around video streaming, which has a much lower acceptable-error-rate requirement than other types of data.

Sorry, folks, I have to rebut this because it's just not true.

Sean, here's one article about the multiple ways video-streaming errors can be compensated for. If it's a movie, the visual quality is degraded. If you are dealing with a spreadsheet or a text file, the output is unusable. Big difference.

Chris is saying that writable DVDs are a poor data storage mechanism, and I'm not arguing against that. But it's not because the generic DVD format is a poor data storage mechanism, it's because the specific media used with writable disks isn't as robust as factory pressed and aluminized disks. Errors that may occur when reading DVD discs (be they pressed or burned) are due to faults or degradation of one kind or another in the physical media that have become so bad as to exceed the capabilities of the error correcting codes, which are quite sophisticated.

My previous post: , "The difference between the two that is relevant to this thread is that the writable disks are much less reliable and stable over time than the factory-pressed disks."

Can we move on?