Best way to "test" integrity of RAW files -- Adobe DNG creator maybe?

Win 11 shows a thumbnail of my RAW files in Explorer. View/Extra large icons is adequate to confirm the file is readable, also is good enough for first level culling. I can see 24 files at a time.

sirhawkeye64 said:

I ran into this with the sample Z8 from DPR since they appear to be corrupt as well but it does render the thumbnail/preview, however even NX Studio won't open them.

Click to expand...

Are any of the corrupted files still there? I'd like to have a look at one or two.

NO, they do not.

sirhawkeye64 said:

s_grins said:

I haven't had any corrupted Raw files, yet,but let me guess that every RAW development software, one way or another, will inform you that the file is corrupt and can't be open.

Click to expand...

Well, yeah they will. Although in the case of ACR, it just shows the corrupted file when you open it. LR does something similar, or if it's really messed up, it will say it can't read it or it will just render the preview and when you click on it, it will say it can't read the file. While I know this is a known issue now, I ran into this with the sample Z8 from DPR since they appear to be corrupt as well but it does render the thumbnail/preview, however even NX Studio won't open them. However I don't know what caused the corruption though (supposedly they do have good RAW files to test but you have to ask for them, but I don't know if they were re-shot or it's an issue with he posting of the files or the files themselves).

Click to expand...

This is not a corruption of a file. Some RAW convertors read full sensor area, including border pixels.

Again, you can use your regular RAW software to find out whether your file is corrupt. I suggest that in wast majority of cases the reason your software can't open the fle is that your camera is not supported.

sirhawkeye64 said:

Regardless, I would think that Nikon at least and probably most mfrs would employ some type of checksum at least for the camera to use to verify the file was written to the card at least.... unless this is NOT done due to he addition time and processing overhead needed to generate a hash for each file. However in the case of NIkon at least, it does dump a .DAT file on the card, which I don't know what this is and have always just deleted it (maybe it contains file info for their proprietary software to use perhaps and/or it contains the mentioned data that their software can read and verify files...? I don't know).

Click to expand...

.DAT file is just a list of files you took. You do not have to copy it on your computer.

--
If you want to be equal, you have to be better...

sirhawkeye64 said:

Regardless, I would think that Nikon at least and probably most mfrs would employ some type of checksum ...

Click to expand...

They don't.

s_grins said:

Digital Nigel said:

Raw files contain a compressed copy of the OOC JPEG. This could be corrupted without the raw data being affected (or vice versa). Ditto the exif data.

Click to expand...

NO, they do not.

Click to expand...

Digital Nigel's comments look correct to me. What do you think is wrong?

s_grins said:

sirhawkeye64 said:

s_grins said:

I haven't had any corrupted Raw files, yet,but let me guess that every RAW development software, one way or another, will inform you that the file is corrupt and can't be open.

Click to expand...

Well, yeah they will. Although in the case of ACR, it just shows the corrupted file when you open it. LR does something similar, or if it's really messed up, it will say it can't read it or it will just render the preview and when you click on it, it will say it can't read the file. While I know this is a known issue now, I ran into this with the sample Z8 from DPR since they appear to be corrupt as well but it does render the thumbnail/preview, however even NX Studio won't open them. However I don't know what caused the corruption though (supposedly they do have good RAW files to test but you have to ask for them, but I don't know if they were re-shot or it's an issue with he posting of the files or the files themselves).

Click to expand...

This is not a corruption of a file. Some RAW convertors read full sensor area, including border pixels.

Again, you can use your regular RAW software to find out whether your file is corrupt. I suggest that in wast majority of cases the reason your software can't open the fle is that your camera is not supported.

Click to expand...

The OP's Nikon camera is supported by NX Studio. If NX Studio won't open the files, something is wrong with the files.

s_grins said:

s_grins said:

Regardless, I would think that Nikon at least and probably most mfrs would employ some type of checksum at least for the camera to use to verify the file was written to the card at least.... unless this is NOT done due to he addition time and processing overhead needed to generate a hash for each file. However in the case of NIkon at least, it does dump a .DAT file on the card, which I don't know what this is and have always just deleted it (maybe it contains file info for their proprietary software to use perhaps and/or it contains the mentioned data that their software can read and verify files...? I don't know).

Click to expand...

.DAT file is just a list of files you took. You do not have to copy it on your computer.

Click to expand...

How do you know what the Nikon .DAT file contains? I couldn't find a reliable explanation when I searched. And why would a list of the files you took need to be saved on the card?

ishwanu said:

s_grins said:

sirhawkeye64 said:

s_grins said:

I haven't had any corrupted Raw files, yet,but let me guess that every RAW development software, one way or another, will inform you that the file is corrupt and can't be open.

Click to expand...

Well, yeah they will. Although in the case of ACR, it just shows the corrupted file when you open it. LR does something similar, or if it's really messed up, it will say it can't read it or it will just render the preview and when you click on it, it will say it can't read the file. While I know this is a known issue now, I ran into this with the sample Z8 from DPR since they appear to be corrupt as well but it does render the thumbnail/preview, however even NX Studio won't open them. However I don't know what caused the corruption though (supposedly they do have good RAW files to test but you have to ask for them, but I don't know if they were re-shot or it's an issue with he posting of the files or the files themselves).

Click to expand...

This is not a corruption of a file. Some RAW convertors read full sensor area, including border pixels.

Again, you can use your regular RAW software to find out whether your file is corrupt. I suggest that in wast majority of cases the reason your software can't open the fle is that your camera is not supported.

Click to expand...

The OP's Nikon camera is supported by NX Studio. If NX Studio won't open the files, something is wrong with the files.

Click to expand...

Maybe OP tried to edit RAW directly in camera? Made a rotation?

ishwanu said:

ishwanu said:

ishwanu said:

Regardless, I would think that Nikon at least and probably most mfrs would employ some type of checksum at least for the camera to use to verify the file was written to the card at least.... unless this is NOT done due to he addition time and processing overhead needed to generate a hash for each file. However in the case of NIkon at least, it does dump a .DAT file on the card, which I don't know what this is and have always just deleted it (maybe it contains file info for their proprietary software to use perhaps and/or it contains the mentioned data that their software can read and verify files...? I don't know).

Click to expand...

.DAT file is just a list of files you took. You do not have to copy it on your computer.

Click to expand...

How do you know what the Nikon .DAT file contains? I couldn't find a reliable explanation when I searched. And why would a list of the files you took need to be saved on the card?

Click to expand...

Just open it.

--
If you want to be equal, you have to be better...

s_grins said:

ishwanu said:

s_grins said:

sirhawkeye64 said:

s_grins said:

I haven't had any corrupted Raw files, yet,but let me guess that every RAW development software, one way or another, will inform you that the file is corrupt and can't be open.

Click to expand...

Well, yeah they will. Although in the case of ACR, it just shows the corrupted file when you open it. LR does something similar, or if it's really messed up, it will say it can't read it or it will just render the preview and when you click on it, it will say it can't read the file. While I know this is a known issue now, I ran into this with the sample Z8 from DPR since they appear to be corrupt as well but it does render the thumbnail/preview, however even NX Studio won't open them. However I don't know what caused the corruption though (supposedly they do have good RAW files to test but you have to ask for them, but I don't know if they were re-shot or it's an issue with he posting of the files or the files themselves).

Click to expand...

This is not a corruption of a file. Some RAW convertors read full sensor area, including border pixels.

Again, you can use your regular RAW software to find out whether your file is corrupt. I suggest that in wast majority of cases the reason your software can't open the fle is that your camera is not supported.

Click to expand...

The OP's Nikon camera is supported by NX Studio. If NX Studio won't open the files, something is wrong with the files.

Click to expand...

Maybe OP tried to edit RAW directly in camera? Made a rotation?

Click to expand...

Maybe anything. That doesn't mean you can say this is not a corruption of a file because you don't know.

s_grins said:

s_grins said:

s_grins said:

s_grins said:

Regardless, I would think that Nikon at least and probably most mfrs would employ some type of checksum at least for the camera to use to verify the file was written to the card at least.... unless this is NOT done due to he addition time and processing overhead needed to generate a hash for each file. However in the case of NIkon at least, it does dump a .DAT file on the card, which I don't know what this is and have always just deleted it (maybe it contains file info for their proprietary software to use perhaps and/or it contains the mentioned data that their software can read and verify files...? I don't know).

Click to expand...

.DAT file is just a list of files you took. You do not have to copy it on your computer.

Click to expand...

How do you know what the Nikon .DAT file contains? I couldn't find a reliable explanation when I searched. And why would a list of the files you took need to be saved on the card?

Click to expand...

Just open it.

Click to expand...

Have you opened it? What did you use to open it? What did you see?

sirhawkeye64 said:

One thing I wish mfrs would develop (or even a third party, since the Codec specs are available) is to create a tool that can be run against RAW files to verify if they are intact or not. I guess it wouldn't be able to tell if a bit got flipped but perhaps if the RAW data portion is incomplete or contains invalid values).

Click to expand...

Some time ago I developed and sold a tool to do this called ImageVerifier, but I haven't supported or sold it for maybe 8 - 10 years. However, as I recall, I checked raw files by running them through DNG Converter. It also checked JPEGs, TIFFs, and other formats using internal algorithms.

I think the issue with manufacturers is that if they had checksums or otherwise provided a way to check raw files, customers would find errors, and then that would result in service issues and, ultimately, a reduction in perceived quality relative to other manufacturers who didn't check. Since a bad card would likely be at fault, and the manufacturers don't even supply cards, this would be a real problem for them.

This is somewhat like the situation with error-correcting (ECC) memory for computers. Mainstream manufacturers don't do it, because it would cause the user to get too many errors. Of course, the errors are still there, but undetected except for crashes and occasional wrong results, none of which can be pinned down. Here are two articles on this subject that I wrote about 17 years ago:



The computer with ECC memory would be considered to be less reliable, because it periodically reported errors, when it is in fact more reliable.

ishwanu said:

s_grins said:

ishwanu said:

s_grins said:

sirhawkeye64 said:

s_grins said:

I haven't had any corrupted Raw files, yet,but let me guess that every RAW development software, one way or another, will inform you that the file is corrupt and can't be open.

Click to expand...

Well, yeah they will. Although in the case of ACR, it just shows the corrupted file when you open it. LR does something similar, or if it's really messed up, it will say it can't read it or it will just render the preview and when you click on it, it will say it can't read the file. While I know this is a known issue now, I ran into this with the sample Z8 from DPR since they appear to be corrupt as well but it does render the thumbnail/preview, however even NX Studio won't open them. However I don't know what caused the corruption though (supposedly they do have good RAW files to test but you have to ask for them, but I don't know if they were re-shot or it's an issue with he posting of the files or the files themselves).

Click to expand...

This is not a corruption of a file. Some RAW convertors read full sensor area, including border pixels.

Again, you can use your regular RAW software to find out whether your file is corrupt. I suggest that in wast majority of cases the reason your software can't open the fle is that your camera is not supported.

Click to expand...

The OP's Nikon camera is supported by NX Studio. If NX Studio won't open the files, something is wrong with the files.

Click to expand...

Maybe OP tried to edit RAW directly in camera? Made a rotation?

Click to expand...

Maybe anything. That doesn't mean you can say this is not a corruption of a file because you don't know.

ishwanu said:

ishwanu said:

ishwanu said:

ishwanu said:

Regardless, I would think that Nikon at least and probably most mfrs would employ some type of checksum at least for the camera to use to verify the file was written to the card at least.... unless this is NOT done due to he addition time and processing overhead needed to generate a hash for each file. However in the case of NIkon at least, it does dump a .DAT file on the card, which I don't know what this is and have always just deleted it (maybe it contains file info for their proprietary software to use perhaps and/or it contains the mentioned data that their software can read and verify files...? I don't know).

Click to expand...

.DAT file is just a list of files you took. You do not have to copy it on your computer.

Click to expand...

How do you know what the Nikon .DAT file contains? I couldn't find a reliable explanation when I searched. And why would a list of the files you took need to be saved on the card?

Click to expand...

Just open it.

Click to expand...

Have you opened it? What did you use to open it? What did you see?

Click to expand...

Notepad

s_grins said:

ishwanu said:

s_grins said:

ishwanu said:

s_grins said:

sirhawkeye64 said:

s_grins said:

I haven't had any corrupted Raw files, yet,but let me guess that every RAW development software, one way or another, will inform you that the file is corrupt and can't be open.

Click to expand...

Well, yeah they will. Although in the case of ACR, it just shows the corrupted file when you open it. LR does something similar, or if it's really messed up, it will say it can't read it or it will just render the preview and when you click on it, it will say it can't read the file. While I know this is a known issue now, I ran into this with the sample Z8 from DPR since they appear to be corrupt as well but it does render the thumbnail/preview, however even NX Studio won't open them. However I don't know what caused the corruption though (supposedly they do have good RAW files to test but you have to ask for them, but I don't know if they were re-shot or it's an issue with he posting of the files or the files themselves).

Click to expand...

This is not a corruption of a file. Some RAW convertors read full sensor area, including border pixels.

Again, you can use your regular RAW software to find out whether your file is corrupt. I suggest that in wast majority of cases the reason your software can't open the fle is that your camera is not supported.

Click to expand...

The OP's Nikon camera is supported by NX Studio. If NX Studio won't open the files, something is wrong with the files.

Click to expand...

Maybe OP tried to edit RAW directly in camera? Made a rotation?

Click to expand...

Maybe anything. That doesn't mean you can say this is not a corruption of a file because you don't know.

s_grins said:

s_grins said:

s_grins said:

s_grins said:

Regardless, I would think that Nikon at least and probably most mfrs would employ some type of checksum at least for the camera to use to verify the file was written to the card at least.... unless this is NOT done due to he addition time and processing overhead needed to generate a hash for each file. However in the case of NIkon at least, it does dump a .DAT file on the card, which I don't know what this is and have always just deleted it (maybe it contains file info for their proprietary software to use perhaps and/or it contains the mentioned data that their software can read and verify files...? I don't know).

Click to expand...

.DAT file is just a list of files you took. You do not have to copy it on your computer.

Click to expand...

How do you know what the Nikon .DAT file contains? I couldn't find a reliable explanation when I searched. And why would a list of the files you took need to be saved on the card?

Click to expand...

Just open it.

Click to expand...

Have you opened it? What did you use to open it? What did you see?

Click to expand...

Notepad

Click to expand...

Please show us the result you saw when you opened it.

This is how Nikon .dat looks like. Panasonic .dat looks different







--
If you want to be equal, you have to be better...

s_grins said:

This is how Nikon .dat looks like. Panasonic .dat looks different

Click to expand...

Ah, I see it's named NC_FLLST.DAT. Now we can do a search for the name and find this:

ishwanu said:

s_grins said:

Digital Nigel said:

Raw files contain a compressed copy of the OOC JPEG. This could be corrupted without the raw data being affected (or vice versa). Ditto the exif data.

Click to expand...

NO, they do not.

Click to expand...

Digital Nigel's comments look correct to me. What do you think is wrong?

Click to expand...

I do not believe in embedded JPGs, as well in compressed copies, that are part of the RAW.

In my department, only header with camera settings is included with RAW data. These camera settings needed for built-in software to create image and feed it into LCD/EVF.

When you shoot OOC JPGs, you add to (or modify) camera settings to commit camera software and produce and save JPG.

--
If you want to be equal, you have to be better...

Thanks for the link. Probably, I need a vodka bottle to read and understand all these attributes.

Cheers

S.

Oh, and not to forget to mention, that this .DAT was produced by Nikon Z 50

s_grins said:

ishwanu said:

s_grins said:

Digital Nigel said:

Raw files contain a compressed copy of the OOC JPEG. This could be corrupted without the raw data being affected (or vice versa). Ditto the exif data.

Click to expand...

NO, they do not.

Click to expand...

Digital Nigel's comments look correct to me. What do you think is wrong?

Click to expand...

I do not believe in embedded JPGs, as well in compressed copies, that are part of the RAW.

Click to expand...

They are included in the raw file, along with the raw data and the exif data.

s_grins said:

In my department, only header with camera settings is included with RAW data. These camera settings needed for built-in software to create image and feed it into LCD/EVF.

When you shoot OOC JPGs, you add to (or modify) camera settings to commit camera software and produce and save JPG.

Click to expand...

Yes, and a compressed version of that OOC JPEG is included in every raw file. Some file viewers show that embedded JPEG (fast, easy), others process the raw data to render an image from it (slower, more difficult, not always possible). A few, like FastStone, give you the choice (but sometimes have to show the embedded JPEG if they can’t process the raw data).

s_grins said:

ishwanu said:

s_grins said:

Digital Nigel said:

Raw files contain a compressed copy of the OOC JPEG. This could be corrupted without the raw data being affected (or vice versa). Ditto the exif data.

Click to expand...

NO, they do not.

Click to expand...

Digital Nigel's comments look correct to me. What do you think is wrong?

Click to expand...

I do not believe in embedded JPGs, as well in compressed copies, that are part of the RAW.

In my department, only header with camera settings is included with RAW data. These camera settings needed for built-in software to create image and feed it into LCD/EVF.

Click to expand...

The statement was a little oddly phrased, but a RAW file does contain an embedded compressed JPEG. That is not something we can choose to disbelieve.

s_grins said:

When you shoot OOC JPGs, you add to (or modify) camera settings to commit camera software and produce and save JPG.

Click to expand...

Even if you don't intentionally shoot JPEGs, you will get them embedded in the RAW files (unless perhaps some specific cameras omit them).

ishwanu said:

s_grins said:

ishwanu said:

s_grins said:

Digital Nigel said:

Raw files contain a compressed copy of the OOC JPEG. This could be corrupted without the raw data being affected (or vice versa). Ditto the exif data.

Click to expand...

NO, they do not.

Click to expand...

Digital Nigel's comments look correct to me. What do you think is wrong?

Click to expand...

I do not believe in embedded JPGs, as well in compressed copies, that are part of the RAW.

In my department, only header with camera settings is included with RAW data. These camera settings needed for built-in software to create image and feed it into LCD/EVF.

Click to expand...

The statement was a little oddly phrased, but a RAW file does contain an embedded compressed JPEG. That is not something we can choose to disbelieve.

ishwanu said:

When you shoot OOC JPGs, you add to (or modify) camera settings to commit camera software and produce and save JPG.

Click to expand...

Even if you don't intentionally shoot JPEGs, you will get them embedded in the RAW files (unless perhaps some specific cameras omit them).

Click to expand...

Firstly, there is no need in embedded JPGs

Secondly, did yoy try to open RAW script? I did, I did not find anything embedded (it was time ago Panasonic RAW files)

s_grins said:

ishwanu said:

s_grins said:

ishwanu said:

s_grins said:

Digital Nigel said:

Raw files contain a compressed copy of the OOC JPEG. This could be corrupted without the raw data being affected (or vice versa). Ditto the exif data.

Click to expand...

NO, they do not.

Click to expand...

Digital Nigel's comments look correct to me. What do you think is wrong?

Click to expand...

I do not believe in embedded JPGs, as well in compressed copies, that are part of the RAW.

In my department, only header with camera settings is included with RAW data. These camera settings needed for built-in software to create image and feed it into LCD/EVF.

Click to expand...

The statement was a little oddly phrased, but a RAW file does contain an embedded compressed JPEG. That is not something we can choose to disbelieve.

s_grins said:

When you shoot OOC JPGs, you add to (or modify) camera settings to commit camera software and produce and save JPG.

Click to expand...

Even if you don't intentionally shoot JPEGs, you will get them embedded in the RAW files (unless perhaps some specific cameras omit them).

Click to expand...

Firstly, there is no need in embedded JPGs

Click to expand...

We are saying they are there whether they're needed or not.

s_grins said:

Secondly, did yoy try to open RAW script? I did, I did not find anything embedded (it was time ago Panasonic RAW files)

Click to expand...

Point to a RAW file somewhere and I'll show you the embedded JPEG.