Best way to "test" integrity of RAW files -- Adobe DNG creator maybe?

[sirhawkeye64]

What's the best way to bulk "test" RAW files after copying from a memory card to check for corrupted images without having to go through each one in Bridge/ACR?

I was thinking that running DNG converter on them and creating DNGs could help identify any corrupted files (as the program would either not read them or it might just create a corrupted image but I can probably preview DNGs faster since they are smaller than my RAW files which are usually 45MP image files so it can take some time to thumb through a lot of them.

Would that be sufficient or is there another method, aside from importing them into LR...?

 

[Digital Nigel]

What's the best way to bulk "test" RAW files after copying from a memory card to check for corrupted images without having to go through each one in Bridge/ACR?

I was thinking that running DNG converter on them and creating DNGs could help identify any corrupted files (as the program would either not read them or it might just create a corrupted image but I can probably preview DNGs faster since they are smaller than my RAW files which are usually 45MP image files so it can take some time to thumb through a lot of them.

Would that be sufficient or is there another method, aside from importing them into LR...?

I suggest FastRawViewer: https://www.fastrawviewer.com

 

[g7_chord]

Do you want to test the integrity of the copying operation, or the integrity of the original file? If the former, the best and fastest way is to compare the two files bit-by-bit. There are many utilities available to do this.

If the latter, then maybe a bulk DNG conversion, although that's probably 100% certain.

 

[sirhawkeye64]

Do you want to test the integrity of the copying operation, or the integrity of the original file? If the former, the best and fastest way is to compare the two files bit-by-bit. There are many utilities available to do this.

If the latter, then maybe a bulk DNG conversion, although that's probably 100% certain.

Well, sort of both. But mostly that the image coming out of the camera is not corrupted (either by the card or the camera). File copy verification I have covered as I use programs like GoodSync which can compare hashes to make sure files are copied successfully from the card. But then again, I guess if the file is corrupt on the card from the camera (not even factoring in file copying yet) then it is what it is. I swap cards, and find one that doesn't have issues I guess. It's just in the past I've had both issues (corruption in-camera) and also after copying (although for the latter, I cannot say whether it was a card, camera, or file copy issue as I only found out after importing into LR). And quite frankly I don't remember if I checked both cards or not (camera was set to write the same to both cards, but this was like 7 years ago).

--
PLEASE NOTE: I usually unsubscribe from forums and comments after a period of time, so if I do not respond, that is likely the reason. Feel free to PM me if you have a questions or need clarification about a comment I made.

 

[lacogada]

But mostly that the image coming out of the camera is not corrupted (either by the card or the camera).

Curious ... why would a file be corrupted coming from card or camera ?

 

[sirhawkeye64]

But mostly that the image coming out of the camera is not corrupted (either by the card or the camera).

Curious ... why would a file be corrupted coming from card or camera ?

If there was some issue when the file was being written to the card. There is the slight possibility that I may have turned off the camera while it was still writing to the card so it's possible any power glitch as a result may have cause the writing to be interrupted or data to be lost. I got about half a usable image but the rest was trashed so corrupted basically. But it was limited to only maybe 3 images that I can recall, which is why I'm not entirely sure if power cycling the camera was the issue (but it was limited to this particular day and location though, so who knows). I only found out later when I got home, so again, many possibilities I guess, but it's enough to make me concerned moving forward and being more cautious.

 

[lacogada]

But mostly that the image coming out of the camera is not corrupted (either by the card or the camera).

Curious ... why would a file be corrupted coming from card or camera ?

If there was some issue when the file was being written to the card. There is the slight possibility that I may have turned off the camera while it was still writing to the card so it's possible any power glitch as a result may have cause the writing to be interrupted or data to be lost. I got about half a usable image but the rest was trashed so corrupted basically. But it was limited to only maybe 3 images that I can recall, which is why I'm not entirely sure if power cycling the camera was the issue (but it was limited to this particular day and location though, so who knows). I only found out later when I got home, so again, many possibilities I guess, but it's enough to make me concerned moving forward and being more cautious.

Thank you ... never had that happen.

 

[sirhawkeye64]

But mostly that the image coming out of the camera is not corrupted (either by the card or the camera).

Curious ... why would a file be corrupted coming from card or camera ?

If there was some issue when the file was being written to the card. There is the slight possibility that I may have turned off the camera while it was still writing to the card so it's possible any power glitch as a result may have cause the writing to be interrupted or data to be lost. I got about half a usable image but the rest was trashed so corrupted basically. But it was limited to only maybe 3 images that I can recall, which is why I'm not entirely sure if power cycling the camera was the issue (but it was limited to this particular day and location though, so who knows). I only found out later when I got home, so again, many possibilities I guess, but it's enough to make me concerned moving forward and being more cautious.

Thank you ... never had that happen.

I think the newer cameras perhaps will sense that data is being written and will wait until it's finished to power off (although good habit is to just wait until it's done and then turn it off, but I was probably in a rush), but again, hard to tell as it was 7 years ago.

 

[g7_chord]

One more thought: Almost all raw files contain a full-size JPEG (not a thumbnail, a full-size image), even if you're shooting raw only and not raw+JPEG. Most simple viewers just show that JPEG so don't look at the raw data at all. These kinds of viewers aren't what you want. (They claim to be raw viewers, but that's not really true.)

 

[sirhawkeye64]

One more thought: Almost all raw files contain a full-size JPEG (not a thumbnail, a full-size image), even if you're shooting raw only and not raw+JPEG. Most simple viewers just show that JPEG so don't look at the raw data at all. These kinds of viewers aren't what you want. (They claim to be raw viewers, but that's not really true.)

This is kind of why I was thinking that DNG creator might be a good tool for this, because it has to read the RAW data to demoasic the image... so if it runs into a problem, it will show up in the resulting embedded JPEG that it creates for the DNG (I assume it creates a new preview, not just copy the one from the RAW file). And my guess is that if it finds a file that is corrupt, it either converts it and generates a corrupt DNG or it flags it and doesn't convert it (that's what LR does when it can't read a particular file, even if it's a supported file type).

One thing i have been doing is once I've verified files are good (not corrupt) by using LR for now, I run a hash on the RAW files and save that to a spreadsheet for future reference.

One thing I wish mfrs would develop (or even a third party, since the Codec specs are available) is to create a tool that can be run against RAW files to verify if they are intact or not. I guess it wouldn't be able to tell if a bit got flipped but perhaps if the RAW data portion is incomplete or contains invalid values).

--
PLEASE NOTE: I usually unsubscribe from forums and comments after a period of time, so if I do not respond, that is likely the reason. Feel free to PM me if you have a questions or need clarification about a comment I made.

 

[Digital Nigel]

One more thought: Almost all raw files contain a full-size JPEG (not a thumbnail, a full-size image),

It depends on the make. For example, Sony camera raws only include a small embedded JPEG.

even if you're shooting raw only and not raw+JPEG. Most simple viewers just show that JPEG so don't look at the raw data at all. These kinds of viewers aren't what you want. (They claim to be raw viewers, but that's not really true.)

FastRawViewer always shows the raw, not the embedded JPEG.

 

[sirhawkeye64]

One more thought: Almost all raw files contain a full-size JPEG (not a thumbnail, a full-size image),

It depends on the make. For example, Sony camera raws only include a small embedded JPEG.

even if you're shooting raw only and not raw+JPEG. Most simple viewers just show that JPEG so don't look at the raw data at all. These kinds of viewers aren't what you want. (They claim to be raw viewers, but that's not really true.)

FastRawViewer always shows the raw, not the embedded JPEG.

But doesn't the camera use the RAW data to create the embedded JPEG? I guess in some respects, the embedded preview is so small that minor "defects" may not show up or be visible perhaps (like a few pixels that are clustered that have erroneous data).

The reason I say this is because on Nikon cameras at least, if you use a JPEG picture profile (even if you don't save JPEG files to the card, I think it still applies the profile to embedded JPEG image in the rAW, even though the RAW data itself is not affected). I guess I need to reverify that but I could have sworn that's how it was done on Nikons.

--
PLEASE NOTE: I usually unsubscribe from forums and comments after a period of time, so if I do not respond, that is likely the reason. Feel free to PM me if you have a questions or need clarification about a comment I made.

 

[ishwanu]

One more thought: Almost all raw files contain a full-size JPEG (not a thumbnail, a full-size image),

It depends on the make. For example, Sony camera raws only include a small embedded JPEG.

even if you're shooting raw only and not raw+JPEG. Most simple viewers just show that JPEG so don't look at the raw data at all. These kinds of viewers aren't what you want. (They claim to be raw viewers, but that's not really true.)

FastRawViewer always shows the raw, not the embedded JPEG.

Actually, it can show either.

 

[Digital Nigel]

One more thought: Almost all raw files contain a full-size JPEG (not a thumbnail, a full-size image),

It depends on the make. For example, Sony camera raws only include a small embedded JPEG.

even if you're shooting raw only and not raw+JPEG. Most simple viewers just show that JPEG so don't look at the raw data at all. These kinds of viewers aren't what you want. (They claim to be raw viewers, but that's not really true.)

FastRawViewer always shows the raw, not the embedded JPEG.

Actually, it can show either.

Ah, I’ve never used it other than to look at the raw image.

 

[Digital Nigel]

One more thought: Almost all raw files contain a full-size JPEG (not a thumbnail, a full-size image),

It depends on the make. For example, Sony camera raws only include a small embedded JPEG.

even if you're shooting raw only and not raw+JPEG. Most simple viewers just show that JPEG so don't look at the raw data at all. These kinds of viewers aren't what you want. (They claim to be raw viewers, but that's not really true.)

FastRawViewer always shows the raw, not the embedded JPEG.

But doesn't the camera use the RAW data to create the embedded JPEG?

Yes, of course. It’s also what you see on the rear screen and EVF. As such, it’s processed, and won’t have the same histogram as the raw data. It also has the lens distortion correction applied.

I guess in some respects, the embedded preview is so small that minor "defects" may not show up or be visible perhaps (like a few pixels that are clustered that have erroneous data).

The reason I say this is because on Nikon cameras at least, if you use a JPEG picture profile (even if you don't save JPEG files to the card, I think it still applies the profile to embedded JPEG image in the rAW, even though the RAW data itself is not affected). I guess I need to reverify that but I could have sworn that's how it was done on Nikons.

Yes, I think that’s correct. The embedded JPEG is what an OOC JPEG would have looked like, but more compressed, and sometimes (not Nikon) with a lower resolution.

 

[s_grins]

I haven't had any corrupted Raw files, yet,but let me guess that every RAW development software, one way or another, will inform you that the file is corrupt and can't be open.

 

[g7_chord]

... file is corrupt and can't be open.

Raw files are really a container for all sorts of data, including image bytes, and it's rare for processing software to look at all of it. Parts not looked at could be bad. Depends on what one means by "corrupt."

Also, the image data could be corrupt in the sense of having the wrong data, although the structure is correct. Raw processing software wouldn't know this, since it doesn't know or care what the correct image is. An example would be a black line across the image due to corruption.

 

[s_grins]

... file is corrupt and can't be open.

Raw files are really a container for all sorts of data, including image bytes, and it's rare for processing software to look at all of it. Parts not looked at could be bad. Depends on what one means by "corrupt."

Also, the image data could be corrupt in the sense of having the wrong data, although the structure is correct. Raw processing software wouldn't know this, since it doesn't know or care what the correct image is. An example would be a black line across the image due to corruption.

If you have seen this black line, post it, please

What about image bites, I do not buy it. There is no duality in RAW files.

 

[Digital Nigel]

... file is corrupt and can't be open.

Raw files are really a container for all sorts of data, including image bytes, and it's rare for processing software to look at all of it. Parts not looked at could be bad. Depends on what one means by "corrupt."

Also, the image data could be corrupt in the sense of having the wrong data, although the structure is correct. Raw processing software wouldn't know this, since it doesn't know or care what the correct image is. An example would be a black line across the image due to corruption.

If you have seen this black line, post it, please

What about image bites, I do not buy it. There is no duality in RAW files.

Raw files contain a compressed copy of the OOC JPEG. This could be corrupted without the raw data being affected (or vice versa). Ditto the exif data.

 

[sirhawkeye64]

I haven't had any corrupted Raw files, yet,but let me guess that every RAW development software, one way or another, will inform you that the file is corrupt and can't be open.

Well, yeah they will. Although in the case of ACR, it just shows the corrupted file when you open it. LR does something similar, or if it's really messed up, it will say it can't read it or it will just render the preview and when you click on it, it will say it can't read the file. While I know this is a known issue now, I ran into this with the sample Z8 from DPR since they appear to be corrupt as well but it does render the thumbnail/preview, however even NX Studio won't open them. However I don't know what caused the corruption though (supposedly they do have good RAW files to test but you have to ask for them, but I don't know if they were re-shot or it's an issue with he posting of the files or the files themselves).

Regardless, I would think that Nikon at least and probably most mfrs would employ some type of checksum at least for the camera to use to verify the file was written to the card at least.... unless this is NOT done due to he addition time and processing overhead needed to generate a hash for each file. However in the case of NIkon at least, it does dump a .DAT file on the card, which I don't know what this is and have always just deleted it (maybe it contains file info for their proprietary software to use perhaps and/or it contains the mentioned data that their software can read and verify files...? I don't know).

--
PLEASE NOTE: I usually unsubscribe from forums and comments after a period of time, so if I do not respond, that is likely the reason. Feel free to PM me if you have a questions or need clarification about a comment I made.