Reading mode: Light Dark

100,000 ARW file nightmare... Is there a program to Batch validate ARW files

Go to last post
J

Johann Kruger

Veteran Member
Messages
1,395
Reaction score
383
I received a ton of ARW files recently and want to archive them to our ZFS servers and tape archive.

I need a shell utility that can validate the contents of ARW files. Preferably Linux. But we have Win 10 and Mac Something.

Thanks.
 
Johann Kruger said:
I received a ton of ARW files recently and want to archive them to our ZFS servers and tape archive.

I need a shell utility that can validate the contents of ARW files. Preferably Linux. But we have Win 10 and Mac Something.

Thanks.
Click to expand...
I'm not sure if this is what you're looking for, but for archival purposes, I batch archive using PackRaw, which losslessly compresses all ARW files smaller than native compressed RAW:

Loading…

encode.ru

This means I can quickly restore the original RAW files from the PackRAW container (similar to Zip, except much higher compression for ARW files).

There should be a command switch to validate the container.

Den
 
That's good information but what I was looking for would validate the contents of the ARW file itself given many of the files will be corrupt from past copying from disk to disk.
 
Johann Kruger said:
That's good information but what I was looking for would validate the contents of the ARW file itself given many of the files will be corrupt from past copying from disk to disk.
Click to expand...
Oh ok,

Try EXIFTOOL

If run from a batch command, it might verify each file, or report corrupted files.

I'm not sure, because I've never needed it for that purpose.

Phil Harvey (the developer) has posted a command for this task at the following page:

Loading…

u88.n24.queensu.ca

exiftool -if '$warning or error' -directory=SOME_OTHER_DIRECTORY -r DIR

Den
 
Last edited:
DenImage said:
Johann Kruger said:
That's good information but what I was looking for would validate the contents of the ARW file itself given many of the files will be corrupt from past copying from disk to disk.
Click to expand...
Oh ok,

Try EXIFTOOL

If run from a batch command, it might verify each file, or report corrupted files.

I'm not sure, because I've never needed it for that purpose.

Phil Harvey (the developer) has posted a command for this task at the following page:

http://u88.n24.queensu.ca/exiftool/forum/index.php?topic=5527.0

exiftool -if '$warning or error' -directory=SOME_OTHER_DIRECTORY -r DIR

Den
Click to expand...
I think EXIFtool only validates EXIF and metadata information.

I need a batch utility that validates that ARW files can be displayed o.k. WITHOUT having to use Sony's ARW Viewer and manually looking at each ARW file to visually inspect that it has not been corrupted by previous file copying.
 
Without doing any research I would import them in Lightroom and export them as jpg, hoping that Lightroom would signal any corruption errors when doing so.
 
addieleman said:
Without doing any research I would import them in Lightroom and export them as jpg, hoping that Lightroom would signal any corruption errors when doing so.
Click to expand...
That's exactly the problem. I don't have a utility can validate for certain that ARW files are good.
 
Johann Kruger said:
DenImage said:
Johann Kruger said:
That's good information but what I was looking for would validate the contents of the ARW file itself given many of the files will be corrupt from past copying from disk to disk.
Click to expand...
Oh ok,

Try EXIFTOOL

If run from a batch command, it might verify each file, or report corrupted files.

I'm not sure, because I've never needed it for that purpose.

Phil Harvey (the developer) has posted a command for this task at the following page:

http://u88.n24.queensu.ca/exiftool/forum/index.php?topic=5527.0

exiftool -if '$warning or error' -directory=SOME_OTHER_DIRECTORY -r DIR

Den
Click to expand...
I think EXIFtool only validates EXIF and metadata information.

I need a batch utility that validates that ARW files can be displayed o.k. WITHOUT having to use Sony's ARW Viewer and manually looking at each ARW file to visually inspect that it has not been corrupted by previous file copying.
Click to expand...
You can load a batch of at least 400 files, possibly as many as 1000, into Adobe Camera Raw. If they load, they are OK.

From there, if they don't need any processing, you could save them as JPGs en masse. Or if the exposures are all over the place, save them as TIF files.
 
D Cox said:
Johann Kruger said:
DenImage said:
Johann Kruger said:
That's good information but what I was looking for would validate the contents of the ARW file itself given many of the files will be corrupt from past copying from disk to disk.
Click to expand...
Oh ok,

Try EXIFTOOL

If run from a batch command, it might verify each file, or report corrupted files.

I'm not sure, because I've never needed it for that purpose.

Phil Harvey (the developer) has posted a command for this task at the following page:

http://u88.n24.queensu.ca/exiftool/forum/index.php?topic=5527.0

exiftool -if '$warning or error' -directory=SOME_OTHER_DIRECTORY -r DIR

Den
Click to expand...
I think EXIFtool only validates EXIF and metadata information.

I need a batch utility that validates that ARW files can be displayed o.k. WITHOUT having to use Sony's ARW Viewer and manually looking at each ARW file to visually inspect that it has not been corrupted by previous file copying.
Click to expand...
You can load a batch of at least 400 files, possibly as many as 1000, into Adobe Camera Raw. If they load, they are OK.

From there, if they don't need any processing, you could save them as JPGs en masse. Or if the exposures are all over the place, save them as TIF files.
Click to expand...
I have 100,000 ARW files and need a batch utility that runs on a shell terminal.
 
D Cox said:
You can load a batch of at least 400 files, possibly as many as 1000, into Adobe Camera Raw. If they load, they are OK.
Click to expand...
That Camera Raw or any other software was able the load the file does not in any way mean that the data was not damaged. It only means that the metadata and the file structure are fine enough to be able to be loaded in that particular application.

Unless the RAW file itself contains some form of integrity check (CRC, MD5 etc.) it is going to be rather hard to detect corruption. A single flipped bit might be completely invisible or may ruin the whole image...
 
  • Like
Reactions: Lan
dcraw is probably your starting point and write some shell script. -i option identifies file, dcraw exits with status 0 if it can decode file else 1

dcraw man page

EDIT: On further thought, just have dcraw convert to jpeg (or TIFF or whatever). If the file is corrupted, dcraw will very likely exit with status 1
 
Last edited:
RoBiK75 said:
D Cox said:
You can load a batch of at least 400 files, possibly as many as 1000, into Adobe Camera Raw. If they load, they are OK.
Click to expand...
That Camera Raw or any other software was able the load the file does not in any way mean that the data was not damaged. It only means that the metadata and the file structure are fine enough to be able to be loaded in that particular application.

Unless the RAW file itself contains some form of integrity check (CRC, MD5 etc.) it is going to be rather hard to detect corruption. A single flipped bit might be completely invisible or may ruin the whole image...
Click to expand...
Yeah. That's what technical people tell me as well.

I wonder if there is some Artificial Intelligence utility can detect corrupted ARW files by looking for a large area of black or missing pixels or something like that.
 
A very handy way to validate a raw file’s image contents (not just the file blindly) is to import them into a fresh, new lightroom catalog and generate 1:1 previews for each image.

Every pixel gets read and used and any problems are flagged up in the summary window at the end of the process. Once you have verified this way, creating a checksum with checksum++ app on Mac or the verifier plugin for lightroom will let you reverify quicker subsequently.

Hope that helps,

Rich.

Johann Kruger said:
I received a ton of ARW files recently and want to archive them to our ZFS servers and tape archive.

I need a shell utility that can validate the contents of ARW files. Preferably Linux. But we have Win 10 and Mac Something.

Thanks.
Click to expand...
 
darktable has the CLI mode. I cull and process the selected RAWs in its GUI, and feed the unselected RAWs through darktable CLI to be converted to JPGs automatically (and with preset/style). If there's corruption, darktable for sure will spit out error. I use Ubuntu, BTW.

Loading…

www.darktable.org
 
Last edited:
Badwater said:
Do one thousand at a time.
Click to expand...
I 'm looking for a shell utility that can read and validate the contents of ARW files so that we don't have to manually load and visually verify that each file is o.k. Preferably a Linux utility with open source so we can hire a consultant to change the program.
 
Johann Kruger said:
Badwater said:
Do one thousand at a time.
Click to expand...
I 'm looking for a shell utility that can read and validate the contents of ARW files so that we don't have to manually load and visually verify that each file is o.k. Preferably a Linux utility with open source so we can hire a consultant to change the program.
Click to expand...
Hi. I use freefilesync to validate my files thats on an external Harddrive. But you would first have to know the files are perfect.

David
 
Johann Kruger said:
Badwater said:
Do one thousand at a time.
Click to expand...
I 'm looking for a shell utility that can read and validate the contents of ARW files so that we don't have to manually load and visually verify that each file is o.k. Preferably a Linux utility with open source so we can hire a consultant to change the program.
Click to expand...
Once again, use dcraw. Your shell script will look something like this

for i in *ARW
do
dcraw -a -c $i > /dev/null
status=$?
if test $status -eq 0
then
echo $i is OK
else
echo $i is not OK
fi
done

This is as good as it's going to get.
 
Is isn't a CLI but Fast Raw Viewer actually reads the RAW file (or can be set) to create a preview. So opening the directory and doing a quick visual scan of the thumbnails (yes, I know with 100,000 images nothing is truly "quick") should be a way to verify the integrity.
 
You must log in or register to reply here.

Similar threads

K
Comparing files across drives, across systems?
Replies
7
Views
8
New Wrycuda
N
B
Folder Mania! How do you manage your pic folders and files?
2
Replies
26
Views
27
vyusseem
V
R
  • User review User review
From Synology to UGREEN: My Journey to a Smarter, Faster NAS with OMV7
Replies
3
Views
13
New Wrycuda
N
I
FOLLOWUP – Switching to G9II from OM1M2
2 3 4
Replies
79
Views
80
HicHic
HicHic
T
LUTs for photography
2 3
Replies
47
Views
48
capsized
C

Keyboard shortcuts

f
Forum
m
My threads
Mobile site
About
Editorial content
Cameras & Lenses
Community
All content, design, and layout are Copyright © 1998–2025 Digital Photography Review All Rights Reserved.
Reproduction in whole or part in any form or medium without specific written permission is prohibited.
When you use DPReview links to buy products, the site may earn a commission.
©GPS Media - Guides, Products, Services.
Back
Top