Lightroom 4 corrupting RAW files / 'Unexpected end-of-file' error'

[SpaceDoc]

no more failures in the last days ...

... but this is more likely due to the fact that I just did not have any time to proceed working with LR4 ...

SpaceDoc

[ Since I have been vocal on this thread so far, just to say I am waiting for more failure observation from either of them before passing further opinion on possible causes and leave it to either poster if they want to raise anything with Adobe. ]

 

[SpaceDoc]

Bingo:
This time it happened to a D700 NEF file

This should rule out the D800 files as the source of the problem, right?

SpaceDoc

 

[Paul_2099]

OK I'll chime in - I have a D800 and two windows 7 x64bit computers and this has been happening very infrequently on BOTH systems. It started on the Lightroom beta 4.1 and carried through to the latest version.

We recently bought a BRAND NEW computer and after many weeks HARD editing, have noticed the problem happen again to a single (yet crucial) file. THe only thing I cant say is if the files are corrupt on import or if as is outlined here it happens at a later stage. I never really thought to check and assumed when finding a corrupt file I had just not noticed until it came time to edit.

What I am curious about is that if FastStone Image Viewer (linked to in the thread: http://nikonrumors.com/forum/topic.php?id=5578 ) can read these supposedly corrupt nef's and has no problem spitting out a 24bit Tiff file (with no errors) then surely something has to be fishy with Lightroom no?

Ahh the joys of bleeding edge technology :-s

EDIT - forgot to point out that prior to the D800 had a D700 and was happily editing away, on one system with LR 3.6 and the other system with LR4 without a single corrupt file. Thus at first I thought it was a D800 fault...now Im not so sure...

 

[Mark K W]

Kind of a background FYI, but Nikon Europe confirmed a potential issue with transferring NEF files from new cameras (like the D800) when using older versions of Nikon Transfer software:
https://nikoneurope-en.custhelp.com/app/answers/detail/a_id/53027

I do not believe this is the problem of the two original posters to this thread though since neither of them specifically identified using Nikon Transfer.

 

[Paul_2099]

I think the crucial thing here is to find out WHAT is happening to these files to supposedly make them corrupt in the eyes of LR. If someone has two versions of the same NEF - one corrupt and one OK then surely a programmer / coder could delve deeper into the files to see where the problem lies?

As I mentioned the 'corruption' cant be that severe if the freely available Faststone image viewer can read and export the 'corrupt' NEF to a perfect 24bit tiff file:

http://www.faststone.org/

Also wondering if the problem would go away if you immediately converted all your NEF's to DNG. In fact would be an interesting experiment to duplicate all imports for a time - converting one lot to DNG and leaving the others as NEF's.

 

[Mark K W]

I think the crucial thing here is to find out WHAT is happening to these files to supposedly make them corrupt in the eyes of LR. If someone has two versions of the same NEF - one corrupt and one OK then surely a programmer / coder could delve deeper into the files to see where the problem lies?

Agree totally. I have asked/suggested that both the OPs look using OS X Finder and Windows Explorer to get the file info. There are checksum utilities and even Hex file comparison utilities out there for both platforms if they wanted to go further - but those are indeed more for engineering eyes to draw any deductions. That would at least help determine if a corruption is consistent (i.e. the same every time the file is opened). My theory is at least the original OP is seeing progressive random read failures, so re-reading and check-summing the file each time would be one way to clarify that.

As I mentioned the 'corruption' cant be that severe if the freely available Faststone image viewer can read and export the 'corrupt' NEF to a perfect 24bit tiff file:

I am not sure there is only one type of corruption. The original OP said he would see the corrupted file show different image patterns and artefacts when opened at different times in different apps (LR and OS X Preview). The thread you point to is discussing systematic corruption by Nikon Transfer during a read-from-card copy-to-disk operation.

http://www.faststone.org/

I read the thread you linked and I could not see anywhere where anyone said that faststone would recover RAW data from a (randomly) corrupted file. I read only a a discussion about possible ways to recover the embedded JPEG thumbnail and preview image, and someone suggesting to contact the faststone authors, and then a comment at the bottom of this thread:
http://nikonrumors.com/forum/topic.php?id=5753

which is not clear whether it is getting the JPEG file from the RAW, or whether faststone knows the systematic corruption that is be caused by using an earlier version of Nikon Transfer used with the RAWs from a later camera and have coded something to specifically correct for that. Either way since neither of the OP here has said they are using Nikon Transfer, and we do not know where (and if) their corruption (meaning overwriting) is taking place then it's unlikely any current utility would be able to correct for what they are seeing.

Also wondering if the problem would go away if you immediately converted all your NEF's to DNG. In fact would be an interesting experiment to duplicate all imports for a time - converting one lot to DNG and leaving the others as NEF's.

Could be something for the OP to try, BUT need to be careful on LR4's XMP setting as that makes LR4 re-write to the DNG when doing adjustment edits; that sort of takes us into a different area of possible corruption causes and we might just get more confused. If there is an underlying hardware problem, playing with multiple software methods on top of that will just create a "smoke-screen" of all sorts of apparent errors and further things to check.

It's much better to set out a logical deduction as per your first suggestion to answer:

• are the RAW files on disk really corrupted? And if so in what precise way?
• and since the original hypothesis is that this is done by LR4 (when it is set to a mode where we think it reads the RAW file only), then how and when does it do this?

Both of the above require engineering level debug using file and OS trace tools.

The alternative is to take the hypothesis based on what the OP has said (especially that the file date-time does not change and that the file contents can be different on each read) and deduce that that in his case is suggestive more of a disk hardware failure, and eliminate that by putting the image files to another disk.

In the absence of more data from you (and also from the two original OPs from this psting) I don't have any theories on what your problem is. Maybe one of you could raise something to an Adobe Forum. The discussion is much better there. I am sure some engineers from Adobe do look here in DPR, but if you want get more attention, go to the Adobe Forums.

 

[Paul_2099]

Im sure i mentioned it but I actually did use Faststone and have successfully recovered a full size 24bit Tiff file from the corrupt D800 NEF that neither lightroom nor Photoshop would read.

For me however it wasnt a solution as the way it rendered colors / skin tones compared to LR with the Nikon preset meant I couldn't get it to match the rest of my shots.

Whats interesting to me is how it reads the D800 NEF at all given the software was last updated mid 2011 - way before the D800 was in the wild?

 

[Mark K W]

Im sure i mentioned it but I actually did use Faststone and have successfully recovered a full size 24bit Tiff file from the corrupt D800 NEF that neither lightroom nor Photoshop would read.

Must have missed that it was you in the threads. Apologies.

By 24bit TIFF, do you mean 24bits per pixel - i.e. 8bit per RGB channel??? That to me may mean FastStone is extracting the 8bit JPEG in the .NEF rather than the RAW sensor data. A "normal" dynamic range conversion from a 12 or 14-bit RAW sensor data would be a TIFF with 16bit per RGB (so 48bits per pixel).

For me however it wasnt a solution as the way it rendered colors / skin tones compared to LR with the Nikon preset meant I couldn't get it to match the rest of my shots.

Maybe because it read the full-res 8bit JPEG from the RAW file???

Whats interesting to me is how it reads the D800 NEF at all given the software was last updated mid 2011 - way before the D800 was in the wild?

If it read the full-res JPEG from the .NEF rather than the RAW sensor data this would also explain how it can do it from a apparently newer version (and corrupted!) .NEF file since the JPEG thumbnail and full-resolution JPEG are I believe in the Nikon header info at the start of the file before the sensor RAW data.

NB everything I say above is from recall or quick reads of other web commentary on .NEF format. I have never used FastStone, and don't shoot Nikon!

 

[SpaceDoc]

I can clearly state that I did not use Nikon transfer.

SpaceDoc
.

Kind of a background FYI, but Nikon Europe confirmed a potential issue with transferring NEF files from new cameras (like the D800) when using older versions of Nikon Transfer software:
https://nikoneurope-en.custhelp.com/app/answers/detail/a_id/53027

I do not believe this is the problem of the two original posters to this thread though since neither of them specifically identified using Nikon Transfer.

 

[SpaceDoc]

Unfortunately not for Mac. ;-(

SpaceDoc

As I mentioned the 'corruption' cant be that severe if the freely available Faststone image viewer can read and export the 'corrupt' NEF to a perfect 24bit tiff file:

http://www.faststone.org/

Also wondering if the problem would go away if you immediately converted all your NEF's to DNG. In fact would be an interesting experiment to duplicate all imports for a time - converting one lot to DNG and leaving the others as NEF's.

 

[SpaceDoc]

thx for your long answer.

probably true that people with more engineering skills might want to look into the problem ... I am not able to process checksums et cetera.

as mentioned: my current workaround is to double copy the data to my harddrive and then - if LR corrupts another file - just recopy over from the original file. this worked well every time until now. and, by the way: in theses cases I copied from the memory card to the harddrive (via Mac OS 'frinder) and than created a subdirectory and copied from the files copied earlier (harddisk copying to harddisk) ... and again: the copies re-transfered / overwriting a corrupted file works fine. still don't believe it's a harddrive failure.

SpaceDoc
.

 

[handymanuk]

I was one of the posters that mentioned this as being a problem. I have processed a lot more files in the following weeks but I haven't experienced this problem again. However, I have not replicated the exact circumstances again (playing video whilst editing).

So I'm not able to provide any more information at this stage, sorry.

 

[handymanuk]

Ok, so this has happened to me again today.

Same situation as before. Watching video from one drive whilst editing photos in LR4 from another. In total three drives involved, one holding my LR catalogue and previews, another holding the actual photo file and the last one held the video that I was playing.

I got an EOF error on export to JPG. Two files were affected. Viewing these in LR in the develop module repeated the EOF effor. I went to Windows explorer and examined the file properties and compared them to my back up copy of the files. There were no differences. I opened the files up in View NX, and there were no issues, I was also able to view the RAW version of the file as well as the embedded JPG. By all accounts, there was nothing wrong with the file itself and it did not appear to have been tampered with or changed at all. I should note that I had paused the video during these inspections.

The only way to recover the image in LR was to copy over the original NEF from my back up drive and replace the 'corrupt' file, overwriting it with the backup. LR was then able to open the photo and all of my previous edits were maintained.

LR problem?

 

[Mark K W]

Ok, so this has happened to me again today.

Same situation as before. Watching video from one drive whilst editing photos in LR4 from another. In total three drives involved, one holding my LR catalogue and previews, another holding the actual photo file and the last one held the video that I was playing.

I got an EOF error on export to JPG. Two files were affected. Viewing these in LR in the develop module repeated the EOF effor. I went to Windows explorer and examined the file properties and compared them to my back up copy of the files. There were no differences. I opened the files up in View NX, and there were no issues, I was also able to view the RAW version of the file as well as the embedded JPG. By all accounts, there was nothing wrong with the file itself and it did not appear to have been tampered with or changed at all. I should note that I had paused the video during these inspections.

Well that's kind of the key info in the context of this thread. There was no evidence of over-writing of the RAW file on the disk.

The only way to recover the image in LR was to copy over the original NEF from my back up drive and replace the 'corrupt' file, overwriting it with the backup. LR was then able to open the photo and all of my previous edits were maintained.

LR re-reads the RAW fully when you reselect it in the develop module. So if you move away to another image and then come back to the "corrupted" one and let it fully open (i.e. wait until after the loading spinner stops), it should all open and appear fine. It re-read fully on Export or Print as well I think.

[The only behavioural oddity I have noticed (and this mechanism is something only Adobe I guess knows) is LR appears to me on my machine and NAS to still hold a file open from an immediately previous or adjacent viewing. I find LR and CS5 can confuse each other in this way - i.e. I think LR should have released the file handle to an image I have deselcted in Develop module to allow CS5 to overwrite a PSD I have imported to LR, but actually has not.]

Your edits are LR instructions in your catalog and get applied as the image is opened, so it's logical those are all there fine.

Only other thing I could think would be if LR's EOF error was actually coming from another file operation (like catalog, or preview files which LR is constantly reading and writing). When LR shows EOF error in the warning dialogue, does it give any more info like the filename??

To me feels more like your system could be running out of file handles or something bizarre at OS level. Maybe something related to video-memory or GPU. Maybe running out of virtual memory, or a paging error. Could be many causes I suppose, but it does feel more like your case is something happening around LR rather than in LR.

LR problem?

Hmmmmm. Based on your evidence, not in the sense of corrupting RAW images on the disk, no. Being vulnerable to some other heavily-loaded system behaviour on your machine, maybe.

 

[alexsid]

Did you see another similar thread in this forum,

http://forums.dpreview.com/forums/readflat.asp?forum=1006&thread=40369558

In particular,

But if I create a copy of the corrupted file, the very same file now works fine.

I dont even have to open the coppied version. The orignal corrupted file is now good as if nothing ever happened.

I suggested this can be potentially explained by CameraRaw cache corruption

Alex

 

[Mark K W]

Yeah, I'd agree that it could be camera RAW cache corruption somehow.

LR should recover from that though by opening the "corrupted" image to completion in Develop module (i.e. until LR's loading progress spinner stops - as I thought LR always works from the underlying RAW source image file when the image is selected in Develop but it's possible there's some logic in the way it loads files, that if the RAW cached version is judged to be fully up-to-date against last-edit time, then it won't regenerate the/a RAW cache image of it until it has to, e.g. by a further adjustment change or creating a virtual copy).

 

[Quoth]

Why anyone would do anything unnecessary that is potentially processor intensive, while processing their images.

Watching videos while processing raw files seems a bit silly, if reducing the potential for corruption is important.

I could play Counter Strike while writing DVD's, but I'm sure my failure rate would go through the roof (on both counts).

 

[PaultheGCnut]

It is not impossible for LR to corrupt the files! When it imports it the RAW file is opened, if any the program has a problem with reading while it in RAM, RAM will write those changes to disk. Error correction gone wrong...aka reverse. I was surprised when I had this explained by CS faculty at a local college.

I sure miss ECC memory
suddie1215 wrote:

It is impossible for Lightroom to corrupt your files. My first guess is that you're having some sort of hardware problem. Have you tried a disk scanning software to look for any bad clusters on your hard drive?

I have an issue with Lightroom 4 which seems to corrupt my Nikon D800 NEF(RAW) files.

Here's my procedure description:

• import NEF-files from memory card (various, not just one, not just one brand) to MacBook through Lexar Card Reader
• import NEF-files into Adobe Lightroom 4

• from time to time a message 'Unexpected end-of-file' error' shows up
• in this case the file can not be opened any more at various levels: sometimes I can still zoom in, sometimes not.
• opening the NEF-file in another program, ie. Mac Preview, leads to a corrupted file at various levels, with coloured lines from the bottom of the picture upwards, at various levels and strengths.
• Export of the files is not possible

• 'date modified' in Mac's "Finder" does not change!

Until this evening I sometimes had the impression that over time the bad part of the photo became bigger over time.

Tonight I had worked on a NEF-file in Lightroom with no problem. Later I went through the different photos for sorting ('star levels') again ... and suddenly one file gave me the 'Unexpected end-of-file' error' message. Opening the file in Preview showed the same symptoms. This time I still had the original file on the memory card so I could copy it over again. After copying the original file into the directory Lightroom 4 accesses the photo opened without any problems, including the changes I had earlier made in Lightroom.

Has anyone experienced the same problem so far? I searched but could not find anything.

I am very scared now and frustrated that a RAW converter as Adobe Lightroom destroys RAW files!

Any help and hints extremely appreciated!
eT

 

[PaultheGCnut]

I believe the open call can corrupt, though almost always does not. I have files like his that corrupt on one computer but I copy over a cd file of same image from earlier and it does not corrupt. However bring a version over from another box and it does. Seems to always happen between importing and when LR refreshes the cache after opening LR to a big catalog or 1st time viewing after import.


Always the 'Nikon .NEF file and not the jpeg. This is a read issue. On my old fuji files it does not happen to (most are jpegs, + a few tiffs)

 

[PaultheGCnut]

I too have this. Way back I had bad RAM (tested bad on software tests) and it would cause Mosiac switches. I could see a flickr when the bit changed...not always leading to immediate corruption.

I replaced everything except CPU and motherboard...Turned those back to Intel under warranty and they sent same equip back as far as I could tell. I don't get those types but I do get partially blank ones in the screen image and next time end of file error 95% of time, I try PS to open and if it will I save as .Tiff and then those few seem good from then on.