Lightroom 4 corrupting RAW files / 'Unexpected end-of-file' error'

[SpaceDoc]

I have an issue with Lightroom 4 which seems to corrupt my Nikon D800 NEF(RAW) files.

Here's my procedure description:

• import NEF-files from memory card (various, not just one, not just one brand) to MacBook through Lexar Card Reader
• import NEF-files into Adobe Lightroom 4

• from time to time a message 'Unexpected end-of-file' error' shows up
• in this case the file can not be opened any more at various levels: sometimes I can still zoom in, sometimes not.
• opening the NEF-file in another program, ie. Mac Preview, leads to a corrupted file at various levels, with coloured lines from the bottom of the picture upwards, at various levels and strengths.
• Export of the files is not possible

• 'date modified' in Mac's "Finder" does not change!

Until this evening I sometimes had the impression that over time the bad part of the photo became bigger over time.

Tonight I had worked on a NEF-file in Lightroom with no problem. Later I went through the different photos for sorting ('star levels') again ... and suddenly one file gave me the 'Unexpected end-of-file' error' message. Opening the file in Preview showed the same symptoms. This time I still had the original file on the memory card so I could copy it over again. After copying the original file into the directory Lightroom 4 accesses the photo opened without any problems, including the changes I had earlier made in Lightroom.

Has anyone experienced the same problem so far? I searched but could not find anything.

I am very scared now and frustrated that a RAW converter as Adobe Lightroom destroys RAW files!

Any help and hints extremely appreciated!
eT

 

[P_e_t_e]

In what you have described I don't see a proof that LR has corrupted your images. A more likely scenario would be that the import from the memory card was at fault.

Have you tried viewing your RAW images in another program before importing them to LR? Something that will actually process the RAW image rather than just displaying the embedded jpeg?
--
Peter

 

[SpaceDoc]

well: after copying over the file as described in my initial posting everything looked fine again. I then closed lightroom (as I updated it to the very recent version (4.0-> 4.1), opened lightroom again ... and the files corrupted again. viewed in preview again the file this time was differently corrupted then the time before! =;-o

also I think that Preview of Mac OS X processes the files.

eT

 

[Mark K W]

Agreed - this feels much more like a HW storage or transfer problem.

Where are you putting the files? MacBook's internal hard-disk? External USB drive? Network drive? Is the drive close to full? I would look at all those areas first... cables, connections, disk-usage, etc., etc.,

How about your memory card reader? The cables and connections for that? Is it on USB hub, directly connected to your MacBook or what? How do you know the files are not being corrupted on transfer from the card? (You do say though that you thought files were being corrupted over time, so that's more suggestive of the disk storage having some problem).

If you suspect LR, what are your LR catalog metadata settings? Make sure LR 4.1 is not set to write data or time changes to the RAW. If that check-box is clear, LR does not write to the RAW files at all, it only reads them - this really would isolate off that LR is not directly the culprit.

How big is your LR RAW cache set to (Under preferences/File Handling)? Is there enough free space on the disk for that? (This does not explain why OS X Preview also shows an error in the file though, but if your disk is full in some way...).

LR does do a lot of disk access (it is constantly writing the catalog, and xmp files if you are using them), and so if any part of your disk hardware is failing LR will be an app that "shows" it.

You also are D800 user, so those are big files to be transferring off your card and storing if you are shooting full resolution.

My suspicion really is it is HW-related rather than LR. LR tells you it is unexpected EOF, so that would suggest disk-related or the transfer from the memory-card rather than RAM memory related, but I guess it is also possible but unlikely it is a RAM issue. How much system RAM do you have?

 

[SpaceDoc]

System Info:
Core2Duo 2.4 GHz - 8 GB RAM on the System
Lightroom on 240 GB SSD
Data on 750 GB HDD (I removed the DVD drive)

Well ...

• imported pictures into Lightroom: everything fine
• worked on pictures in Lightroom: everything fine
• looked through the pictures in Lightroom again: 'end of file error'
• looked at the .NEF file in Mac Preview: colored stripes
• copied .NEF from CF-card (hadn't deleted yet this time) to harddisk again (new directory) and overwrote the corrupted .NEF-file from this copy (while Lightroom was open)
• one picture forward, one backward viewing in Lightroom: all fine now.
• closed Lightroom (and updated from LR 4.0 to LR 4.1)
• restarted Lightroom: the same picture now gives me an 'end of file error again'
• looked at the .NEF file in Mac Preview: colored stripes BUT different from the first time ...

so how is that ?

btw:

• recopied the back-up .NEF file into Lightroom's folder again: everything is fine again

 

[Denis de Gannes]

I would suggest that you have your ram checked. I experienced this a couple of years ago and eventually had to change the ram as one of the modules was bad.
--
Denis de Gannes

 

[Uwe1]

I had a similar problem with an earlier version of Lightroom and CR2 files: corruption of images after loading them into the program and generating a preview. Some people vehemently denied that this was a Lightroom bug, but since I've stopped using LR altogether, the image corruption has never happened again. And I'm still using the same system and memory cards.

In my case it was interesting to see that only the RAW information of the file was corrupted while the preview image was always left intact.

-Uwe

 

[Mark K W]

Is it always and only the same one .NEF file which is showing corruption?

[ Inside every RAW file is JPEG thumbnail. That is what LR first loads, until it has fully generated a preview from the rest of the RAW data. So, if a RAW file is corrupted, then the image may at first look OK from that JPEG thumbnail, but in reality the RAW sensor data in the file is corrupted. ]

How about if you open the original .NEF file from the card directly in OS X Preview without importing it at all into and without launching LR? What about opening that .NEF file in Nikon's own processing software?

What is the status of the setting of your EXIF "Write date or time changes into proprietary raw files" in LR's Catalog Settings.. Metadata configuration?

 

[suddie1215]

It is impossible for Lightroom to corrupt your files. My first guess is that you're having some sort of hardware problem. Have you tried a disk scanning software to look for any bad clusters on your hard drive?

I have an issue with Lightroom 4 which seems to corrupt my Nikon D800 NEF(RAW) files.

Here's my procedure description:

• import NEF-files from memory card (various, not just one, not just one brand) to MacBook through Lexar Card Reader
• import NEF-files into Adobe Lightroom 4

• from time to time a message 'Unexpected end-of-file' error' shows up
• in this case the file can not be opened any more at various levels: sometimes I can still zoom in, sometimes not.
• opening the NEF-file in another program, ie. Mac Preview, leads to a corrupted file at various levels, with coloured lines from the bottom of the picture upwards, at various levels and strengths.
• Export of the files is not possible

• 'date modified' in Mac's "Finder" does not change!

Until this evening I sometimes had the impression that over time the bad part of the photo became bigger over time.

Tonight I had worked on a NEF-file in Lightroom with no problem. Later I went through the different photos for sorting ('star levels') again ... and suddenly one file gave me the 'Unexpected end-of-file' error' message. Opening the file in Preview showed the same symptoms. This time I still had the original file on the memory card so I could copy it over again. After copying the original file into the directory Lightroom 4 accesses the photo opened without any problems, including the changes I had earlier made in Lightroom.

Has anyone experienced the same problem so far? I searched but could not find anything.

I am very scared now and frustrated that a RAW converter as Adobe Lightroom destroys RAW files!

Any help and hints extremely appreciated!
eT

 

[alexsid]

I had a similar problem with an earlier version of Lightroom and CR2 files: corruption of images after loading them into the program and generating a preview. Some people vehemently denied that this was a Lightroom bug, but since I've stopped using LR altogether, the image corruption has never happened again. And I'm still using the same system and memory cards.

I'd like to comment on that, being a software engineer. Here are some of unusual problems related to hardware I remember:

• CPU failure (register bit flipping) triggered only by specific machine instructions pattern
• RAM failure triggered only by specific memory access patterns
• memory corruption by hardware drivers, once again manifesting itself only under special conditions

What I want to say, the fact that there is no image corruption when you stopped using LR does not prove that RAM is fine. I had a dual-boot PC (Linux and Windows) that worked fine with Linux for a week without reboot, but every time I booted to Windows, I saw strange things (corrupted FS!) in 1-2 hours after boot. Running RAM test showed problems indeed and after replacing RAM there were no more problems with Windows.

LR should open RAW files read-only, so I am 99.9% sure this is not a LR bug. This can be bad RAM, faulty motherboard, bad drivers for some of devices and so on. It is quite possible that running LR just creates a specific RAM/VM/disk driver access pattern that triggers the problem.

Please check your PC thoroughly:

• run RAM test for at least 12 hours
• check your FS
• run tests for disks
• run some CPU stress tests (e.f. IntelBurnTest)

Alex

 

[Uwe1]

Might be an exotic hardware issue, I agree, but it could be something else as well:

http://forums.adobe.com/message/4279358 (post #8)

Also, I know for a fact that Adobe is doing all sorts of sh*t behind the scenes that can even affect other programs/source code in a negative way. Just take a look at this for example (here CS5 was the culprit):

http://stackoverflow.com/questions/9577540/tactionmainmenubar-and-tactiontoolbar-lose-settings

-Uwe

 

[Simon Garrett]

It is impossible for Lightroom to corrupt your files.

If not impossible, then certainly pretty darned unlikely. Even if you decide never to use LR again, I'd still treat your system as suspect, as you may have an incipient failure.

As others have said, I'd leave your system running whatever tests you have for a long time. Maybe leave your system cycling tests whenever you go out for the next few days.
--
Simon

 

[alexsid]

Might be an exotic hardware issue, I agree, but it could be something else as well:

http://forums.adobe.com/message/4279358 (post #8)

LR4 uses multithreading intensively. It is theoretically possible it does not protect some structures properly and as a result threads might write to a wrong file. Or it can be a bug in OS (e.g. in pthreads library). But in this case I would expect to see more users affected by that problem. There can be rare corner-case bugs in anything: OS, drivers, LR. Such bugs are extremely difficult to find and fix, until someone finds a way to duplicate the issue in a simple standalone test.

Also, I know for a fact that Adobe is doing all sorts of sh*t behind the scenes that can even affect other programs/source code in a negative way. Just take a look at this for example (here CS5 was the culprit):

http://stackoverflow.com/questions/9577540/tactionmainmenubar-and-tactiontoolbar-lose-settings

Yes, but this affects GUI stuff only - quite possible.

File corruption is a completely different matter. To manipulate files, all applications use system calls provided by OS: open, read, write, fcntl and so on. So if a file gets corrupted, it is either the application is writing to it explicitly or it triggers a bug in one of kernel subsystems (Virtual Memory, drivers, system calls etc.)

If a user-space application (such as LR) opens a file read-only (as I assume LR should do), there is no way it can corrupt this file, no mater how buggy it is. Now the question is whether LR opens RAW as R/O or as R/W - even though there is no need to do this (we can trace this ourselves using NTTrace - it should print how file is being opened).

Still, it makes sense to start from running thorough hardware tests.

Alex

 

[Uwe1]

If not impossible, then certainly pretty darned unlikely. Even if you decide never to use LR again, I'd still treat your system as suspect, as you may have an incipient failure.

Let's turn that argument around: why would I suspect a malfunctioning system if LR is the only software on my PC that corrupts images?

-Uwe

 

[Simon Garrett]

If not impossible, then certainly pretty darned unlikely. Even if you decide never to use LR again, I'd still treat your system as suspect, as you may have an incipient failure.

Let's turn that argument around: why would I suspect a malfunctioning system if LR is the only software on my PC that corrupts images?

Because many many thousands of people have been using Lightroom (more than use Photoshop), and I have heard of no confirmed cases of Lightroom corrupting raw files. I'm not doubting what you say, but I regard it as "unconfirmed" until it can be repeated on another machine and can be shown not to be hardware-related.
--
Simon

 

[suddie1215]

Do you have another card reader you can try? How about connecting the camera directly to the Mac and transferring that way. All the signs point to a hardware problem and you need to systematically go through your workflow to pinpoint the possible cause. I suspect either a bad card reader or USB cable.

 

[Uwe1]

I'm not doubting what you say, but I regard it as "unconfirmed" until it can be repeated on another machine and can be shown not to be hardware-related.

There can be countless reasons why a corruption of images happens - a buggy software, malfunctioning hardware components, the person that sits in front of the screen, ... you name it. I would therefore take a more pragmatic, "common sense" approach: since I have yet to see a software that is bug free, and since none of my images had been corrupted after I ditched LR, I simply assume that the latter was the culprit. In the end it really doesn't matter whether I'm right or wrong with my assumption, because that approach has worked for me for almost half a year now.

-Uwe

 

[alexsid]

If not impossible, then certainly pretty darned unlikely. Even if you decide never to use LR again, I'd still treat your system as suspect, as you may have an incipient failure.

Let's turn that argument around: why would I suspect a malfunctioning system if LR is the only software on my PC that corrupts images?

-Uwe

"When you have eliminated the impossible, whatever remains, however improbable, must be the truth" (Sherlock Holmes

A normal (i.e. not part of OS) application accesses memory (each application has its own address space - protected from all other applications), does computations (CPU) and uses OS-provided services via system calls.

If you open a file in read-only mode, something like (in C):

fd = open("my.CR2", O_RDONLY);

you manipulate this file after that using this 'fd' variable (file descriptor). No matter what you do with this file descriptor via other system calls, you should be unable to change this file contents or corrupt it. If you could, this would mean OS bug (Windows or OS X or whatever).

All modern operational systems are designed to protect other applications and OS itself from misbehaving/buggy applications.

So assuming that LR opens your raw file read-only, it is impossible for LR to corrupt/modify this file - if it is possible, this means OS bug (no matter how buggy is LR itself).

Now to your question - why LR only? From programmer's point of view LR4 looks like that:

• it is heavily multithreaded
• it is very graphics-intensive
• it is reading/writing many files (caches, catalog, temporary files)

Now if the user having the problem had several hundred other applications of similar design, I would be very much surprised indeed if it was only LR4 that corrupted his files. But what other applications of the same complexity is a typical photographer running - maybe Photoshop. (Plugins are not counted, they are much simpler).

So this should be either a hardware failure or a an OS bug (OS itself plus all 3rd-party drivers - most drivers are running in kernel space)

Alex

 

[Marcin 3M]

I am using nx view to download images into two locations, then nx loader runs lr and images are added to the lr library.
I also noticed images corruption, but backup images were always o.k.

Things are somewhat better since I changed USB port, where hdd-s are connected (via hub). I suspected some problems with usb2/usb3 compatibility. However, no scientific tests results so far...
--
Marcin_3M

 

[alexsid]

I would therefore take a more pragmatic, "common sense" approach: since I have yet to see a software that is bug free, and since none of my images had been corrupted after I ditched LR, I simply assume that the latter was the culprit. In the end it really doesn't matter whether I'm right or wrong with my assumption, because that approach has worked for me for almost half a year now.

-Uwe

This is usually called a "workaround" - hiding an existing bug/problem by avoiding conditions triggering it! You wouldn't believe how many applications working fine for years have hidden bugs. Then something changes (e.g. shared libc library) and they start crashing intermittently. And surely users/developers start blaming the new version of libc until you point to a bug in their application...

Alex