Lightroom 4 corrupting RAW files / 'Unexpected end-of-file' error'

alexsid said:

Assuming that LR opens raw files R/O (I don't have access to LR sources but this is a very reasonable assumption), I cannot suggest a mechanism for a user-space application running without administrative privileges to corrupt R/O files.

Click to expand...

Well, there seem to have been issues in the past with JPEG's:

http://feedback.photoshop.com/photoshop_family/topics/lightroom_3_4_is_corrupting_my_source_jpegs_no_really

Please note the official response in the first answer.

Uwe1 said:

alexsid said:

Assuming that LR opens raw files R/O (I don't have access to LR sources but this is a very reasonable assumption), I cannot suggest a mechanism for a user-space application running without administrative privileges to corrupt R/O files.

Click to expand...

Well, there seem to have been issues in the past with JPEG's:

http://feedback.photoshop.com/photoshop_family/topics/lightroom_3_4_is_corrupting_my_source_jpegs_no_really

Please note the official response in the first answer.

Click to expand...

Yes, and please note: " I can confirm that corruption occurs when I hit CMD-S to save the meta-data to the file"

In that case, the user was working with JPG. But:

http://forums.adobe.com/message/4306185

"You can't save metadata directly to the raw file in Lightroom."

There are many documents from Adobe saying explicitly that medata can be saved in TIFF, DNG, JPEF, PSD - but not in proprietary files.

So yes, buggy LR could corrupt JPG - but not CR2 (proprietary format)

Alex

I agree it is extremely unlikely to be LR that is corrupting the files.

Thx for the link. but please not that my problem concerns Ligthroom 4 .

Also my problem is about RAW/NEF files rather than JPG.

eT

Uwe1 said:

alexsid said:

Assuming that LR opens raw files R/O (I don't have access to LR sources but this is a very reasonable assumption), I cannot suggest a mechanism for a user-space application running without administrative privileges to corrupt R/O files.

Click to expand...

Well, there seem to have been issues in the past with JPEG's:

http://feedback.photoshop.com/photoshop_family/topics/lightroom_3_4_is_corrupting_my_source_jpegs_no_really

Please note the official response in the first answer.

Click to expand...

The option to write meta-data into the RAW file is deactiveated on my installation.

eT

Mark K W said:

To the OP (not sure you have given us this info):

• How many RAW files have you seen corrupted in this way?

Click to expand...

Already a couple of them ... estimated > 10. :-(

Mark K W said:

• What is the status of the setting of your EXIF "Write date or time changes into proprietary raw files" in LR's Catalog Settings Metadata configuration?

Click to expand...

Deactivated (from the beginning).

Mark K W said:

• When you say "worked on pictures in LR", what kind of things were you doing in the Develop Module to the image which first appeared OK, and then after was corrupted? Are you sure you are fully generating and manipulating a 1:1 preview of this exact corrupted image? (I am trying to determine the possibility that your RAW file(s) are actually corrupt in the RAW data part of the file either already on the card or in the card=> HDD transfer process).

Click to expand...

Mostly basic development settings plus cropping

Mark K W said:

• You say the corrupted image file date/time had not changed on your disk, and is the same as the date/time on the image on the card - yes? How about the file-size? If all are the same, that's very key information to assist in the debugging of your problem.

Click to expand...

NEF-RAW files in full (36 MPx) resolution: about 40-50 MB. (NEF-lossless-compressed)

Mark K W said:

• What app are you using to copy the images from the card (reader) to your HDD? How are you doing that copy?

Click to expand...

Lexar USB-Card-Reader on MacBook (unibody late 2008), copied by Mac OS X Finder 'move' or 'copy'.

eT

40-50 MB / 36 MPx NEF-lossless-compressed RAW file.

Time: no clue. Not timing it. Takes a while: it's only a Core-2-Duo 2.4 GHz system with Lightroom on the SSP and the photo data on the 2.5" HDD (DVD taken out).

eT

Mark K W said:

I guess other pieces of key info - these are .NEF from a D800 which maybe is in 36MP mode, so:

• how big are these D800 .NEF files you are importing to LR typically? How many MByte?

• how long does LR4.1 take to load a good image (render to 1:1 sharpness in the Develop Module)?

Click to expand...

did the same ... all old entries from the LR3 times ... therefore not applicable to the current problem.

eT

Focus Photoeditor User said:

google "raw files corrupted in Lightroom"

Click to expand...

Last night it happend again.

• NEF copied to harddrive in back-up-no-touch-directory (Lexar Card Reader)
• NEF copied from harddrive back-up-no-touch-directory to directory LR4 accesses
• NEF imported into LR4
• worked on the photos ...

... - initially / while editing no problems ...
... - when exporting error message on two files that they could not be exported

... - when trying to look at those photos now in LR4: "unexpected end-of-file-error"-message
... - when looking at those NEF now in Apple Preview: corrupted

... - when copying another copy of the NEF-file into LR4 directory, closing and re-opening LR4 and exporting again: this time no problems, export finished fine.

And that should not be Lightroom?

Thx.
eT

the Ram.
CPU temp.
Reseating components and leads.
Lightroom settings...

Images are on an Internal hard drive so Lightroom is not accessing an external USB drive.
Low disk space on OS.
Other running software.

Minimise the possible variables and hopefully the cause can be isolated.

Its pointless offering advice; the OP clearly believes Lightroom is the cause of his problems.

I haven't read all of the posts so my idea may not apply here. I'm wondering if it could be the sidecar file that is being corrupted. Would it be helpful to delete the sidecar file and see if the anomaly goes away?

Ronny

OK - I have read all you said. Thanks for your replies to my questions.

I honestly do not believe there is a problem in LR. You have it set such that it does not write EXIF to RAW files on editing, and you are not using it to copy your files from the memory card. Hence - I agree with others that using it the way you are - LR will (or certainly should be!) opening the source RAW files via OS X calls in read-only mode.

Based on what you have said (i.e. you are copying the files to your MacBook and then recovering uncorrupted versions from alternate directories on that), then I also am not suspicious of your card reader for the time being.

So - a couple more questions to help investigate further:

• when a RAW file is corrupted and you compare a corrupt version with an uncorrupted copy in OS X finder, what do you see for the Get Info - i.e. size, date, last open, last modified? How do each of those file attributes compare between the corrupted and non-corrupted versions? Is it possible to use OS X finder to copy the corrupted file to another file, and then again check and compare the file Get Info parameters, and also to check that corrupted file copy for RAW readability in Preview?

• where are your lrcat and lrdata files and the raw cache? Are those on your 240GB SSD? (lrcat and lrdata normally are in your Pictures directory, and the camera raw cache is normally in your Library/Caches folder)

• your images are then on a 750GB HDD - is that right?? What else is on that drive? Based on what you have said that drive was a swap out for a DVD drive - is that correct? When was it swapped out - and how long have you been using it? I have never done a MAC drive swap out, but how about the needed HW drivers, etc.,??

Quoth said:

the Ram.
CPU temp.

Click to expand...

I am waiting now when you'll suggest to put one hand on the screen and one on the wall, turn 90 degrees to the left, maybe he should check this too, it could be the cause..

it's ridiculously obvious LR has to do with the problem, now if there is also some "other" causes, yes possible, but it's the interaction with LR that triggers it. I am a logical person, and I also trust much more the original poster's attitude than the ones of others who posted on this thread.

And even if it was just a hard disk problem or other (I dont think ram is even possible given the description of problem), LR is still very responsible to overcharge the hard-disk / the O.S. or whatever hardware interested in the process.

LR should adapt to the machine, not the other way around, unless adobe strats in the business of O.S. in that case he can invent the ideal O.S. for running its monstrous software.

not to disappoint you, but if it's from the times of LR3 (and some said they had the same with LR 1 as well) it's likely it0s still there in LR4

Quoth said:

Ram - poor quality - broken - badly seated.
USB leads and ports.
Cheap / broken card reader.
Cheap corrupt CF/SD cards
Incorrect drive removal.
Sata/ide leads and connections.
Other running software.
Magnetic or radio interference.
Voltage spikes / cheap PSU
Overheating CPU
Dust / Static / shorting out
Bios
Dust / Static / shorting out
Bios

Click to expand...

Sounds like you would blame anything but adobe...are you sure it's me who has a downer for LR or is it you who are blind intentionally

You are obviously intent on blaming Lightroom when no-one else here has these issues and it is highly likely to be a hardware issue somewhere in the chain.

Every piece of data passes through Ram, so it is quite likely that a memory error would corrupt any data it touches.

Memory corruption of running processes can also occur with radio or magnetic interference.

Similarly a red hot CPU can begin throwing out data errors, and if the fan has failed, or the system is clogged with dust, then overheating is a possible cause.

Lets throw a couple of new things into the equation.
Overheating drives. Damaged read heads. corrupt filesystem.
Incorrectly removed drives, damaged sata connectors.

Removing all of the variables is the only way the OP is going to solve this, but if you are happier ignoring all the potential variables and just blaming the software, then good luck.

That's not correct many others have had problems with LR corrupting files (check internet pls), and in this forum, in this same thread as well, please read the other 2 people who had same problem. Now I can only think there is an overstressing by LR of the hardware which in my opinion in this case is mainly the hard-disk. The memory works just fine, he never got any problem editing the picture (where memory is far more important than when writing to file).

Also his hardware may be perfectly valid, it's just LR asking too much from it. I think LR will on the long term destroy your hard-drive because it has to keep such a huge database of files and it has to continually read and write on the disk. This is more evident in import and export operations and with big sized files. The only question I would have asked the OP would be how long have you been using LR on that disk?

Focus Photoeditor User said:

That's not correct many others have had problems with LR corrupting files (check internet pls), and in this forum, in this same thread as well, please read the other 2 people who had same problem. Now I can only think there is an overstressing by LR of the hardware which in my opinion in this case is mainly the hard-disk. The memory works just fine, he never got any problem editing the picture (where memory is far more important than when writing to file).

Also his hardware may be perfectly valid, it's just LR asking too much from it. I think LR will on the long term destroy your hard-drive because it has to keep such a huge database of files and it has to continually read and write on the disk. This is more evident in import and export operations and with big sized files. The only question I would have asked the OP would be how long have you been using LR on that disk?

Click to expand...

I have seen this happen as the drive that LR is using for a scratch disk starts to fail. Light Room might be one of the few programs the OP is using that puts real stress on the scratch disk.....and why it is seen as the problem. I suspect that if he where to switch to Photoshop (large scratch disk use) he might also start seeing the corruption there also.

Seems you are away for a while, so let me summarise based on what I think I have understood from you.

Firstly, I am not sure there is evidence that any corruption (i.e. over-writing of data on the magnetic surface of the HDD) is actually taking place. What you do have is LR reporting that it cannot always read the contents of a RAW file that it has previously read off your HDD. You have other applications (OS X Preview) similarly unable to read the data.

As far as I can determine, nothing so far says any data was corrupted (i.e. changed) on the HDD. In fact you have the converse that OS X apparently reports same file date/time for a "corrupted" vs non-corrupted version of a file; that means if anything is corrupting it is a real low-level write operation, beneath the OS. LR definitely does not do that because it can equally work image files on local drives, network drives, USB drives, etc., etc., and I really don't believe LR is stacked full of low-level drivers for every network architecture and storage device it might encounter. In fact I know it uses the OS services because I can trace that on my own MacBook installation of LR4.1 with iosnoop, etc..

So - I would suggest as a theory that your HDD where the images are stored is failing or behaving intermittently on file reads , possibly just on data reads from certain sectors. It's possible a HW failure or miss-connection is inadvertantly writing to the disk during what was supposed to be a read operation, but I would more believe you are actually seeing failing reading. The fact that the corruption is inconsistent (you say the file can look different each time) would again maybe suggest read failure. When you re-copy the non-corrupted file, I think that will write it to new sectors on the disk.

I am presuming your LR catalog, preview files, and camera raw cache are sensibly on your SSD. So LR will often show the correct image from it's own earlier cache or previews. As far as I know LR reads the original RAW when it is selected for edit in Develop Module or during Export. I think in Library module, and at other times it uses the Previews or 1:1 camera RAW cached version.

One way for you to prove this failing HDD theory is to either put the images being worked on onto your SSD, or onto another USB HDD, and just re-observe the behaviour when re-working the images in LR. It is easy to copy the RAW files using the OS into a new folder on a new drive, and then just update the folder location in LR's library module. All LR adjustments to date will be preserved that way because they are in the catalog database.

Provided there is space on the SSD or a convenient USB HDD that would be my next step anyway.

If not I would be checking your HDD connection cables and the physical HDD installation in general.

[ If you still sees RAW corruption on another disk, then that's a deeper case and suspicion could/would swing back to LR, but I just believe there are so many installations of LR in place around the world, even now being used with D800's larger files, that there would be more noise in the Adobe forums and elsewhere about it. ]