- Messages
- 30,182
- Solutions
- 112
- Reaction score
- 12,428
Well said and OP might be well served to take this advice.
-
Welcome to the new forums! Please read this first. For known issues we are working to resolve, click here.
Lightroom 4 corrupting RAW files / 'Unexpected end-of-file' error'
- Thread starter SpaceDoc
- Start date
-
- Tags
- nikon d800
that Lightroom doesn't stress the system very much at all, compared to a great many other programs.
The huge database idea is not necessarily true and depends on how you use Lightroom. My largest lrcat file is 48 Mb, but I do create new ones for each specific job, as I don't need a single huge one for the tens of thousands of images I have.
If Lightroom is identifying a hardware fault then, it is only by luck that other programs have not fallen foul of it so far.
Please don't mistake me for a Lightroom fanboy, as before LR3 I thought it was quite poor in it's image quality, but I do think they have done a pretty good job with the more recent versions (with a few reservations).
The huge database idea is not necessarily true and depends on how you use Lightroom. My largest lrcat file is 48 Mb, but I do create new ones for each specific job, as I don't need a single huge one for the tens of thousands of images I have.
If Lightroom is identifying a hardware fault then, it is only by luck that other programs have not fallen foul of it so far.
Please don't mistake me for a Lightroom fanboy, as before LR3 I thought it was quite poor in it's image quality, but I do think they have done a pretty good job with the more recent versions (with a few reservations).
handymanuk
Active member
I'm also having this issue and it only started yesterday. I imported around 1800 Nikon D800 NEF files into LR4 from my holiday last week. I've previously imported many D800 files into LR and have never seen this issue before. My PC setup was exactly the same then as it is now and I've not installed any updates to LR.
The behaviour is as the OP described. Photos imported and saved on my hard drive. I have one HDD for photos, one for documents where the LR catalogue is and my system drive is an SSD. I then go ahead and edit the photos, cropping, white balance correction, noise reduction, brushes etc. This has all worked fine until export when some photos fail and give the error "Unexpected end of file error". If I then copy the original NEF back from my portable HDD (backup drive) and copy it over the 'corrupted' NEF and re-open LR then everything is fine, my edits are still in place and I can export.
This seems to happen randomly, images that I've just worked on, images that I worked on yesterday but haven't touched in this session.
What I haven't done is try to open a 'corrupted' NEF in another program, like ViewNX to see if the file really is affected.
I'm running Windows 7 Home Premium SP1, 64 Bit, Intel i7-3820 CPU @ 3.6Ghz with 16GB of RAM.
I'll be honest and say that I'm worried. My photos are precious to me (photos of my niece as a baby whom I won't see for another 6 months) and the thought of losing them is a very worrying prospect. This is something that I've never had to contend with before. My day to day backup solution wouldn't cover this since I'd have to be aware that there was an issue with a file before copying it back. If the corruption occurs outside of LR then my files could become corrupted and then that damaged file is copied to the backup disk overnight, overwriting the original undamaged copy.
One other thing, should I move my LR catalogue to the SSD?
The behaviour is as the OP described. Photos imported and saved on my hard drive. I have one HDD for photos, one for documents where the LR catalogue is and my system drive is an SSD. I then go ahead and edit the photos, cropping, white balance correction, noise reduction, brushes etc. This has all worked fine until export when some photos fail and give the error "Unexpected end of file error". If I then copy the original NEF back from my portable HDD (backup drive) and copy it over the 'corrupted' NEF and re-open LR then everything is fine, my edits are still in place and I can export.
This seems to happen randomly, images that I've just worked on, images that I worked on yesterday but haven't touched in this session.
What I haven't done is try to open a 'corrupted' NEF in another program, like ViewNX to see if the file really is affected.
I'm running Windows 7 Home Premium SP1, 64 Bit, Intel i7-3820 CPU @ 3.6Ghz with 16GB of RAM.
I'll be honest and say that I'm worried. My photos are precious to me (photos of my niece as a baby whom I won't see for another 6 months) and the thought of losing them is a very worrying prospect. This is something that I've never had to contend with before. My day to day backup solution wouldn't cover this since I'd have to be aware that there was an issue with a file before copying it back. If the corruption occurs outside of LR then my files could become corrupted and then that damaged file is copied to the backup disk overnight, overwriting the original undamaged copy.
One other thing, should I move my LR catalogue to the SSD?
Seeing the same filelength and timestamp does not guarantee the file has not been modified - an application can write to it (changing the contents without length) and then change the timestamp explicitly. Could you please do two following experiments:
1. Copy the 'corrupted' NEF to another disk and then copy it back. If this fixes the problem, this means that its contents is OK
2. Set all NEF files in one of directories that gets corrupted to Read-Only
http://www.techtin.com/windows-7/make-files-and-folder-read-only-in-windows-7/
and see whether this hides the problem (if not, this is definitely very low-level)
Alex
1. Copy the 'corrupted' NEF to another disk and then copy it back. If this fixes the problem, this means that its contents is OK
2. Set all NEF files in one of directories that gets corrupted to Read-Only
http://www.techtin.com/windows-7/make-files-and-folder-read-only-in-windows-7/
and see whether this hides the problem (if not, this is definitely very low-level)
Alex
beaujolais76
Member
why not just make RAW file read-only in OS ?
That should prove that it wasn't changed by lightroom in case an issue will appear again
That should prove that it wasn't changed by lightroom in case an issue will appear again
Interesting further input for sure.
I think we need the original OP from the thread to come back and give us update re. his own configuration to further know what his problem may be. The fact that you are seeing pretty much the same is definitely of note.
In your case, I would start (sorry to be tedious) by asking you same questions re. the setting of your LR write EXIF data to raw configuration. If that is not set (as per this threads OP's case), then again LR should only be opening the files in read-only mode, so should not be the cause of corruption. (Just saying should not , and not yet exonerating LR).
Other question to you is also the same as to the OP - when you look in Windows Explorer to see File Properties for a corrupt vs non-corrupt version of a file, what do you see? What does your back-up software tell-you? Are you using incremental backup software that detects file differences? Does the backup software say that any raw file has been changed in any way?
Your catalog and previews are on a 2nd HDD, so we can pretty-much discount LR over-stressing your 1st HDD image disk as being a possible cause. As far as I understand LR is only reading from that image disk when you actually load an image (i.e. when you see LR's loading progress spinner). That's pretty light use, but your D800 files are quite large and will take time to load. Does not mean your image HDD does not have a problem though! Hate to ask, but how old is that image HDD you are using?
In library, grid, filmstrip, views, etc., etc., LR is using the previews from your 2nd catalog HDD. And yes, LR will be working this 2nd disk very hard, e.g. whenever you move around in LR and it has to show new images of any kind to the screen (in filmstrip, anywhere), then it is constantly opening and updating those preview files, as well as the opening and constant updating of the active catalog file. LR is also reading/writing the camera raw cache files, and by default I think those are on the application disk (I am Mac user, so not 100% sure where but it's in your LR preferences menu).
It is often recommended to put catalog and preview files on SSD for access speed reasons, but SSD also wear out, so for longevity and disk stress reasons maybe the answer is different. You have them on a different HDD to your images and that is a good way. I can't give any other opinion than that, other than make sure you also back up the lrcat file to another disk occassionally. Back up your lrdata previews also if you want, but LR can re-generate those if needed. Camera raw cache is a cache, so LR handles that all on it's own and no need to back up and if it's on your SSD I guess it is fine there although it is being constantly written.
Your edits in LR will always be in place so long as your lrcat file is robust, hence why you can overwrite the underlying RAW and still see all the adjustments you made to that image, and how LR can then export, print, etc., the image from the new copy of the file.
I think we need both you and the OP to do more comparisons of corrupted vs non-corrupted versions of a file in your OS (Alex is correct an app can change a file's content keeping the timestamp and file size, but that has to be a clearly coded operation).
It's interesting that both of you and the OP have D800 files, which I think must be the largest SLR RAWs out there. That D800 may be some factor in the equation, so there may come a point soon to raise an issue with Adobe, but they will (I am pretty certain) ask you and the OP all the same disk- and file-related questions as we are.
[ I am a user of LR for about the last 3 years and have an engineering background. I enjoy the hunt of tracking down a problem... and that's my only position in this - other than I might one day also get a D800! ]
I think we need the original OP from the thread to come back and give us update re. his own configuration to further know what his problem may be. The fact that you are seeing pretty much the same is definitely of note.
In your case, I would start (sorry to be tedious) by asking you same questions re. the setting of your LR write EXIF data to raw configuration. If that is not set (as per this threads OP's case), then again LR should only be opening the files in read-only mode, so should not be the cause of corruption. (Just saying should not , and not yet exonerating LR).
Other question to you is also the same as to the OP - when you look in Windows Explorer to see File Properties for a corrupt vs non-corrupt version of a file, what do you see? What does your back-up software tell-you? Are you using incremental backup software that detects file differences? Does the backup software say that any raw file has been changed in any way?
Your catalog and previews are on a 2nd HDD, so we can pretty-much discount LR over-stressing your 1st HDD image disk as being a possible cause. As far as I understand LR is only reading from that image disk when you actually load an image (i.e. when you see LR's loading progress spinner). That's pretty light use, but your D800 files are quite large and will take time to load. Does not mean your image HDD does not have a problem though! Hate to ask, but how old is that image HDD you are using?
In library, grid, filmstrip, views, etc., etc., LR is using the previews from your 2nd catalog HDD. And yes, LR will be working this 2nd disk very hard, e.g. whenever you move around in LR and it has to show new images of any kind to the screen (in filmstrip, anywhere), then it is constantly opening and updating those preview files, as well as the opening and constant updating of the active catalog file. LR is also reading/writing the camera raw cache files, and by default I think those are on the application disk (I am Mac user, so not 100% sure where but it's in your LR preferences menu).
It is often recommended to put catalog and preview files on SSD for access speed reasons, but SSD also wear out, so for longevity and disk stress reasons maybe the answer is different. You have them on a different HDD to your images and that is a good way. I can't give any other opinion than that, other than make sure you also back up the lrcat file to another disk occassionally. Back up your lrdata previews also if you want, but LR can re-generate those if needed. Camera raw cache is a cache, so LR handles that all on it's own and no need to back up and if it's on your SSD I guess it is fine there although it is being constantly written.
Your edits in LR will always be in place so long as your lrcat file is robust, hence why you can overwrite the underlying RAW and still see all the adjustments you made to that image, and how LR can then export, print, etc., the image from the new copy of the file.
I think we need both you and the OP to do more comparisons of corrupted vs non-corrupted versions of a file in your OS (Alex is correct an app can change a file's content keeping the timestamp and file size, but that has to be a clearly coded operation).
It's interesting that both of you and the OP have D800 files, which I think must be the largest SLR RAWs out there. That D800 may be some factor in the equation, so there may come a point soon to raise an issue with Adobe, but they will (I am pretty certain) ask you and the OP all the same disk- and file-related questions as we are.
[ I am a user of LR for about the last 3 years and have an engineering background. I enjoy the hunt of tracking down a problem... and that's my only position in this - other than I might one day also get a D800! ]
handymanuk
Active member
Thank you for the great advice and tips on what to try next.
I'll post more when I've done some further investigation.
Thanks again.
After reading through this thread this is the first thing that I looked at. This is not checked so LR will not write meta data changes to the RAW files.
I use plain and simple synctoy which states that it does detect file changes but I'm not sure what attributes it bases this on. I'll try to get another file to error tonight to perform some analysis as I was more focussed on recovering the image at the time rather than looking for causes.
Well the RAW file store HDD isn't too old, maybe a couple of years however the preview drive was my old, original system drive back from 2008. I've now moved the catalogue and preview cache to the SSD.
Thanks, I've moved the entire LR directory over, cache, previews and catalog.
I'll definitely try tonight to gather some more information. Something that I did notice is that at the time that the 'corruption' happened I was playing video from the drive that held the catalogue and previews. Perhaps that is relevant?
I think that they probably are the largest files at the moment, yes.
I'll post more when I've done some further investigation.
Thanks again.
Let's see how you get on with looking into the errors, and the original OP has not posted back yet.
I am wondering whether both of you are seeing hard-disk failures - and it is the fact that .NEF files from D800 are large which is the factor on why you are seeing it. I do not know the .NEF format at all, but maybe that file format is simply not at all resilient to errors, and this is all just pure statistcs around failing disk sectors being most evident in larger files, particularly if those are non-streamable formats without any higher-level error correction.
[ Still not discounting LR is to blame, but the evidence so far is against - for the original OP the file date/times were the same, and for both of you LR is not set in a mode where it needs to write to the file... ]
I would suggest you try reading the corrupt files multiple times in some filecheck type utility and see whether you get the same (checksum) result each time. Also run disk check utilities on the image disk. In my view at this point disk failure remains the most likely for both you and the OP.
It is vaguely possible the video-playing has some relation, but it was from another disk, so it would have to be pretty tenuous link, like a power-supply problem (video-playing runs your CPUs and your other video disk hard, so it could case the power supply to not be as smooth or at the exact voltage as it should be). Could also be weird RAM corruption making LR think it should write to the original RAW, but that is soooo unlikely.
I am wondering whether both of you are seeing hard-disk failures - and it is the fact that .NEF files from D800 are large which is the factor on why you are seeing it. I do not know the .NEF format at all, but maybe that file format is simply not at all resilient to errors, and this is all just pure statistcs around failing disk sectors being most evident in larger files, particularly if those are non-streamable formats without any higher-level error correction.
[ Still not discounting LR is to blame, but the evidence so far is against - for the original OP the file date/times were the same, and for both of you LR is not set in a mode where it needs to write to the file... ]
I would suggest you try reading the corrupt files multiple times in some filecheck type utility and see whether you get the same (checksum) result each time. Also run disk check utilities on the image disk. In my view at this point disk failure remains the most likely for both you and the OP.
It is vaguely possible the video-playing has some relation, but it was from another disk, so it would have to be pretty tenuous link, like a power-supply problem (video-playing runs your CPUs and your other video disk hard, so it could case the power supply to not be as smooth or at the exact voltage as it should be). Could also be weird RAM corruption making LR think it should write to the original RAW, but that is soooo unlikely.
hi, thx!
one of my first thougths as well ...
... but Apple Preview is not looking at those side-car files ...
... and when I open the files in Preview they are corrupted. (and have not been before)
eT
one of my first thougths as well ...
... but Apple Preview is not looking at those side-car files ...
... and when I open the files in Preview they are corrupted. (and have not been before)
eT
thx, Mark,
for this detailed answer and posting! very much appreciated! (sorry, I could not read this earlier as my current studies with ISU @ FIT/KSC are extremely time demanding)
Interesting theory you brought up and I will definitely look further into it. Though still interesting that LR4 (in some cases) corrupted the same picture twice (if it was LR).
Will look further into it!
eT
.
for this detailed answer and posting! very much appreciated! (sorry, I could not read this earlier as my current studies with ISU @ FIT/KSC are extremely time demanding)
Interesting theory you brought up and I will definitely look further into it. Though still interesting that LR4 (in some cases) corrupted the same picture twice (if it was LR).
Will look further into it!
eT
.
you too?! ... OMG
so maybe it's the combo D800 & LR4???
b/c I doubt that it's probable that a couple of people have the same problems arising due to hardware issues on their computer in such a setting ...
... did you talk to Adobe or Nikon about it?
eT
so maybe it's the combo D800 & LR4???
b/c I doubt that it's probable that a couple of people have the same problems arising due to hardware issues on their computer in such a setting ...
... did you talk to Adobe or Nikon about it?
eT
thx, alex.
next time the problem will occur I will do like you suggested in (1).
also one moment will try out your suggestion in (2) - though not on win7 or winXYZ but Mac. ;-)
eT
next time the problem will occur I will do like you suggested in (1).
also one moment will try out your suggestion in (2) - though not on win7 or winXYZ but Mac. ;-)
eT
thx again, Mark. (I like your 'hunting down' ;-) )
some more detail on my hardware configuration / age:
SSD = 240 GB OCZ Vertex3 (LR4 and Catalog on this one) - about 1 year old
HDD = 750 GB WDC WD7500BPKT (RAW files on this one) - a couple of months old.
MacBook Unibody 'late 2008', 2.4GHz, 8 GB RAM (RAM about 1/2 year old)
eT
ps: dSLR's with big files ... well: there's also the Pentax 645D ;-) - which I do not own but adore it's viewfinder!
some more detail on my hardware configuration / age:
SSD = 240 GB OCZ Vertex3 (LR4 and Catalog on this one) - about 1 year old
HDD = 750 GB WDC WD7500BPKT (RAW files on this one) - a couple of months old.
MacBook Unibody 'late 2008', 2.4GHz, 8 GB RAM (RAM about 1/2 year old)
eT
ps: dSLR's with big files ... well: there's also the Pentax 645D ;-) - which I do not own but adore it's viewfinder!
Unfortunately too busy for my studies of ISU here at FIT & KSC ...
Good question ... I use NEF-lossless-compressed files on my D800 (as well as on the D700). I don't know what kind of compression algorythm Nikon is using.
eT
well: somebody else actually has the same problem ...
http://forums.dpreview.com/forums/read.asp?forum=1006&message=41775591
eT
http://forums.dpreview.com/forums/read.asp?forum=1006&message=41775591
eT
I'm really not trying to be argumentative and the problem must be worrying, not knowing how and why images are being corrupted, but out of the hundreds of people on here who will be using Lightroom 4 daily,only two of you using Apple Macs are getting the file corruption.
That is a few variables removed though (Windows, file system, likelihood of viruses...)
Are all the images on internal drives?
I see above you jumped on the fact of you both using LR4 and D800, but it could just as easily be any combination of D800 and Mac OS, or Mac OS and LR4, or none of the above.
Back in the 90's we used to get a few hardware/driver issues on our Macs, which was invariably solved by removing quicktime from the equation.
Things are better now, but the cause and solution of your problem may not be logical or obvious.
That is a few variables removed though (Windows, file system, likelihood of viruses...)
Are all the images on internal drives?
I see above you jumped on the fact of you both using LR4 and D800, but it could just as easily be any combination of D800 and Mac OS, or Mac OS and LR4, or none of the above.
Back in the 90's we used to get a few hardware/driver issues on our Macs, which was invariably solved by removing quicktime from the equation.
Things are better now, but the cause and solution of your problem may not be logical or obvious.
Unless I am confusing who you were refering to, I understood second poster is running Windows:
http://forums.dpreview.com/forums/read.asp?forum=1006&message=41775591
"I'm running Windows 7 Home Premium SP1, 64 Bit, Intel i7-3820 CPU @ 3.6Ghz with 16GB of RAM."
I believe probabilities and evidence gathered so far suggests disk failings in both cases.
Both posters have stated where there files are - the "corrupted" RAWs, and also LR catalog, preview, the LR program and implicitly ACR cache files. In both cases the corrupted files are on separate internal drives from the OS+apps and from their LR catalog, preview, and cache files.
It was me who first suggested possible obscure statistical correlation of effect of disk failures to D800's larger .NEF files.
[ Since I have been vocal on this thread so far, just to say I am waiting for more failure observation from either of them before passing further opinion on possible causes and leave it to either poster if they want to raise anything with Adobe. ]
http://forums.dpreview.com/forums/read.asp?forum=1006&message=41775591
"I'm running Windows 7 Home Premium SP1, 64 Bit, Intel i7-3820 CPU @ 3.6Ghz with 16GB of RAM."
I believe probabilities and evidence gathered so far suggests disk failings in both cases.
Both posters have stated where there files are - the "corrupted" RAWs, and also LR catalog, preview, the LR program and implicitly ACR cache files. In both cases the corrupted files are on separate internal drives from the OS+apps and from their LR catalog, preview, and cache files.
It was me who first suggested possible obscure statistical correlation of effect of disk failures to D800's larger .NEF files.
[ Since I have been vocal on this thread so far, just to say I am waiting for more failure observation from either of them before passing further opinion on possible causes and leave it to either poster if they want to raise anything with Adobe. ]
Keyboard shortcuts
- f
- Forum