raw file cd-r backup warning

Evan Lauber

Active member
Messages
71
Reaction score
0
Location
NY, US
Heres a good reason to shoot jpeg instead of RAW. only kidding...

I just wasted a lot of time archiving RAW files to CD-R and found out an amazing discovery - many of them were defective! It looks like RAW files need to be precisely backed up with zero errors, to the exact bit or byte or CRC check, or whatever. I have used my new 32 speed cd burner with no noticable problems what so ever when backing up tiff files. (they must be more forgiving than raw files if the data is not saved 100% true)

Basically I wasted all this time making bad cd's unknowingly, and thinking I had a virus. I thought I had good backup cd-r's, but I was shocked to find out they are not! Right now I am lucky i backed up tiff files at these high burn speeds as they can be loaded up where RAW files can not.

The raw files would not convert with breeze browser or canon's software. Luckily I did not erase my compact flash card, and was able to do some testing. My findings are that CRW files will error out on cd-r backups showing errors in the cd-r backup that even roxio's ezcd with data verify can not catch! When I burn tiff files at the same speed and same cd-r blanks, I get no errors. So basically there appears to be no forgiveness for even the slightest error in a RAW file. Maybe canon can update their raw file converter to allow for some forgiveness for minor data errors.

Lesson Learned: buy CD-R blanks that are rated higher than your max burn speed that you use, as the manufacturer that I used claimed 32 x speed, but i did not get a reliable burn untill i dropped down to 16 x speed. Also do not rely on the verify option alone, as it did not work for me, as it did not report errors in the RAW files. Always do a preview with the canon software (if you can figure out how to get it to show specific directorys, or breeze browser) and see if you can see the thumnails on the burned cd, to be sure you have a good backup of your raw files, as they are real finicky. With this in mind, I now make two cd backups, burn at slower speeds, and also backup the converted tiffs.

with all this work, its just one more reason for the jpeg camp to say its not worth all the work to shoot raw... however I still shoot raw for all the right reasons! just a little bit more careful now backing up before I erase my compact flash cards!

for those of you who are into details. the burned cd-r that had defective CRW files never had any problems with any other burns at all. this has been verified on several cd-r's, and the byte count on the crw's was the same, and they passed roxio ezcd creator 5's verify. not sure if this uses a crc check or not. the file i checked had a file modified date which was 4 hours earlier than the original.

Hope this saves you guys some time, and some images from being lost!

Evan Lauber
http://www.portfolios.com/evan
 
Thanks for the info.
I've found loosing photos is a serious problem :-)

Thanh
Heres a good reason to shoot jpeg instead of RAW. only kidding...

I just wasted a lot of time archiving RAW files to CD-R and found
out an amazing discovery - many of them were defective! It looks
like RAW files need to be precisely backed up with zero errors, to
the exact bit or byte or CRC check, or whatever. I have used my
new 32 speed cd burner with no noticable problems what so ever when
backing up tiff files. (they must be more forgiving than raw files
if the data is not saved 100% true)

Basically I wasted all this time making bad cd's unknowingly, and
thinking I had a virus. I thought I had good backup cd-r's, but I
was shocked to find out they are not! Right now I am lucky i
backed up tiff files at these high burn speeds as they can be
loaded up where RAW files can not.

The raw files would not convert with breeze browser or canon's
software. Luckily I did not erase my compact flash card, and was
able to do some testing. My findings are that CRW files will error
out on cd-r backups showing errors in the cd-r backup that even
roxio's ezcd with data verify can not catch! When I burn tiff
files at the same speed and same cd-r blanks, I get no errors. So
basically there appears to be no forgiveness for even the slightest
error in a RAW file. Maybe canon can update their raw file
converter to allow for some forgiveness for minor data errors.

Lesson Learned: buy CD-R blanks that are rated higher than your max
burn speed that you use, as the manufacturer that I used claimed 32
x speed, but i did not get a reliable burn untill i dropped down to
16 x speed. Also do not rely on the verify option alone, as it
did not work for me, as it did not report errors in the RAW files.
Always do a preview with the canon software (if you can figure out
how to get it to show specific directorys, or breeze browser) and
see if you can see the thumnails on the burned cd, to be sure you
have a good backup of your raw files, as they are real finicky.
With this in mind, I now make two cd backups, burn at slower
speeds, and also backup the converted tiffs.

with all this work, its just one more reason for the jpeg camp to
say its not worth all the work to shoot raw... however I still
shoot raw for all the right reasons! just a little bit more
careful now backing up before I erase my compact flash cards!

for those of you who are into details. the burned cd-r that had
defective CRW files never had any problems with any other burns at
all. this has been verified on several cd-r's, and the byte count
on the crw's was the same, and they passed roxio ezcd creator 5's
verify. not sure if this uses a crc check or not. the file i
checked had a file modified date which was 4 hours earlier than the
original.

Hope this saves you guys some time, and some images from being lost!

Evan Lauber
http://www.portfolios.com/evan
--

o
 
Good news Frank, glad your not having problems! Thats because you are most likely using higher quality cd-r's then i used, or your burn speed is slower.

i still feel that there is a chance that others may have this issue at some point with some cd-r's and at some faster burn speeds.

i just want people to know that there is a chance that they could be having this problem, and to check their burned cd-r's by converting the raw files off the burned cd-r's a few times. this way you know you have a good cd burning speed for your blank cd-r's with your specific burner. that and the fact that for some reason the files must be precise, so minor data errors that you may had that you never noticed as they don't show on tiff files and in audio applications, will show up with bad and unusable raw files.

maybe someone can make some software for converting bad raw files!

Evan
My CRW files are perfectly readable, no matter what burning speed I
use.

Frank Vandenberghe
 
I had a 4x Yamaha CD writer whch never lost me a file with any CDs it could write to. I now have a much cheaper LiteOn 24x and it often corrupts files - even with quality CDs. I think I will buy anothewr Yamaha. A bit more expensive but likely worth it.

LCD
 
You're not using DirectCD (or whatever it's called now) or any other packet writing software, are you? I've never had great success with that, and I've seen plenty of other negative reports on it.

My burning software of preference is Nero, and I've never had any problem with the files. I always check my CDs before deleting the original files with Windiff, a bit-for-bit file comparer that comes on the Windows98 CD (just which isn't automatically installed).

KRs
Chris
 
Heres a good reason to shoot jpeg instead of RAW. only kidding...
I just wasted a lot of time archiving RAW files to CD-R and found
out an amazing discovery - many of them were defective! It looks
like RAW files need to be precisely backed up with zero errors, to
the exact bit or byte or CRC check, or whatever. I have used my
new 32 speed cd burner with no noticable problems what so ever when
backing up tiff files. (they must be more forgiving than raw files
if the data is not saved 100% true)
The probable reason for this is that RAW is compressed (albeit non-lossily). Therefore any block errors in the CD-R files will have much more serious consequences than if the files were TIFF.

Therefore, the solution is to always, I repeat always, convert the RAW files to lossless TIFF before you burn them. This also has another advantage: With TIFF, you have certainty that your precious photographs can be opened and used in the future. This alone is good enough reason why you should always convert the RAW's into TIFF before storage.

You mentioned JPEG. It has to be said that those who say that JPEG is as good as lossless "because the difference is so small" or "the difference cannot be seen with my eyes" are mistaken. If their eyes cannot see it, others' might. Also, in the future there will be developed better tools for display and better printers, so it is always best to store the best possible quality for posterity. The deplorably widespread but nevertheless wrong assumption that JPEG is as good as lossless does great damage to much of humanity's photographic works, and a more conscious approach to this is needed.

Per Inge Oestmoen
http://www.coldsiberia.org/
 
Good news Frank, glad your not having problems! Thats because you
are most likely using higher quality cd-r's then i used, or your
burn speed is slower.
Another factor is that there are great differences between CD-burners' ability to write CD's accurately with a minimum of block errors. Many believe that burning is either-or, so that if a file is burned it is as good as the files from any other burner.

Such is not the case. Two factors come into play:

1. The accuracy of the laser beam that writes the data. If accuracy is low, there will be many block errors, and if accuracy is high, correspondingly fewer.

2. The power of the laser beam itself

Over time, both deteriorate, but the best CD burners like the Plextor SCSI machines (which are made for longer and heavier use than the IDE versions) are certified to last for 2000 hours of burning before any appreciable loss in accuracy or laser power occurs.

Per Inge Oestmoen
http://www.coldsiberia.org/
 
Heres a good reason to shoot jpeg instead of RAW. only kidding...
I just wasted a lot of time archiving RAW files to CD-R and found
out an amazing discovery - many of them were defective! It looks
like RAW files need to be precisely backed up with zero errors, to
the exact bit or byte or CRC check, or whatever. I have used my
new 32 speed cd burner with no noticable problems what so ever when
backing up tiff files. (they must be more forgiving than raw files
if the data is not saved 100% true)
The probable reason for this is that RAW is compressed (albeit
non-lossily). Therefore any block errors in the CD-R files will
have much more serious consequences than if the files were TIFF.

Therefore, the solution is to always, I repeat always, convert the
RAW files to lossless TIFF before you burn them. This also has
another advantage: With TIFF, you have certainty that your precious
photographs can be opened and used in the future. This alone is
good enough reason why you should always convert the RAW's into
TIFF before storage.

You mentioned JPEG. It has to be said that those who say that JPEG
is as good as lossless "because the difference is so small" or "the
difference cannot be seen with my eyes" are mistaken. If their eyes
cannot see it, others' might. Also, in the future there will be
developed better tools for display and better printers, so it is
always best to store the best possible quality for posterity. The
deplorably widespread but nevertheless wrong assumption that JPEG
is as good as lossless does great damage to much of humanity's
photographic works, and a more conscious approach to this is needed.
I had almost decided to take all my D60 images in RAW and extract the JEPEG and store the RAW files on a CDR or DVD+R. Now that this problem with burning RAWs to CDRs has come up I am not sure I am prepared to invest all the time converting every RAW to TIFF before I store it on CDR/ DVD. Unless I can find a way to reliably store the orignal RAW images on CDR/DVD, I might stay with JPEG. The trade-off between image quality and my available time will probab;y swing it against RAW...
 
If you don't test the results, then you don't have a valid backup.

I had a situation 6 months ago, where I performed a backup to CD-R. Luckily I tested the backup before deleting the files on the PC as only around 20% of the CD was readable. With something as valuable as
photos, I usually make 2 backups.... To be sure, to be sure!!!
Heres a good reason to shoot jpeg instead of RAW. only kidding...

I just wasted a lot of time archiving RAW files to CD-R and found
out an amazing discovery - many of them were defective! It looks
like RAW files need to be precisely backed up with zero errors, to
the exact bit or byte or CRC check, or whatever. I have used my
new 32 speed cd burner with no noticable problems what so ever when
backing up tiff files. (they must be more forgiving than raw files
if the data is not saved 100% true)

Basically I wasted all this time making bad cd's unknowingly, and
thinking I had a virus. I thought I had good backup cd-r's, but I
was shocked to find out they are not! Right now I am lucky i
backed up tiff files at these high burn speeds as they can be
loaded up where RAW files can not.

The raw files would not convert with breeze browser or canon's
software. Luckily I did not erase my compact flash card, and was
able to do some testing. My findings are that CRW files will error
out on cd-r backups showing errors in the cd-r backup that even
roxio's ezcd with data verify can not catch! When I burn tiff
files at the same speed and same cd-r blanks, I get no errors. So
basically there appears to be no forgiveness for even the slightest
error in a RAW file. Maybe canon can update their raw file
converter to allow for some forgiveness for minor data errors.

Lesson Learned: buy CD-R blanks that are rated higher than your max
burn speed that you use, as the manufacturer that I used claimed 32
x speed, but i did not get a reliable burn untill i dropped down to
16 x speed. Also do not rely on the verify option alone, as it
did not work for me, as it did not report errors in the RAW files.
Always do a preview with the canon software (if you can figure out
how to get it to show specific directorys, or breeze browser) and
see if you can see the thumnails on the burned cd, to be sure you
have a good backup of your raw files, as they are real finicky.
With this in mind, I now make two cd backups, burn at slower
speeds, and also backup the converted tiffs.

with all this work, its just one more reason for the jpeg camp to
say its not worth all the work to shoot raw... however I still
shoot raw for all the right reasons! just a little bit more
careful now backing up before I erase my compact flash cards!

for those of you who are into details. the burned cd-r that had
defective CRW files never had any problems with any other burns at
all. this has been verified on several cd-r's, and the byte count
on the crw's was the same, and they passed roxio ezcd creator 5's
verify. not sure if this uses a crc check or not. the file i
checked had a file modified date which was 4 hours earlier than the
original.

Hope this saves you guys some time, and some images from being lost!

Evan Lauber
http://www.portfolios.com/evan
 
The more I think about this the more convinced I am to "compromise on quality" and go with JPEG as my standard format for the D60. If it is necessary to convert every RAW image to TIFF before it is stored to a CDR this poses two problems that I think will affect most people:

1. A D60 TIFF is something like 38MB, so you can only get 19 on a CDR !!

2. The time to do all the conversions. I think it would get past the point of having fun pretty quickly.

... just a personal opinion. I hope someone can help change my mind as I'm keen to make a decision on which format to standardize on.
I had a situation 6 months ago, where I performed a backup to CD-R.
Luckily I tested the backup before deleting the files on the PC as
only around 20% of the CD was readable. With something as valuable
as
photos, I usually make 2 backups.... To be sure, to be sure!!!
Heres a good reason to shoot jpeg instead of RAW. only kidding...

I just wasted a lot of time archiving RAW files to CD-R and found
out an amazing discovery - many of them were defective! It looks
like RAW files need to be precisely backed up with zero errors, to
the exact bit or byte or CRC check, or whatever. I have used my
new 32 speed cd burner with no noticable problems what so ever when
backing up tiff files. (they must be more forgiving than raw files
if the data is not saved 100% true)

Basically I wasted all this time making bad cd's unknowingly, and
thinking I had a virus. I thought I had good backup cd-r's, but I
was shocked to find out they are not! Right now I am lucky i
backed up tiff files at these high burn speeds as they can be
loaded up where RAW files can not.

The raw files would not convert with breeze browser or canon's
software. Luckily I did not erase my compact flash card, and was
able to do some testing. My findings are that CRW files will error
out on cd-r backups showing errors in the cd-r backup that even
roxio's ezcd with data verify can not catch! When I burn tiff
files at the same speed and same cd-r blanks, I get no errors. So
basically there appears to be no forgiveness for even the slightest
error in a RAW file. Maybe canon can update their raw file
converter to allow for some forgiveness for minor data errors.

Lesson Learned: buy CD-R blanks that are rated higher than your max
burn speed that you use, as the manufacturer that I used claimed 32
x speed, but i did not get a reliable burn untill i dropped down to
16 x speed. Also do not rely on the verify option alone, as it
did not work for me, as it did not report errors in the RAW files.
Always do a preview with the canon software (if you can figure out
how to get it to show specific directorys, or breeze browser) and
see if you can see the thumnails on the burned cd, to be sure you
have a good backup of your raw files, as they are real finicky.
With this in mind, I now make two cd backups, burn at slower
speeds, and also backup the converted tiffs.

with all this work, its just one more reason for the jpeg camp to
say its not worth all the work to shoot raw... however I still
shoot raw for all the right reasons! just a little bit more
careful now backing up before I erase my compact flash cards!

for those of you who are into details. the burned cd-r that had
defective CRW files never had any problems with any other burns at
all. this has been verified on several cd-r's, and the byte count
on the crw's was the same, and they passed roxio ezcd creator 5's
verify. not sure if this uses a crc check or not. the file i
checked had a file modified date which was 4 hours earlier than the
original.

Hope this saves you guys some time, and some images from being lost!

Evan Lauber
http://www.portfolios.com/evan
 
That is not correct. If you open a TIF file that is corrupted it will be just as undreadable as a CRW file that is corrupted.

The problem is that much CD writing software spews out data at a pre-determined rate but has no way to know whether it actually got written correctly.

This problem affects ANY kind of data. I have a friend who also wrote many unusable CDs with data (not images).

The solution is not to change the format of your data.

The solutions are:

1. Use good quality media

2. Use software which does a Verify operation - that is does a read and compare of what has just been written

3.(Belt and braces) do the above twice and keep the 2nd CD in a safe place (e.g. bank safe deposit) away from the first copy

4. (Belt and braces 2) Scan your CD with Thumbsplus on another computer to ensure it is readable

Cheers
Ian
The probable reason for this is that RAW is compressed (albeit
non-lossily). Therefore any block errors in the CD-R files will
have much more serious consequences than if the files were TIFF.

Therefore, the solution is to always, I repeat always, convert the
RAW files to lossless TIFF before you burn them. This also has
another advantage: With TIFF, you have certainty that your precious
photographs can be opened and used in the future. This alone is
good enough reason why you should always convert the RAW's into
TIFF before storage.

You mentioned JPEG. It has to be said that those who say that JPEG
is as good as lossless "because the difference is so small" or "the
difference cannot be seen with my eyes" are mistaken. If their eyes
cannot see it, others' might. Also, in the future there will be
developed better tools for display and better printers, so it is
always best to store the best possible quality for posterity. The
deplorably widespread but nevertheless wrong assumption that JPEG
is as good as lossless does great damage to much of humanity's
photographic works, and a more conscious approach to this is needed.

Per Inge Oestmoen
http://www.coldsiberia.org/
--
Ian S
http://www.rainpalm.com
 
Heres a good reason to shoot jpeg instead of RAW. only kidding...

I just wasted a lot of time archiving RAW files to CD-R and found
out an amazing discovery - many of them were defective! It looks
like RAW files need to be precisely backed up with zero errors, to
the exact bit or byte or CRC check, or whatever. I have used my
new 32 speed cd burner with no noticable problems what so ever when
backing up tiff files. (they must be more forgiving than raw files
if the data is not saved 100% true)

Basically I wasted all this time making bad cd's unknowingly, and
thinking I had a virus. I thought I had good backup cd-r's, but I
was shocked to find out they are not! Right now I am lucky i
backed up tiff files at these high burn speeds as they can be
loaded up where RAW files can not.

The raw files would not convert with breeze browser or canon's
software. Luckily I did not erase my compact flash card, and was
able to do some testing. My findings are that CRW files will error
out on cd-r backups showing errors in the cd-r backup that even
roxio's ezcd with data verify can not catch! When I burn tiff
files at the same speed and same cd-r blanks, I get no errors. So
basically there appears to be no forgiveness for even the slightest
error in a RAW file. Maybe canon can update their raw file
converter to allow for some forgiveness for minor data errors.

Lesson Learned: buy CD-R blanks that are rated higher than your max
burn speed that you use, as the manufacturer that I used claimed 32
x speed, but i did not get a reliable burn untill i dropped down to
16 x speed. Also do not rely on the verify option alone, as it
did not work for me, as it did not report errors in the RAW files.
Always do a preview with the canon software (if you can figure out
how to get it to show specific directorys, or breeze browser) and
see if you can see the thumnails on the burned cd, to be sure you
have a good backup of your raw files, as they are real finicky.
With this in mind, I now make two cd backups, burn at slower
speeds, and also backup the converted tiffs.

with all this work, its just one more reason for the jpeg camp to
say its not worth all the work to shoot raw... however I still
shoot raw for all the right reasons! just a little bit more
careful now backing up before I erase my compact flash cards!

for those of you who are into details. the burned cd-r that had
defective CRW files never had any problems with any other burns at
all. this has been verified on several cd-r's, and the byte count
on the crw's was the same, and they passed roxio ezcd creator 5's
verify. not sure if this uses a crc check or not. the file i
checked had a file modified date which was 4 hours earlier than the
original.

Hope this saves you guys some time, and some images from being lost!

Evan Lauber
http://www.portfolios.com/evan
Evan,

Your experience (although painful) is a valuable lesson for all of us regarding backup.

However, the solution is to ensure that the backup of the CRW is valid, not to change to TIFF for long term storgage. If there is a brun problem with your media or burner, then fix it and check the discs as needed, but to not back up the CRW or to back up a TIFF in which you have lost half the advantages of the CRW (control over future conversions), is in my oipinon the wrong solution.

So thanks a lot for sharing your experience, but the lesson that I will walk away with is to make sure that my burns are good, not to decide not to archive my RAW files.

Just my opinion. YMMV.
--
Regards,

Michael Tapes
http://www.YarcPlus.com
http://www.michaeltapes.com
 
That is not correct. If you open a TIF file that is corrupted it
will be just as undreadable as a CRW file that is corrupted.
This is a bit imprecise.

The point is that a in a compressed file like RAW or ZIP, the data will be much more densely packed. Therefore any errors will have more severe xonsequences, and the threshold for unreadability will be lower.
The problem is that much CD writing software spews out data at a
pre-determined rate but has no way to know whether it actually got
written correctly.
That depends on the quality and accuracy of the CD/DVD writer and of the media used.
This problem affects ANY kind of data. I have a friend who also
wrote many unusable CDs with data (not images).
This is true, but it becomes more acute with any kind of compression because of the increased density of data, which causes block errors to destroy more data per area than when data are not so densely packed. Otherwise what you say is correct.
The solution is not to change the format of your data.
Why? This is of no consequence and moreover one does not change anything after the first burning.
The solutions are:
1. Use good quality media
That is important, but it is actually more important to have a CD or DVD burner which is capable of writing accurately with few block errors. Also, the laser intensity will be reduced over time, and cheap writers will likely experience a more rapid deterioration.
2. Use software which does a Verify operation - that is does a read
and compare of what has just been written
Recommendable.
3.(Belt and braces) do the above twice and keep the 2nd CD in a
safe place (e.g. bank safe deposit) away from the first copy
This is VERY important. At least two backups of every important file is a necessity.
4. (Belt and braces 2) Scan your CD with Thumbsplus on another
computer to ensure it is readable
Also recommendable, but be aware that the correction mechanisms in all computers and all sofware may deceive you. Good quality media and a burner of similarly high standard is a prerequisite to safe storage.

Per Inge Oestmoen
http://www.coldsiberia.org/
 
If you don't test the results, then you don't have a valid backup.
I had a situation 6 months ago, where I performed a backup to CD-R.
Luckily I tested the backup before deleting the files on the PC as
only around 20% of the CD was readable. With something as valuable
as photos, I usually make 2 backups.... To be sure, to be sure!!!
Agreed.

But be aware that different CD writers may have vastly different block error rates, and that it is important to have as few block errors as possible. This depends on BOTH the burner and the media.

For more information:

http://www.cd-info.com/CDIC/Technology/CD-R/Media/testing.html

Per Inge Oestmoen
http://www.coldsiberia.org/
 
However, the solution is to ensure that the backup of the CRW is
valid, not to change to TIFF for long term storgage. If there is a
brun problem with your media or burner, then fix it and check the
discs as needed, but to not back up the CRW or to back up a TIFF in
which you have lost half the advantages of the CRW (control over
future conversions), is in my oipinon the wrong solution.
The only problem with RAW is that RAW is a proprietary format. Hence we cannot have certainty that our files can be opened in 20 years or more from now. With an established standard like TIFF we have that certainty.

One may easily make the basic adjustments of color and brightness on the RAW before conversion to TIFF.

Per Inge Oestmoen
http://www.coldsiberia.org/
 
However, the solution is to ensure that the backup of the CRW is
valid, not to change to TIFF for long term storgage. If there is a
brun problem with your media or burner, then fix it and check the
discs as needed, but to not back up the CRW or to back up a TIFF in
which you have lost half the advantages of the CRW (control over
future conversions), is in my oipinon the wrong solution.
The only problem with RAW is that RAW is a proprietary format.
Hence we cannot have certainty that our files can be opened in 20
years or more from now. With an established standard like TIFF we
have that certainty.
Somehow I think I'll have a copy of YarcPlus around > :)
One may easily make the basic adjustments of color and brightness
on the RAW before conversion to TIFF.
Yes, of course, but what if I want to change what I had thought was right back then...or better conversion techniques are established (as they will be) > :)

In my opinion keeping the RAW negative is essential, but your points not assumming that your archives are burbed correctly are valid and appreciated.

Thanks..

--
Regards,

Michael Tapes
http://www.YarcPlus.com
http://www.michaeltapes.com
 
We were going nuts trying to trouble-shoot a friend's new CD-RW drive. It worked just fine except for the last 10 or 20 percent of the disc. Any files stored there were corrupted. Didn't matter if it was text, RAW, or music files.

We finally narrowed it down to the media.

Moral:

Buy good media.

Burn a few COMPLETE CD's (no space left over), and check all the files to make sure they're not corrupted.

--
The Unofficial Photographer of The Wilkinsons
http://thewilkinsons.crosswinds.net
Photography -- just another word for compromise
 

Keyboard shortcuts

Back
Top