Software to process CHDK raw files from Canon compacts?

Started Feb 15, 2017 | Discussions
NAwlins Contrarian Veteran Member • Posts: 3,198
Software to process CHDK raw files from Canon compacts?

What software do you use to process the raw files you create using the CHDK firmware add-on with Canon compact cameras that don't normally produce raw files? Why do you use it / prefer it? What alternatives do you not / no longer use, and why?

It appears that at the moment, Adobe Lightroom (at least version 6.8) will process DNGs created with the latest stable CHDK (1.4.1), but DxO Optics Pro (at least Elite version 11.3.1) will not. I just downloaded the new RawTherapee 5, which I may try too.

Background: I've used various versions of the CHDK firmware add-on with various Canon compacts over the years, and had differing degrees of luck processing the raw files. One takeaway is that saving them as DNGs seemed to do better than saving them as CR2s or CRWs. Preparing for a trip where I would not have the space to bring a 'real' camera, I installed the latest stable version of CHDK (1.4.1, I think) on our PowerShot SX230 HS. Upon return, my usual favorite raw converter, DxO Optics Pro, simply would not load the files. Lightroom did load them, and let me apply lens corrections and perspective distortion modifiers as wanted / needed, but the output seems to have odd artifacts. Specifically, there are blue and greed dots that don't look to me like color noise. Here's an example with 100% crops--which, alas, DPR seems to be recompressing or something and masking the problem:

The artifacts are circles in white.

And here they are at 400%:

So: any idea what's causing this? Any suggestions for alternatives?

Thanks!

 NAwlins Contrarian's gear list:NAwlins Contrarian's gear list
Nikon Coolpix S30 Canon PowerShot S110 Sony Alpha DSLR-A580 Tamron SP AF 90mm F/2.8 Di Macro Tamron SP 70-300mm F/4-5.6 Di USD +5 more
Canon PowerShot SX230 HS
If you believe there are incorrect tags, please send us this post using our feedback form.
Enders Shadow
Enders Shadow Senior Member • Posts: 2,489
Re: Software to process CHDK raw files from Canon compacts?

High ISO long exposure? If that's the case, I'd guess those are hot pixels. The camera's JPEG engine has a map of the dead/hot pixels and avoids using them. CHDK apparently doesn't use the map when creating the RAW file.

-- hide signature --

Phil

 Enders Shadow's gear list:Enders Shadow's gear list
Canon PowerShot G12 Panasonic Lumix DMC-FZ200 Sony Cyber-shot DSC-RX10 Olympus Tough TG-3 Fujifilm X30 +22 more
MOD Dale Buhanan Veteran Member • Posts: 4,280
Re: Software to process CHDK raw files from Canon compacts?

So then these happen through the conversion process, and they are not in the original?  If that is the case, abandon that procedure and find anorher one.  DPP does a good job and won't produce those strange artifacts

-- hide signature --

kind regards
Dale
Moderator Canon Powershot, Feedback and 7D/XXD forums

 Dale Buhanan's gear list:Dale Buhanan's gear list
Canon PowerShot SX50 HS Canon G1 X II Canon EOS 70D Canon PowerShot SD500 Canon PowerShot G5 +35 more
OP NAwlins Contrarian Veteran Member • Posts: 3,198
Re: Software to process CHDK raw files from Canon compacts?

High ISO long exposure? If that's the case, I'd guess those are hot pixels. The camera's JPEG engine has a map of the dead/hot pixels and avoids using them. CHDK apparently doesn't use.

Thanks for the reply. The example was taken at ISO 100. The exposure was 1 second, which I don't really consider long,* but do you? Also, does the fact that all of the spots I noted were green or blue, not red, tell anything useful?

You raise another issue that I need to look at, though: CHDK has some facility to map hot pixels / noise / whatever, and I need to figure out how to turn it on / use it.

*As an aside, I was trying to achieve blurring of the people in the picture, and also to keep noise moderate, so I went for ISO 100 and as long of an exposure as I could get. Later I realized that the SX230 HS's aperture-priority mode will not select a shutter speed slower than 1 second. I could and maybe should have tried going into manual exposure mode, stopped down to f/5.6 or even f/8 (the smallest), and used maybe a 3 second or 5 second exposure.

 NAwlins Contrarian's gear list:NAwlins Contrarian's gear list
Nikon Coolpix S30 Canon PowerShot S110 Sony Alpha DSLR-A580 Tamron SP AF 90mm F/2.8 Di Macro Tamron SP 70-300mm F/4-5.6 Di USD +5 more
OP NAwlins Contrarian Veteran Member • Posts: 3,198
Re: Software to process CHDK raw files from Canon compacts?

So then these happen through the conversion process, and they are not in the original? If that is the case, abandon that procedure and find anorher one. DPP does a good job and won't produce those strange artifacts

Thanks for the reply. The JPEGs don't have the green and blue dots, but of course JPEGs aren't really originals. In fact, comparing the SOOC JPEGs to the DNGs that CHDK creates shows just how extreme are some of the JPEG processing activities, like correcting the lens's huge geometric distortion. Last I tried, DPP would not process CHDK raw files that were tagged as CR2s or CRWs. I have processed them with Lightroom, and in the past with UFRaw, which I think was a port of DCRaw to a GIMP plugin.

 NAwlins Contrarian's gear list:NAwlins Contrarian's gear list
Nikon Coolpix S30 Canon PowerShot S110 Sony Alpha DSLR-A580 Tamron SP AF 90mm F/2.8 Di Macro Tamron SP 70-300mm F/4-5.6 Di USD +5 more
Geodude Contributing Member • Posts: 565
Re: Software to process CHDK raw files from Canon compacts?

Def hot pixels which get removed by the JPEG engine during conversion in camera....there are a fair number here in the Raw file sample which seems high must admit. Do you have Photoshop/ or elements? Open in Adobe Camera Raw and see if replicated there? Or DPP as suggested. Obviously, if you see in other Raw converters these hot pixels do exist and not a product of software conversion...can you post the original Raw file before conversion in LR?

Try the freeware GIMP and see if that's any good?

 Geodude's gear list:Geodude's gear list
Canon PowerShot G1 X
Enders Shadow
Enders Shadow Senior Member • Posts: 2,489
Re: Software to process CHDK raw files from Canon compacts?

The problem is this camera doesn't natively produce Canon RAW files. He's using CHDK that produces a DNG.

-- hide signature --

Phil

 Enders Shadow's gear list:Enders Shadow's gear list
Canon PowerShot G12 Panasonic Lumix DMC-FZ200 Sony Cyber-shot DSC-RX10 Olympus Tough TG-3 Fujifilm X30 +22 more
OP NAwlins Contrarian Veteran Member • Posts: 3,198
Update and explanation ... but no fix.
1

First off, thanks to everyone who has tried to help.

After extensive research and some experimentation, I'm 89% confident of the following:

* The problem is not Lightroom, or anything raw converter-related. Conversely, telling CHDK to write CR2s or CRWs and then processing in DPP (even if you could, which in the past I could not) or some other program would not help.

* These are bad pixels. Apparently on my camera, around 37,000 of them! Supposedly 18,000 is pretty typical. I guess 0.3% bad is not a crisis. On the other hand, it's far too many to fix manually.

* CHDK has a function whereby it keeps a list of bad pixels, and then instead of using their readings, takes an average from the adjacent pixels and writes that instead. I had it set to use this option.

* However, to make the foregoing option work, you have to first manually call for an operation that creates the bad pixel list--which appears as a file named badpixel.bin in the CHDK directory on the memory card. Bust just telling CHDK to create the bad pixel list does not necessarily create the list. In my experience, the 'create a bad pixel list' function fails more than 90% of the time. The CHDK folks basically acknowledge this can be a problem. Initially I didn't notice the failure warning or the lack of the bad pixel list file in the CHDK directory on the memory card. I found references that the best chance of the creation process working are with ISO 100 or whatever is base and an exposure between 1/5 s and 0.8 s. Using that, eventually I got it to work, and now badpixel.bin exists where it should.

* However, the DNG I took as a test after getting badpixel.bin created and verifying that the correct ('average the area') option was set still shows the bad pixels in Lightroom.

So I'm still trying to make it work. I guess as way to use / fix the existing DNGs, I could take a black shot and then use that to create a mask and with layers black out the bad pixels, which would be a lot less intrusive than leaving them blue and green. Of course, to make this work you'd need to make the particular lens geometry adjustments for the mask every time, which would get really cumbersome.

Ah the frustrations of using a free firmware hack.

 NAwlins Contrarian's gear list:NAwlins Contrarian's gear list
Nikon Coolpix S30 Canon PowerShot S110 Sony Alpha DSLR-A580 Tamron SP AF 90mm F/2.8 Di Macro Tamron SP 70-300mm F/4-5.6 Di USD +5 more
reyalp Regular Member • Posts: 147
Re: Update and explanation ... but no fix.

NAwlins Contrarian wrote:

First off, thanks to everyone who has tried to help.

After extensive research and some experimentation, I'm 89% confident of the following:

You've basically got it right, but there are a few details that may help.

When you select DNG version 1.3 in CHDK, it uses a DNG feature called Opcode lists to tell the raw processors which pixel values are "bad". Most raw software doesn't support this feature, but Adobe's DNG converter should. I'm surprised lightroom doesn't.

Raw therapee still doesn't support opcodes AFAIK, but in the 4.2.x versions at least the "dead pixel filter" in the raw tab should do a pretty good job of patching them.

IIRC the SX230 has very little free RAM with CHDK running, which may explain the badpixel.bin method not working even after you've managed to generate the file.

Personally, I use chdkptp to fix bad pixels in CHDK DNGs, but it's not the most user friendly approach https://app.assembla.com/spaces/chdkptp/wiki/DNG_Processing

OP NAwlins Contrarian Veteran Member • Posts: 3,198
Re: Update and explanation ... but no fix.

When you select DNG version 1.3 in CHDK, it uses a DNG feature called Opcode lists to tell the raw processors which pixel values are "bad". Most raw software doesn't support this feature, but Adobe's DNG converter should. I'm surprised lightroom doesn't.

Raw therapee still doesn't support opcodes AFAIK, but in the 4.2.x versions at least the "dead pixel filter" in the raw tab should do a pretty good job of patching them.

IIRC the SX230 has very little free RAM with CHDK running, which may explain the badpixel.bin method not working even after you've managed to generate the file.

Personally, I use chdkptp to fix bad pixels in CHDK DNGs, but it's not the most user friendly approach https://app.assembla.com/spaces/chdkptp/wiki/DNG_Processing

Not only am I pretty sure it's set to DNG v. 1.3, I seem to recall getting an error when I tried to even explore what were the other options. As far as Lightroom not fixing the bad pixels even after I succeeded in creating the bad pixel file, I don't know what to say except that I'm up-to-date in the standalone version, that is, v. 6.8. I may have to try Raw Therapee 5 to see about this 'bad pixel filter'--or does Lightroom have something familiar of which I'm unaware?

Last and maybe most: please tell me more about chdkptp. Can I download an executable or installer from somewhere? In any case I will have to follow your link later.

Thanks!

 NAwlins Contrarian's gear list:NAwlins Contrarian's gear list
Nikon Coolpix S30 Canon PowerShot S110 Sony Alpha DSLR-A580 Tamron SP AF 90mm F/2.8 Di Macro Tamron SP 70-300mm F/4-5.6 Di USD +5 more
reyalp Regular Member • Posts: 147
Re: Update and explanation ... but no fix.

NAwlins Contrarian wrote:

Not only am I pretty sure it's set to DNG v. 1.3, I seem to recall getting an error when I tried to even explore what were the other options.

Right, to use DNG 1.1, you must successfully create the badpixel.bin file. This repeatedly fails on some cameras for reasons we don't understand, and also can take up a lot of memory, so 1.3 is the default. I wasn't clear from your post whether you had been able to create badpixel.bin successfully or not.

I may have to try Raw Therapee 5 to see about this 'bad pixel filter'--or does Lightroom have something familiar of which I'm unaware?

I'm not a Lightroom user, so I'll leave that for others to answer. If it has an option to correct "dead" or zero valued pixels, that's what you want. (you see saturated color spots because only one color component is 0)

Last and maybe most: please tell me more about chdkptp. Can I download an executable or installer from somewhere? In any case I will have to follow your link later.

There are executables for windows, see first link in https://app.assembla.com/spaces/chdkptp/wiki/Install

As I mentioned, this isn't a very user friendly option. I wrote the program so I find it convenient, but YMMV. Even if you only use it to process DNGs, you'll need to install libusb-win32 to satisfy DLL dependencies, and the actual conversion is done using command line commands.

If you are looking for something more user friendly, Adobe's free DNG converter will do a DNG to DNG conversion, which results in a new DNG that has the bad pixels fixed and can probably be opened by programs like DxO Optics. CHDK DNGs conform to the DNG standard, but they use some uncommon options that aren't supported in some programs.

PhilM oz Senior Member • Posts: 1,308
Re: Software to process CHDK raw files from Canon compacts?

I've processed a lot of CHDK DNG files in Lightroom and I've not seen a problem like this, it should automatically correct for dead pixels.

I added the DNG 1.3 bad pixel handling code to CHDK a few years ago, as reyalp has said, non-Adobe software does not always handle the CHDK DNG files; but Photoshop and LR should work.

Can you upload a sample DNG file to a file sharing site and post the link here so we can examine the contents more closely to see what is going on.

Phil.

OP NAwlins Contrarian Veteran Member • Posts: 3,198
Re: Update and explanation ... but no fix.

Right, to use DNG 1.1, you must successfully create the badpixel.bin file. This repeatedly fails on some cameras for reasons we don't understand, and also can take up a lot of memory, so 1.3 is the default. I wasn't clear from your post whether you had been able to create badpixel.bin successfully or not.

Thanks. I had originally set it to DNG 1.3, thinking (1) newer is probably better and (2) they made it the default for a reason. (I did not originally succeed in creating a badpixel.bin file.) Lightroom does not seem to correct the bad pixels on the shots recorded in DNG 1.3 format. See more below.

 NAwlins Contrarian's gear list:NAwlins Contrarian's gear list
Nikon Coolpix S30 Canon PowerShot S110 Sony Alpha DSLR-A580 Tamron SP AF 90mm F/2.8 Di Macro Tamron SP 70-300mm F/4-5.6 Di USD +5 more
OP NAwlins Contrarian Veteran Member • Posts: 3,198
DNG 1.1 now working in LR, DNG 1.3 still has bad pixels.

As a further update: as of my latest test, pictures recorded in DNG 1.1 format now work fine in Lightroom 6.8. Yes, I was finally able to create a badpixel.bin file, on about the twentieth attempt, shooting at ISO 100 into a white piece of paper, where A mode selected a 1/5 s shutter speed.

However, I still cannot get Lightroom to process out the bad pixels in pictures recorded in DNG 1.3 format. Maybe CHDK should write DNG 1.3 files that pass Lightroom the information with which to correct for bad pixels, and maybe Lightroom should correct for bad pixels using information it finds in the DNG 1.3 files that CHDK created. But it just is not happening, at least for me.

Below are two samples, of which you should be able to call up the full files, both processed from DNGs to JPEGs in Lightroom 6.8, one from a DNG 1.1 and one from a DNG 1.3.

DNG version 1.1 from CHDK converted to JPEG in Lightroom 6.8

And here's the other:

DNG version 1.3 from CHDK converted to JPEG in Lightroom 6.8; view original and see the bad pixels

As you can see if you click on "View: original size" and then zoom in to 100%, the second version still has all the bad pixels.

Oh, and in case you're wondering, these pictures make it obvious just how much software lens correction the SX230 HS is applying, to correct the bad geometric distortion and vignetting.

 NAwlins Contrarian's gear list:NAwlins Contrarian's gear list
Nikon Coolpix S30 Canon PowerShot S110 Sony Alpha DSLR-A580 Tamron SP AF 90mm F/2.8 Di Macro Tamron SP 70-300mm F/4-5.6 Di USD +5 more
OP NAwlins Contrarian Veteran Member • Posts: 3,198
Re: Software to process CHDK raw files from Canon compacts?

Phil, thanks very much for the explanation and offer. As my recent post below (in threaded view) explains, now I've gotten Lightroom to convert the DNG 1.1 files so that the bad pixels are removed, but still no luck with the DNG 1.3 files.

If I've understood the explanations correctly, CHDK needs the badpixel.bin file, but uses it to 'bake in' the bad pixel corrections to the DNG 1.1 files, so Lightroom or whatever does not have correct them; however, with DNG 1.3 files, CHDK justs passes along a list of where bad pixel corrections need to be made, and lets Lightroom or whatever make them as it deems best--at least theoretically allowing for more sophisticated bad pixel correction algorithms, including even better ones in the future.

Of course I have a vested interest in getting Lightroom to correct the bad pixels in DNG 1.3 files because I have dozens of them from a recent trip, and there are far too many bad pixels to correct manually.

The DNG 1.3 for the sample picture in my recent post is about 18 MB. Would it be easier or quicker for you to send me a private message with your e-mail address, and I'll e-mail it to you? I'm not very familiar with Dropbox-type uploading, although I have done it before and could do it if you prefer.

 NAwlins Contrarian's gear list:NAwlins Contrarian's gear list
Nikon Coolpix S30 Canon PowerShot S110 Sony Alpha DSLR-A580 Tamron SP AF 90mm F/2.8 Di Macro Tamron SP 70-300mm F/4-5.6 Di USD +5 more
OP NAwlins Contrarian Veteran Member • Posts: 3,198
Re: Software to process CHDK raw files from Canon compacts?

I've uploaded to a file-share the sample DNGs I used in Lightroom to make the sample JPEGs that I posted previously. This link has the DNG version 1.1, DNG version 1.3, and badpixel.bin:

https://app.box.com/s/s0vqvd0g28qb4aic2nlm5nzopurlpgqh

I've discovered another oddity: the lowly Windows Photo Viewer appears to render the DNG ver. 1.3 with the bad pixels fixed. Why Lightroom is not fixing them remains a mystery.

 NAwlins Contrarian's gear list:NAwlins Contrarian's gear list
Nikon Coolpix S30 Canon PowerShot S110 Sony Alpha DSLR-A580 Tamron SP AF 90mm F/2.8 Di Macro Tamron SP 70-300mm F/4-5.6 Di USD +5 more
PhilM oz Senior Member • Posts: 1,308
Re: Software to process CHDK raw files from Canon compacts?
2

Thanks for the files.

There was a bug in the DNG 1.3 code in CHDK that I have now fixed. The new version should be available on the autobuild server in a few hours. This should ensure new images are corrected in LR as expected.

Existing images can be fixed; but unfortunately the Adobe DNG converter wont fix them. If you have a lot of affected images I can probably create a command line program to help fix the issue.

Phil.

OP NAwlins Contrarian Veteran Member • Posts: 3,198
Re: Software to process CHDK raw files from Canon compacts?

There was a bug in the DNG 1.3 code in CHDK that I have now fixed. The new version should be available on the autobuild server in a few hours. This should ensure new images are corrected in LR as expected.

Thanks for the update. I'm glad to learn that it wasn't wholly my misuse or ignorance that caused the problem. It's also nice to play a small role in the improvement of CHDK, which is a great tool.

Existing images can be fixed; but unfortunately the Adobe DNG converter wont fix them. If you have a lot of affected images I can probably create a command line program to help fix the issue.

That is a very generous offer! I have two dozen pictures from a recent short trip. It would be great if I could process them correctly in Lightroom. I think I could handle a command-line interface. (Although my computer skills have atrophied greatly, command lines were the norm for my first half-dozen years of computing, and there was even a time when I had to do some work with 8086 assembly language and computers with hex keypads.) I don't want to put you to much trouble, though, so I guess it depends on how much time and effort would be involved on your end.

Now here is the part I don't understand: my existing DNG 1.3 files were created before there was a badpixel.bin on the card. How does Windows Photo Viewer know--apparently WPV does know, because the DNGs from the trip display in it without bad pixels--which are the bad pixels? Clearly it's not a case of CHDK having created the DNG 1.3 files by using an existing badpixel.bin to put into them a list of bad pixels for WPV to fix.

Thanks a mil for everything!

 NAwlins Contrarian's gear list:NAwlins Contrarian's gear list
Nikon Coolpix S30 Canon PowerShot S110 Sony Alpha DSLR-A580 Tamron SP AF 90mm F/2.8 Di Macro Tamron SP 70-300mm F/4-5.6 Di USD +5 more
PhilM oz Senior Member • Posts: 1,308
Re: Software to process CHDK raw files from Canon compacts?
2

I've created a simple command line program to fix existing DNG files.

Send me a PM with your email address so I can send it to you. Let me know what OS you are running.

Phil.

OP NAwlins Contrarian Veteran Member • Posts: 3,198
Re: Software to process CHDK raw files from Canon compacts?

Thanks Phil, that's very generous of you, and I will do so momentarily!

 NAwlins Contrarian's gear list:NAwlins Contrarian's gear list
Nikon Coolpix S30 Canon PowerShot S110 Sony Alpha DSLR-A580 Tamron SP AF 90mm F/2.8 Di Macro Tamron SP 70-300mm F/4-5.6 Di USD +5 more
Keyboard shortcuts:
FForum MMy threads