Magenta edge tinging at high ISO and a possible solution...

Started Aug 17, 2009 | Discussions
BudH Contributing Member • Posts: 543
Re: Magenta edge tinging at high ISO and a possible solution...

The improvement demonstrated here is just incredible!

-- hide signature --

GX-1S, K20D (x2), AF-540FGZ, EF500 Super, DA 12-24, DA 18-55, DA 50-200, DA 55-300, Sigma 17-70, A 28/f2.8, A 135/f2.8, FA 50/f1.4, A 28-80, A 70-210, Sigma 600 Mirror, Sigma AF1.4 TC

Marius Shanzer Forum Member • Posts: 53
My Results with Raw Border Correcter v. 9

A huge thank you to Gordon for his ongoing work on Raw Border Correcter. I guesstimate that I have 2000+ shots on my K20D taken at ISO 3200 that are candidates for RBC. Not everyone needs to shoot at 3200 or higher but if you do, you owe it to yourself to try out RBC. I've created a small gallery on my website that demonstrates my results with RBC.

Version A -- run through ACR with 30 Luminance and 50 Color Noise Reduction and identical across all ten images.
Version B -- identical settings as Version A but run through RBC

Version C -- same noise reduction in ACR as Versions A & B + Noiseware in Photoshop. In addition, color balance and exposure were tweaked on an individual basis. For this reason, there is actually more noise in Version C than in Version B for a few a images.

http://www.marius-shanzer.com/gbg-rbc-demo/index.html

This is better than upgrading my current equipment (not that I wouldn't mind that). It's another bite at the apple for my older photos.

Thanks again Gordon.

Marius

-- hide signature --
 Marius Shanzer's gear list:Marius Shanzer's gear list
Nikon D700 Nikon AF-S Nikkor 14-24mm f/2.8G ED Nikon AF-S Nikkor 24-70mm f/2.8G ED Nikon AF-S Nikkor 50mm f/1.4G Tokina AT-X Pro 100mm f/2.8 Macro +4 more
photogerald Senior Member • Posts: 1,798
Re: My Results with Raw Border Correcter v. 9

Marius Shanzer wrote:

A huge thank you to Gordon for his ongoing work on Raw Border Correcter. I guesstimate that I have 2000+ shots on my K20D taken at ISO 3200 that are candidates for RBC. Not everyone needs to shoot at 3200 or higher but if you do, you owe it to yourself to try out RBC. I've created a small gallery on my website that demonstrates my results with RBC.

Version A -- run through ACR with 30 Luminance and 50 Color Noise Reduction and identical across all ten images.
Version B -- identical settings as Version A but run through RBC

Version C -- same noise reduction in ACR as Versions A & B + Noiseware in Photoshop. In addition, color balance and exposure were tweaked on an individual basis. For this reason, there is actually more noise in Version C than in Version B for a few a images.

http://www.marius-shanzer.com/gbg-rbc-demo/index.html

Thanks for posting these - that last image (version c) really cleaned up nicely, didn't it? By the way, where were these taken? It looks like it was an interesting performance.

I'm also working on some examples which I'll be posting soon. I've actually discovered something surprising (in a good way).

This is better than upgrading my current equipment (not that I wouldn't mind that). It's another bite at the apple for my older photos.

I think this tool does more than many people realize. Details to follow...

photogerald Senior Member • Posts: 1,798
My results with 0.9.0.0!

Firstly, I wanted to thank Dominique (the OP) for coming up with the idea of RAW border correction, and of course GordonBGood for providing us with a tool to perform this correction with ease.

The test which I am going to present here is a bit different from the others - I never shoot at ISO 3200 or higher, instead, I wanted to see what the RAW border correction could do for my images.

For my test I have selected an image that was taken on my recent trip to HK. It was shot at ISO 1000 and is representative of the type of image that I would take at what I consider to be "high" ISO. I realize that this same image could have been taken at ISO 100 (and with better results), but I don't like lugging around a tripod while on vacation.

The RAW file (PEF) was first run through GordonBGood's RAW Border Correcter tool (v0.9.0.0). Both the "corrected" and original RAW files were processed using Pentax Photo Lab 3.61 with default settings and all noise reduction disabled. A series of exposure adjustments were made in Photo Lab (+1ev, +2ev, +3ev) for each. The resulting JPGs were then resized to 2000 pixel width using IrfanView, before being uploaded to PBase. The images shown here have been further resized by PBase, however if you would like to see the larger 2000 pixel versions, just change "large.jpg" to "original.jpg" in the image URLs (if anyone would like to see the full-sized originals, just let me know).

Here is the "before" image (no exposure adjustment):

This image looks pretty good as-is. However, once you compare it directly with the "after" image (with Raw Border Correcter applied), you'll see that even it benefits from the correction (you might need to download both images and switch back and forth between them to see the difference):

Next, let's look at what happens when we apply a +1EV adjustment (the original image might look better with a bit of an exposure boost anyways):

Here the colour cast in the dark area along the bottom should be apparent if you're using a calibrated monitor. If it isn't, just compare it with the corrected version:

At +2EV things start to get ugly:

Fortunately, K20D RAW Border Correcter saves the day:

Still not convinced? How about +3EV:

Surely there's no hope, right?

Wrong!

Holy smokes, Batman! This is ISO 1000 pushed 3 stops! Ignoring the overexposed areas, this would be equivalent to ISO 8000 right? This image is actually quite usable juding by the original (unfortunately, the sharpening applied during resizing has exaggerated the noise in the images shown here).

What's a bit surprising to me after looking at all these shots is the effect that the tool has on the non-black areas (sky and buildings). I know you mentioned earlier that the correction shouldn't affect the colour balance, but the buildings do appear to be slightly changed. Or does this merely indicate that the buildings were also affected by the magenta cast? This isn't a complaint, by the way - I actually prefer this look. I'm just curious to know if I'm interpreting the results correctly.

Anyways, overall, it's as if a layer of grit (magenta-coloured!) has been removed from the images.

Taking this idea further, I decided to try it with the following ISO 400 image:

Let's jump straight to +3EV:

And the corrected version:

The difference is subtle, but it's there (look closely at original.jpg). Admittedly, this isn't a very good example as the dark areas aren't really that dark. That said, I've run into colour casts with photos taken at this ISO under artificial lighting which required some large white balance corrections, and I would expect these to benefit more from the RAW Border Correcter tool. I plan to look into this some more.

So, to wrap up this rather long post, I'll summarize here my thoughts regarding GordonBGood's K20D Raw Border Correcter tool:

-this tool is a must have for images taken at ISO 3200 and 6400, no doubt about it (look at the examples from the other posters)

-this tool will even benefit images taken at moderately-high ISOs (such as ISO 800 and 1600), and is highly recommend especially if the images require any sort of PP

-it remains to be seen how much of a benefit there will be below ISO 800, but I have a strong feeling it will prove useful whenever large exposure or white balance adjustments need to be made (in PP)

I would strongly encourage every K20D shooter to give this tool a try, even those like me who don't usually shoot above 1600 ISO. I'm confident you'll find this to be a must-have. And let us know your findings.

Thanks again to everyone who has contributed to this idea. I can't tell you how excited I am after trying this tool - it feels like as if I am shooting with a new camera (K20D Super)!

photogerald Senior Member • Posts: 1,798
Re: PROGRESS: Version 0.8.0.0 has one more tiny fix...

jamesm007 wrote:

Now if only we could disable (or tone-down) the RAW NR!

If you speak of the built in NR for RAW files that the K20D and K-7 applies I have a different opinion. As you now know Nikon applies in many of their models RAW NR at base iso - I would much rather have it at iso3200 than iso200!

However the real point, member parcarfan asked if it would be better to shoot at iso1600 and -1EV and avoid the NR in RAW files with the K-7 at iso3200. The K20D starts at iso1600. I explored this a bit (with my K20D) and it seems the NR does nothing you can see or destructive to details, it may in fact bring out details otherwise lost in noise, if you turned it off suddenly details would not appear, its very slight at best there would be no benefit with it off IMO. I tried shooting a iso800 pic at -1EV to reproduce iso1600 (starting ISO for K20D NR to) and found the iso1600 pic the same perhaps better... hard to judge. What I really found is its serious pixel peeping stuff you don't see it at normal or 100% viewing.

Hi James - I had missed your reply earlier. I recall seeing a more "processed" look to images taken at higher ISO, and I attributed this to the RAW NR. However, I should go back and test this again.

Your idea about shooting at lower ISO and pushing in PP is very interesting - in fact, my tests at ISO 1000 (see my last post) seem to support this approach. You might want to repeat your ISO 800 test but at -2EV, and compare the results with shots at ISO 3200.

IMO: Bottom line the RAW NR is nothing to worry about and probably applies corrections in the PRIME engine you would have a hard time doing with NR software, also there is no need to 'tone-down' this NR as its very subtle, like the NR used at iso200 in the Nikon DC5000. This noise is global and not the type GB program fixes, removing the NR would only put the burden on us to remove it, I don't believe it hurts details as the pics I have at iso6400 are as good as they get for an APS sensor IMO with plenty of detail I don't desire at all to be able to disable NR at ISO6400 as I don't believe especially at iso1600 and iso3200 that you would see a much of a difference and or any need to

The shots at 6400 are definitely a lot better after using RAW Border Corrector.

photogerald Senior Member • Posts: 1,798
> 1 stop improvement

jamesm007 wrote:

Bottom line it works, it drastically (removes) reduces the corner tint with DNG files from the K20D, this alone greatly improves the K20D. ISO6400 is now usable, and it helps all ISO that have the tint even ISO1600 if needed, IMO this program give K20D owners about a one stop or more improvement in low light high ISO shooting, a very nice improvement indeed . All K20D owners who shoot RAW need this program.

My tests at ISO 1000 back this up, easily.

GordonBGood Veteran Member • Posts: 6,308
Re: My results with 0.9.0.0!

photogerald wrote:

What's a bit surprising to me after looking at all these shots is the effect that the tool has on the non-black areas (sky and buildings). I know you mentioned earlier that the correction shouldn't affect the colour balance, but the buildings do appear to be slightly changed. Or does this merely indicate that the buildings were also affected by the magenta cast? This isn't a complaint, by the way - I actually prefer this look. I'm just curious to know if I'm interpreting the results correctly.

Thanks, Gerald, for doing these tests showing the effectiveness of the correction algorithm vertically (landscape orientation).

If you read some of my posts to this thread carefully, you will see that I have predicted a marginal improvement at least down to ISO 200, which improvements will be visible when you do these extreme exposure boosts. As an aside, for the K20D I don't recommend dropping ISO below about ISO 800 and then recovering with an exposure boost as one then runs into the problems of the K20D's (and K-7's) limited Dynamic Range (DR) in the form of relatively increased deep shadow noise at those lower ISO's. However, reducing ISO to between ISO 800 to less than ISO 1600 (ISO 1600 has raw Noise Reduction - NR) and then boosting in post processing will give you the advantage of processing files without NR and thus maximizing detail while still getting equivalent to high ISO's, while leaving all of options of best optimizing NR in post processing to trade of detail vs. noise as the user sees fit.

As to your perception of White Balance (WB) changing with the application of the RBC tool, I said that overall WB has been designed not to change , but didn't say that actual variation of WB won't change in that an average removal of magenta at the edges will still increase the green in the center of the image. I did this because Auto WB (AWB) calculations done by the camera are generally done on a global image basis and I didn't want to disturb the overall look of the images based on these calculations. So what you are seeing isn't completely unexpected. And yes, as the correction is applied as an offset on a row by row basis (landscape orientation), it does apply to bright tones as well as dark, although the effect is a little less for the bright tones than the dark due to the way Tone Response Curves (TRC's) are generally applied and the way out human vision works. However, obviously the correction is enough to affect our perception of local WB.

Regards, GordonBGood

photogerald Senior Member • Posts: 1,798
Re: My results with 0.9.0.0!

GordonBGood wrote:

photogerald wrote:

What's a bit surprising to me after looking at all these shots is the effect that the tool has on the non-black areas (sky and buildings). I know you mentioned earlier that the correction shouldn't affect the colour balance, but the buildings do appear to be slightly changed. Or does this merely indicate that the buildings were also affected by the magenta cast? This isn't a complaint, by the way - I actually prefer this look. I'm just curious to know if I'm interpreting the results correctly.

Thanks, Gerald, for doing these tests showing the effectiveness of the correction algorithm vertically (landscape orientation).

If you read some of my posts to this thread carefully, you will see that I have predicted a marginal improvement at least down to ISO 200, which improvements will be visible when you do these extreme exposure boosts. As an aside, for the K20D I don't recommend dropping ISO below about ISO 800 and then recovering with an exposure boost as one then runs into the problems of the K20D's (and K-7's) limited Dynamic Range (DR) in the form of relatively increased deep shadow noise at those lower ISO's. However, reducing ISO to between ISO 800 to less than ISO 1600 (ISO 1600 has raw Noise Reduction - NR) and then boosting in post processing will give you the advantage of processing files without NR and thus maximizing detail while still getting equivalent to high ISO's, while leaving all of options of best optimizing NR in post processing to trade of detail vs. noise as the user sees fit.

Thanks for the suggestions - I will do more experimentation in the range of ISO 800-1250 to find the "sweet spot" for high-ISO/low-light shooting.

As to your perception of White Balance (WB) changing with the application of the RBC tool, I said that overall WB has been designed not to change , but didn't say that actual variation of WB won't change in that an average removal of magenta at the edges will still increase the green in the center of the image. I did this because Auto WB (AWB) calculations done by the camera are generally done on a global image basis and I didn't want to disturb the overall look of the images based on these calculations. So what you are seeing isn't completely unexpected. And yes, as the correction is applied as an offset on a row by row basis (landscape orientation), it does apply to bright tones as well as dark, although the effect is a little less for the bright tones than the dark due to the way Tone Response Curves (TRC's) are generally applied and the way out human vision works. However, obviously the correction is enough to affect our perception of local WB.

I have gone back and found the post where you originally mentioned the effects of the tool on WB, and it all makes sense to me. As always, the time you have taken to explain these advanced topics is very much appreciated.

KHRoN New Member • Posts: 23
nighttime hand-held ISO 6400 and manual f4

Today (tonight) I have shot some photos to test RBC... and I'm amazed with the real-life difference!

Pentax K20D, ISO 6400, RAW + manual Raynette (?) f2.8 @ f4 + RBC 0.9 + ACR 5.4

both pictures (unprocessed and processed by RBC 0.9.0.0) was developed with exactly the same settings (tweaked for processed one)

first two pictures are shot at 1/10, third one is shot at 1/2

-- hide signature --

Cuiusvis hominis est errare, nullius nisi insipientis in errore perseverare.

photogerald Senior Member • Posts: 1,798
Re: nighttime hand-held ISO 6400 and manual f4

Amazing results! Thanks for sharing these!

GordonBGood Veteran Member • Posts: 6,308
Re: nighttime hand-held ISO 6400 and manual f4

KHRoN wrote:

Today (tonight) I have shot some photos to test RBC... and I'm amazed with the real-life difference!
Pentax K20D, ISO 6400, RAW + manual Raynette (?) f2.8 @ f4 + RBC 0.9 + ACR 5.4

both pictures (unprocessed and processed by RBC 0.9.0.0) was developed with exactly the same settings (tweaked for processed one)

Thanks for the examples, and glad to see RBC is doing what it is supposed to do. You camera looks like a good sample, too, in that I don't see any wide pattern vertical banding.

Just out of curiousity, what kind of "tweaks" did you apply to the RBC processed images? If they are simple and would seem to be something that would be generally preferred, I might be able to incorporate them into the program.

To all, I haven't dropped this project with version 0.9 but have been working on several things, some in collaboration with other people, as follows:

1) I now have the "drag and drop" capability working, so that one can drop a folder, an image file, or several image files from the same folder onto the '.exe' file (or onto a shortcut to it in Windows, including one in the "Send to" folder) and it will automatically process those with the usual user interface (Correct button automatically activated), with the destination folder set to be the same as the source.

2) This works with Linux as well as Windows, and one has the same capability, not with the '.exe' file directly but with a "launcher" text '.desktop' file which is a special feature of variants of Linux supporting the XDG http://www.freedesktop.org specification, which is most of them these days, including any new Gnome, KDE, or Xfce Desktop Environments. I will include an example of a "launcher" '.desktop' file in the next distribution, along with instructions on how to install the program for the local user only or globally for all users. This will also work with links or "symlinks".

3) It looks like I will be able to support Mac OSX in the same way with the '.app' folder "Bundle" construction that become so-called applications when it embeds a script file properly within its structure. Again, I believe that drag-and-drop capability will work with these and they can be perhaps be installed either locally or globally in the Applications folder with full drag and drop capability either directly to the "application" or to a link to it. I haven't been able to test this directly as I don't have a working Mac available to me just now, so have had to work through someone else. I will be documenting this OSX installation in the to-be-released version, as well.

4) Finally, I have been working with the capability to add lossless compression/decompression to the file support so as to support camera DNG's that have been converted to compressed DNG's by Adobe Lightroom or the Adobe DNG Converter. This is pretty over half complete as I can just paste in some of the work I did recently on opening CR2 files from the Canon 7D, with only the compression to be converted from the PEF file compression and tested.

Then there are a few more refinements I would like to add, including investigating this wide pattern vertical banding noise removal, but they will likely come slower.

Regards, GordonBGood

photogerald Senior Member • Posts: 1,798
Re: nighttime hand-held ISO 6400 and manual f4

GordonBGood wrote:

To all, I haven't dropped this project with version 0.9 but have been working on several things, some in collaboration with other people, as follows:

Hi Gordon - I realize that you have a lot on your plate right now, but I was just thinking the other day how it would be nice to be able to specify the prefix and/or suffix used in the naming of the processed files. The program could still use the prefix of "mc_" as the default, and it would have to use the default anyways when it's invoked through drag-and-drop (without adding the complexity of storing settings in a configuration file). Perhaps you could consider this if it can be implemented failry quickly. Thanks!

GordonBGood Veteran Member • Posts: 6,308
Re: nighttime hand-held ISO 6400 and manual f4

photogerald wrote:

GordonBGood wrote:

To all, I haven't dropped this project with version 0.9 but have been working on several things, some in collaboration with other people, as follows:

Hi Gordon - I realize that you have a lot on your plate right now, but I was just thinking the other day how it would be nice to be able to specify the prefix and/or suffix used in the naming of the processed files. The program could still use the prefix of "mc_" as the default, and it would have to use the default anyways when it's invoked through drag-and-drop (without adding the complexity of storing settings in a configuration file). Perhaps you could consider this if it can be implemented failry quickly. Thanks!

Gerald, I've been thinking of changing the prefix to "RBC_" as that seems to be the default reference for the program around here I've avoided adding features but could add this to a little text input box if others think it would be useful. OTOH, users could just use a batch renaming program to rename these to anything they like. Others opinions?

Regards, GordonBGood

Marius Shanzer Forum Member • Posts: 53
Demo Photos

Thanks. These were actually shot in New York last April. I've photographed the same dancers in a number of venues, always with (un)available light.
--
http://www.marius-shanzer.com

 Marius Shanzer's gear list:Marius Shanzer's gear list
Nikon D700 Nikon AF-S Nikkor 14-24mm f/2.8G ED Nikon AF-S Nikkor 24-70mm f/2.8G ED Nikon AF-S Nikkor 50mm f/1.4G Tokina AT-X Pro 100mm f/2.8 Macro +4 more
Marius Shanzer Forum Member • Posts: 53
RBC -- Additional Functionality for Version 1.0

A time-shifting algorithm to rescue indecisive moments?

http://www.marius-shanzer.com

 Marius Shanzer's gear list:Marius Shanzer's gear list
Nikon D700 Nikon AF-S Nikkor 14-24mm f/2.8G ED Nikon AF-S Nikkor 24-70mm f/2.8G ED Nikon AF-S Nikkor 50mm f/1.4G Tokina AT-X Pro 100mm f/2.8 Macro +4 more
KHRoN New Member • Posts: 23
Re: nighttime hand-held ISO 6400 and manual f4

Just out of curiousity, what kind of "tweaks" did you apply
to the RBC processed images?

A bit of exaggeration, it was well past midnight when I was writing that

My workflow was: ACR 5.4 (only white balance, blacks, and as I've shot more photos some of them also brightness and contrast, always chroma noise to 100, luma to 0), Resize, Smart Sharpen, Save for web.

So there is nothing really crucial/revolutional here, just ordinary post-processing. Only strong reduction of chroma noise without reducing luma noise (I like grainy photos).

-- hide signature --

Cuiusvis hominis est errare, nullius nisi insipientis in errore perseverare.

dosdan Contributing Member • Posts: 536
ISO 1600 boosted 2 stops

I do not have any ISO 3200 or 6400 from my K20D to test with v0.9 of RBC, so I took an ISO 1600 from a theatre shoot and boosted it 2 stops in post.

Original resized shot:

The lighting was quite yellow in the theatre. This is the white-balanced version with 2 stops of exposure boost & 2 steps increase in DR. It's too bright, but I wanted to work with the dark parts in the corner.

100% comparison - Top Left. You can see the decrease in blue noise. WB OK. (No extra NR applied during conversion.)

100% comparison - Bottom Right under the table. The WB goes greenish towards the bottom of the image.

Dan.

 dosdan's gear list:dosdan's gear list
Pentax K100D Super Pentax K20D Pentax K-5 Pentax K-01 Pentax K-3 +1 more
dosdan Contributing Member • Posts: 536
Re: ISO 1600 boosted 2 stops

The "original" version was already white-balanced.

Dan.

 dosdan's gear list:dosdan's gear list
Pentax K100D Super Pentax K20D Pentax K-5 Pentax K-01 Pentax K-3 +1 more
OP dlacoutu Contributing Member • Posts: 811
Re: nighttime hand-held ISO 6400 and manual f4

Hello!

Gordon, did you try the DISABLEBLC script command yet? Has it something to do with the black level correction you complained about?

 dlacoutu's gear list:dlacoutu's gear list
Samsung NX100 Pentax K-5 Samsung NX200 Pentax smc FA 50mm F1.4 Samsung NX 30mm F2 Pancake +6 more
GordonBGood Veteran Member • Posts: 6,308
Re: nighttime hand-held ISO 6400 and manual f4

dlacoutu wrote:

Gordon, did you try the DISABLEBLC script command yet? Has it something to do with the black level correction you complained about?

No, I didn't know about this and the DISABLENR command to put in the SYSPARMS.TXT file until your posts, as I haven't been monitoring the Pentax Hacker site lately. It's very likely that this may have to do with the Black Level Correction and that one can output raw files without these, which would make them completely raw and unprocessed. I need to find time to try this with my K200D, not that it has any particular problems.

Regards, GordonBGood

Keyboard shortcuts:
FForum MMy threads