Working around GR III Adobe AI NR limitations

To start, this is probably not a very practical technique to use in the field. It is more a proof of concept and an illustration of just how much Ricoh's choice to apply noise reduction to RAW files is reducing the usefulness of the GR III/IIIx in low light now that we have very effective modern AI based NR tools.

As many are familiar, Adobe's AI NR really does next to nothing for GR III files. It works fine with GR II files. So folks with the newest Ricoh cameras can't take advantage of the newest post processing techniques to reduce noise.

The root cause appears to be the heavy handed and impossible to turn off RAW noise reduction applied to GR III files at ISO 200 and above. Other manufacturers have been guilty of this, but the NR applied in the GR III RAW files (i.e. DNG) is fairly extreme in magnitude and starts at a very low ISO (just ISO 200).

As an experiment I've compared using Adobe AI NR on two different files:

• The DPR Low Light Studio Shot at ISO 6400
• The DPR Raw DR Studio Shot at ISO 100 + 6EV

I downloaded the DNGs from DPR and in LR reset all development settings, equalized exposures with the exposure compensation slider (-1 EV for the Low Light shot and +5 EV for the Raw DR shot), and set WB on the grey scale patch in the studio image. Next I applied Adobe's AI NR at 75%.

The results (as usual click to see full resolution):



On the left is the ISO 6400 shot, on the right is the ISO 100 shot 6 EV underexposed.

This clearly demonstrates Adobe AI NR works great on GR III files as long as Ricoh's ham-fisted RAW NR isn't gumming up the works.

It also suggests a not so useful workaround: Don't use higher ISO settings, instead use exposure compensation at base ISO and matching but opposite exposure compensation in post processing.

Indeed that works in theory as illustrated above, but comes with some severe handicaps. The obvious one is all JPGs and RAW preview images being grossly underexposed. The other is that depending on your camera settings you may end up in a situation in which the liveview is also horribly underexposed and thus unusable for composition. (I believe the workaround described here: https://www.dpreview.com/forums/post/63806942 can help in certain situations).

My GR IIIx is still in the mail on the way to me so I can't evaluate how workable this is in real life on the actual camera, but the topic just came up on the forum today so I thought I'd share early "results".

Please feel free to ask any questions if I've done a poor job explaining, and of course offer any corrections if I got something wrong!

--
Ken W
See profile for equipment list

The issue is adobe. Other tools work just fine.

maltmoose said:

The issue is adobe. Other tools work just fine.

Click to expand...

Yes, Adobe's implementation is restricted to unmolested RAW files and that limits what it can be applied to making it a less flexible tool than some alternatives. Some other tools can work even on JPEGs and can fall back on those techniques when dealing with heavy RAW NR as Ricoh has chosen for the GR III.

That said, those other techniques do not work as well as when they start with unmolested RAW data. So they could do even better if we could turn off RAW NR, but they don't just completely fall flat on their face as the Adobe implementation does.

Either way, RAW NR has always been poison in every camera it has been put in. Ricoh's decision to add RAW NR to the GR III, and not just RAW NR but pretty heavy handed RAW NR at that, makes the camera less useful overall than it would be with untouched RAW data (and they aren't the only manufacturer to do this, but their implementation is quite severe compared to most). And that's true regardless of what post processing tools you use. How bad the impact is depends on the particular tools you use, but it isn't good for any of them.

Saying the issue is just Adobe is an oversimplification. RAW NR is bad - period. That's a Ricoh problem. Adobe's AI NR implementation restrictions is also not good, and that's an Adobe problem. Thing is, a camera manufacturer already has to make a bad decision (doing RAW NR) in order for the Adobe limitation to manifest itself as a problem to the user. So I'm going to lean towards more of a Ricoh problem than an Adobe problem in this case, but that's just an opinion and not one to put much weight into. Since both are at fault it probably isn't worth wasting time debating who is more at fault.

"The root cause appears to be the heavy handed and impossible to turn off RAW noise reduction applied to GR III files at ISO 200 and above."

Am I misreading what else has been said in this thread but I think the above is wrong?

You can turn off noise reduction as applied to those files. According to the manual anyway.

TonyGamble said:

"The root cause appears to be the heavy handed and impossible to turn off RAW noise reduction applied to GR III files at ISO 200 and above."

Am I misreading what else has been said in this thread but I think the above is wrong?

You can turn off noise reduction as applied to those files. According to the manual anyway.

Click to expand...

Sorry, it is a bit confusing! My fault for not explaining better.

The noise reduction control in the camera menu is for JPEG noise reduction. As you say you can turn it on or off. It only affects JPEG files.

Separately there is noise reduction applied to the RAW data itself starting at ISO 200 and there is no way to turn that off. This RAW NR was new in the GR III. The GR II did not do RAW NR.

I probably made things more confusing by using the phrase “heavy handed” for the RAW NR. It is quite mild compared to the JPEG NR we are all used to seeing. I only meant “heavy handed for RAW NR”. It is actually visually quite mild compared to typical JPEG NR settings. But it is strong enough to break some AI NR algorithms.

And again, to emphasize, having RAW NR that can’t be turned off is something multiple manufacturers have done at various times. Fortunately most have been moving away from doing it over time. Usually it isn’t a big deal expect for obsessive pixel peepers, but recently with the way AI NR algorithms work best it has become more apparent to more users when it breaks AI NR processing.

"Separately there is noise reduction applied to the RAW data itself starting at ISO 200 and there is no way to turn that off."

How do you know?

Where does it say that in the Manual?

TonyGamble said:

"Separately there is noise reduction applied to the RAW data itself starting at ISO 200 and there is no way to turn that off."

How do you know?

Where does it say that in the Manual?

Click to expand...

It is not in the Manual but Ricoh engineers have talked about it. It is the same for the Pentax cameras since ~2016/2017. There has been lots of discussion on the Pentax forum. You can find references to the "accelerator chip". Silly name.

Maybe not "proof" but "evidence" is this... The Photons to Photos site (https://www.photonstophotos.net/Charts/PDR.htm) measures the dynamic range for cameras at different ISOs. There is a correlation by dynamic range and noise. If you look at the chart for the GR III you see that the dynamic range starts to go down as ISO increases but then takes a step up. This is the hardware base noise reduction kicking in.

Some people dislike this approach, other like myself are just happy with the image quality.



Doug

DougOB said:

TonyGamble said:

"Separately there is noise reduction applied to the RAW data itself starting at ISO 200 and there is no way to turn that off."

How do you know?

Where does it say that in the Manual?

Click to expand...

It is not in the Manual but Ricoh engineers have talked about it. It is the same for the Pentax cameras since ~2016/2017. There has been lots of discussion on the Pentax forum. You can find references to the "accelerator chip". Silly name.

Maybe not "proof" but "evidence" is this... The Photons to Photos site (https://www.photonstophotos.net/Charts/PDR.htm) measures the dynamic range for cameras at different ISOs. There is a correlation by dynamic range and noise. If you look at the chart for the GR III you see that the dynamic range starts to go down as ISO increases but then takes a step up. This is the hardware base noise reduction kicking in.

Some people dislike this approach, other like myself are just happy with the image quality.



Doug

Click to expand...

Yep. And it is in fact also mentioned in the GR III review right here on DPR:


Again, RAW noise reduction has historically been something that only obsessive pixel peepers care about. Or, to be fair, folks doing unusual things like astrophotography exposure stacking. Bottom line, not an issue for the vast, vast majority of users.

But with the advent of AI based noise reduction in RAW converters it can have some side effects that impact more people. In this particular case anyone who uses Adobe LR/ACR and would like to do AI NR.

As already mentioned there are other NR tools that while they would work a bit better with RAW files that don't have NR can still handle the Ricoh NR and provide some decent NR. So that's another way to dodge the problem, but of course it means adding in another piece of software.

--
Ken W
See profile for equipment list

kenw said:

maltmoose said:

The issue is adobe. Other tools work just fine.

Click to expand...

Yes, Adobe's implementation is restricted to unmolested RAW files and that limits what it can be applied to making it a less flexible tool than some alternatives. Some other tools can work even on JPEGs and can fall back on those techniques when dealing with heavy RAW NR as Ricoh has chosen for the GR III.

That said, those other techniques do not work as well as when they start with unmolested RAW data. So they could do even better if we could turn off RAW NR, but they don't just completely fall flat on their face as the Adobe implementation does.

Either way, RAW NR has always been poison in every camera it has been put in. Ricoh's decision to add RAW NR to the GR III, and not just RAW NR but pretty heavy handed RAW NR at that, makes the camera less useful overall than it would be with untouched RAW data (and they aren't the only manufacturer to do this, but their implementation is quite severe compared to most). And that's true regardless of what post processing tools you use. How bad the impact is depends on the particular tools you use, but it isn't good for any of them.

Saying the issue is just Adobe is an oversimplification. RAW NR is bad - period. That's a Ricoh problem. Adobe's AI NR implementation restrictions is also not good, and that's an Adobe problem. Thing is, a camera manufacturer already has to make a bad decision (doing RAW NR) in order for the Adobe limitation to manifest itself as a problem to the user. So I'm going to lean towards more of a Ricoh problem than an Adobe problem in this case, but that's just an opinion and not one to put much weight into. Since both are at fault it probably isn't worth wasting time debating who is more at fault.

Click to expand...

It makes sense that working on RAW RGB values (as opposed to processed DNG values) will give better results as powerful software is able to extract (reconstruct) more details from raw sensor data (assuming that this raw data is indeed raw, like in R, G and B photo sites data, not pixels).

I was amazed what can be achieved on Sony a6000 high ISO raws, but I was very disappointed with denoise results from GR IIIx.

As you said, this points to Ricoh NR applied to DNG as the root cause.

Also, fun question: if (internal) NR is applied by camera to DNG, is it applied on data before or after demosaicing/debayer? (I'm guessing after)

barret2023 said:

kenw said:

maltmoose said:

The issue is adobe. Other tools work just fine.

Click to expand...

Yes, Adobe's implementation is restricted to unmolested RAW files and that limits what it can be applied to making it a less flexible tool than some alternatives. Some other tools can work even on JPEGs and can fall back on those techniques when dealing with heavy RAW NR as Ricoh has chosen for the GR III.

That said, those other techniques do not work as well as when they start with unmolested RAW data. So they could do even better if we could turn off RAW NR, but they don't just completely fall flat on their face as the Adobe implementation does.

Either way, RAW NR has always been poison in every camera it has been put in. Ricoh's decision to add RAW NR to the GR III, and not just RAW NR but pretty heavy handed RAW NR at that, makes the camera less useful overall than it would be with untouched RAW data (and they aren't the only manufacturer to do this, but their implementation is quite severe compared to most). And that's true regardless of what post processing tools you use. How bad the impact is depends on the particular tools you use, but it isn't good for any of them.

Saying the issue is just Adobe is an oversimplification. RAW NR is bad - period. That's a Ricoh problem. Adobe's AI NR implementation restrictions is also not good, and that's an Adobe problem. Thing is, a camera manufacturer already has to make a bad decision (doing RAW NR) in order for the Adobe limitation to manifest itself as a problem to the user. So I'm going to lean towards more of a Ricoh problem than an Adobe problem in this case, but that's just an opinion and not one to put much weight into. Since both are at fault it probably isn't worth wasting time debating who is more at fault.

Click to expand...

It makes sense that working on RAW RGB values (as opposed to processed DNG values) will give better results as powerful software is able to extract (reconstruct) more details from raw sensor data (assuming that this raw data is indeed raw, like in R, G and B photo sites data, not pixels).

I was amazed what can be achieved on Sony a6000 high ISO raws, but I was very disappointed with denoise results from GR IIIx.

As you said, this points to Ricoh NR applied to DNG as the root cause.

Also, fun question: if (internal) NR is applied by camera to DNG, is it applied on data before or after demosaicing/debayer? (I'm guessing after)

Click to expand...

The GR III DNG files are not demosaiced, they are still RGB photosite data.

Doug

barret2023 said:

As you said, this points to Ricoh NR applied to DNG as the root cause.

Also, fun question: if (internal) NR is applied by camera to DNG, is it applied on data before or after demosaicing/debayer? (I'm guessing after)

Click to expand...

The DNGs are not demosaiced, they contain the Bayer data nearly strait from the sensor. So this particular stage of NR is applied before demosaicing. And that matches what happens in most other manufacturer's cameras that apply some sort of NR to their RAW files.

How NR is done for JPEGs is less clear, it could in theory happen at a variety of stages in the processing pipeline and there isn't a good way to figure it out.

Note also that it isn't unusual for "RAW" files, whatever their format, to contain data that has been somewhat modified from the exact numbers originally read from the sensor. Some manufacturers do their vignetting corrections to the RAW data, which is typically not too much of a problem. Others do some white balance scaling to the RAW data as well, which again is usually not a big deal. Probably the most common thing these days is that for cameras that have on sensor PDAF the RAW file contains interpolated data for the PDAF pixels.

There's kind of a balance to be struck between making some camera features apply to RAW files regardless of the RAW processor you use and modifying the RAW data so much that it causes other problems. There isn't always a "right" or "best" answer for every situation.

Purists would want the RAW data itself unmodified with metadata instructing the RAW converter what corrections to do, but then that depends on the RAW converter knowing how to read and apply that metadata. That's quite common and widely supported for distortion, vignetting, and lateral CA these days and so almost all compact and mirrorless cameras take that approach for those parameters.

maltmoose said:

The issue is adobe. Other tools work just fine.

Click to expand...

What other software have you tried? (or read about)

Zorgon2 said:

maltmoose said:

The issue is adobe. Other tools work just fine.

Click to expand...

What other software have you tried? (or read about)

Click to expand...

I have and use capture one, photo ai, photolab, skilkypix, darktable, affinity photo.

They all work to some degree or another,i like them all for different things. Best results can be obtained by using combinations of tools. I dont mind some noise though and also shoot some film and also pretty happy using or editing jpegs as well.

maltmoose said:

Zorgon2 said:

maltmoose said:

The issue is adobe. Other tools work just fine.

Click to expand...

What other software have you tried? (or read about)

Click to expand...

I have and use capture one, photo ai, photolab, skilkypix, darktable, affinity photo.

They all work to some degree or another,i like them all for different things. Best results can be obtained by using combinations of tools. I dont mind some noise though and also shoot some film and also pretty happy using or editing jpegs as well.

Click to expand...

So which one works the best, specifically on GRIII RAW files?

Zorgon2 said:

maltmoose said:

Zorgon2 said:

maltmoose said:

The issue is adobe. Other tools work just fine.

Click to expand...

What other software have you tried? (or read about)

Click to expand...

I have and use capture one, photo ai, photolab, skilkypix, darktable, affinity photo.

They all work to some degree or another,i like them all for different things. Best results can be obtained by using combinations of tools. I dont mind some noise though and also shoot some film and also pretty happy using or editing jpegs as well.

Click to expand...

So which one works the best, specifically on GRIII RAW files?

Click to expand...

Well they all do well with processing GRIII files. Its down to the skills of the user.

Noise reduction is really a low consideration as part of the processing i do and some noise rarely makes or breaks a photo and i really dont like the smeared plastic noise free look. so upto iso 1600 any of these work ok for me. im happy with what capture one does overall and has functional and speed advantages over photolab.

If yove got some really nasty color noise and high iso luma noise at say iso 12800 then i think photolab exported to tiff into photo ai will clean it up well and also silkypix does a great job on hot pixels and color noise and demosaicing then turn luma noise reduction off in silkypix and export to photo ai with some manual settings and will look good as well.

Photo ai on its own is not great at color noise reduction or hot pixel removal and the defaults are never optimal imo.

I really dont recommend iso 12800 for anything other than black and white though. i cap at 6400 max as i do with all my apsc and really try to keep it below 1600 so i just dont have to worry about it. I got a fulframe as well for a reason

maltmoose said:

Zorgon2 said:

maltmoose said:

Zorgon2 said:

maltmoose said:

The issue is adobe. Other tools work just fine.

Click to expand...

What other software have you tried? (or read about)

Click to expand...

I have and use capture one, photo ai, photolab, skilkypix, darktable, affinity photo.

They all work to some degree or another,i like them all for different things. Best results can be obtained by using combinations of tools. I dont mind some noise though and also shoot some film and also pretty happy using or editing jpegs as well.

Click to expand...

So which one works the best, specifically on GRIII RAW files?

Click to expand...

Well they all do well with processing GRIII files. Its down to the skills of the user.

Noise reduction is really a low consideration as part of the processing i do and some noise rarely makes or breaks a photo and i really dont like the smeared plastic noise free look. so upto iso 1600 any of these work ok for me. im happy with what capture one does overall and has functional and speed advantages over photolab.

If yove got some really nasty color noise and high iso luma noise at say iso 12800 then i think photolab exported to tiff into photo ai will clean it up well and also silkypix does a great job on hot pixels and color noise and demosaicing then turn luma noise reduction off in silkypix and export to photo ai with some manual settings and will look good as well.

Photo ai on its own is not great at color noise reduction or hot pixel removal and the defaults are never optimal imo.

I really dont recommend iso 12800 for anything other than black and white though. i cap at 6400 max as i do with all my apsc and really try to keep it below 1600 so i just dont have to worry about it. I got a fulframe as well for a reason

Click to expand...

After doing some research I can see that Topaz Photo AI has exactly the same problem as Adobe. It cannot remove the noise from the Ricoh GRIII/x RAW files. Of course you can convert it to TIFF first, then use Photo AI, but the results are significantly less good.

From what others are saying, DxO seems to be the software of choice for denoising Ricoh/Pentax RAW files (that do not work with Adobe).

Just live with the noise, it adds character. Too much theory here.

I suggest Luminar Neo.

Has nobody given it a try?

T

Zorgon2 said:

maltmoose said:

Zorgon2 said:

maltmoose said:

Zorgon2 said:

maltmoose said:

The issue is adobe. Other tools work just fine.

Click to expand...

What other software have you tried? (or read about)

Click to expand...

I have and use capture one, photo ai, photolab, skilkypix, darktable, affinity photo.

They all work to some degree or another,i like them all for different things. Best results can be obtained by using combinations of tools. I dont mind some noise though and also shoot some film and also pretty happy using or editing jpegs as well.

Click to expand...

So which one works the best, specifically on GRIII RAW files?

Click to expand...

Well they all do well with processing GRIII files. Its down to the skills of the user.

Noise reduction is really a low consideration as part of the processing i do and some noise rarely makes or breaks a photo and i really dont like the smeared plastic noise free look. so upto iso 1600 any of these work ok for me. im happy with what capture one does overall and has functional and speed advantages over photolab.

If yove got some really nasty color noise and high iso luma noise at say iso 12800 then i think photolab exported to tiff into photo ai will clean it up well and also silkypix does a great job on hot pixels and color noise and demosaicing then turn luma noise reduction off in silkypix and export to photo ai with some manual settings and will look good as well.

Photo ai on its own is not great at color noise reduction or hot pixel removal and the defaults are never optimal imo.

I really dont recommend iso 12800 for anything other than black and white though. i cap at 6400 max as i do with all my apsc and really try to keep it below 1600 so i just dont have to worry about it. I got a fulframe as well for a reason

Click to expand...

After doing some research I can see that Topaz Photo AI has exactly the same problem as Adobe. It cannot remove the noise from the Ricoh GRIII/x RAW files. Of course you can convert it to TIFF first, then use Photo AI, but the results are significantly less good.

Click to expand...

I never use photo ai on raw because then you dont get the color and tonality and edits from raw convertor. Photo ai does better on tiff from raw convertors regardless of what they claim.

Zorgon2 said:

From what others are saying, DxO seems to be the software of choice for denoising Ricoh/Pentax RAW files (that do not work with Adobe).

Click to expand...

Photolab does well on some but not all high noise gr files. Can introduce nasty artifacts. On bad gr noise you can beat photolab on its own with say silkypix and photo ai. Its just not a popular tool. Its key to not use defaults though.

Great info from the OP, I was just thinking about this the other day.

I really do find the situation annoying , neither Ricoh or Adobe will address it. To me the issue lies predominantly with Ricoh.

For lower light indoor shots , my smart phone works better which is a shame. It's another nail in the coffin for carrying a dedicated small form factor cam unfortunately.