How do I processing RAW images in the same way my camera processes JPEGs?

[stephband]

I have a Panasonic Lumix S5. It captures a RAW file and saves both that and a JPEG version of the same image. I have filters switched off. When I import the two files into an image editor they look completely different. Sometimes, I want to recreate the way my camera has processed the JPEG while editing the RAW image but I'm struggling to understand what it is doing and why.

An example. Here's an image I shot last night. It's not terribly good, but it illustrates the problem...




the JPEG produced by the camera




A screenshot of the RAW image



We can see that that lava river in the JPEG produced by the camera is yellow, as it was on the night, whereas the RAW image has regions that are bleached white. Presumably it still carries the data to unbleach those white patches.

At first I thought this was simply a question of white balance, which I assume is not applied to the RAW image. But adjusting the white balance in an image editor does not solve the problem.

So what is the camera doing and how can I recreate what it is doing?

 

[Kameratrollet]

I have a Panasonic Lumix S5. It captures a RAW file and saves both that and a JPEG version of the same image. I have filters switched off. When I import the two files into an image editor they look completely different. Sometimes, I want to recreate the way my camera has processed the JPEG while editing the RAW image but I'm struggling to understand what it is doing and why.

An example. Here's an image I shot last night. It's not terribly good, but it illustrates the problem...


the JPEG produced by the camera


A screenshot of the RAW image

We can see that that lava river in the JPEG produced by the camera is yellow, as it was on the night, whereas the RAW image has regions that are bleached white. Presumably it still carries the data to unbleach those white patches.
At first I thought this was simply a question of white balance, which I assume is not applied to the RAW image. But adjusting the white balance in an image editor does not solve the problem.

So what is the camera doing and how can I recreate what it is doing?

Are the lava river highlights in the raw file clipped or not clipped?

 

[BobKnDP]

I expect that a raw file would have a much larger dynamic range than a JPEG, so it shouldn't be saturated.

Perhaps the issue is with your monitor or the software you're using to display the image.

 

[Erik Baumgartner]

What software are you using? You will want to bring down the exposure slider with the RAW to get the highlight detail back in play, and then bring back up whatever else needs to be brighter.

Sharing a link to your RAW file here could bring about more specific suggestions.

 

[stephband]

I'm on an MacBook Pro M1 Max, it has a high dynamic range screen. Also, the monitor would not affect the quality of the screenshot. I don't think that's the problem.

 

[Erik Baumgartner]

I'm on an MacBook Pro M1 Max, it has a high dynamic range screen. Also, the monitor would not affect the quality of the screenshot. I don't think that's the problem.

There is dynamic range contained in the RAW file that you aren't taking advantage of. If the jpeg processing could retain the highlight color, careful RAW processing can do at least as well or likely better.

 

[stephband]

The application RAW Power tells me the red channel clips in the river, but the other channels do not.

However, I just discovered the control 'Gamut Map'. When I set that to OFF the river becomes yellow.

In Pixelmator, setting EDR (Extended Dynamic Range) to ON appears to do the same thing.

I am on a recent Macbook Pro with a high dynamic range screen. This is clearly something to do with mapping gamut to the dynamic range of my screen.

 

[stephband]

Aha. You are on to something.

In RAW Power, I just discovered the control 'Gamut Map'. When I set that to OFF the river becomes yellow.

In Pixelmator, setting the control 'EDR' (Extended Dynamic Range) to ON appears to do the same thing.

 

[Doug Haag]

Nikon and Canon have free software that is designed to convert raw files from their cameras. Perhaps Panasonic does not.

If it does and it it works the same way as my Nikon software (and I assume like Canon's too) in default mode it converts the raw to be identical (and I do mean absolutely IDENTICAL)) to the jpeg from the camera.

The camera takes the captured data (which is raw data) and converts it to the jpeg you see -- using all the camera settings regarding picture styles, etc. in the process. The manufacturer's raw conversion software does exactly the same thing. That's why the converted result (when saved as an 8-bit jpeg and not a 16-bit tiff) is identical.

If Panasonic provides such conversion software, I suggest you try that and see how the results compare. It may operate in the same way as Nikon and Canon's software does.

 

[stephband]

The weird thing about Pixelmator is that as soon as you have more than one layer it switches EDR mode off. It appears editing together bracketed photos can only be done with clipped colours.

 

[BobKnDP]

I think that Panasonic does not provide a raw file processing utility.

The freeware made available by Canon for their cameras is supposed to offer a duplicate of the in-camera processing, but I've never verified that. I mainly use DCO Photolab 5.

 

[NAwlins Contrarian]

How do I processing RAW images in the same way my camera processes JPEGs?

I have a Panasonic Lumix S5. It captures a RAW file and saves both that and a JPEG version of the same image. I have filters switched off. When I import the two files into an image editor they look completely different. Sometimes, I want to recreate the way my camera has processed the JPEG while editing the RAW image but I'm struggling to understand what it is doing and why.

Realistically, your best chance to achieve this is with whatever raw conversion software Panasonic includes with / for your S5. That may well have controls that, if set correctly, duplicate the in-camera processing used to create the JPEG, and then you can tweak from there--and/or export as a 16-bit TIFF, preferably in a wider color space, for further editing in your image processing software of choice.

Turning the output of the analog-to-digital converters for each of the camera's single-color pixels (being behind a red, green, or blue filter) into full color involves a large degree of interpretation. Then often other processing occurs on top of that: geometric distortion correction, noise reduction, sharpening, etc. There is no single, fixed, 'right' way or process, and it's as much art as science. Even the very basic issue of how the camera and/or the software determines, or is set to, the white balance parameters hugely affects the result.

Personally I find that usually I don't want to closely emulate the SOOC JPEG, but sometimes I do. And when I do, using my usual raw converter, DxO PhotoLab, sometimes I can, and sometimes I can't. And on rare occasions I go back to the Sony or Canon software.

 

[stephband]

This is interesting. I was under the impression that many of these processes would be deterministic, but you are saying that there is a degree of interpretation involved.

For example, I had thought that when I set lens correction to ON in the camera, that I am relying on the camera to distort (or undistort) the image. And that is true for the in-camera JPEG. But I have just found a switch in RAW Power, 'Apple Lens Correction', and it comes on by default when viewing a RAW file ... and I guess that is using a different process to do lens correction.

The original issue appears to be about mapping the gamut of the RAW image to the wide gamut of my MacBook's screen. I am effectively using Apple's RAW processes, as that is what both the application RAW Power and Pixelmator Pro rely upon.

 

[BobKnDP]

Camera RAW is unprocessed. The computer software has to do all corrections, including converting the image (commonly a Bayer mosaic) to RGB for all pixels, and applying lens corrections. (Lens corrections typically include distortion and vignetting. Lateral color correction, which gives one type of color fringing, may also be done.)

 

[stephband]

I sort of understood that to be the case, but I hadn't fully realised that ALL of these processes are being done by the computer.

When editing RAW photos with lens correction, it's ON by default in Pixelmator. There is no control to switch it off. But that's only the case if you had it ON in the camera. That's what led me to believe the camera was doing that. Instead, I guess it must be simply flagged as ON in the RAW data so that Pixelmator/Mac can redo it.




Just to be clear, you are saying this applies to ALL processes? I can completely ignore White Balance in the camera because I am processing it myself in RAW editing? This would be useful to me because on the Lumix I can assign the WB button to a more useful function.

 

[s_grins]

I have a Panasonic Lumix S5. It captures a RAW file and saves both that and a JPEG version of the same image. I have filters switched off. When I import the two files into an image editor they look completely different. Sometimes, I want to recreate the way my camera has processed the JPEG while editing the RAW image but I'm struggling to understand what it is doing and why.

An example. Here's an image I shot last night. It's not terribly good, but it illustrates the problem...


the JPEG produced by the camera


A screenshot of the RAW image

We can see that that lava river in the JPEG produced by the camera is yellow, as it was on the night, whereas the RAW image has regions that are bleached white. Presumably it still carries the data to unbleach those white patches.
At first I thought this was simply a question of white balance, which I assume is not applied to the RAW image. But adjusting the white balance in an image editor does not solve the problem.

So what is the camera doing and how can I recreate what it is doing?

To start from the beginning:

RAW is not a image file. RAW is a data file. Your camera (as well as all others) process this data in certain way to represent an image (mostly JEPG).

The reason you you do not see exactly the same picture is simple to understand. Software in your camera that process the RAW file lacks the sophistication of computer based software. Also settings of your camera (filters included) affect the JEPG image while RAW image stays intact.

My guess is that when you use SilkyPix RAW editor with initial settings, you will get very close image (in my mind piece of software that works inside your camera is made by the same company)

--
If you want to be equal, you have to be better...

 

[Erik Baumgartner]

I sort of understood that to be the case, but I hadn't fully realised that ALL of these processes are being done by the computer.

When editing RAW photos with lens correction, it's ON by default in Pixelmator. There is no control to switch it off. But that's only the case if you had it ON in the camera. That's what led me to believe the camera was doing that. Instead, I guess it must be simply flagged as ON in the RAW data so that Pixelmator/Mac can redo it.

Just to be clear, you are saying this applies to ALL processes? I can completely ignore White Balance in the camera because I am processing it myself in RAW editing? This would be useful to me because on the Lumix I can assign the WB button to a more useful function.

Yes, a RAW file is just that, a file, not an image at all. It contains all the RAW unprocessed sensor data and must be completely processed by the computer from scratch. There are various metadata tags about shooting settings (like WB etc.). for the RAW processor to use in it's initial default processing, but all that can be changed by you. Also, the histogram you're seeing in your application is only representative of what you're seeing on the screen with your current setting, NOT what is contained in the RAW file. There is often significant highlight detail still available to you despite your histogram/screen suggesting otherwise.

A link to your RAW would go a long way in helping us help you.

 

[Digital Nigel]

I sort of understood that to be the case, but I hadn't fully realised that ALL of these processes are being done by the computer.

When editing RAW photos with lens correction, it's ON by default in Pixelmator. There is no control to switch it off. But that's only the case if you had it ON in the camera. That's what led me to believe the camera was doing that. Instead, I guess it must be simply flagged as ON in the RAW data so that Pixelmator/Mac can redo it.

Yes. With some cameras, the lens correction data is stored in the raw file, and applied by the raw processor. Some processors let you choose not to apply it, some don't. And they may sometimes apply different, possibly superior, lens corrections. For example, with PhotoLab, it's often possible to get a wider image than the in-camera JPEG.

Just to be clear, you are saying this applies to ALL processes? I can completely ignore White Balance in the camera because I am processing it myself in RAW editing? This would be useful to me because on the Lumix I can assign the WB button to a more useful function.

Yes, the camera WB setting is recorded in the raw file exif data, but doesn't affect the image data, and can be used or ignored during processing. I leave it at AWB in all my cameras, and override as needed during processing in PhotoLab, which gives an option to process WB 'as shot'. I often use the picked to set the WB based on a grey area in the shot.

 

[NAwlins Contrarian]

Camera RAW is unprocessed. The computer software has to do all corrections, including converting the image (commonly a Bayer mosaic) to RGB for all pixels, and applying lens corrections. (Lens corrections typically include distortion and vignetting. Lateral color correction, which gives one type of color fringing, may also be done.)

Jim Kasson (a sometimes-moderator and regular participant at various times in the medium format, Nikon, and Sony forums) has shown pretty well that very few camera 'raw' files are truly raw, and the real question is how much and what type of processing has occurred. The most infamous current cooking of 'raw' files is noise reduction applied to Canon R-series 'raw' files, although in the past other companies have done similar things. Other processing includes, but is not limited to, substituting and interpolating to fill PDAF pixels. So although pretty much any 'raw' is much less cooked than a JPEG, it's rarely actually raw, and the question is what and how much has been done.

 

[stephband]

Thanks, that is useful. Although about the histogram, in RAW Power it does change when you switch Gamut Map OFF, and in Pixelmator it does not when you do the equivalent and switch EDR mode ON. I am beginning to suspect Pixelmator may be at fault here.

A link to your RAW would go a long way in helping us help you.

Yes, sorry, it is here...

stephen.band/test/PANA4186.RW2

My question now is, how do you guys compose bracketed photos together, for example to create a HDR edit of several exposures? The software I have will process one RAW file at a time. In Pixelmator Pro, you can switch wide gamut on (EDR - ON) for a single RAW layer, but as soon as you add another layer it is automatically switched off. Do the RAW files need to be processed and 'flattened' before you edit them together?