Still puzzled about some basics

Status
Not open for further replies.
Triangles are where the noise reduction is applied to raw data. Canon has it in several cameras.
Applied to RAW data, not part of RAW data.
Applied to the section with raw data before the raw file is written to your SD card.
Another example - 'long exposure noise reduction' option that captures a dark frame and subtracts it from the raw data.
Subtracts it from RAW data, not part of RAW data.
Not part of readings from ADC, but a part of raw data section in a raw file. So there's no original raw readings from ADC in those raw files.
Also Sony raw files used to be known for 'star eater effect' that's supposed to reduce the amount of hot pixels.
Raw files, not RAW data.
The stars were 'eaten' in raw data.
Nikon applies scaling to blue channel (IIRC) but it's not destructive (not lossy unlike noise reduction).
RAW data doesn't have channels.
Oh yes it does. Typically there's four channels: red, two green channels and blue. The data is structured so that the processing software can distinguish the channels (data for pixels of different colours), how else would you render the colours?

The 'rawest' digital data we could've had - digital values coming from ADC (during the sensor readout after exposure is taken). But those values are not guaranteed to be actually written in the raw data section. That data gets scaled quite frequently (non-destructive), sometimes gets noise reduction applied (destructive) and other unknown/unspecified processing applied before it gets written to a raw file.

You might find this text interesting if you understand the technical language:

https://github.com/lclevy/canon_cr3

Just an example of a raw file format. There's quite a lot of different 'tags' with some data under each tag. The 'raw per-pixel' data is under a tag, and metadata comes under other tags. Embedded jpeg is just one of the tags (and there will be metadata inside the jpeg, too).

Which reminds me, by the way, the 'raw data' in question is almost always compressed and sometimes it's lossy compression (o again we get even farther from the original values from ADC).

--
https://www.instagram.com/quarkcharmed/
https://500px.com/quarkcharmed
 
Last edited:
You're now telling us that the OP wrote one thing but meant a different thing.
I'm telling you the OP didn't write what you said he did and what you put in quotes, Little deceit there huh?
This might help:
You responded to me after I responded to the OP's 2nd question. You're confused about how to follow the thread.
In the 2nd follow-up message, the OP was also asking about raw files and you were talking about raw files too. 'Raw data' wasn't even mentioned.
You can't do it can you. You can't let go of that nerd ego. Yes, the OP used the term RAW files but it's clear from reading his question that what he wants to know is if the raw data in the raw file is affected by the camera's picture controls.

Is it at all possible that any of us have heard someone use the term raw file when what they were in fact talking about was the raw data in the file? Possible? Could this be one of those cases? I'm pretty sure it is. Stop stroking your nerd ego and answer the OP's question.

And I'm still waiting for documentation of that misinformation you presented.
https://www.dpreview.com/forums/post/67167581

5676a17aeb2945e7b4450101a8f71514.jpg.png

Bye, I'm not continuing this conversation.
 
OP here. Well, I thought I was asking some simple questions! But I'm reminded by this thread that very little in digital photography is simple. In any case, despite some posters getting hot under the collar, I've learned a lot from all your responses.

At the same time, I'm frustrated. I've been doing digital photography for a long time (15+ years) now. And in general I'm pleased with my results. But I always feel like someone who is driving a car without knowing very much about what's going on under the hood. (As my questions here prove.)

So my final question is, how did you all get there? (Or better, how can I get there?) Is there an online resource that can explain some of the more complex aspects of digital that you all seem to have mastered? Should I be reading something like Jeff Schewe's The Digital Negative, which seems sort of intimidating. Or, when all is said and done, should I just stop worrying about my technical backwardness and enjoy my photography in blissful ignorance -- since I manage to get results I'm happy with?

I know this has wondered far from my original questions, but I'd appreciate your thoughts. And please., no fighting!
 
....... Or, when all is said and done, should I just stop worrying about my technical backwardness and enjoy my photography in blissful ignorance -- since I manage to get results I'm happy with?
If your results please you then there is not much more to say.

If you end up with some problem and need help, then ask about it in the forum here that is relevant to whatever camera model you own.

Otherwise, just enjoy!
 
OP here. Well, I thought I was asking some simple questions! But I'm reminded by this thread that very little in digital photography is simple. In any case, despite some posters getting hot under the collar, I've learned a lot from all your responses.

...
Conceptually it's simple, as long as you don't get bogged down in the details.

.

Subject lighting, aperture, and shutter determine how much light reaches the camera's sensor. We call the light reaching the sensor the "exposure" (not that this is not how light or dark the image looks).

More light means less noise.

Metering is often used to guide how much light to capture. When the camera is in an automatic mode, the camera will adjust shutter and/or aperture based on it's internal metering.

While not actually how it works, it can help to think about the pixels on the sensor as "counting" photons. Roughly speaking, the output of the pixel is related to the number of photons the pixel sees.

The ISO setting serves three basic functions:
  1. It sets a target exposure for the camera's metering system (which may guide aperture/shutter settings).
  2. It gives a context for interpreting the raw data. At high ISO settings a particular photon count may result in a bright pixel in the resulting JPEG, while at a low ISO that same photon count might produce a dark pixel.
  3. It gives the camera the opportunity to optimize itself for the corresponding exposure.
.

What confuses a lot of people is the interactions between the multiple functions of ISO. When you change the ISO setting, you change the target exposure for the metering system. Typically, the camera (or the photographer) alter aperture, shutter, or lighting in order to hit that target exposure. So while ISO does not directly effect exposure, it has an indirect affect which often influences exposure.
 
I'm actually not a beginner, but two things about digital photography have always puzzled me. I assumed I'd work them out eventually. Then decided to see if this forum could help.

1. What camera functions affect RAW files?

2. Is the histogram really useful when shooting RAW?
What are your answers to your above questions?
 
They show the RAW data but the use the settings for JPEG. Usually the adjustments are saved separately from RAW and they are applied to show the image. If you didn't make any adjustments yet they will show a very similar image with the JPEG.
 
OP here. Well, I thought I was asking some simple questions! But I'm reminded by this thread that very little in digital photography is simple. In any case, despite some posters getting hot under the collar, I've learned a lot from all your responses.

At the same time, I'm frustrated. I've been doing digital photography for a long time (15+ years) now. And in general I'm pleased with my results. But I always feel like someone who is driving a car without knowing very much about what's going on under the hood. (As my questions here prove.)

So my final question is, how did you all get there?
[/QUOTE]
I learned to test my hardware; to set up a feedback loop between test results and further testing and keep tightening the loop.

My first job in photo was I think either 1975 or 76. From that point till today photography has been my sole source of income. My most critical milestone along the journey was meeting and buying a used camera from Jim. Jim was the staff photographer for the St. Louis baseball Cardinals -- we became friends and I learned photography from Jim. These were of course film days.

Eventually I went back to school to get a graduate degree in photo -- good experience but Jim had already taught me everything I really needed to know. The first time I sat down with a densitometer was under Jim's guidance. You can test cameras and film buy looking at the photos but that's soft and fudgy. Too often you wake up the next morning, look at the results and wonder what the bleep was wrong with you yesterday. A densitometer replaces the soft and fudgy with hard facts. I leaned to expose and process a roll of film, take that film to a densitometer and read it and graph the results, examine the results then re-test and refine to improve the results. I had a closed loop and I was able to tighten it.

Jim taught me some really critical lessons. One of the most important: You never stop testing. If you don't maintain a practice of regular equipment testing you lose control.

Along came the digital age. It was exciting but I had no way to close a testing loop. I freaked out, digital took away my densitometer! ARGHHH! I was going nuts because sh*t was going on inside the camera and computer with software that hid what it was doing from me. I had a couple frustrating bad years there -- got the rug pulled out from under me because I couldn't make a test exposure and examine hard results.

Thank heaven along came LibRaw, and RawDigger. RawDigger gave me my densitometer back and now all is right again. I can make a test exposure, examine hard results, re-test and refine.
(Or better, how can I get there?)
Start using your copy of RawDigger and establish a regular practice of testing your hardware.
Is there an online resource that can explain some of the more complex aspects of digital that you all seem to have mastered? Should I be reading something like Jeff Schewe's The Digital Negative, which seems sort of intimidating.
Start by reading the articles and tutorial material on the RawDigger website but do it with your copy of RawDigger and test files that you've created to follow along with.
Or, when all is said and done, should I just stop worrying about my technical backwardness and enjoy my photography in blissful ignorance -- since I manage to get results I'm happy with?
Ignorance is not really blissful. Here's another story; I eventually became a college prof teaching photography. I've had a lot of memorable students over the years. One day will just call hm John, shows up in my class. He really wanted to be a photographer and he was out there hustling small event coverage jobs. He took multiple classes so I got to know him after awhile. One day in the lab he was at the processing sink developing film and I saw he was holding his hands with crossed fingers over the film canister and kinda chanting.

Me: "What's with the crossed fingers?"

John: "These are important I really hope they come out."

This was classic John. He resisted doing the hard work of getting control over his equipment and processing and so he regularly put himself into to this state of anxiety at the processing sink -- not blissful.

Seriously: RawDigger -- best money in photo you will ever spend.
I know this has wondered far from my original questions, but I'd appreciate your thoughts. And please., no fighting!
 
Last edited:
One of the big issues is that there isn't a universal standard for what's in raw files. Every manufacturer does it differently.

For the sake of clarity, I am going to use the term "raw data" to mean the data in the raw file that derives from the pixels on the sensor, and "metadata" to mean just about everything else in the raw file.

For the sake of this discussion, let's assume that all images taken have the same exposure (light reaching the sensor while the shutter is open).

Most cameras record in the raw file various metadata, which often includes some combination of shutter speed, aperture, ISO setting, lens mounted, focus distance, and many camera specific settings.

On some cameras, the raw data (representing the readings from the sensor) is not affected by the ISO setting. On these cameras (assuming the same exposure) you get the same raw data, no matter what the ISO is set to.

On most cameras, the ISO setting will affect the raw data. It might be scaled to a more reasonable range, or be processed in a different manner.

For instance, some cameras will apply additional noise reduction with higher ISO settings. This may involve techniques like dark frame subtraction.

So when asking the question what settings affect the raw file, the answer it depends.

Just about every setting will affect the metadata.

Settings that affect exposure will change the raw data.

Setting that change how the camera massages the data from the sensor will affect the raw data. For many cameras the ISO setting will affect how the sensor data is scaled.
 
One of the big issues is that there isn't a universal standard for what's in raw files. Every manufacturer does it differently.
Many of the formats are based on (or rather derived from) TIFF (see DNG, CR3, ARW, NEF)
For the sake of clarity, I am going to use the term "raw data" to mean the data in the raw file that derives from the pixels on the sensor, and "metadata" to mean just about everything else in the raw file.
Metadata is usually something logically related to the data. There's pixel data, there's metadata for that data, and there's 'everything else'. I'm not sure if the embedded JPEG can be called 'metadata', but it also has its own metadata inside which partially relates to JPEG only.

In case of the TIFF-derived formats, everything is under 'tags', and some tags have per-pixel data attached, and some tags have metadata. Tags also take space in the raw file (generally - file format markup is part of the file in this case), so it won't be technically correct to say 'metadata is everything else'.

For simplicity I'd say a raw file typically has

( <embedded jpeg>, <metadata for pixel data>, <raw pixel data>, <some other irrelevant for us stuff> )

Some of the 'metadata' and 'raw pixel data' will be used by raw editors to render the initial image.

For even greater simplicity one could say that raw files have the maximum possible information for postprocessing.
So when asking the question what settings affect the raw file, the answer it depends.

Just about every setting will affect the metadata.
So it affects the raw file.
Settings that affect exposure will change the raw data.
That's right. Also, as above, ISO setting and settings like 'long exposure noise reduction'. Plus general noise reduction in some cases. Also, lossy compression will affect the raw data.
 
There is RAW data (light as recorded by the camera's sensor);
Could you please explain what do you mean by the above?
As an independent observer, it seemed obvious that he meant the raw data was a representation of the light that was seen by the camera's sensor.

Obviously, he did not mean that the raw data was itself "light". Yes, perhaps he could have worded it in a better fashion, but the meaning seems to be clear. I guess if one wanted to argue over grammar (rather than photography) one could argue over the choice of words.

But I think it's helpful to make it clear when one is disagreeing over the wording rather than disagreeing with the underlying concepts.

As always, when there is a disagreement, it's helpful to state what you think is the correct answer.
 
There is RAW data (light as recorded by the camera's sensor);
Could you please explain what do you mean by the above?
he meant the raw data was a representation of the light that was seen by the camera's sensor.
So is JPEG (one can take an OOC JPEG and "calculate back" the light values in the scene), and in a sense, anything else that faithfully (subjective) gives a viewer the impression of the scene.

If highlights are clipped in raw, how to describe the representation, as limited? What if shadows are plugged? In the absence of light, given exposure is long enough and ISO setting is high enough all that gets out is noise. What does raw data represent in this case? I'm not even going to mention many other factors, distorting the "representation".

One could say that raw represents the scene better than JPEG, but even that isn't an absolute given.

There is no simple or single answer to the question what is raw data. As of today, raw data is what the camera manufacturer says it is.

And even than...

Adobe are suggesting the following: "RAW files contain uncompressed and unprocessed image data", but what is image data? Is normalization (artificial black level, ISO, white balance pre-conditioning, noise reduction, etc) allowed here and isn't considered processing? Is lossless compression considered no compression? Are files recorded in a lossy compressed mode are raw in name only?

Consider what Nikon are saying: "RAW image files ... contain all the image information captured by the camera's sensor, along with the image's metadata (the camera's identification and its settings, the lens used and other information). The NEF file is written to the memory card in either an uncompressed or 'lossless' compressed form." OK, how are we supposed to understand 'lossless' here? As true lossless, or as visually lossless? What is "image information", and what is not?

And before I bow out, some types of raw data (most notably, with Canon) include non-image data (buffers, optical black, etc.).

Ciao.
 
The OP probably left long ago to go take photos, but meanwhile the arguments continue, as always. :-)

If some absolute beginner asks me about the files, I say...
  • The raw file contains what the sensor sees.
  • The jpeg file is result of the camera user's choice of quality and style settings working on that raw file.
If they ask further questions then follow up with better info but usually the eyes glaze over in the first few seconds so it ends there.
 
There is RAW data (light as recorded by the camera's sensor);
Could you please explain what do you mean by the above?
he meant the raw data was a representation of the light that was seen by the camera's sensor.
There is no simple or single answer to the question what is raw data. As of today, raw data is what the camera manufacturer says it is.

Adobe are suggesting the following: "RAW files

Consider what Nikon are saying: "RAW image files ...
My understanding is raw files are not RAW data, and RAW data is part of raw files.
 
There is RAW data (light as recorded by the camera's sensor);
Could you please explain what do you mean by the above?
he meant the raw data was a representation of the light that was seen by the camera's sensor.
There is no simple or single answer to the question what is raw data. As of today, raw data is what the camera manufacturer says it is.

Adobe are suggesting the following: "RAW files

Consider what Nikon are saying: "RAW image files ...
My understanding is raw files are not RAW data, and RAW data is part of raw files.
And so it goes on, and on, and on.....

Time to leave was days ago.
 
There is RAW data (light as recorded by the camera's sensor);
Could you please explain what do you mean by the above?
he meant the raw data was a representation of the light that was seen by the camera's sensor.
So is JPEG (one can take an OOC JPEG and "calculate back" the light values in the scene), and in a sense, anything else that faithfully (subjective) gives a viewer the impression of the scene.
Yes, that's a reasonable statement. As an overview, values in both raw files and JPEG files flow from the light hitting the sensor.

But I think we are losing sight of the context of the statement. I believe he was trying to differentiate between the data that flows from the pixels and the metadata. In that context, it seems to me, that his usage was correct.
If highlights are clipped in raw, how to describe the representation, as limited? What if shadows are plugged? In the absence of light, given exposure is long enough and ISO setting is high enough all that gets out is noise. What does raw data represent in this case? I'm not even going to mention many other factors, distorting the "representation".
Again, you are losing site of the context. You seem to be suggesting that one can't generally refer to the data derived from the sensor unless one completely describes the technicalities and limitations of that data.

Such a level of detail may very well be critical if you were writing software that processed raw files. However, it seems needlessly complicated if one is simply trying to describe in a beginners forum that some data flows from the sensor pixels, and some does not (i.e. the metadata tag that contains the photographer's name).

One could say that raw represents the scene better than JPEG, but even that isn't an absolute given.
Again, I am not sure how this is relevant to a discussion as to which camera settings affect the raw file.

How, or even if, those setting affect the resulting JPEG isn't really on topic.
There is no simple or single answer to the question what is raw data. As of today, raw data is what the camera manufacturer says it is.
Again, I think you are trying to get into the weeds on a topic where a general overview is called for.
And even than...

Adobe are suggesting the following: "RAW files contain uncompressed and unprocessed image data", but what is image data? Is normalization (artificial black level, ISO, white balance pre-conditioning, noise reduction, etc) allowed here and isn't considered processing? Is lossless compression considered no compression? Are files recorded in a lossy compressed mode are raw in name only?
I think in the context of the OP's question, it is sufficient to say something like "the image data reflects what the sensor's pixels saw. The exact format, content, and meaning of the data varies with camera models."

If you want to go into a little more detail, you can certainly mention that the various ways the data might have been manipulated (lossy compressed, lossless compressed, noise reduction, scaling, etc.).

Consider what Nikon are saying: "RAW image files ... contain all the image information captured by the camera's sensor, along with the image's metadata (the camera's identification and its settings, the lens used and other information). The NEF file is written to the memory card in either an uncompressed or 'lossless' compressed form." OK, how are we supposed to understand 'lossless' here? As true lossless, or as visually lossless? What is "image information", and what is not?
And before I bow out, some types of raw data (most notably, with Canon) include non-image data (buffers, optical black, etc.).

Ciao.
Imagine that you were trying to teach someone how to drive a typical new model car. You can tell them that when the car is in gear, pushing on the gas pedal makes the car go faster. Many people would think that this is a sufficient generalization for someone who wants to learn to drive.

On the other hand, you could point out that this isn't really the case, Most cars have some sort of transmission. The gas pedal generally feeds more gas to the engine, this may make the engine turn faster. Depending on how the gears change, a faster turning engine may result in a slower moving car. You could even point out that some cars have automatic transmissions, some have manual transmissions, and some electric cars have neither. In some hybrid cars, the gas pedal is connected to electrical motors, and the gas engine is controlled by a different system. While all of these are true (and I have left a lot of details out), the beginner may be better off with the rule of thumb that pressing the gas pedal makes the car go faster.

In the context of the OP's question, I don't think there is a benefit to a long discussion on how simply referring to data from the sensor can be misconstrued.

But's that just my opinion. I hope that if the OP found your detailed corrections helpful, he will chime in and correct me, and thank you for the clarifications.
 
There is RAW data (light as recorded by the camera's sensor);
Could you please explain what do you mean by the above?
he meant the raw data was a representation of the light that was seen by the camera's sensor.
There is no simple or single answer to the question what is raw data. As of today, raw data is what the camera manufacturer says it is.

Adobe are suggesting the following: "RAW files

Consider what Nikon are saying: "RAW image files ...
My understanding is raw files are not RAW data, and RAW data is part of raw files.
And so it goes on, and on, and on.....

Time to leave was days ago.
And yet it is so difficult to tear oneself away.

They are starting to argue over the grammar now. :-|
 
You are quite right…This has gone too far. Thread locked.
 
Status
Not open for further replies.

Keyboard shortcuts

Back
Top