Some raw-related things we often read that are wrong

[mcabato]

Fujifilm have created what seems to be their own ISO standard

This one, above.

 

[just Tony]

Below is a short list of things that spring to mind. Please add your favorites.

• The readability of raw files is not guaranteed in future

Do you feel confident that my descendants will be able to use my raw files in 4017AD? The horrific rate of extinctions of both encoding formats and media forms in just the last 50 years alone makes me at least a little bit wary.

Perhaps you would agree to the following: if you are mistaken about this, your descendants will pay mine a sum of either a) an amount of currency equivalent to a median annual income of that year, or b) 3 goats, whichever is greater. This contract will be recorded in four languages: English and French (because of their current global ubiquity), and ancient Greek and ancient Latin (because of their proven track records of already over 2000 years). The text shall be directly human readable and etched onto two media types: slabs of granite, and plates of Hastelloy-C metal.

Have your lawyer call mine at your earliest convenience. ;-)

 

[Leonard Shepherd]

Below is a short list of things that spring to mind. Please add your favorites.

• The readability of raw files is not guaranteed in future

Do you feel confident that my descendants will be able to use my raw files in 4017AD? The horrific rate of extinctions of both encoding formats and media forms in just the last 50 years alone makes me at least a little bit wary.

In some ways this is similar to archival print work I do. 99% of it is based on a surviving print as there is no surviving negative.

The chances of a son/daughter or grandson/daughter accessing a computer hard drive that (due to the parents old age) has not been used for a decade can be close to nil.

 

[Iliah Borg]

Below is a short list of things that spring to mind. Please add your favorites.

• The readability of raw files is not guaranteed in future

Do you feel confident that my descendants will be able to use my raw files in 4017AD?

Yes.

ancient Greek and ancient Latin (because of their proven track records of already over 2000 years).

I prefer Aramaic, with track record of over 3000 years. It is more raw, too.

 

[just Tony]

My question is theoretical in nature. The stronger CFA crowd talks about better separation of the R, G, B wavelengths (less wide).

Let's say one made a theoretical CFA that was perfect in the sense that each color captured only a narrow band of wavelength, zero capture of light outside this narrow band for each color and there was absolutely no overlap. Let's even say that the discrimination is so narrow that there are gaps between the R, G, B filters.

Now wouldn't this lead to severe color problems with various light sources and a variety of subject matter?

Imagine that we have subject details that reflect or emit within only very narrow wavelength bands, so small that they could fit entirely within one of the receiving color channel bands, or entirely within a gap between channels.

• Any stimulus wavelength, no matter where it falls within any one particular color band, can be recorded in one and only one way. There would be no information available to distinguish between 590 nm and 610 nm for example.
• Any wavelength that unluckily falls in a gap will not be recorded at all.

Overlap conditions on the other hand provide gradual changes of the response ratio between at least two receiving channels, no matter what wavelength is being received. You could then reproduce a "610 nm subject" and a "590 nm subject" with different displayed hues.

 

[just Tony]

Below is a short list of things that spring to mind. Please add your favorites.

• The readability of raw files is not guaranteed in future

Do you feel confident that my descendants will be able to use my raw files in 4017AD?

Yes.

And I am confident that I will win a lottery next week.

ancient Greek and ancient Latin (because of their proven track records of already over 2000 years).

I prefer Aramaic, with track record of over 3000 years. It is more raw, too.

I accept that addition. And I will also archive some recipes for goat, as I am not very confident that there will be any monetary systems in use in 4017.

 

[Iliah Borg]

ancient Greek and ancient Latin (because of their proven track records of already over 2000 years).

I prefer Aramaic, with track record of over 3000 years. It is more raw, too.

I accept that addition

*puts a flower on casket of irony*

 

[dwight3]

...Perhaps you would agree to the following: if you are mistaken about this, your descendants will pay mine a sum of either a) an amount of currency equivalent to a median annual income of that year, or b) 3 goats, whichever is greater. This contract will be recorded in four languages: English and French (because of their current global ubiquity), and ancient Greek and ancient Latin (because of their proven track records of already over 2000 years). The text shall be directly human readable and etched onto two media types: slabs of granite, and plates of Hastelloy-C metal.
Have your lawyer call mine at your earliest convenience...

I can't be sure, not being trained in the law, but I think you would have trouble enforcing a contract that binds a third party who is not a signatory to the contract (because his grandparents haven't been born yet).

 

[SolB]

Below is a short list of things that spring to mind. Please add your favorites.

• The readability of raw files is not guaranteed in future

Do you feel confident that my descendants will be able to use my raw files in 4017AD?

Yes.

ancient Greek and ancient Latin (because of their proven track records of already over 2000 years).

I prefer Aramaic, with track record of over 3000 years. It is more raw, too.

 

[fishy wishy]

Below is a short list of things that spring to mind. Please add your favorites.

• The readability of raw files is not guaranteed in future

Do you feel confident that my descendants will be able to use my raw files in 4017AD? The horrific rate of extinctions of both encoding formats and media forms in just the last 50 years alone makes me at least a little bit wary.

In some ways this is similar to archival print work I do. 99% of it is based on a surviving print as there is no surviving negative.

The chances of a son/daughter or grandson/daughter accessing a computer hard drive that (due to the parents old age) has not been used for a decade can be close to nil.

At last a realist. Although theoretically people could keep old technology around for a lifetime, most people don't have the psychology of hoarders or collectors and value their space. As well as keeping the right bits around, people are going to have to be savvy about where to find software converters in some instances. All I know is that when someone hands me a linux hard drive with files to recover, even though I know I can boot with a liveCD, I don't relish the hassle. Dealing with old photo formats going to only get more obscure in the years to come. There's a difference between what people can do theoretically and what they do in practice. That's why people walk into camera shops and ask for old video tapes to be converted. They could get a USB box for their PC, but in practice a lot don't.

 

[Simon Garrett]

Below is a short list of things that spring to mind. Please add your favorites.

• The readability of raw files is not guaranteed in future

Do you feel confident that my descendants will be able to use my raw files in 4017AD?

Yes.

I agree with Iliah, but as I said above, to treat "readability of raw files in future" as a binary yes/no question is IMHO misleading.

I work with small, volunteer museum/heritage organisations with material that is in theory easy to access, but for them it might as well be in some yet-to-be-deciphered Babylonian script. These are people that find resizing jpegs a challenge, recognise tif but need help, and raw: what's that?

Such organisations are the mainstream of preserving heritage in the UK, and I suspect many countries. Raw files will be usable to them in 4017, but in practice raw stuff will get put on one side until they have the time and money (i.e. possibly never), and they'll concentrate on stuff they can access easily. It happens already: hard-to-use material is given lower priority. "We don't know what's in it, but we do have other stuff that we know is valuable, so let's park the hard stuff."

One can think of other examples: when photographers die, if their children are not photographers, how hard will they try to go through stuff they don't know how to access? But jpegs: they're much more likely to go through those.

--

Simon

 

[Iliah Borg]

Below is a short list of things that spring to mind. Please add your favorites.

• The readability of raw files is not guaranteed in future

Do you feel confident that my descendants will be able to use my raw files in 4017AD?

Yes.

ancient Greek and ancient Latin (because of their proven track records of already over 2000 years).

I prefer Aramaic, with track record of over 3000 years. It is more raw, too.

Me too. In fact will be using it soon

Ha Lachma Anya

And it is in Aramaic because it was thought that when we recite HLA at the open door those that were unable to get proper education because of poverty will understand the common language and enter

 

[Tony Beach]

Below is a short list of things that spring to mind. Please add your favorites.

• The readability of raw files is not guaranteed in future

Do you feel confident that my descendants will be able to use my raw files in 4017AD?

Yes.

I agree with Iliah, but as I said above, to treat "readability of raw files in future" as a binary yes/no question is IMHO misleading.

I work with small, volunteer museum/heritage organisations with material that is in theory easy to access, but for them it might as well be in some yet-to-be-deciphered Babylonian script. These are people that find resizing jpegs a challenge, recognise tif but need help, and raw: what's that?

Such organisations are the mainstream of preserving heritage in the UK, and I suspect many countries. Raw files will be usable to them in 4017, but in practice raw stuff will get put on one side until they have the time and money (i.e. possibly never), and they'll concentrate on stuff they can access easily. It happens already: hard-to-use material is given lower priority. "We don't know what's in it, but we do have other stuff that we know is valuable, so let's park the hard stuff."

One can think of other examples: when photographers die, if their children are not photographers, how hard will they try to go through stuff they don't know how to access? But jpegs: they're much more likely to go through those.

Realistically no one is going to care about every photo or picture anyone ever took any more than they will care about every scrap of paper anyone ever scribbled on. Instead we can expect that historians in the future will look for the work of specific individuals. If someone is believed to be significant enough to care what they took a photo of in the past then there will be those that have the means and are willing to go to the effort to view that photo just as today there are those that learn Latin just so they can read the original works of ancient scholars.

 

[Simon Garrett]

Below is a short list of things that spring to mind. Please add your favorites.

• The readability of raw files is not guaranteed in future

Do you feel confident that my descendants will be able to use my raw files in 4017AD?

Yes.

I agree with Iliah, but as I said above, to treat "readability of raw files in future" as a binary yes/no question is IMHO misleading.

I work with small, volunteer museum/heritage organisations with material that is in theory easy to access, but for them it might as well be in some yet-to-be-deciphered Babylonian script. These are people that find resizing jpegs a challenge, recognise tif but need help, and raw: what's that?

Such organisations are the mainstream of preserving heritage in the UK, and I suspect many countries. Raw files will be usable to them in 4017, but in practice raw stuff will get put on one side until they have the time and money (i.e. possibly never), and they'll concentrate on stuff they can access easily. It happens already: hard-to-use material is given lower priority. "We don't know what's in it, but we do have other stuff that we know is valuable, so let's park the hard stuff."

One can think of other examples: when photographers die, if their children are not photographers, how hard will they try to go through stuff they don't know how to access? But jpegs: they're much more likely to go through those.

Realistically no one is going to care about every photo or picture anyone ever took any more than they will care about every scrap of paper anyone ever scribbled on. Instead we can expect that historians in the future will look for the work of specific individuals. If someone is believed to be significant enough to care what they took a photo of in the past then there will be those that have the means and are willing to go to the effort to view that photo just as today there are those that learn Latin just so they can read the original works of ancient scholars.

 

[Tony Beach]

Below is a short list of things that spring to mind. Please add your favorites.

• The readability of raw files is not guaranteed in future

Do you feel confident that my descendants will be able to use my raw files in 4017AD?

Yes.

I agree with Iliah, but as I said above, to treat "readability of raw files in future" as a binary yes/no question is IMHO misleading.

I work with small, volunteer museum/heritage organisations with material that is in theory easy to access, but for them it might as well be in some yet-to-be-deciphered Babylonian script. These are people that find resizing jpegs a challenge, recognise tif but need help, and raw: what's that?

Such organisations are the mainstream of preserving heritage in the UK, and I suspect many countries. Raw files will be usable to them in 4017, but in practice raw stuff will get put on one side until they have the time and money (i.e. possibly never), and they'll concentrate on stuff they can access easily. It happens already: hard-to-use material is given lower priority. "We don't know what's in it, but we do have other stuff that we know is valuable, so let's park the hard stuff."

One can think of other examples: when photographers die, if their children are not photographers, how hard will they try to go through stuff they don't know how to access? But jpegs: they're much more likely to go through those.

Realistically no one is going to care about every photo or picture anyone ever took any more than they will care about every scrap of paper anyone ever scribbled on. Instead we can expect that historians in the future will look for the work of specific individuals. If someone is believed to be significant enough to care what they took a photo of in the past then there will be those that have the means and are willing to go to the effort to view that photo just as today there are those that learn Latin just so they can read the original works of ancient scholars.

To say that the important stuff will be preserved (because "there will be those that have the means and are willing to go to the effort to view that photo") is at best only part of the story.

There will always be researchers that will know what they are looking for, and have the time and expertise to access material (or the money to pay someone else to do it). They will be a minority, and concentrate as you say on specific individuals known to be significant.

If we're only concerned about preserving what we know to be the most important stuff then no problem.

But much of our history is not preserved this way, and most of our "historians" are not like that at all. They don't learn Latin and they know nothing about raw conversion. Hence my comments about small poorly-resourced groups of volunteers that are collectively responsible for preserving much of the heritage in the UK. In many cases they simply don't have the expertise to access difficult material, and are unlikely to prioritise the money for someone else to do it unless they know for certain that there's something really important there.

I speak from experience: the harder it is for the people doing the work to access the material, the more likely it is to be left.

What makes you think JPEG will be any less archaic in the future?

 

[Iliah Borg]

...Perhaps you would agree to the following: if you are mistaken about this, your descendants will pay mine a sum of either a) an amount of currency equivalent to a median annual income of that year, or b) 3 goats, whichever is greater. This contract will be recorded in four languages: English and French (because of their current global ubiquity), and ancient Greek and ancient Latin (because of their proven track records of already over 2000 years). The text shall be directly human readable and etched onto two media types: slabs of granite, and plates of Hastelloy-C metal.

Have your lawyer call mine at your earliest convenience...

I can't be sure, not being trained in the law, but I think you would have trouble enforcing a contract that binds a third party who is not a signatory to the contract (because his grandparents haven't been born yet).

File storage and AI are advancing fast. In about 30 years AI will be capable to open and convert any raw file that exists now. That's where we heading.

As to enforcing, it is not a matter of that, it is a matter of honour. That's why Tony's phrase "Have your lawyer call mine at your earliest convenience" is so grotesque.

 

[Simon Garrett]

Below is a short list of things that spring to mind. Please add your favorites.

• The readability of raw files is not guaranteed in future

Do you feel confident that my descendants will be able to use my raw files in 4017AD?

Yes.

I agree with Iliah, but as I said above, to treat "readability of raw files in future" as a binary yes/no question is IMHO misleading.

I work with small, volunteer museum/heritage organisations with material that is in theory easy to access, but for them it might as well be in some yet-to-be-deciphered Babylonian script. These are people that find resizing jpegs a challenge, recognise tif but need help, and raw: what's that?

Such organisations are the mainstream of preserving heritage in the UK, and I suspect many countries. Raw files will be usable to them in 4017, but in practice raw stuff will get put on one side until they have the time and money (i.e. possibly never), and they'll concentrate on stuff they can access easily. It happens already: hard-to-use material is given lower priority. "We don't know what's in it, but we do have other stuff that we know is valuable, so let's park the hard stuff."

One can think of other examples: when photographers die, if their children are not photographers, how hard will they try to go through stuff they don't know how to access? But jpegs: they're much more likely to go through those.

Realistically no one is going to care about every photo or picture anyone ever took any more than they will care about every scrap of paper anyone ever scribbled on. Instead we can expect that historians in the future will look for the work of specific individuals. If someone is believed to be significant enough to care what they took a photo of in the past then there will be those that have the means and are willing to go to the effort to view that photo just as today there are those that learn Latin just so they can read the original works of ancient scholars.

To say that the important stuff will be preserved (because "there will be those that have the means and are willing to go to the effort to view that photo") is at best only part of the story.

There will always be researchers that will know what they are looking for, and have the time and expertise to access material (or the money to pay someone else to do it). They will be a minority, and concentrate as you say on specific individuals known to be significant.

If we're only concerned about preserving what we know to be the most important stuff then no problem.

But much of our history is not preserved this way, and most of our "historians" are not like that at all. They don't learn Latin and they know nothing about raw conversion. Hence my comments about small poorly-resourced groups of volunteers that are collectively responsible for preserving much of the heritage in the UK. In many cases they simply don't have the expertise to access difficult material, and are unlikely to prioritise the money for someone else to do it unless they know for certain that there's something really important there.

I speak from experience: the harder it is for the people doing the work to access the material, the more likely it is to be left.

What makes you think JPEG will be any less archaic in the future?

 

[michaeladawson]

My question is theoretical in nature. The stronger CFA crowd talks about better separation of the R, G, B wavelengths (less wide).

Let's say one made a theoretical CFA that was perfect in the sense that each color captured only a narrow band of wavelength, zero capture of light outside this narrow band for each color and there was absolutely no overlap. Let's even say that the discrimination is so narrow that there are gaps between the R, G, B filters.

Now wouldn't this lead to severe color problems with various light sources and a variety of subject matter?

Imagine that we have subject details that reflect or emit within only very narrow wavelength bands, so small that they could fit entirely within one of the receiving color channel bands, or entirely within a gap between channels.

• Any stimulus wavelength, no matter where it falls within any one particular color band, can be recorded in one and only one way. There would be no information available to distinguish between 590 nm and 610 nm for example.
• Any wavelength that unluckily falls in a gap will not be recorded at all.

Overlap conditions on the other hand provide gradual changes of the response ratio between at least two receiving channels, no matter what wavelength is being received. You could then reproduce a "610 nm subject" and a "590 nm subject" with different displayed hues.

Yes. My question was a bit leading as I knew what the answer was going to be when I posed it. So why did I pose it?

The argument had been made that the Canon 5D II (or III, I forget which now) has wider R,G,B filter response shoulders than the original 5D. It was stated that this was due to weaker CFA and it was further opined that the color was worse due to the weaker CFA.

So my question was intended to establish that you need overlap of the R,G,B filter response in order get good color. So then the question becomes one of "how much overlap is desired?" Is the original Canon 5D CFA somehow the perfect overlap? I doubt it. Is the Mark II or III too much overlap? In the opinion of some it is.

This question goes along with my second question of whether the peak wavelength sensitivities also impact the color rendering and SMI. Do peak sensitivities need to align closely with human eye sensitivities? The wavelength peak sensitivities were also different between the two Canon 5D models.

No answers from me. I only posed the questions for thought.

 

[michaeladawson]

Not sure that it answers my questions. But it was a good read into the work that goes into developing good color output from a sensor. Less instrumentation decision making and more human judgement than I would have thought.

Thanks for the link.

 

[Leonard Shepherd]

So my question was intended to establish that you need overlap of the R,G,B filter response in order get good color. So then the question becomes one of "how much overlap is desired?"

A digital camera (at the most basic technical level) can record millions of colours from just a basic R, a basic G (actually 2) and a basic B filter over different pixels. To do this the system looks at what is going on at several nearby pixels before deciding what colour and density to alocate to a particular pixel site.

I presume this is what you mean by "need overlap of the R, G,B filter response".

How good colour is in part depends on how good the camera is at accurately processing colour information. Colour gamut and accuracy is something that has improved noticeably since I got my first digital camera 12 years ago.