Can we trust DxOMark (after they rated their own DxO One, 1" sensor at 1657 ISO)?

walkaround wrote:

TrojMacReady wrote:

ChristianHass wrote:

If you read their review the sensor gets a normal ISO score of 506 which is basicly the same as the other 1" sensors. It's only in their "superraw" mode it gets 1657.

Superraw basicly means that the camera takes 4 shots and averages them out to reduce the noise, so it's not going to work for anything moving.

Sony RX100 also has multishot NR, but DxO didn't test that. If they had it'd probably get a similar result.

Sony's version doesn't work in RAW and they test RAW data, before demosaicing, gamma curves, color profiles etc. are applied.

DXO doesn't say anywhere on their site that they test raw "data" before demosaicing, etc.

"All sensor scores reflect only the RAW sensor performance of a camera body. All measurements are performed on the RAW image file BEFORE demosaicing or other processing prior to final image delivery."

from: http://www.dxomark.com/About/Sensor-scores

They use raw files straight from the camera, and do in fact convert them to "images":

Well, of course.  But RAW files straight from the camera aren't demosaiced.  That's something that most RAW converters automatically do when you open a RAW file.  Most of the popular ones also automatically apply a colour profile, a tone curve, some lens corrections and sometimes a bunch of other tweaks as well.  But it's perfectly possible to analyze the RAW data before all of these things take place, and that's what DxO Labs do.

walkaround wrote:

TrojMacReady wrote:

ChristianHass wrote:

If you read their review the sensor gets a normal ISO score of 506 which is basicly the same as the other 1" sensors. It's only in their "superraw" mode it gets 1657.

Superraw basicly means that the camera takes 4 shots and averages them out to reduce the noise, so it's not going to work for anything moving.

Sony RX100 also has multishot NR, but DxO didn't test that. If they had it'd probably get a similar result.

Sony's version doesn't work in RAW and they test RAW data, before demosaicing, gamma curves, color profiles etc. are applied.

DXO doesn't say anywhere on their site that they test raw "data" before demosaicing, etc. They use raw files straight from the camera, and do in fact convert them to "images":

"As we do not have access to intermediate outputs on the sensor, DxOMark measures RAW images — the very same images that can be accessed by photographers who use cameras that shoot in RAW."

Like posted by cainn24, they specifically state RAW data before demosaicing etc., basically what I said above. They do not convert to viewable images, they test binary data. Which is quite different from jpegs (Sony multi shot NR).

cainn24 wrote:

Well, of course. But RAW files straight from the camera aren't demosaiced. That's something that most RAW converters automatically do when you open a RAW file. Most of the popular ones also automatically apply a colour profile, a tone curve, some lens corrections and sometimes a bunch of other tweaks as well. But it's perfectly possible to analyze the RAW data before all of these things take place, and that's what DxO Labs do.

which is why it's incorrect doing company to company comparisons.

rrccad wrote:

cainn24 wrote:

Well, of course. But RAW files straight from the camera aren't demosaiced. That's something that most RAW converters automatically do when you open a RAW file. Most of the popular ones also automatically apply a colour profile, a tone curve, some lens corrections and sometimes a bunch of other tweaks as well. But it's perfectly possible to analyze the RAW data before all of these things take place, and that's what DxO Labs do.

which is why it's incorrect doing company to company comparisons.

I don't understand your point.  Could you elaborate?

cainn24 wrote:

walkaround wrote:

TrojMacReady wrote:

ChristianHass wrote:

If you read their review the sensor gets a normal ISO score of 506 which is basicly the same as the other 1" sensors. It's only in their "superraw" mode it gets 1657.

Superraw basicly means that the camera takes 4 shots and averages them out to reduce the noise, so it's not going to work for anything moving.

Sony RX100 also has multishot NR, but DxO didn't test that. If they had it'd probably get a similar result.

Sony's version doesn't work in RAW and they test RAW data, before demosaicing, gamma curves, color profiles etc. are applied.

DXO doesn't say anywhere on their site that they test raw "data" before demosaicing, etc.

"All sensor scores reflect only the RAW sensor performance of a camera body. All measurements are performed on the RAW image file BEFORE demosaicing or other processing prior to final image delivery."

from: http://www.dxomark.com/About/Sensor-scores

Ok, I missed that one statement, thanks. But it seems to be contradicted in other places on the site, where they mention "raw image" over and over. Not "raw image file", but "raw image". I don't think this is splitting hairs when talking about a scientific subject. There is actually no such thing as a "raw image". Canon CR2 and Nikon NEF files are not images. You can't view them in any way until they are either converted, or their embedded jpegs are displayed. So are dxo just sloppy with their website copy?

In the "About DXOMark" section, they say this about their tests:

"All published DxOMark measurements have been made using DxO Analyzer.

DxO Analyzer includes the following three key elements:

• A dedicated camera testing lab specifically equipped with dedicated test targets, lighting systems, light-boxes, light-meters, telemeters, spectrometers, etc.
• A set of precisely-described and bias-free test protocols for each measurement category which:
  • Strictly accounts for all physical parameters that influence measurements.
  • Ensures repeatability of the measurements.
• Software that automatically analyzes test target images, performs quality controls, and reports all measurements in graphic and data formats.

The basic protocol for all measurements is:

• Adjust test targets, lights, and the selected camera and lens.
• Shoot a set of images of the target at different camera and lens settings.
• Process the target images with DxO Analyzer (machine vision algorithms enable automatic processing).
• Evaluate the data and graphs reported through the DxO Analyzer interface."

This is as clear as mud. "Process the images" sounds like conversion to me. If they were testing data before demosaicing or other processing why wouldn't they just say "DXO Analyzer tests the raw file data"?

They use raw files straight from the camera, and do in fact convert them to "images":

Well, of course. But RAW files straight from the camera aren't demosaiced. That's something that most RAW converters automatically do when you open a RAW file. Most of the popular ones also automatically apply a colour profile, a tone curve, some lens corrections and sometimes a bunch of other tweaks as well. But it's perfectly possible to analyze the RAW data before all of these things take place, and that's what DxO Labs do.

How is it "perfectly possible" to analyze proprietary data for dozens of different manufacturers' raw sensor data? Image sensors don't all record data in the same way, in the same format. And demosaicing isn't some add-on post process, it's intrinsic to the output of the sensor.

cainn24 wrote:

rrccad wrote:

cainn24 wrote:

Well, of course. But RAW files straight from the camera aren't demosaiced. That's something that most RAW converters automatically do when you open a RAW file. Most of the popular ones also automatically apply a colour profile, a tone curve, some lens corrections and sometimes a bunch of other tweaks as well. But it's perfectly possible to analyze the RAW data before all of these things take place, and that's what DxO Labs do.

which is why it's incorrect doing company to company comparisons.

I don't understand your point. Could you elaborate?

Various companies will leave a raw file from the camera at a particular state of "polish" or finish.

by simply reading the raw data and not doing any level interpretations, you never get an equal look at the data itself.

an example of this is black point offsets.  if a raw file has a floating offset on row / column - this has to be calculated back in before noise evaluation versus a raw file that does this internally in the camera. post raw conversion all things will be equal as the raw convertor will handle that during demoisicing.

walkaround wrote:

cainn24 wrote:

walkaround wrote:

TrojMacReady wrote:

ChristianHass wrote:

If you read their review the sensor gets a normal ISO score of 506 which is basicly the same as the other 1" sensors. It's only in their "superraw" mode it gets 1657.

Superraw basicly means that the camera takes 4 shots and averages them out to reduce the noise, so it's not going to work for anything moving.

Sony RX100 also has multishot NR, but DxO didn't test that. If they had it'd probably get a similar result.

Sony's version doesn't work in RAW and they test RAW data, before demosaicing, gamma curves, color profiles etc. are applied.

DXO doesn't say anywhere on their site that they test raw "data" before demosaicing, etc.

"All sensor scores reflect only the RAW sensor performance of a camera body. All measurements are performed on the RAW image file BEFORE demosaicing or other processing prior to final image delivery."

from: http://www.dxomark.com/About/Sensor-scores

Ok, I missed that one statement, thanks. But it seems to be contradicted in other places on the site, where they mention "raw image" over and over. Not "raw image file", but "raw image". I don't think this is splitting hairs when talking about a scientific subject. There is actually no such thing as a "raw image". Canon CR2 and Nikon NEF files are not images. You can't view them in any way until they are either converted, or their embedded jpegs are displayed. So are dxo just sloppy with their website copy?

In the "About DXOMark" section, they say this about their tests:

"All published DxOMark measurements have been made using DxO Analyzer.

DxO Analyzer includes the following three key elements:

• A dedicated camera testing lab specifically equipped with dedicated test targets, lighting systems, light-boxes, light-meters, telemeters, spectrometers, etc.
• A set of precisely-described and bias-free test protocols for each measurement category which:
  • Strictly accounts for all physical parameters that influence measurements.
  • Ensures repeatability of the measurements.
• Software that automatically analyzes test target images, performs quality controls, and reports all measurements in graphic and data formats.

The basic protocol for all measurements is:

• Adjust test targets, lights, and the selected camera and lens.
• Shoot a set of images of the target at different camera and lens settings.
• Process the target images with DxO Analyzer (machine vision algorithms enable automatic processing).
• Evaluate the data and graphs reported through the DxO Analyzer interface."

This is as clear as mud. "Process the images" sounds like conversion to me. If they were testing data before demosaicing or other processing why wouldn't they just say "DXO Analyzer tests the raw file data"?

No one said that DxO Labs isn't analyzing image data. I think what you're missing is the fact that image data exists even before demosaicing takes place.

Here, have a quick read: https://en.wikipedia.org/wiki/Demosaicing

The reason they do this is presumably because there is really no such thing as a standardized demosaicing method, and different methods can impact slightly upon noise.  Generally the differences are insignificant for the most part but DxO Labs are simply trying to work with data that is as "raw" as possible.

They use raw files straight from the camera, and do in fact convert them to "images":

Well, of course. But RAW files straight from the camera aren't demosaiced. That's something that most RAW converters automatically do when you open a RAW file. Most of the popular ones also automatically apply a colour profile, a tone curve, some lens corrections and sometimes a bunch of other tweaks as well. But it's perfectly possible to analyze the RAW data before all of these things take place, and that's what DxO Labs do.

How is it "perfectly possible" to analyze proprietary data for dozens of different manufacturers' raw sensor data? Image sensors don't all record data in the same way, in the same format.

Get a RAW file.  Open it in RawTherapee, or RawDigger, or any number of other similarly capable apps that support the RAW format in question.  Disable demosaicing.

Done

The reason there is support for "proprietary" RAW formats in third-party applications is because someone reverse engineered them.

And demosaicing isn't some add-on post process, it's intrinsic to the output of the camera.

It's only intrinsic to JPEG output, not RAW output.  Again, a RAW file straight from the camera is not demosaiced.

rrccad wrote:

cainn24 wrote:

rrccad wrote:

cainn24 wrote:

Well, of course. But RAW files straight from the camera aren't demosaiced. That's something that most RAW converters automatically do when you open a RAW file. Most of the popular ones also automatically apply a colour profile, a tone curve, some lens corrections and sometimes a bunch of other tweaks as well. But it's perfectly possible to analyze the RAW data before all of these things take place, and that's what DxO Labs do.

which is why it's incorrect doing company to company comparisons.

I don't understand your point. Could you elaborate?

Various companies will leave a raw file from the camera at a particular state of "polish" or finish.

"Cooking" RAW data to any significant degree isn't really that common, and in most cases it has been detected pretty quickly and identified as such.

The underlying argument you seem to be making here is that the sensor measurements that DxO Labs publish can't possibly correlate with what people will see in the real world when comparing actual images.  But that's simply not true.  If you run a bunch of RAW files through a standardized development process you will discover a strong tendency for the relative differences identified by DxO Labs to manifest in your final images.

cainn24 wrote:

walkaround wrote:

cainn24 wrote:

walkaround wrote:

TrojMacReady wrote:

ChristianHass wrote:

If you read their review the sensor gets a normal ISO score of 506 which is basicly the same as the other 1" sensors. It's only in their "superraw" mode it gets 1657.

Superraw basicly means that the camera takes 4 shots and averages them out to reduce the noise, so it's not going to work for anything moving.

Sony RX100 also has multishot NR, but DxO didn't test that. If they had it'd probably get a similar result.

Sony's version doesn't work in RAW and they test RAW data, before demosaicing, gamma curves, color profiles etc. are applied.

DXO doesn't say anywhere on their site that they test raw "data" before demosaicing, etc.

"All sensor scores reflect only the RAW sensor performance of a camera body. All measurements are performed on the RAW image file BEFORE demosaicing or other processing prior to final image delivery."

from: http://www.dxomark.com/About/Sensor-scores

Ok, I missed that one statement, thanks. But it seems to be contradicted in other places on the site, where they mention "raw image" over and over. Not "raw image file", but "raw image". I don't think this is splitting hairs when talking about a scientific subject. There is actually no such thing as a "raw image". Canon CR2 and Nikon NEF files are not images. You can't view them in any way until they are either converted, or their embedded jpegs are displayed. So are dxo just sloppy with their website copy?

In the "About DXOMark" section, they say this about their tests:

"All published DxOMark measurements have been made using DxO Analyzer.

DxO Analyzer includes the following three key elements:

• A dedicated camera testing lab specifically equipped with dedicated test targets, lighting systems, light-boxes, light-meters, telemeters, spectrometers, etc.
• A set of precisely-described and bias-free test protocols for each measurement category which:
  • Strictly accounts for all physical parameters that influence measurements.
  • Ensures repeatability of the measurements.
• Software that automatically analyzes test target images, performs quality controls, and reports all measurements in graphic and data formats.

The basic protocol for all measurements is:

• Adjust test targets, lights, and the selected camera and lens.
• Shoot a set of images of the target at different camera and lens settings.
• Process the target images with DxO Analyzer (machine vision algorithms enable automatic processing).
• Evaluate the data and graphs reported through the DxO Analyzer interface."

This is as clear as mud. "Process the images" sounds like conversion to me. If they were testing data before demosaicing or other processing why wouldn't they just say "DXO Analyzer tests the raw file data"?

No one said that DxO Labs isn't analyzing image data. I think what you're missing is the fact that image data exists even before demosaicing takes place.

Here, have a quick read: https://en.wikipedia.org/wiki/Demosaicing

The reason they do this is presumably because there is really no such thing as a standardized demosaicing method, and different methods can impact slightly upon noise. Generally the differences are insignificant for the most part but DxO Labs are simply trying to work with data that is as "raw" as possible.

I know what demosaicing is. The idea of testing a sensor/camera before demosaicing is absurd, and any result you do get has less practical value for the end user of the camera, who will have to convert any raw files in order to use them.

They use raw files straight from the camera, and do in fact convert them to "images":

Well, of course. But RAW files straight from the camera aren't demosaiced. That's something that most RAW converters automatically do when you open a RAW file. Most of the popular ones also automatically apply a colour profile, a tone curve, some lens corrections and sometimes a bunch of other tweaks as well. But it's perfectly possible to analyze the RAW data before all of these things take place, and that's what DxO Labs do.

How is it "perfectly possible" to analyze proprietary data for dozens of different manufacturers' raw sensor data? Image sensors don't all record data in the same way, in the same format.

Get a RAW file. Open it in RawTherapee, or RawDigger, or any number of other similarly capable apps that support the RAW format in question. Disable demosaicing.

Done

This is just showing you their best guess for how the image should look. The only company that knows exactly how a Canon CR2 file should be converted, is Canon. Same for all the other companies.

The reason there is support for "proprietary" RAW formats in third-party applications is because someone reverse engineered them.

And demosaicing isn't some add-on post process, it's intrinsic to the output of the camera.

It's only intrinsic to JPEG output, not RAW output. Again, a RAW file straight from the camera is not demosaiced.

I changed what I said to "instrinsic to the output of the sensor", but you were obviously replying already and it isn't in your quote. But it still stands that there is no practical thing as "image data before demosaicing". There is no image before conversion, period.

cainn24 wrote:

rrccad wrote:

cainn24 wrote:

rrccad wrote:

cainn24 wrote:

Well, of course. But RAW files straight from the camera aren't demosaiced. That's something that most RAW converters automatically do when you open a RAW file. Most of the popular ones also automatically apply a colour profile, a tone curve, some lens corrections and sometimes a bunch of other tweaks as well. But it's perfectly possible to analyze the RAW data before all of these things take place, and that's what DxO Labs do.

which is why it's incorrect doing company to company comparisons.

I don't understand your point. Could you elaborate?

Various companies will leave a raw file from the camera at a particular state of "polish" or finish.

"Cooking" RAW data to any significant degree isn't really that common, and in most cases it has been detected pretty quickly and identified as such.

The underlying argument you seem to be making here is that the sensor measurements that DxO Labs publish can't possibly correlate with what people will see in the real world when comparing actual images. But that's simply not true. If you run a bunch of RAW files through a standardized development process you will discover a strong tendency for the relative differences identified by DxO Labs to manifest in your final images.

There is no such thing as a "standardized development process". Where do you get that idea? This or that website can choose to convert every camera using ACR, but then your results are dependent on the ability of ACR to render every raw file with equal ability and quality.

There isn't even an industry accepted standard test for "dynamic range" in digital cameras.

cainn24 wrote:

rrccad wrote:

cainn24 wrote:

rrccad wrote:

cainn24 wrote:

Well, of course. But RAW files straight from the camera aren't demosaiced. That's something that most RAW converters automatically do when you open a RAW file. Most of the popular ones also automatically apply a colour profile, a tone curve, some lens corrections and sometimes a bunch of other tweaks as well. But it's perfectly possible to analyze the RAW data before all of these things take place, and that's what DxO Labs do.

which is why it's incorrect doing company to company comparisons.

I don't understand your point. Could you elaborate?

Various companies will leave a raw file from the camera at a particular state of "polish" or finish.

"Cooking" RAW data to any significant degree isn't really that common,

actually what i discribed it most certainly is common.

and in most cases it has been detected pretty quickly and identified as such.

actually that's not the case at all. for instance, dxo never publishes / identifies that over around ISO 3200 sony bakes in NR, and around 6400 I believe that nikon does it.

they never identify compression related errors, or even lossless versus lossy raws, nor any indication of spatial filtering that occurs on long exposures from some companies.

Also they assume that all raw data as at the same point of "finish"

The underlying argument you seem to be making here is that the sensor measurements that DxO Labs publish can't possibly correlate with what people will see in the real world when comparing actual images. But that's simply not true. If you run a bunch of RAW files through a standardized development process you will discover a strong tendency for the relative differences identified by DxO Labs to manifest in your final images.

perhaps, however we are taking about the fine degree of scientific measurement. if we want to get right down to it, there's so little difference in any sensor from any camera from any manufacturer (of the same sensor size) right now, they all work very well.

walkaround wrote:

cainn24 wrote:

walkaround wrote:

cainn24 wrote:

walkaround wrote:

TrojMacReady wrote:

ChristianHass wrote:

If you read their review the sensor gets a normal ISO score of 506 which is basicly the same as the other 1" sensors. It's only in their "superraw" mode it gets 1657.

Superraw basicly means that the camera takes 4 shots and averages them out to reduce the noise, so it's not going to work for anything moving.

Sony RX100 also has multishot NR, but DxO didn't test that. If they had it'd probably get a similar result.

Sony's version doesn't work in RAW and they test RAW data, before demosaicing, gamma curves, color profiles etc. are applied.

DXO doesn't say anywhere on their site that they test raw "data" before demosaicing, etc.

"All sensor scores reflect only the RAW sensor performance of a camera body. All measurements are performed on the RAW image file BEFORE demosaicing or other processing prior to final image delivery."

from: http://www.dxomark.com/About/Sensor-scores

Ok, I missed that one statement, thanks. But it seems to be contradicted in other places on the site, where they mention "raw image" over and over. Not "raw image file", but "raw image". I don't think this is splitting hairs when talking about a scientific subject. There is actually no such thing as a "raw image". Canon CR2 and Nikon NEF files are not images. You can't view them in any way until they are either converted, or their embedded jpegs are displayed. So are dxo just sloppy with their website copy?

In the "About DXOMark" section, they say this about their tests:

"All published DxOMark measurements have been made using DxO Analyzer.

DxO Analyzer includes the following three key elements:

• A dedicated camera testing lab specifically equipped with dedicated test targets, lighting systems, light-boxes, light-meters, telemeters, spectrometers, etc.
• A set of precisely-described and bias-free test protocols for each measurement category which:
  • Strictly accounts for all physical parameters that influence measurements.
  • Ensures repeatability of the measurements.
• Software that automatically analyzes test target images, performs quality controls, and reports all measurements in graphic and data formats.

The basic protocol for all measurements is:

• Adjust test targets, lights, and the selected camera and lens.
• Shoot a set of images of the target at different camera and lens settings.
• Process the target images with DxO Analyzer (machine vision algorithms enable automatic processing).
• Evaluate the data and graphs reported through the DxO Analyzer interface."

This is as clear as mud. "Process the images" sounds like conversion to me. If they were testing data before demosaicing or other processing why wouldn't they just say "DXO Analyzer tests the raw file data"?

No one said that DxO Labs isn't analyzing image data. I think what you're missing is the fact that image data exists even before demosaicing takes place.

Here, have a quick read: https://en.wikipedia.org/wiki/Demosaicing

The reason they do this is presumably because there is really no such thing as a standardized demosaicing method, and different methods can impact slightly upon noise. Generally the differences are insignificant for the most part but DxO Labs are simply trying to work with data that is as "raw" as possible.

I know what demosaicing is. The idea of testing a sensor/camera before demosaicing is absurd, and any result you do get has less practical value for the end user of the camera, who will have to convert any raw files in order to use them.

Yet the data that DxO Labs publish correlates strongly with what I see myself when running RAW output from various cameras through any number of different workflows, and plenty of other people find the same to be true.

They use raw files straight from the camera, and do in fact convert them to "images":

Well, of course. But RAW files straight from the camera aren't demosaiced. That's something that most RAW converters automatically do when you open a RAW file. Most of the popular ones also automatically apply a colour profile, a tone curve, some lens corrections and sometimes a bunch of other tweaks as well. But it's perfectly possible to analyze the RAW data before all of these things take place, and that's what DxO Labs do.

How is it "perfectly possible" to analyze proprietary data for dozens of different manufacturers' raw sensor data? Image sensors don't all record data in the same way, in the same format.

Get a RAW file. Open it in RawTherapee, or RawDigger, or any number of other similarly capable apps that support the RAW format in question. Disable demosaicing.

Done

This is just showing you their best guess for how the image should look. The only company that knows exactly how a Canon CR2 file should be converted, is Canon. Same for all the other companies.

Yet in this context as well the data that I can view correlates strongly what I see when running RAW output from various cameras through any number of different workflows, and plenty of other people find the same to be true.

The reason there is support for "proprietary" RAW formats in third-party applications is because someone reverse engineered them.

And demosaicing isn't some add-on post process, it's intrinsic to the output of the camera.

It's only intrinsic to JPEG output, not RAW output. Again, a RAW file straight from the camera is not demosaiced.

I changed what I said to "instrinsic to the output of the sensor", but you were obviously replying already and it isn't in your quote. But it still stands that there is no practical thing as "image data before demosaicing". There is no image before conversion, period.

You claimed you understood what demosaicing is but if you don't understand how there can be image data before that particular process takes place then you really should in fact read up on it.

walkaround wrote:

cainn24 wrote:

rrccad wrote:

cainn24 wrote:

rrccad wrote:

cainn24 wrote:

Well, of course. But RAW files straight from the camera aren't demosaiced. That's something that most RAW converters automatically do when you open a RAW file. Most of the popular ones also automatically apply a colour profile, a tone curve, some lens corrections and sometimes a bunch of other tweaks as well. But it's perfectly possible to analyze the RAW data before all of these things take place, and that's what DxO Labs do.

which is why it's incorrect doing company to company comparisons.

I don't understand your point. Could you elaborate?

Various companies will leave a raw file from the camera at a particular state of "polish" or finish.

"Cooking" RAW data to any significant degree isn't really that common, and in most cases it has been detected pretty quickly and identified as such.

The underlying argument you seem to be making here is that the sensor measurements that DxO Labs publish can't possibly correlate with what people will see in the real world when comparing actual images. But that's simply not true. If you run a bunch of RAW files through a standardized development process you will discover a strong tendency for the relative differences identified by DxO Labs to manifest in your final images.

There is no such thing as a "standardized development process".

Sure there is.  I take two RAW files, open them up in the same RAW converter, and develop them with the exact same settings.  My standardized development process might be different from someone else's but as long as it is the same for all the output in question there there will be a strong tendency for the relative differences that DxO Labs measure and report on to manifest through that development process to a similar degree.  That's the point.  There is in fact correlation, and that makes the data useful.

cainn24 wrote:

walkaround wrote:

cainn24 wrote:

walkaround wrote:

cainn24 wrote:

walkaround wrote:

TrojMacReady wrote:

ChristianHass wrote:

If you read their review the sensor gets a normal ISO score of 506 which is basicly the same as the other 1" sensors. It's only in their "superraw" mode it gets 1657.

Superraw basicly means that the camera takes 4 shots and averages them out to reduce the noise, so it's not going to work for anything moving.

Sony RX100 also has multishot NR, but DxO didn't test that. If they had it'd probably get a similar result.

Sony's version doesn't work in RAW and they test RAW data, before demosaicing, gamma curves, color profiles etc. are applied.

DXO doesn't say anywhere on their site that they test raw "data" before demosaicing, etc.

"All sensor scores reflect only the RAW sensor performance of a camera body. All measurements are performed on the RAW image file BEFORE demosaicing or other processing prior to final image delivery."

from: http://www.dxomark.com/About/Sensor-scores

Ok, I missed that one statement, thanks. But it seems to be contradicted in other places on the site, where they mention "raw image" over and over. Not "raw image file", but "raw image". I don't think this is splitting hairs when talking about a scientific subject. There is actually no such thing as a "raw image". Canon CR2 and Nikon NEF files are not images. You can't view them in any way until they are either converted, or their embedded jpegs are displayed. So are dxo just sloppy with their website copy?

In the "About DXOMark" section, they say this about their tests:

"All published DxOMark measurements have been made using DxO Analyzer.

DxO Analyzer includes the following three key elements:

• A dedicated camera testing lab specifically equipped with dedicated test targets, lighting systems, light-boxes, light-meters, telemeters, spectrometers, etc.
• A set of precisely-described and bias-free test protocols for each measurement category which:
  • Strictly accounts for all physical parameters that influence measurements.
  • Ensures repeatability of the measurements.
• Software that automatically analyzes test target images, performs quality controls, and reports all measurements in graphic and data formats.

The basic protocol for all measurements is:

• Adjust test targets, lights, and the selected camera and lens.
• Shoot a set of images of the target at different camera and lens settings.
• Process the target images with DxO Analyzer (machine vision algorithms enable automatic processing).
• Evaluate the data and graphs reported through the DxO Analyzer interface."

This is as clear as mud. "Process the images" sounds like conversion to me. If they were testing data before demosaicing or other processing why wouldn't they just say "DXO Analyzer tests the raw file data"?

No one said that DxO Labs isn't analyzing image data. I think what you're missing is the fact that image data exists even before demosaicing takes place.

Here, have a quick read: https://en.wikipedia.org/wiki/Demosaicing

The reason they do this is presumably because there is really no such thing as a standardized demosaicing method, and different methods can impact slightly upon noise. Generally the differences are insignificant for the most part but DxO Labs are simply trying to work with data that is as "raw" as possible.

I know what demosaicing is. The idea of testing a sensor/camera before demosaicing is absurd, and any result you do get has less practical value for the end user of the camera, who will have to convert any raw files in order to use them.

Yet the data that DxO Labs publish correlates strongly with what I see myself when running RAW output from various cameras through any number of different workflows, and plenty of other people find the same to be true.

We discussed this point earlier. I don't believe you are systematically shooting "various cameras" side by side, and then evaluating them in controlled testing. Are you? So, "correlates strongly" means your camera tests well, and you like its output... yes?

And full disclosure: what is your camera(s) of choice?

They use raw files straight from the camera, and do in fact convert them to "images":

Well, of course. But RAW files straight from the camera aren't demosaiced. That's something that most RAW converters automatically do when you open a RAW file. Most of the popular ones also automatically apply a colour profile, a tone curve, some lens corrections and sometimes a bunch of other tweaks as well. But it's perfectly possible to analyze the RAW data before all of these things take place, and that's what DxO Labs do.

How is it "perfectly possible" to analyze proprietary data for dozens of different manufacturers' raw sensor data? Image sensors don't all record data in the same way, in the same format.

Get a RAW file. Open it in RawTherapee, or RawDigger, or any number of other similarly capable apps that support the RAW format in question. Disable demosaicing.

Done

This is just showing you their best guess for how the image should look. The only company that knows exactly how a Canon CR2 file should be converted, is Canon. Same for all the other companies.

Yet in this context as well the data that I can view correlates strongly what I see when running RAW output from various cameras through any number of different workflows, and plenty of other people find the same to be true.

The reason there is support for "proprietary" RAW formats in third-party applications is because someone reverse engineered them.

And demosaicing isn't some add-on post process, it's intrinsic to the output of the camera.

It's only intrinsic to JPEG output, not RAW output. Again, a RAW file straight from the camera is not demosaiced.

I changed what I said to "instrinsic to the output of the sensor", but you were obviously replying already and it isn't in your quote. But it still stands that there is no practical thing as "image data before demosaicing". There is no image before conversion, period.

You claimed you understood what demosaicing is but if you don't understand how there can be image data before that particular process takes place then you really should in fact read up on it.

It's you who are missing the distinction, not me. We both agree there is data present before demosaicing, but it's not image data until after demosaicing. How can you claim otherwise, when a great deal of what you see when you look at a digital camera image is interpreted and interpolated by the demosaicing process. It's literally filling in missing information in order to piece together what you and I see as an "image".

cainn24 wrote:

walkaround wrote:

cainn24 wrote:

rrccad wrote:

cainn24 wrote:

rrccad wrote:

cainn24 wrote:

Well, of course. But RAW files straight from the camera aren't demosaiced. That's something that most RAW converters automatically do when you open a RAW file. Most of the popular ones also automatically apply a colour profile, a tone curve, some lens corrections and sometimes a bunch of other tweaks as well. But it's perfectly possible to analyze the RAW data before all of these things take place, and that's what DxO Labs do.

which is why it's incorrect doing company to company comparisons.

I don't understand your point. Could you elaborate?

Various companies will leave a raw file from the camera at a particular state of "polish" or finish.

"Cooking" RAW data to any significant degree isn't really that common, and in most cases it has been detected pretty quickly and identified as such.

The underlying argument you seem to be making here is that the sensor measurements that DxO Labs publish can't possibly correlate with what people will see in the real world when comparing actual images. But that's simply not true. If you run a bunch of RAW files through a standardized development process you will discover a strong tendency for the relative differences identified by DxO Labs to manifest in your final images.

There is no such thing as a "standardized development process".

Sure there is. I take two RAW files, open them up in the same RAW converter, and develop them with the exact same settings. My standardized development process might be different from someone else's but as long as it is the same for all the output in question there there will be a strong tendency for the relative differences that DxO Labs measure and report on to manifest through that development process to a similar degree. That's the point. There is in fact correlation, and that makes the data useful.

You snipped off what I said after that:

This or that website can choose to convert every camera using ACR, but then your results are dependent on the ability of ACR to render every raw file with equal ability and quality.

If I don't use the converter you use, the results your "standardized process" is useless to me.

rrccad wrote:

cainn24 wrote:

rrccad wrote:

cainn24 wrote:

rrccad wrote:

cainn24 wrote:

Well, of course. But RAW files straight from the camera aren't demosaiced. That's something that most RAW converters automatically do when you open a RAW file. Most of the popular ones also automatically apply a colour profile, a tone curve, some lens corrections and sometimes a bunch of other tweaks as well. But it's perfectly possible to analyze the RAW data before all of these things take place, and that's what DxO Labs do.

which is why it's incorrect doing company to company comparisons.

I don't understand your point. Could you elaborate?

Various companies will leave a raw file from the camera at a particular state of "polish" or finish.

"Cooking" RAW data to any significant degree isn't really that common,

actually what i discribed it most certainly is common.

and in most cases it has been detected pretty quickly and identified as such.

actually that's not the case at all. for instance, dxo never publishes / identifies that over around ISO 3200 sony bakes in NR

Actually I've seen DxO report on "smoothing" of RAW data for several Sony cameras at high ISO. This is what you need to look at:

, and around 6400 I believe that nikon does it.

Evidence please.

they never identify compression related errors, or even lossless versus lossy raws, nor any indication of spatial filtering that occurs on long exposures from some companies.

Also they assume that all raw data as at the same point of "finish"

Yet as I have stated in other replies here now, the data that DxO Labs publish correlates strongly with what I see myself when running RAW output from various cameras through any number of different workflows, and plenty of other people find the same to be true.

There will always be some exceptions, but that's the bathwater, not the baby.

The underlying argument you seem to be making here is that the sensor measurements that DxO Labs publish can't possibly correlate with what people will see in the real world when comparing actual images. But that's simply not true. If you run a bunch of RAW files through a standardized development process you will discover a strong tendency for the relative differences identified by DxO Labs to manifest in your final images.

perhaps, however we are taking about the fine degree of scientific measurement. if we want to get right down to it, there's so little difference in any sensor from any camera from any manufacturer (of the same sensor size) right now, they all work very well.

Your minor differences are another persons major ones. We shouldn't try to force our own subjective judgements about such things onto everyone else.

walkaround wrote:

cainn24 wrote:

walkaround wrote:

cainn24 wrote:

walkaround wrote:

cainn24 wrote:

walkaround wrote:

TrojMacReady wrote:

ChristianHass wrote:

If you read their review the sensor gets a normal ISO score of 506 which is basicly the same as the other 1" sensors. It's only in their "superraw" mode it gets 1657.

Superraw basicly means that the camera takes 4 shots and averages them out to reduce the noise, so it's not going to work for anything moving.

Sony RX100 also has multishot NR, but DxO didn't test that. If they had it'd probably get a similar result.

Sony's version doesn't work in RAW and they test RAW data, before demosaicing, gamma curves, color profiles etc. are applied.

DXO doesn't say anywhere on their site that they test raw "data" before demosaicing, etc.

"All sensor scores reflect only the RAW sensor performance of a camera body. All measurements are performed on the RAW image file BEFORE demosaicing or other processing prior to final image delivery."

from: http://www.dxomark.com/About/Sensor-scores

Ok, I missed that one statement, thanks. But it seems to be contradicted in other places on the site, where they mention "raw image" over and over. Not "raw image file", but "raw image". I don't think this is splitting hairs when talking about a scientific subject. There is actually no such thing as a "raw image". Canon CR2 and Nikon NEF files are not images. You can't view them in any way until they are either converted, or their embedded jpegs are displayed. So are dxo just sloppy with their website copy?

In the "About DXOMark" section, they say this about their tests:

"All published DxOMark measurements have been made using DxO Analyzer.

DxO Analyzer includes the following three key elements:

• A dedicated camera testing lab specifically equipped with dedicated test targets, lighting systems, light-boxes, light-meters, telemeters, spectrometers, etc.
• A set of precisely-described and bias-free test protocols for each measurement category which:
  • Strictly accounts for all physical parameters that influence measurements.
  • Ensures repeatability of the measurements.
• Software that automatically analyzes test target images, performs quality controls, and reports all measurements in graphic and data formats.

The basic protocol for all measurements is:

• Adjust test targets, lights, and the selected camera and lens.
• Shoot a set of images of the target at different camera and lens settings.
• Process the target images with DxO Analyzer (machine vision algorithms enable automatic processing).
• Evaluate the data and graphs reported through the DxO Analyzer interface."

This is as clear as mud. "Process the images" sounds like conversion to me. If they were testing data before demosaicing or other processing why wouldn't they just say "DXO Analyzer tests the raw file data"?

No one said that DxO Labs isn't analyzing image data. I think what you're missing is the fact that image data exists even before demosaicing takes place.

Here, have a quick read: https://en.wikipedia.org/wiki/Demosaicing

The reason they do this is presumably because there is really no such thing as a standardized demosaicing method, and different methods can impact slightly upon noise. Generally the differences are insignificant for the most part but DxO Labs are simply trying to work with data that is as "raw" as possible.

I know what demosaicing is. The idea of testing a sensor/camera before demosaicing is absurd, and any result you do get has less practical value for the end user of the camera, who will have to convert any raw files in order to use them.

Yet the data that DxO Labs publish correlates strongly with what I see myself when running RAW output from various cameras through any number of different workflows, and plenty of other people find the same to be true.

We discussed this point earlier. I don't believe you are systematically shooting "various cameras" side by side, and then evaluating them in controlled testing. Are you?

I certainly do. But I don't exactly have thousands of cameras so I also analyse output provided by other people who have tested different cameras in properly controlled conditions.

So, "correlates strongly" means your camera tests well, and you like its output... yes?

No. It means that what I see when I develop images from RAW correlates strongly with the relative differences that DxO Labs report on. This is in much the same way that the RAW data visualizations produced by DPR and made available in their studio comparison tool also correlate strongly with that data.

And full disclosure: what is your camera(s) of choice?

I shoot and have shot with everything from a smartphone to a bunch of different Micro Four Thirds cameras. I would probably be most aptly described as a small format enthusiast. It's not because I think they are objectively better in any way, just better suited to maximizing my own enjoyment.

They use raw files straight from the camera, and do in fact convert them to "images":

Well, of course. But RAW files straight from the camera aren't demosaiced. That's something that most RAW converters automatically do when you open a RAW file. Most of the popular ones also automatically apply a colour profile, a tone curve, some lens corrections and sometimes a bunch of other tweaks as well. But it's perfectly possible to analyze the RAW data before all of these things take place, and that's what DxO Labs do.

How is it "perfectly possible" to analyze proprietary data for dozens of different manufacturers' raw sensor data? Image sensors don't all record data in the same way, in the same format.

Get a RAW file. Open it in RawTherapee, or RawDigger, or any number of other similarly capable apps that support the RAW format in question. Disable demosaicing.

Done

This is just showing you their best guess for how the image should look. The only company that knows exactly how a Canon CR2 file should be converted, is Canon. Same for all the other companies.

Yet in this context as well the data that I can view correlates strongly what I see when running RAW output from various cameras through any number of different workflows, and plenty of other people find the same to be true.

The reason there is support for "proprietary" RAW formats in third-party applications is because someone reverse engineered them.

And demosaicing isn't some add-on post process, it's intrinsic to the output of the camera.

It's only intrinsic to JPEG output, not RAW output. Again, a RAW file straight from the camera is not demosaiced.

I changed what I said to "instrinsic to the output of the sensor", but you were obviously replying already and it isn't in your quote. But it still stands that there is no practical thing as "image data before demosaicing". There is no image before conversion, period.

You claimed you understood what demosaicing is but if you don't understand how there can be image data before that particular process takes place then you really should in fact read up on it.

It's you who are missing the distinction, not me. We both agree there is data present before demosaicing, but it's not image data until after demosaicing.

Yes, it is. It's just isn't a "full colour" image, if you know what I mean.

How can you claim otherwise, when a great deal of what you see when you look at a digital camera image is interpreted and interpolated by the demosaicing process. It's literally filling in missing information in order to piece together what you and I see as an "image".

OK, I see. You're just being very specific with your definition of "image data".

But my point is, and always has been, that DxO Labs are indeed analysing something that could rightly be defined as "image data" in a broader sense, and that most importantly their methodology does in fact allow their data to correlate strongly with what the typical RAW shooter will see in their efforts to maximize IQ.

walkaround wrote:

cainn24 wrote:

walkaround wrote:

cainn24 wrote:

rrccad wrote:

cainn24 wrote:

rrccad wrote:

cainn24 wrote:

Well, of course. But RAW files straight from the camera aren't demosaiced. That's something that most RAW converters automatically do when you open a RAW file. Most of the popular ones also automatically apply a colour profile, a tone curve, some lens corrections and sometimes a bunch of other tweaks as well. But it's perfectly possible to analyze the RAW data before all of these things take place, and that's what DxO Labs do.

which is why it's incorrect doing company to company comparisons.

I don't understand your point. Could you elaborate?

Various companies will leave a raw file from the camera at a particular state of "polish" or finish.

"Cooking" RAW data to any significant degree isn't really that common, and in most cases it has been detected pretty quickly and identified as such.

The underlying argument you seem to be making here is that the sensor measurements that DxO Labs publish can't possibly correlate with what people will see in the real world when comparing actual images. But that's simply not true. If you run a bunch of RAW files through a standardized development process you will discover a strong tendency for the relative differences identified by DxO Labs to manifest in your final images.

There is no such thing as a "standardized development process".

Sure there is. I take two RAW files, open them up in the same RAW converter, and develop them with the exact same settings. My standardized development process might be different from someone else's but as long as it is the same for all the output in question there there will be a strong tendency for the relative differences that DxO Labs measure and report on to manifest through that development process to a similar degree. That's the point. There is in fact correlation, and that makes the data useful.

You snipped off what I said after that:

This or that website can choose to convert every camera using ACR, but then your results are dependent on the ability of ACR to render every raw file with equal ability and quality.

If I don't use the converter you use, the results your "standardized process" is useless to me.

My point has never been that someone else's development process will be useful to you if you use a different one.  Rather my point has always been that if you use your own standardized development process you will, to some significant extent, be able to observe the relative differences that DxO Labs report on.  And the reason I am making this point is, again, to demonstrate that DxO Labs are telling you something real about comparative sensor performance.