DxOMark still silent on the E-M5

Started Sep 1, 2012 | Discussions
jim stirling
jim stirling Veteran Member • Posts: 7,356
Re: Dxo is a questionable "organization" and . . .

woof woof wrote:

jim stirling wrote:

woof woof wrote:

I'm a bit late to this but I'm interested in the view that they wont release the results as there may have been some in camera "cheating."

I'm retired now but product testing was one of the things that I used to do and in DxO's position I would simply state the test method, the measuring equipment or method and detail the result. It really is that simple and the tester shouldn't IMVHO care how the equipment under test generates the end result. It's just a test and the result.

Would that apply to athletes using drugs surely the tester shouldn't care how fast the the athlete runs if only the result matters .
Jim

Well, that's not really the same thing or even like a remotely sensible analogy now is it?

Well that would depend if for example DXO was suspicious some behind the scenes jiggery pokery {technical term} was influencing results putting camera A at an advantage over camera B. I am not saying this is necessarily the case just giving a possible scenario. As an example RAW level built in NR can skew DR results due to the way that DR is measured :).
Jim

 jim stirling's gear list:jim stirling's gear list
Panasonic FZ1000 Panasonic Lumix DMC-GF1 Panasonic Lumix DMC-GX7 Panasonic Lumix DMC-GH4 Nikon D810 +12 more
Anders W Forum Pro • Posts: 21,468
Re: Dxo is a questionable "organization" and . . .

3DrJ wrote:

Makinations wrote:

How is any of that cheating?

Of course it's not at all "cheating", merely toying with the use of that term. Some in this forum have "accused" cameras of "cooking" RAW output in order to get higher "scores" from testing organizations like DxO.

Those who accuse camera manufacturers of "cooking" RAW typically refer to types of processing that are not required in order to produce a RAW file (noise reduction by means of some kind of averaging across pixels). Unless that's what you mean, using terms like "cheating" or "cooking" is just misleading.

The implication is by processing the data produced by the sensor camera makers have "cheated".

No that's a misunderstanding on your part for reasons spelled out above. "Cheating" or "cooking" is not the same as just any kind of processing.

They think RAW files do, or should, reflect exactly what signals the sensor itself generated.

Yes, that's what they think. And I agree with them.

People with any sense at all will realize it doesn't work that way. There's no such thing as a "true" RAW file, the camera by necessity has to process, or "cook", sensor signals in order to record anything at all. Fact is, all RAW images have been sliced, diced and considerably processed.

No, that's not the case. The only kind of "processing" required to generate a RAW file is to convert analog data (the electric charge in each pixel) to digital data (a number corresponding to the magnitude of the charge).

That's all I've been saying on the subject. As I see it, the better camera makers are at "cooking", the tastier will be the RAW images the camera hands us.

On the contrary, the less they "cook" the better.

 Anders W's gear list:Anders W's gear list
Panasonic Lumix DMC-G1 Olympus OM-D E-M5 Olympus E-M1 Panasonic Lumix G Vario 14-45mm F3.5-5.6 ASPH OIS Panasonic Lumix G Vario 7-14mm F4 ASPH +28 more
3DrJ Senior Member • Posts: 1,027
Re: Dxo is a questionable "organization" and . . .

Anders W wrote:

The only processing required in taking the step from the charge stored in each pixel on the sensor to the RAW output file is to convert from an analog signal of a certain magnitude (an electric charge) to a digital signal of corresponding magnitude (a number). Please tell me how you arrive at the conclusion that this step must necessarily involve some kind of cheating.

Of course it's not really "cheating", but that's been a term others have applied. All I was saying was that of necessity some processing of the sensor's signals will occur. Exactly how it's done may not be revealed to us, wherein some "corrections" may be applied, but still, unless there is spurious camera output, no way to know if some "magic" has occurred. No doubt best to make no assumptions.

Anyone can measure sensor output in the form of a RAW file. As indicated above, the conversion from analog signal to RAW digital data need not (and typically does not) involve more than a per-pixel conversion from an analog signal to a digital one.

I don't know that we can say that for sure. Who knows, at the silicon level that "per-pixel conversion" could be modified or more complex that theory would suggest.

Now lets doff our chef hats to the camera vendors who "cheat". The more they are willing and able to do so, the better the results we get to enjoy.

I am not particularly fond of camera manufacturers who do "cheat" and thus find no reason to "doff my chef hat" to them. AFAIK, "cheating" in the stage from analog to RAW digital output, when it occurs, takes the form of noise reduction, i.e., resolution is traded for lower noise. DxOMark regularly tests for this kind of "cheating" and reports it when they observe it. There may have been cameras who slipped by this control in the past, but I think it unlikely that they would do so at this stage.

Outside of a few at Sony or Olympus, no one can say what "tricks" might be employed to optimize output parameters, such as noise, etc. And if it's the case, we can't describe it as cheating, just process improvements.

What we can tell are the characteristics of the output. If resolution is only so much under particular conditions, that's a fact. We don't have to account for the how it got to be the way it is, all we have to do is see what it is.

As you say, actually "cheating", or trying to, is not likely to pass unnoticed by sharp observers. Manufacturers who attempt to cheat are likely to lose out rather quickly.

Camera makers won't fool Mother Nature.

Here is further information on how DxOMark handles the issue:
...

Cameras known to do NR in RAW include the Pentax K-5 (from ISO 3200 on), the Nikon series 1 cameras (from ISO 800 on), and the Leica M9 (from ISO 160 on). As you can see if you go to for example the DR graph for the K-5 via the link below, this is indicated by DxOMark by using a "hollow" rather than solid dot for the "smoothed" data.

No doubt there could be many ways output could be less than ideal. What can be done is to document the result, the facts are what they are. We don't have to "explain" it, in fact, it may not even be possible to do so. If "smoothed" output is the best, or all, a camera can do, it's a characteristic to document. Ultimately, it's users who determine the camera's acceptability for their particular purposes.

We can't argue with facts, nor with Nature herself. The arguments, such as they are, reflect opinions about what facts mean , and what does that really accomplish?

JRA

-- hide signature --

Artists must not only see, but see what they are seeing.

 3DrJ's gear list:3DrJ's gear list
Olympus E-1 Olympus E-3 Olympus PEN E-P3 Olympus OM-D E-M5 Olympus E-M1 +5 more
3DrJ Senior Member • Posts: 1,027
Re: Dxo is a questionable "organization" and . . .

Well, the terms here like "cheating" and "cooking" arouse strong responses, which was not my intention to evoke.

Actually, I was using the terms ironically. Of course we wouldn't condone cheating. But it's likely enough that there are levels of processing embedded in the silicon or firmware that we just can't detect.

Such processing, or modification of signals would not necessarily imply unfavorable alteration of image characteristics. Rather, the ability to improve the result is exactly what we anticipate technological development to accomplish. It's certainly the case for so many other products in the realm of computers and their components.

That is a kind of "cooking" you'd endorse, wouldn't you? In that sense, and only in that sense, we might say "cooking" is good. That's all I was getting at and meant to convey.

A tangential thought is that process improvements for sensors, A/D converters, etc., might well alter certain relations between absolute sensor signalling and raw file data. I wonder if certain changes beginning to arrive will. if not immediately, be a target of silicon-level processing. For example, embedding PDAF arrays within the CDAF sensor would leave "holes" to fill. If this is done automatically with silicon circuitry, the sensor data would not have to reflect the solution.

That's just my own brain ramp, just speculation, nothing to take seriously there. But built-in processing is hardly unknown, a zillion CPU's out there would testify to it.

JRA

-- hide signature --

Artists must not only see, but see what they are seeing.

 3DrJ's gear list:3DrJ's gear list
Olympus E-1 Olympus E-3 Olympus PEN E-P3 Olympus OM-D E-M5 Olympus E-M1 +5 more
Anders W Forum Pro • Posts: 21,468
Re: Dxo is a questionable "organization" and . . .

3DrJ wrote:

Anders W wrote:

The only processing required in taking the step from the charge stored in each pixel on the sensor to the RAW output file is to convert from an analog signal of a certain magnitude (an electric charge) to a digital signal of corresponding magnitude (a number). Please tell me how you arrive at the conclusion that this step must necessarily involve some kind of cheating.

Of course it's not really "cheating", but that's been a term others have applied. All I was saying was that of necessity some processing of the sensor's signals will occur. Exactly how it's done may not be revealed to us, wherein some "corrections" may be applied, but still, unless there is spurious camera output, no way to know if some "magic" has occurred. No doubt best to make no assumptions.

The point is that there are ways to reveal whether "magic" has occurred in the sense of "cooking" or "cheating".

Anyone can measure sensor output in the form of a RAW file. As indicated above, the conversion from analog signal to RAW digital data need not (and typically does not) involve more than a per-pixel conversion from an analog signal to a digital one.

I don't know that we can say that for sure. Who knows, at the silicon level that "per-pixel conversion" could be modified or more complex that theory would suggest.

Because there are ways to test statistically for that.

Now lets doff our chef hats to the camera vendors who "cheat". The more they are willing and able to do so, the better the results we get to enjoy.

I am not particularly fond of camera manufacturers who do "cheat" and thus find no reason to "doff my chef hat" to them. AFAIK, "cheating" in the stage from analog to RAW digital output, when it occurs, takes the form of noise reduction, i.e., resolution is traded for lower noise. DxOMark regularly tests for this kind of "cheating" and reports it when they observe it. There may have been cameras who slipped by this control in the past, but I think it unlikely that they would do so at this stage.

Outside of a few at Sony or Olympus, no one can say what "tricks" might be employed to optimize output parameters, such as noise, etc. And if it's the case, we can't describe it as cheating, just process improvements.

We can draw the line between "processing improvements" and "cheating/cooking" and that suffices for me. For example, the manufacturers make strong efforts to reduce read noise (a kind of electronic noise added in the step between electric charge and a digital data point corresponding to that charge). This is not "cheating" or "cooking". On the contrary, it makes the raw material cleaner by reducing a disturbance.

Noise reduction by means of averaging pixel values, on the other hand, just covers up one problem (noise) at the expense of another (loss of resolution). I don't like this being done by the camera since a) I prefer to be in control of when and whether to apply NR, b) I would like to choose the tools myself, c) the tools are likely to become better over time so that I can go back and reprocess my RAWs at a later stage if I care too.

What we can tell are the characteristics of the output. If resolution is only so much under particular conditions, that's a fact. We don't have to account for the how it got to be the way it is, all we have to do is see what it is.

Sites like DxO don't try to measure actual sensor resolution. They only report the pixel count. Therefore, I am grateful that they report whether a manufacturer applies NR in RAW, thus a) demonstrating that the sensor at certain ISOs has less resolution than you might expect and b) accounting for that fact. If they didn't do this, their data on other things (SNR, DR, etc) would be misleading in the comparison between different cameras.

As you say, actually "cheating", or trying to, is not likely to pass unnoticed by sharp observers. Manufacturers who attempt to cheat are likely to lose out rather quickly.

Camera makers won't fool Mother Nature.

Noone can fool Mother Nature. Fortunately, they are not likely, at this stage, to fool DxOMark either.

Here is further information on how DxOMark handles the issue:
...

Cameras known to do NR in RAW include the Pentax K-5 (from ISO 3200 on), the Nikon series 1 cameras (from ISO 800 on), and the Leica M9 (from ISO 160 on). As you can see if you go to for example the DR graph for the K-5 via the link below, this is indicated by DxOMark by using a "hollow" rather than solid dot for the "smoothed" data.

No doubt there could be many ways output could be less than ideal. What can be done is to document the result, the facts are what they are. We don't have to "explain" it, in fact, it may not even be possible to do so. If "smoothed" output is the best, or all, a camera can do, it's a characteristic to document. Ultimately, it's users who determine the camera's acceptability for their particular purposes.

As to explaining it, see above.

 Anders W's gear list:Anders W's gear list
Panasonic Lumix DMC-G1 Olympus OM-D E-M5 Olympus E-M1 Panasonic Lumix G Vario 14-45mm F3.5-5.6 ASPH OIS Panasonic Lumix G Vario 7-14mm F4 ASPH +28 more
zxaar Veteran Member • Posts: 4,246
Re: DxO Labs lays down some E-M5 tested performance limits ...

Detail Man wrote:

zxaar wrote:

Detail Man wrote:

Maybe if you wish hard enough, whatever you imagine will come true.

All we need do is disbelieve .

That seems to be the motto of m43 forum.

It seemed to be what you, yourself were endeavoring to do ...

Naa. Well if you find a statement made by me which is not factually true let me know.

This sentence was specially satisfying to my sadist soul.

DxoMark Wrote:

And even though the Olympus OM-D E-M5 can take advantage of some very nice micro 4/3-compatible lenses (Olympus M.Zuiko Digital and Panasonic Lumix G lines), the Pentax K-01 uses the much broader array of Pentax K-mount lenses .

Like some deluded fanboys here say about NEX, there are no lenses in m43 compared to k01.

Since it uses APS-C format, it's not hard to imagine that it does not use M43 lenses. Surprised ?

I think you were replying to someone else's post. I read my post five times and could not see where i said that it (k-01) uses m43 lenses.

-- hide signature --

::> Knowledge is mother of efficiency.

_sem_ Veteran Member • Posts: 5,023
Re: Dxo is a questionable "organization" and . . .

3DrJ wrote:

Well, the terms here like "cheating" and "cooking" arouse strong responses, which was not my intention to evoke.

Actually, I was using the terms ironically. Of course we wouldn't condone cheating. But it's likely enough that there are levels of processing embedded in the silicon or firmware that we just can't detect.

Such processing, or modification of signals would not necessarily imply unfavorable alteration of image characteristics. Rather, the ability to improve the result is exactly what we anticipate technological development to accomplish. It's certainly the case for so many other products in the realm of computers and their components.

That is a kind of "cooking" you'd endorse, wouldn't you? In that sense, and only in that sense, we might say "cooking" is good. That's all I was getting at and meant to convey.

Aside from the cheating aspect, cooking of raw data is no good even if merely favourable alterations are involved. There are different ways (sequences of operations) to process raw data. If raw data is half-cooked, this means certain treatment is baked in & cannot be undone. This means 3rd-party converters (among them DxO OP) don't get the kind the kind of input they expect and can't do processing their preferred way (possibly even a bit better than stock processing). Why then claim raw output at all, they could do lossless tiffs instead.

But I admit it may be sometimes difficult to draw the line because raw would imply the output of the hardware without additional software processing, while the line between hardware and software may not be straight. For example one might be able to implement certain filtering algorithms in hardware such as FPGA in order to speed up processing rather than cheat.

exdeejjjaaaa
exdeejjjaaaa Veteran Member • Posts: 8,263
Re: 16-bit TIFFs are about as good as you can get

boggis the cat wrote:

exdeejjjaaaa wrote:

Adventsam wrote:

Check out techradar, no agenda! they review every camera and the results seem pretty accurate and not that different to dxo in many respects.

any site that measures DR of JPG or TIFF files is worthless

The way that DR is measured is based on a statistical averaging, so even an 8-bit JPEG could yield DR information that would agree with a raw file.

As to TIFF -- well, you must use the raw to produce an output of some sort, and a 16 bit per channel TIFF is about as good as you'll get.

not exactly - I can use various levels/methods of NR for example (that is on top of what may or may not be done before raw files was written by firmware) - hence the details are much more tainted.

The caveat with using OOC JPEGs is of course that some manufacturers are better than others at maximising OOC JPEG output. So comparing e.g. Canon's lacklustre OOC JPEGs to Olympus' may not be valid.

the same is with TIFF...

 exdeejjjaaaa's gear list:exdeejjjaaaa's gear list
Sony Alpha a7R II Sony FE 55mm F1.8 Phase One Capture One Pro +25 more
exdeejjjaaaa
exdeejjjaaaa Veteran Member • Posts: 8,263
Re: 16-bit TIFFs are about as good as you can get

correlation between DxOMark tests and tests by a well known (in certain circles) William Claff ( http://www.dpreview.com/members/9263714680/forums , http://home.comcast.net/~NikonD70/ ) = http://home.comcast.net/~NikonD70/Charts/PDR_Landscape.htm

interested parties can contact William, supply him w/ EM5 raw files prepared per his specifications and enjoy the results of his research.

 exdeejjjaaaa's gear list:exdeejjjaaaa's gear list
Sony Alpha a7R II Sony FE 55mm F1.8 Phase One Capture One Pro +25 more
barb42 Forum Member • Posts: 66
Re: DxOMark still silent on the E-M5

I am using the E-M5. I will judge it by the images I get out of it, according to my skill level. I do not care what score anyone gives to it. Some people will get poor images out of the 'best' or most expensive camera, some will get magic out of the cheapest. Make good, creative images and forget the rest.

 barb42's gear list:barb42's gear list
Olympus PEN E-PL2 Olympus OM-D E-M5
exdeejjjaaaa
exdeejjjaaaa Veteran Member • Posts: 8,263
Re: DxOMark still silent on the E-M5

then what are you doing in this thread, dude ?

 exdeejjjaaaa's gear list:exdeejjjaaaa's gear list
Sony Alpha a7R II Sony FE 55mm F1.8 Phase One Capture One Pro +25 more
SHood Veteran Member • Posts: 5,078
RX100 just scored higher than any previous m43
 SHood's gear list:SHood's gear list
Sony RX100 Panasonic Lumix DMC-GH3 Olympus E-M1 II Panasonic Lumix G Vario 100-300mm F4-5.6 OIS Olympus M.Zuiko 300mm F4 IS Pro
Bob Tullis
Bob Tullis Forum Pro • Posts: 35,923
We need an Ignore Topic feature (NT)
-- hide signature --

...Bob, NYC
http://www.bobtullis.com

"Well, sometimes the magic works. . . Sometimes, it doesn't." - Little Big Man
.

 Bob Tullis's gear list:Bob Tullis's gear list
Fujifilm X-Pro2 Fujifilm X-H1 Fujifilm X100F Fujifilm X-T2
MichaelKJ Veteran Member • Posts: 3,466
Re: RX100 just scored higher than any previous m43

At least we know that they are back from vacation and have resumed testing.

DR for the RX100 measured 12.4 versus 11.3 for the GH2. DR for the two measures were essentially equal from 160 up, with the higher score reflecting the fact that the RX100 measured 12.4 at its base of 80.

 MichaelKJ's gear list:MichaelKJ's gear list
Sony RX100 III Olympus PEN E-PL1 Olympus OM-D E-M5 +1 more
Everdog Veteran Member • Posts: 4,837
Wow, Rx100 ISO is off by almost 1 stop!

People complain about the EM5, but the RX100 (Which I own) is far worse.

zxaar Veteran Member • Posts: 4,246
Re: Wow, Rx100 ISO is off by almost 1 stop!

Everdog wrote:

People complain about the EM5, but the RX100 ( Which I own ) is far worse.

No surprise that you buy worse and inferior things.

-- hide signature --

::> Knowledge is mother of efficiency.

Everdog Veteran Member • Posts: 4,837
Why DxO score are a bad joke

The charts and actual numbers are useful, but the DxO "scores" are a bad joke. The GH1 and GH2 consistantly score higher than the RX100 through out most of the ranges on the graphs, but because the RX100 wins at ISO80, it somehow scores a lot higher....except in low light where it is dismal (still bought one though ).

jalywol
jalywol Veteran Member • Posts: 9,445
Excellent idea +1 (nt)

No Text

highwave Senior Member • Posts: 1,268
Re: RX100 just scored higher than any previous m43

MichaelKJ wrote:

At least we know that they are back from vacation and have resumed testing.

DR for the RX100 measured 12.4 versus 11.3 for the GH2. DR for the two measures were essentially equal from 160 up, with the higher score reflecting the fact that the RX100 measured 12.4 at its base of 80.

Odd the RX100 giving better dynamic range at ISO80. I thought Dpreview mentioned that the ISO80 gives no advantage at all because it's not native.

DxO is getting more and more suspicious. I'm starting to lose credibility for their work.

 highwave's gear list:highwave's gear list
Olympus PEN E-PL1 Olympus E-M5 II Panasonic Lumix G 20mm F1.7 ASPH Panasonic Lumix G 14mm F2.5 ASPH Olympus M.Zuiko Digital 45mm F1.8 +5 more
Everdog Veteran Member • Posts: 4,837
Re: Wow, Rx100 ISO is off by almost 1 stop!

zxaar wrote:

Everdog wrote:

People complain about the EM5, but the RX100 ( Which I own ) is far worse.

No surprise that you buy worse and inferior things.

Worse and inferior to which P&S?

It is "worse and inferior" to the E-M5, but what camera isn't?
Heck, it's off the charts at DxO!

Keyboard shortcuts:
FForum MMy threads