DxOMark still stumped by the E-M5?

Started Jul 23, 2012 | Discussions
sderdiarian
Senior MemberPosts: 3,071Gear list
Like?
DxOMark still stumped by the E-M5?
Jul 23, 2012

The E-M5's been on the street for a couple of months now and all major camera sites have tested it, including DxOMark, yet their results have yet to be posted.

One could conclude they're first waiting to report on Canon's new MILC, but that can't be the case, being a highly professional site. So where's the hitch, DxO? Has this new Olympus mFT camera really got you that stumped?

It's gotten to be pretty much a matter of curiosity for me, having seen enough image comparisons to realize Olympus pulled off some magic with the new Sony sensor. I can only hope Sony also shares their RX100 sensor with the upcoming XZ-2, making it the perfect compliment to the E-M5, right down to Olympus colors and IBIS.
--
Sailin' Steve

 sderdiarian's gear list:sderdiarian's gear list
Olympus OM-D E-M5 Panasonic Lumix G 20mm F1.7 ASPH Olympus M.Zuiko Digital ED 12-50mm 1:3.5-6.3 EZ Olympus M.Zuiko ED 75-300mm 1:4.8-6.7 II
Banger
Contributing MemberPosts: 575Gear list
Like?
Highly professional site ????
In reply to sderdiarian, Jul 23, 2012

I don't think so ...

Rgds, Rob

 Banger's gear list:Banger's gear list
Olympus E-30 Olympus PEN E-P3 Olympus E-M1
Reply   Reply with quote   Complain
3DrJ
Senior MemberPosts: 1,027Gear list
Like?
Re: DxOMark still stumped by the E-M5?
In reply to sderdiarian, Jul 23, 2012

sderdiarian wrote:

The E-M5's been on the street for a couple of months now and all major camera sites have tested it, including DxOMark, yet their results have yet to be posted.

One could conclude they're first waiting to report on Canon's new MILC, but that can't be the case, being a highly professional site. So where's the hitch, DxO? Has this new Olympus mFT camera really got you that stumped?

Maybe they are confused and frightened by the EM-5, but that itself is puzzling since as you say, it's been evaluated by at least a couple of dozen reviewers already. There are those who question how "professional" their results happen to be, and maybe that's what you are getting at. If really "professional" what's the big deal? The EM-5 can't be all that different from other cameras, can it?

OTOH, we could also ask if it matters what DXO publishes. Does it change any actual characteristic of the camera? Seems to me there isn't a whole lot of reason to care whatever they finally report. Looks like they just missed the boat.

JRA

-- hide signature --

Artists must not only see, but see what they are seeing.

 3DrJ's gear list:3DrJ's gear list
Olympus E-1 Olympus E-3 Olympus PEN E-P3 Olympus OM-D E-M5 Olympus E-M1 +5 more
Reply   Reply with quote   Complain
3DrJ
Senior MemberPosts: 1,027Gear list
Like?
Re: Highly professional site ????
In reply to Banger, Jul 23, 2012

Curiously, that's what I was writing about when you wrote the above. What leads you to doubt their "professionalism"?

JRA

-- hide signature --

Artists must not only see, but see what they are seeing.

 3DrJ's gear list:3DrJ's gear list
Olympus E-1 Olympus E-3 Olympus PEN E-P3 Olympus OM-D E-M5 Olympus E-M1 +5 more
Reply   Reply with quote   Complain
sderdiarian
Senior MemberPosts: 3,071Gear list
Like?
Re: Highly professional site ????
In reply to Banger, Jul 23, 2012

Banger wrote:

I don't think so ...

Must admit to being tongue in cheek, knowing how their results punish cameras that start at ISO200 as Olympus does. I would absolutely love to be proven very wrong by some positive results.

Their continued silence does makes me wonder, either they're afraid to publish poor results for a camera that's received universal praise, proving their methods are of questionable fairness, or they know Olympus managed to game their testing system and are afraid to admit it.

Either way, entertaining. Please pass the popcorn...
--
Sailin' Steve

 sderdiarian's gear list:sderdiarian's gear list
Olympus OM-D E-M5 Panasonic Lumix G 20mm F1.7 ASPH Olympus M.Zuiko Digital ED 12-50mm 1:3.5-6.3 EZ Olympus M.Zuiko ED 75-300mm 1:4.8-6.7 II
Reply   Reply with quote   Complain
Banger
Contributing MemberPosts: 575Gear list
Like?
Re: DxOMark still stumped by the E-M5?
In reply to 3DrJ, Jul 23, 2012

I've always wondered about their methodology for many of their measurements and felt that much is irelevant to real world photography and image quality. There have been many debates about this on DPR which you can search for if you are interested.

In any event the delay in getting out testing results on the E-M5 (I don't have one), kinda makes me wonder what they are up to, not that I would put much credence in anything they say......good or bad.

Rgds, Rob

 Banger's gear list:Banger's gear list
Olympus E-30 Olympus PEN E-P3 Olympus E-M1
Reply   Reply with quote   Complain
3DrJ
Senior MemberPosts: 1,027Gear list
Like?
Re: DxOMark still stumped by the E-M5?
In reply to Banger, Jul 23, 2012

Banger wrote:

I've always wondered about their methodology for many of their measurements and felt that much is irelevant to real world photography and image quality. There have been many debates about this on DPR which you can search for if you are interested.

In any event the delay in getting out testing results on the E-M5 (I don't have one), kinda makes me wonder what they are up to, not that I would put much credence in anything they say......good or bad.

Yes, I've seen and read (some) of those threads. In one fairly recent discussion, participants came to agree DXO's results are ambiguous, since basic data, such as sample size and sample source, is not provided so we are unable to apply results to the run of EM-5's in general.

Relevance of the testing results aside, the issue becomes "if you look guilty, people will think you are guilty". The unaccountable delay when consumers have been clamoring for it, leaves the result appearing tainted and untrustworthy, even if heavenly pure. By saying nothing at all and leaving the matter in doubt, it becomes a case of "shooting oneself in the foot".

JRA

-- hide signature --

Artists must not only see, but see what they are seeing.

 3DrJ's gear list:3DrJ's gear list
Olympus E-1 Olympus E-3 Olympus PEN E-P3 Olympus OM-D E-M5 Olympus E-M1 +5 more
Reply   Reply with quote   Complain
highwave
Senior MemberPosts: 1,037Gear list
Like?
Re: DxOMark still stumped by the E-M5?
In reply to sderdiarian, Jul 23, 2012

DxO has been absolutely despicable when it comes to the OM-D results

They provide support for RAW and yet refuse to publish their results which they clearly have and probably had for a long while.

And they claim they have some technical issues they wanted to check about the camera!

They should respect consumers who give them credit for their work and not hide information they have for whatever motives.

They loose credibility

 highwave's gear list:highwave's gear list
Olympus OM-D E-M5 Panasonic Lumix G 20mm F1.7 ASPH Panasonic Lumix G 14mm F2.5 ASPH Olympus M.Zuiko Digital 45mm 1:1.8 Panasonic Lumix G Vario 45-150mm F4-5.6 ASPH Mega OIS +3 more
Reply   Reply with quote   Complain
bobn2
Forum ProPosts: 26,095
Like?
Re: DxOMark still stumped by the E-M5?
In reply to highwave, Jul 23, 2012

highwave wrote:

DxO has been absolutely despicable when it comes to the OM-D results

They provide support for RAW and yet refuse to publish their results which they clearly have and probably had for a long while.

And they claim they have some technical issues they wanted to check about the camera!

They should respect consumers who give them credit for their work and not hide information they have for whatever motives.

They loose credibility

It all begs the question of why a group of people who so dislike DxOmark's tests and proclaim them to be irrelevant to real photography are so keen for them to be published and so critical of DxO for the delay. DxO have said on the forum that they had some surprising results and are retesting to check. That sounds to me what any professional testing lab would do if they got unexpected results.
--
Bob

Reply   Reply with quote   Complain
boggis the cat
Veteran MemberPosts: 6,267Gear list
Like?
DxOMark impartiality in question
In reply to bobn2, Jul 23, 2012

bobn2 wrote:

highwave wrote:

DxO has been absolutely despicable when it comes to the OM-D results

They provide support for RAW and yet refuse to publish their results which they clearly have and probably had for a long while.

And they claim they have some technical issues they wanted to check about the camera!

They should respect consumers who give them credit for their work and not hide information they have for whatever motives.

They loose credibility

It all begs the question of why a group of people who so dislike DxOmark's tests and proclaim them to be irrelevant to real photography are so keen for them to be published and so critical of DxO for the delay.

Well, Bob, my view is that it begs the question of DxO's impartiality.

I noticed that they were quite happy to rubbish the Canon 5D III improvements over the 5D II. Yet they never met a Sony sensor they didn't like.

DxO have said on the forum that they had some surprising results and are retesting to check.

How many weeks do they need to re-test?

That sounds to me what any professional testing lab would do if they got unexpected results.

So what did they "expect"?

Did Sony not let them know they were using one of their sensors?

As it happens, I used to work within a product testing laboratory. When you find questionable or suspicious results you double-check then retest immediately using the same methodology. If the results still seem "unexpected" then your methodology is flawed or the product is somehow gaming the testing. If the test method is published and prescriptive then you may be able to game the test (but any decent lab will catch you out anyway, as they don't trust narrowly prescriptive tests for that very reason) -- but DxO's methodology is murky, so how would you game it?

 boggis the cat's gear list:boggis the cat's gear list
Olympus E-5 Olympus E-M1 Olympus Zuiko Digital ED 12-60mm 1:2.8-4.0 SWD Olympus Zuiko Digital ED 50-200mm 1:2.8-3.5 SWD Olympus 12-40mm F2.8 +10 more
Reply   Reply with quote   Complain
zxaar
Veteran MemberPosts: 3,122
Like?
Re: DxOMark impartiality in question
In reply to boggis the cat, Jul 23, 2012

boggis the cat wrote:

bobn2 wrote:

highwave wrote:

DxO has been absolutely despicable when it comes to the OM-D results

They provide support for RAW and yet refuse to publish their results which they clearly have and probably had for a long while.

And they claim they have some technical issues they wanted to check about the camera!

They should respect consumers who give them credit for their work and not hide information they have for whatever motives.

They loose credibility

It all begs the question of why a group of people who so dislike DxOmark's tests and proclaim them to be irrelevant to real photography are so keen for them to be published and so critical of DxO for the delay.

Well, Bob, my view is that it begs the question of DxO's impartiality.

I noticed that they were quite happy to rubbish the Canon 5D III improvements over the 5D II. Yet they never met a Sony sensor they didn't like.

DxO have said on the forum that they had some surprising results and are retesting to check.

How many weeks do they need to re-test?

Sure thats the only thing these guys are busy doing.

That sounds to me what any professional testing lab would do if they got unexpected results.

So what did they "expect"?

Something that is "normal" from a sensor.

Did Sony not let them know they were using one of their sensors?

You have to ask this to Sony. This is not the forum to get answer on this.

As it happens, I used to work within a product testing laboratory.

You seems to have only 1 testing job based on your comment above.

When you find questionable or suspicious results you double-check then retest immediately using the same methodology.

Shouldn't you get the same result with same methodology.

If the results still seem "unexpected" then your methodology is flawed

May be it is, but this does not tell what the flaw is and probably they are working on finding the answer to that question. As any impartial and professional site would do.

or the product is somehow gaming the testing. If the test method is published and prescriptive then you may be able to game the test (but any decent lab will catch you out anyway, as they don't trust narrowly prescriptive tests for that very reason) -- but DxO's methodology is murky, so how would you game it?

They are open to change you, have to convince or prove to them that they are wrong. I am sure the way you talk you would have no difficulty in showing them that they are wrong. Go ahead.

Everyone wants perfect methodology, teach them that.

-- hide signature --

::> Knowledge is mother of efficiency.

Reply   Reply with quote   Complain
Leo
Leo
Veteran MemberPosts: 5,820
Like?
Highly professional - one of the best. I think ...
In reply to Banger, Jul 23, 2012

Banger wrote:

I don't think so ...

Rgds, Rob
--

That it is Olympus who slowed the DXO review.
Leo

Reply   Reply with quote   Complain
rrr_hhh
Veteran MemberPosts: 5,894Gear list
Like?
Re: DxOMark still stumped by the E-M5?
In reply to bobn2, Jul 23, 2012

bobn2 wrote:

It all begs the question of why a group of people who so dislike DxOmark's tests and proclaim them to be irrelevant to real photography are so keen for them to be published and so critical of DxO for the delay. DxO have said on the forum that they had some surprising results and are retesting to check. That sounds to me what any professional testing lab would do if they got unexpected results.

Plus one ! I agree with you. However, it is taking them a lot of time to retest ! At one point it may be better for them to publish their results and tell us why they think those results may be wrong, or what they need to retest.

May be that they are like me : not able to get E-M5s to test !
--
rrr_hhh

 rrr_hhh's gear list:rrr_hhh's gear list
Olympus PEN E-P3 Olympus OM-D E-M5 Canon EOS 6D Canon EF 50mm f/1.4 USM Canon EF 16-35mm f/2.8L USM +18 more
Reply   Reply with quote   Complain
bobn2
Forum ProPosts: 26,095
Like?
Re: DxOMark impartiality in question
In reply to boggis the cat, Jul 23, 2012

boggis the cat wrote:

bobn2 wrote:

highwave wrote:

DxO has been absolutely despicable when it comes to the OM-D results

They provide support for RAW and yet refuse to publish their results which they clearly have and probably had for a long while.

And they claim they have some technical issues they wanted to check about the camera!

They should respect consumers who give them credit for their work and not hide information they have for whatever motives.

They loose credibility

It all begs the question of why a group of people who so dislike DxOmark's tests and proclaim them to be irrelevant to real photography are so keen for them to be published and so critical of DxO for the delay.

Well, Bob, my view is that it begs the question of DxO's impartiality.

My view is that your view begs the question of your impartiality.

I noticed that they were quite happy to rubbish the Canon 5D III improvements over the 5D II. Yet they never met a Sony sensor they didn't like.

You probably should take advice from someone who has expertise in lab work and making measurements. The process of measuring produces measurements. They have nothing to do with liking or not liking something. Now, one can choose the set of measurements that favours one item over another, but I haven't seen any suggestion that DxO is going to change what it measures just for the E-M5.

DxO have said on the forum that they had some surprising results and are retesting to check.

How many weeks do they need to re-test?

I don't know. It depends on what they have going through their labs, and what is the nature of what they have found. I would think they have no commercial advantage in delaying the test any more than necessary.

That sounds to me what any professional testing lab would do if they got unexpected results.

So what did they "expect"?

They have tested many sensors and understand how they work. That gives them the base knowledge to know the expected envelope of performance and recognise an outlier when they see it.

Did Sony not let them know they were using one of their sensors?

That is quite a despicable slur on DxO, and shows you up more than it does them. There is no reason from their results to conclude any Sony bias. When Sony sensors weren't as good as the competition, DxO measured them not as goo. Look at the D2X, for instance. When Sony put in the R&D effort to start making the best sensors there are, DxO measured them as the best. There is no reason to suspect anything other than Sony sensors measure well at DxO because they are very good.

As it happens, I used to work within a product testing laboratory.

What was your job description?

When you find questionable or suspicious results you double-check then retest immediately using the same methodology. If the results still seem "unexpected" then your methodology is flawed or the product is somehow gaming the testing.

No-one has yet said that the E-M5 results were 'questionable' or 'suspicious', because those are both very valued judgements. Most labs would retest if they found an outlier, simply to check that there hadn't been some deviation from protocol. If you get the same result again, then it seems that you truly have an outlier. For product testing sometimes then, I would think, you would find another sample, just in case the outlier results were the result of a 'Friday afternoon job'. If the second sample confirms outlier status, then possibly you'd go and publish.

We already know from DM's investigation that the raw files in the E-M5 are scaled very low (that is that the saturation 'ISO' that DxO measures is very low with respect to the metering ISO). That to an extent is gaming DxO's system, because if, for instance, you are testing a 200 ISO setting with an exposure nominal for 100 ISO the results will look much better than they would be in real life. Suppose the E-M5 metered ISO's were more than a stop higher than the 'saturation' ISO - that would beg a difficult question for DxO, not about the testing, but how to present the results.

If the test method is published and prescriptive then you may be able to game the test (but any decent lab will catch you out anyway, as they don't trust narrowly prescriptive tests for that very reason) -- but DxO's methodology is murky, so how would you game it?

DxO's methodology is not 'murky'. The explain it at length and in considerably more detail than any other test site. Certainly, the detail of the experimental methodology is about the same as would be expected in a refereed paper (even if the style of narrative is different).

-- hide signature --

Bob

Reply   Reply with quote   Complain
texinwien
Senior MemberPosts: 2,529Gear list
Like?
Beg The Question - Get It Right
In reply to bobn2, Jul 23, 2012

bobn2 wrote:

boggis the cat wrote:

bobn2 wrote:

highwave wrote:

It all begs the question of why a group of people who so dislike DxOmark's tests and proclaim them to be irrelevant to real photography are so keen for them to be published and so critical of DxO for the delay.

Well, Bob, my view is that it begs the question of DxO's impartiality.

My view is that your view begs the question of your impartiality.

Suppose the E-M5 metered ISO's were more than a stop higher than the 'saturation' ISO - that would beg a difficult question for DxO, not about the testing, but how to present the results.

http://begthequestion.info/

"Begging the question" is a form of logical fallacy in which a statement or claim is assumed to be true without evidence other than the statement or claim itself.

To beg the question does not mean "to raise the question." (e.g. "It begs the question, why is he so dumb?") This is a common error of usage made by those who mistake the word "question" in the phrase to refer to a literal question.

I beg you all - please stop misusing this phrase.

 texinwien's gear list:texinwien's gear list
Canon EOS 300D Olympus OM-D E-M5 Canon EF-S 18-55mm f/3.5-5.6 Panasonic Lumix G 20mm F1.7 ASPH Panasonic Lumix G Vario 100-300mm F4-5.6 OIS +14 more
Reply   Reply with quote   Complain
highwave
Senior MemberPosts: 1,037Gear list
Like?
Re: DxOMark still stumped by the E-M5?
In reply to bobn2, Jul 23, 2012

bobn2 wrote:

highwave wrote:

DxO has been absolutely despicable when it comes to the OM-D results

They provide support for RAW and yet refuse to publish their results which they clearly have and probably had for a long while.

And they claim they have some technical issues they wanted to check about the camera!

They should respect consumers who give them credit for their work and not hide information they have for whatever motives.

They loose credibility

It all begs the question of why a group of people who so dislike DxOmark's tests and proclaim them to be irrelevant to real photography are so keen for them to be published and so critical of DxO for the delay. DxO have said on the forum that they had some surprising results and are retesting to check. That sounds to me what any professional testing lab would do if they got unexpected results.
--
Bob

I personally hold DxO with utmost respect for their work. I'm not technically familiar with their work to the point of being able to criticize it but I do have to admit it does seem to have more attention to detail than any other work I see on the internet. So I give them more credit than others out there.

But honestly, doesn't their delay of publishing results (even after they provide RAW support) raise your suspicion even the slightest? Bare in mind there are a lot of claims spread around forums that there are tie ups between Sony and DxO. I'm not suggesting that DxO would falsify results in favor of Sony but is it really absolutely unquestionable that DxO might delay publishing results if that would please Sony?

 highwave's gear list:highwave's gear list
Olympus OM-D E-M5 Panasonic Lumix G 20mm F1.7 ASPH Panasonic Lumix G 14mm F2.5 ASPH Olympus M.Zuiko Digital 45mm 1:1.8 Panasonic Lumix G Vario 45-150mm F4-5.6 ASPH Mega OIS +3 more
Reply   Reply with quote   Complain
Detail Man
Forum ProPosts: 14,837
Like?
Re: DxOMark still stumped by the E-M5?
In reply to highwave, Jul 23, 2012

highwave wrote:

... there are a lot of claims spread around forums that there are tie ups between Sony and DxO.

I have looked (then as well as now), and all that I can find is our own conversation starting here:

http://forums.dpreview.com/forums/read.asp?forum=1041&message=41801420

Reply   Reply with quote   Complain
highwave
Senior MemberPosts: 1,037Gear list
Like?
Re: DxOMark still stumped by the E-M5?
In reply to Detail Man, Jul 23, 2012

Detail Man wrote:

highwave wrote:

... there are a lot of claims spread around forums that there are tie ups between Sony and DxO.

I have looked (then as well as now), and all that I can find is our own conversation starting here:

http://forums.dpreview.com/forums/read.asp?forum=1041&message=41801420

If I recall properly, you're one of the posters who said they read about the Sony/DxO tie up news on the internet before it vanished.

I hope I'm not mixing you up with someone else DM.

 highwave's gear list:highwave's gear list
Olympus OM-D E-M5 Panasonic Lumix G 20mm F1.7 ASPH Panasonic Lumix G 14mm F2.5 ASPH Olympus M.Zuiko Digital 45mm 1:1.8 Panasonic Lumix G Vario 45-150mm F4-5.6 ASPH Mega OIS +3 more
Reply   Reply with quote   Complain
Detail Man
Forum ProPosts: 14,837
Like?
Re: DxOMark still stumped by the E-M5?
In reply to highwave, Jul 23, 2012

highwave wrote:

Detail Man wrote:

highwave wrote:

... there are a lot of claims spread around forums that there are tie ups between Sony and DxO.

I have looked (then as well as now), and all that I can find is our own conversation starting here:

http://forums.dpreview.com/forums/read.asp?forum=1041&message=41801420

If I recall properly, you're one of the posters

That would be me. I do not know of one single other poster on the internet. I believe that your memory is perhaps failing you. You (and any other interested readers) need to read this post:

http://forums.dpreview.com/forums/read.asp?forum=1041&message=41807334

... who said they read about the Sony/DxO tie up news on the internet before it vanished.

I would suggest that you read our previous conversation, where we already discussed these things

You expressed suspicions (of some sort) when it was not understood that Sony manufactures the image-sensor in the E-M5. Now that it is understood that Sony does manufacture the image sensor in the E-M5, you seem to harbor new suspicions (of some entirely undefined sort). Not interested.

Reply   Reply with quote   Complain
Jorginho
Veteran MemberPosts: 6,312Gear list
Like?
I am not into conspiracy theory's, but here's one.
In reply to sderdiarian, Jul 23, 2012

The sensor is very good. If DxO shows it, this means some of their important clients (Nikon, Sony) lose marketshare. So you wait and wait and wait until a newer (and better) APS-c is due to come out. You publish the results and as soon as you can, you publish the results of the new APS-c sensor in the new NEX. And Oly's advantage has dissaepeared in no time...

It is actually the only negative thing I can come up with that could make some sense in my view.

Other than that I trust them and may be they are very thorough and indeed have some problems. is that so impossible? Let's wait and see.

 Jorginho's gear list:Jorginho's gear list
Panasonic Lumix DMC-G1 Panasonic Lumix DMC-GH2 Olympus PEN E-PL5 Panasonic Lumix G Vario 14-45mm F3.5-5.6 ASPH OIS Panasonic Lumix G 20mm F1.7 ASPH +7 more
Reply   Reply with quote   Complain
Keyboard shortcuts:
FForum MMy threads