A Level Playing Field? Not for Sony.

In the recent blog posting regarding the use of Adobe Camera RAW as a standard converter for comparing the performance of different cameras ( http://blog.dpreview.com/editorial/2008/11/adobe-camera-ra.html#more ), reference was made to trying to "create a level playing field" and making "results - as much as possible - comparable between cameras." Also, reference was made to wanting "to show a camera's image output in its "purest" form, i.e. as captured by the sensor and as little as possible altered by sharpening algorithms, noise reduction or any other forms of image processing."

I agree with this approach and have no issue with this approach as described in the blog posting.

But, this approach is not similarly applied when it comes to the use of camera created jpegs, which Phil has stated in another post are "the most relevant and important" ( http://forums.dpreview.com/forums/read.asp?forum=1000&message=29861966 ).

For example, in the recent review of the Sony A900, "fine" jpegs are used for analyzing the image quality of the A900. For the A900, this is a middle level jpeg. The best quality jpeg in the camera is "extra fine." Thus, in the case of the A900 (and other Sony DSLRs), the reviews/tests are not using the image output with as little image processing as possible. By contrast, for each of the other three cameras to which the A900 was compared, the D700, 1DS Mk III and 5D, the use of "fine" jpegs for those cameras represented their highest quality jpegs (according to the dpreview tests for each one). Thus, for all three cameras against which the A900 was compared, the jpegs were the highest quality jpeg the cameras produced, but not the A900.

Nowhere in the review was this stated or described. I have no idea whether there are any visible differences between an "extra fine" and "fine" jpeg from the A900 (or any other Sony DSLR). But, if there are such differences due to the higher compression used to create the smaller file size "fine" jpeg, they would most likely show up in the actual pixel views that are used to compare the image quality of different cameras.

Now, there may be a logical reason why this was done (such as the fact that when shooting RAW+jpeg in the A900 and other Sony's you are limited to a "fine" jpeg, and one assumes that for efficiency reasons the camera testers are shooting RAW+jpeg in order to do a test shot once to get both a RAW and jpeg file). But, this is not in keeping with the previously stated desire to compare cameras on a level playing field.

I have tried raising this privately (through the feedback link), and in questions posted in other threads in which the subject arose, all to no avail - i.e., I have not seen or received a response as to why this different test procedure is used for the Sony cameras. Given that I shoot only RAW, this doesn't really have an impact on me personally, but for the many other site readers that led Phil to state that jpegs are the "most relevant and important," it does.

So, in the absence of an explanation for this anomalous test methodology, I decided to post this comment in a new thread and see what happens.

While it is a criticism of the dpreview tests, it is intended as a constructive one. That is, with the hope that it will lead to more consistent tests which will only serve to improve the reliability and usefulness of the tests.

Finally, as I am on the road and will have no access to the internet for several days after tonight, I will not be able to monitor this thread or respond to any posts made for about six days. So, for those who are interested, I look forward to seeing your comments when I am again able to access the internet.

--
Mark Van Bergh

I chalked the usage of the mid-level JPEG of Sony ("fine") instead of using Sony's highest level of JPEG (termed by Sony as "Extra fine") to compare against the highest-level JPEG in case of the other manufacturers (termed by them as "fine"), including Canon and Nikon, as the lack of understanding of the reviewer or refusal to understand the product terminology used by the manufacturer well enough, either prior to the review or even during the review process.

As we know, when shooting RAW+JPEG, Sony does not allow the shooting of the "Extra Fine" JPEG option and only allows "Fine", and I am assuming that the reviewer shot in RAW+JPEG, rather than taking a RAW shot and another shot with JPEG (Extra Fine)....an extra step that was side-stepped for convenience but in the process compromising the credibility of the review.

He made several other snide comments about the MS stick slot of the A900 being useless etc. and for gathering dust, and (I believe) deliberately (or due to lack of knowledge of the availability of fast MS cards) conducted the camera speed tests with slow 4MB/Sec MS Pro Duo cards, when there were 30MB/Sec MS HG Pro Duo cards available - probably intended to re-inforce his earlier contention that the MS slot provided by Sony (in ADDITION to the CF-slot already present) is useless.

I think his earliest conclusion was understandable (due to lack of knowledge of the availability of fast MS products) but refusal to correct the review, even after people pointed out the obvious mistakes, was a bit difficult to comprehend. Currently, the speed tests conducted with the 4MB/Sec ancient MS pro Duo cards, is amusing to anyone with some understanding of the speeds available with fast MS formats.

The pertinent question is, if the reviewer is unfamilar with the product and/or has not taken the time to study the product adequately, either before or during the review, then has the review itself been compromised ? Are the conclusions then relevant ?

It is not my intent in starting this thread to "bash" the review of the A900 or get into personal criticism of the reviewer. It is more about a methodological flaw in the dpreview testing process that seems to apply only to Sony DSLRs (though I have not checked with respect to Pentax or Olympus and what quality jpeg file is used for those camera reviews). While I may agree that there were other flaws in the A900 review, that isn't really the focus of my comments.

Rather, I prefer to raise the question I did regarding the failure to use comparable level jpeg files for comparison purposes. This, in light of the comments made in the new blog about the stated desire to compare cameras on a level playing field and to maximizing the ability to look at what the sensor is capable of providing by minimizing possible image degradation from in-camera processing, as stated when explaining the use of ACR as a standard RAW converter.

Other possible flaws of the A900 review are not really germane to this limited topic.

I hope people will keep the discussion on track while I'm out of touch.

--
Mark Van Bergh

Which is the default setting? I think that you have two way of testing a camera, serching the best settings can be a time consuming approach, and you can address this in a separate paragraph of the review, using camera defaults instead is a fair way of doing tests because that's the way the manufacturer wants the camera to behave.

Back in the days of cameras like the Fuji S602, similar things occurred in reviews. The "default" settings or the recommended settings from the manual, did not produce the best results. I'm not sure it's the reviewer's job to try to dig through all of the possible combinations of settings and software products to find something better than the manufacturer did or recommends.

Craig Gillette said:

I'm not sure it's the reviewer's job to try to dig through all of the
possible combinations of settings and software products to find
something better than the manufacturer did or recommends.

Click to expand...

I can't accept that rather forgiving point of view. It is the reviewers job to try to get the best from any camera using his knowledge of cameras & photography.

I could excuse a new user if they used the basic bog standard Jpeg settings but anyone reviewing for a living should know better. If the results that arise from using standard settings gives compromised quality then at the very least that should be pointed out & some suggestion given as to how they might be improved. This approach is particularly appropriate for the evaluation of DSLRs - a more casual approach to reviewing might, arguably, be acceptable for reviewing P&S models.

It is just not correct to dismiss the problems by saying the basic settings are the responsibility of the manufacturer who sensibly caters for non-standard conditions by providing the additional settings on the assumption that they will be used when appropriate.

The evaluation of RAW files is a little more problematic as different converter programs behave differently. In addition not all RAW files give the same quality at standard 'default' settings. However to use a variety of settings in various converters would like crossing a minefield & lead to accusations of bias.

Keith-C

I'm not really interested in current SONY products, so I didn't read the A900 review closely...I just skimmed over it.

What I would suggest someone who IS concerned about this apparent bias and/or ignorance do several things:

1) Compare the file sizes of comparable JPEG images from various cameras. If the file size of the SONY "fine" file is comparable to the "fine" files from other manufacturers, then it's appropriate to use it instead of the "extra fine" setting.

2) Use JPEGsnoop and see how the Quality Factor of the A900 "fine" files compares to the competition. If it is comparable, then to use the "extra fine" setting on the A900 would be unfair to the others. ;-)

3) Test the speed of MS vs CF media in the A900. My R1 has two slots just like the A900 and we've found that SONY crippled the CF slot...the MS media works significantly faster. :-( This is one reason I'm not a SONY fan anymore...

--
Charlie Davis
Nikon 5700, Sony R1, Nikon D300
HomePage: http://www.1derful.info
Bridge Blog: http://www.here-ugo.com/BridgeBlog/
'Experience: Discovering that a claw hammer will bend nails.
Epiphany: Discovering that a claw hammer is two tools...'

As a former R1 user, I agree that the CF-slot of the R1 was crippled from a speed perspective and slower than the MS-slot.

However, in case of the dSLRs, the A700 and the A900 write to the CF-card, faster than any other camera in the market, bar none. They write faster than the Nikon D3/D300/D700 and WAY faster than any of the Canons, including uber cameras like the 1DSMKIII and 1DMKIII or even newer ones like the 50D.

Check out Rob Galbraith who posts the write speeds of every "serious" camera in the market, with detailed speed comparisons between various CF-cards.

So you can drop that particular portion of the argument, that prevents you from being a Sony fan. j/k

chuxter said:

3) Test the speed of MS vs CF media in the A900. My R1 has two slots
just like the A900 and we've found that SONY crippled the CF
slot...the MS media works significantly faster. :-( This is one
reason I'm not a SONY fan anymore...

Click to expand...

Craig, there is nothing to dig through.

When they do JPEG tests between cameras, why would you choose a mid-grade JPEG in one model and choose the highest quality JPEG in another model, to compare against ?

If they want to switch to the highest quality JPEG in the Sony (when comparing it to the highest quality JPEG in a Canon or a Nikon), it would take 2 seconds in the menu - I hope a reviewer would know how to do that in a camera he/she is reviewing.

As a RAW-only shooter, JPEG results from in-camera are irrelevant to me but several people seem to make judgement calls even in such high-end cameras, based on the results from a JPEG and so the hope is that a reviewer will spend that extra 2 seconds to make sure that the COMPARISON is done on a level playing field.

Craig Gillette said:

I'm not
sure it's the reviewer's job to try to dig through all of the
possible combinations of settings and software products to find
something better than the manufacturer did or recommends.

Click to expand...

I agree.

Cameras should be tested at their highest resolution settings without any further tweaks. If this gives an "unfair advantage" to more expensive cameras, then so be it. That is precisely why they are more expensive.

The relevancy of testing "basic" Jpeg output at all on the top models is an interersting subject.... it would be a lot like testing race cars in 10 MPH speed zones.

While it is probably true that the majority of entry level DSLR users shoot mostly jpegs, my personal belief is that those people who spend so much money for the top models PROBABLY use the best files that produce the best quality. Why else would they spend so much money for high grade lenses and expensive top end bodies, if image quality wasn't their goal?

Every maker offers multiple choices for jpeg compression. Cannon offers 2 kinds, everyone else offers 3 on their high end models.

On the Canon 5D and 1Ds it's..... normal and fine
On the Nikon D3 it's.................. basic, normal, and fine
On the Olymps E3 it's................ standard, high, and super high
On the Pentax K20D it's............. normal, fine, and super fine
On the Sony A900 it's................ standard, fine and extra fine

Would anyone buy an $8,000 Canon 1Ds III and then shoot jpegs in "normal" to conserve space on a $16 memory card? Or would Nikon D3 owners really use the "basic" setting for jpegs?

I sure don't think so. In fact, I even doubt if 1Ds owners and D3 owners shoot very many jpegs at all.

I own a lowly Olympus E510, and when I shoot jpegs, which is almost always, I only use the "super high" setting.

The Sony A900 costs $3000.... without a lens! It should be tested the way most buyers would use it.... and that means with raw files and highest quality jpeg files only.

--
Marty
http://marty4650.zenfolio.com/p82379129/slideshow#h275db7e6

Panasonic FZ7, FZ20, FZ30, LX2
Olympus C-4000, C-7000
Olympus E-510, Zuiko 12-42mm, 40-150mm

roweraay said:

As a former R1 user, I agree that the CF-slot of the R1 was crippled
from a speed perspective and slower than the MS-slot.

However, in case of the dSLRs, the A700 and the A900 write to the
CF-card, faster than any other camera in the market, bar none. They
write faster than the Nikon D3/D300/D700 and WAY faster than any of
the Canons, including uber cameras like the 1DSMKIII and 1DMKIII or
even newer ones like the 50D.

Click to expand...

Yes, but my point was NOT that SONY didn't have a fast CF write speed...it was that SONY thinks that it's OK to cripple the write speed of non-MS cards...at least they did in the past. I see no data to suggest that the A900 acts this way. But I still believe that SONY is capable of doing dumb, nasty things like this. :-(

roweraay said:

Check out Rob Galbraith who posts the write speeds of every "serious"
camera in the market, with detailed speed comparisons between various
CF-cards.

Click to expand...

Yes, that's a good resource. But I wish he or someone equally competent would compare CF vs MS for the SONY products. The data that dpr publishes is totally worthless. :-(

roweraay said:

So you can drop that particular portion of the argument, that
prevents you from being a Sony fan. j/k

Click to expand...

I like the R1. I don't like SONY. Sorry.

roweraay said:

chuxter said:

3) Test the speed of MS vs CF media in the A900. My R1 has two slots
just like the A900 and we've found that SONY crippled the CF
slot...the MS media works significantly faster. :-( This is one
reason I'm not a SONY fan anymore...

Click to expand...

Click to expand...

--
Charlie Davis
Nikon 5700, Sony R1, Nikon D300
HomePage: http://www.1derful.info
Bridge Blog: http://www.here-ugo.com/BridgeBlog/
'Experience: Discovering that a claw hammer will bend nails.
Epiphany: Discovering that a claw hammer is two tools...'

roweraay said:

Craig, there is nothing to dig through.

When they do JPEG tests between cameras, why would you choose a
mid-grade JPEG in one model and choose the highest quality JPEG in
another model, to compare against ?

Click to expand...

One reason might be that camera A's highest Q JPEG is comparable to camera B's middle Q JPEG? It's also not "fair" to compare the speed of Camera A (that has small JPEG files at it's highest setting) with the speed of camera B (that has large JPEG files at it's highest setting). If dpr did that, then people would complain that they were biased! Unfortunately, dpr can't win this one.

BTW, the A900 has 13.7 MB extra-fine JPEGs and the D700 has 5.3 MP fine JPEGs. The A900 has a faster processor than the D700. Wanna guess why?

--
Charlie Davis
Nikon 5700, Sony R1, Nikon D300
HomePage: http://www.1derful.info
Bridge Blog: http://www.here-ugo.com/BridgeBlog/
'Experience: Discovering that a claw hammer will bend nails.
Epiphany: Discovering that a claw hammer is two tools...'

If you save in Photoshop then you get to chose between quality level 1 (lowest) to 12 (highest). Most DSLRs save at quality level 8 to 10 for Fine JPEGs and you'll have trouble spotting the difference.

chuxter said:

BTW, the A900 has 13.7 MB extra-fine JPEGs and the D700 has 5.3 MP
fine JPEGs. The A900 has a faster processor than the D700. Wanna
guess why?

Click to expand...

The A900 (24.6MP) has over twice the megapixels of the D700 (12.1 MP), which means the files from the A900 (and the detail contained with it) would be much larger than the lower resolution D700 (my first blush guess on this).

For a move even-handed comparison (which clears things up a bit), the file sizes in the Nikon D300 are as follows:

JPEG Fine: 4.2MB
Loss-less 12-bit uncompressed RAW: 18.7MB.

The sizes in the similar Sony A700 are as follows:

JPEG Fine: 4.1MB (comparable size to the Nikon Fine !)

JPEG X-Fine: 9.1MB (which means the X-fine has higher quality, which is MIA in case of Nikon)
Loss-less 12-bit uncompressed RAW: 18.6MB

So it seems like the folks in dpreview (who have clearly shot with X-fine, since their timing tests have indicated that), have deliberately used the mid-level JPEG of the Sony cameras, for comparison to the competition's top-level JPEG, just because the competition does not even provide the highest level of JPEG provided in Sony cameras. So dpreview does not want Sony one-upping their blue-eyed-boys, by even attempting this comparison ?

The A900's RAW files are around 38-40MB in size and when shooting at its 5FPS max, RAW+JPEG, the A900 is moving around 240-250MB of data, per SECOND vs around 155MB/Sec in case of the 1DSMKIII, when shooting at 5FPS, RAW+JPEG.

That means the data pipeline architecture and the electronics of the A900, are comparable to the D3 rather than the cheaper D700 and the 1DSMKIII rather than the cheaper 5DII. Also note that the RAW files of the $8000 1DSMKIII at full resolution, is only 24-25MB in size, which means even the 1DSMKIII is not processing or moving as much data as the A900, when both cameras are shooting at their max speed of 5FPS.

The Sony is not only writing to the CF-card faster than any of these, but the internal architecture of the camera is also proportionately superior.

The 100% viewfinder of the A900 is also larger than that of the D3 and the Canon 1DSMKII, let alone that of the sub-100% little brothers, the D700 or the 5DII.

you show us an obvious and visible difference between Sony Fine and Extra Fine JPEGs and we'll concede that you may have a point.

Why on earth would you compare cameras based on file size and not on optical results?

It's not like memory cards are terribly expensive, and we always select cameras to get the best performance from smaller files.

You seem to be suggesting that a Sony "middle level file size" SHOULD be compared to someone else's "high level file" because the file sizes are similar. And I think it makes far more sense to compare the best file a camera can produce against the best file another camera can produce. And that means, RAW vs RAW, and "best jpegs" vs. "best jpegs." And then the reviewer should note that one brands files are larger than the other, since that might matter to someone.

You can craft an argument for comparing similar file sizes, but that won't be very useful for folks who simply want the best results.
--
Marty
http://marty4650.zenfolio.com/p82379129/slideshow#h275db7e6

Panasonic FZ7, FZ20, FZ30, LX2
Olympus C-4000, C-7000
Olympus E-510, Zuiko 12-42mm, 40-150mm

It seems everywhere you turn, there is a Sony user moaning about unfair treatment.

I'll never have thought any group could out do the Olympus bunch in the chip on the shoulder sakes but jeeze, these guys really take the biscuit.

The Memory Shtick must be put to pasture so yes, it is good that the reviewer took a pop at it.

GET OVER IT AND PUT A FREAKING SOCK IN IT!

--
http://dakanji.com

'I make statements based on fact not predictions.'
KMSEA: 12:33:17 PM, Saturday, November 12, 2005 (GMT)

DPR have two options here:

1. Shoot in the highest level of JPEG each camera supports
2. Shoot in the highest level of JPEG all cameras compared support

Now, it seems very likely to me that there would be no visible difference in any case, but lets suppose there is.

If they take the first route, then the 'better' pictures are a mixture of lens, sensor, processor and jpeg compression level.

If they take the second route then the 'better' pictures are a mixture of lens, sensor, and processor.

Now, what kind of user would benefit from 1, and what kind would benefit from 2?

A user who never, ever, shot in RAW, but wanted to know what the best possible IQ output from the camera was, would prefer to see 1.

A user who shot in a mixture of formats, or a user who shot only in RAW, would benefit from option 2, because that would be showing them strengths and weakness in factors that affect all image output formats, without confusing the comparison by introducing JPEG compression rates into the mix.

Now, I suggest that while there are some new DSLR users who don't want to get into RAW, and just want to know what the 'best' jpegs they'll get from the camera are, I think more people fall into category 2.

All that said - if it were a review of cheap point and shoot compacts, there would be much more point comparing the manufacturers 'best' jpeg output of each camera, as people will be less interested in comparing the nuances of one sensor or processor, and will just want to know "what do the best pictures come out like?"

J

JonP said:

DPR have two options here:

1. Shoot in the highest level of JPEG each camera supports
2. Shoot in the highest level of JPEG all cameras compared support

Click to expand...

[snip]

What you're assuming here is that one manufacturer's definition of 'fine' is the same as another's.

What if Sony had called their highest level 'fine' instead of 'Xtra Fine'?

It's just a name. And a file size. Sony's top level jpg runs about 20meg and their next level down about 9meg.

I have to admit I don't see any difference between the two. But I'd rather shoot cRAW anyway (approx 24meg file size).

All Sony has to do is run all their images through the strongest, detail-killing in-camera NR and it'll match the others and make the reviewers happy.
--



'I cried because I had no E-3. Then I met a man with no E-510'

Olympus E-410, Nikon D100 (IR) & Pentax K20D.
57 lenses of various types from most brands.