"That's Not a Good Test!"

[chkproductions]

It seems to me that no matter what forum I'm on, when someone, out of the goodness of their heart and taking up of their time, post some test - camera to camera, sensor to sensor, lens to lens, iso to iso, whatever - someone posts, "That's Not a Good Test!"

Then what is?

Can you all help those who put in their heart and time with some guidelines to create valid tests that will be accepted by the majority of DP members so you don't get 3 posts in and see the dreaded "That's Not a Good Test!" post.

Even when using DP's own test charts there is a post that will say "That's Not a Good Test!"

I'm interested because I'd like to see what the majority accepts as a valid test so when I'm reading through a post I can feel confident in the results.

Thanks

chk

 

[Dennis]

It seems to me that no matter what forum I'm on, when someone, out of the goodness of their heart and taking up of their time, post some test - camera to camera, sensor to sensor, lens to lens, iso to iso, whatever - someone posts, "That's Not a Good Test!"

Then what is?

Can you all help those who put in their heart and time with some guidelines to create valid tests that will be accepted by the majority of DP members so you don't get 3 posts in and see the dreaded "That's Not a Good Test!" post.

It's easier to find fault in a test than to devise a good test. In fact, the ease of finding fault illustrates that. It's so difficult to design a good test that it's probably just best left to those who do it well. Why does some newbie feel obligated to throw up (pun intended) snapshots taken with two cameras he just happens to own "just in case" it "might be beneficial" to some mythical buyer who's in a quandry trying to decide between those two cameras based on how they handle snapshots in P mode ? (I saw one yesterday where the photos were taken handheld, with slightly differing composition, leading one to see bigger light sources which could have affected the metering ... this could lead someone to think that that camera underexposes).

I'm interested because I'd like to see what the majority accepts as a valid test so when I'm reading through a post I can feel confident in the results.

It really depends on what you're trying to accomplish. No comparison is going to be equally meaningful to everyone, because everyone shoots differently. Consider a low light test between the EM1 and D7100. You probably want to keep the shutter speed the same. Then, do you shoot both at the same f-stop ? Are you trying to maintain the same depth of field with your low light shooting, in which case you need appropriate f-stops ? Arbitrarily equal f-stops just to be "fair" ? Whatever f-stop you need to get some degree of sharpness ? (Maybe one lens is better wide open than the other ?) Or whatever f-stop is available to you in the lenses you have, because no matter which system you shoot, you'll just open up the lens in low light ? Or do you equalize the ISO ? Again, how do you shoot ? I shoot Auto ISO in M mode, so I have no interest in same ISO - I want same brightness from whatever exposure settings I'd use, based on which lenses I'd use. (At least dxomark maps out SNR versus measured ISO so you can compare based on knowledge of what lenses you'd use). I know, for instance, that I shoot an f/2.8 zoom and a couple f/1.8 primes with my D7000 and that if I were to use an EM1, I'd use an f/2.8 zoom and a couple f/1.8 primes, so I'd be happy to see a comparison at same f-stop without worrying about stopping down the APS-C option to match DOF. I just look for how relevant the test is to me (P mode isn't; same ISO at different exposure settings usually isn't; in-camera jpeg isn't) and then look for sources of error that could indicate that the results aren't necessarily what I'll see. Really, between dpreview, imaging-resource, dxomark, slrgear, lenstip, photozone I see no need for amateur tests, no matter how well-intentioned. At best, they tend to be redundant, and at worst, misleading.

• Dennis
  --
  Gallery at http://kingofthebeasts.smugmug.com

 

[darklamp]

As you say the posts are made by people who want to help others, but so are the posts warning people that the tests are not well done.

Both are equally valid reasons to post.

Why don't they explain how to test properly ? Because it would simply take too long. Good tests are complex and there are lots of pitfalls for the unwary.

In fact hood tests are often detailed on the internet by testing firms like ImaTest. The reaction of the "helpful amateur" is these are too complex. They're complex for a reason - good tests are complex and they yield complex results that are often hard to interpret if you're not experienced.

 

[Klaus dk]

A good test is a test showing the gear I own is better than the rest!

Even the best tests are dismissed because they did or did not count in ergonomics, economy, child labor, weight, sensor size, high ISO performance, low ISO noise, dynamic range, flare, resolution, jpeg-engine, size, file size, RAW results, distortion, distortion correction ... ad nauseam ad infinitum.

Just tell me my gear is allright so I can get on with my life.

Have a nice weekend. :-D

 

[Dennis]

I hereby recall my answer ! Klaus has spoken the truth - nicely done !

• Dennis
  --
  Gallery at http://kingofthebeasts.smugmug.com

 

[enemjii]

It seems to me that no matter what forum I'm on, when someone, out of the goodness of their heart and taking up of their time, post some test - camera to camera, sensor to sensor, lens to lens, iso to iso, whatever - someone posts, "That's Not a Good Test!"

Then what is?

Can you all help those who put in their heart and time with some guidelines to create valid tests that will be accepted by the majority of DP members so you don't get 3 posts in and see the dreaded "That's Not a Good Test!" post.

Even when using DP's own test charts there is a post that will say "That's Not a Good Test!"

I'm interested because I'd like to see what the majority accepts as a valid test so when I'm reading through a post I can feel confident in the results.

Thanks

chk

 

[Richard]

But not all tests are valid. Even comparing dissimilar cameras, lenses settings. It lets us see what happens under those conditions, but it may not be a valid test comparing one DOF to another using different format.

A statement like my previous Nikon was cooler in color but not this one but it is not as warm as the canon. Even subjective tests like this help us understand the differences but are not scientific. You cannot then say

That makes the Nikon better than the canon. Then someone else is going to say that is a bad test, when they really mean it is not valid.

chkproductions wrote:

It seems to me that no matter what forum I'm on, when someone, out of the goodness of their heart and taking up of their time, post some test - camera to camera, sensor to sensor, lens to lens, iso to iso, whatever - someone posts, "That's Not a Good Test!"

Then what is?

Can you all help those who put in their heart and time with some guidelines to create valid tests that will be accepted by the majority of DP members so you don't get 3 posts in and see the dreaded "That's Not a Good Test!" post.

Even when using DP's own test charts there is a post that will say "That's Not a Good Test!"

I'm interested because I'd like to see what the majority accepts as a valid test so when I'm reading through a post I can feel confident in the results.

Thanks

chk

--
http://chkphotography.zenfolio.com/

 

[DenWil]

Most of the tests I see are not relevant to what i do for a living so they are de facto, not good tests.

Take all the FF 35 mm cameras. Fill the entire frame with a great face, using great natural light, all with the same 100mm Zeiss at f4, process all in Capture 1 and enlarge to 20 x 30, place side by side on the wall and compare. That is a valid test for me.

 

[Bill Robb]

It seems to me that no matter what forum I'm on, when someone, out of the goodness of their heart and taking up of their time, post some test - camera to camera, sensor to sensor, lens to lens, iso to iso, whatever - someone posts, "That's Not a Good Test!"

Then what is?

A good test is repeatable, and is only testing for one variable.
Most ad hoc tests fail in both of these parameters.

 

[Bill Robb]

Most of the tests I see are not relevant to what i do for a living so they are de facto, not good tests.

Take all the FF 35 mm cameras. Fill the entire frame with a great face, using great natural light, all with the same 100mm Zeiss at f4, process all in Capture 1 and enlarge to 20 x 30, place side by side on the wall and compare. That is a valid test for me.

Right away, your test fails. Natural light is anything but consistent.

 

[69chevy]

It seems to me that no matter what forum I'm on, when someone, out of the goodness of their heart and taking up of their time, post some test - camera to camera, sensor to sensor, lens to lens, iso to iso, whatever - someone posts, "That's Not a Good Test!"

Then what is?

Can you all help those who put in their heart and time with some guidelines to create valid tests that will be accepted by the majority of DP members so you don't get 3 posts in and see the dreaded "That's Not a Good Test!" post.

Even when using DP's own test charts there is a post that will say "That's Not a Good Test!"

I'm interested because I'd like to see what the majority accepts as a valid test so when I'm reading through a post I can feel confident in the results.

Thanks

chk

 

[D Cox]

It seems to me that no matter what forum I'm on, when someone, out of the goodness of their heart and taking up of their time, post some test - camera to camera, sensor to sensor, lens to lens, iso to iso, whatever - someone posts, "That's Not a Good Test!"

Then what is?

A good test is one with only one variable. For instance, if comparing two cameras, a good test would use the same individual lens (not two copies of "the same" lens) on both.

Or if comparing two lenses, they would be put on the same camera, so that the lens is the only variable.

Lighting must of course be exactly the same.

Can you all help those who put in their heart and time with some guidelines to create valid tests that will be accepted by the majority of DP members so you don't get 3 posts in and see the dreaded "That's Not a Good Test!" post.

Even when using DP's own test charts there is a post that will say "That's Not a Good Test!"

I'm interested because I'd like to see what the majority accepts as a valid test so when I'm reading through a post I can feel confident in the results.

 

[jvkelley]

It's not really possible to satisfy everyone with a test. If you go to great lengths to control as many variables as possible, then people will view your test as too scientific and not relevant to real world shooting.

If you compare two products based purely on real world shooting, people will complain that you have too many uncontrolled variables and your comparison is invalid.

--

J.V.

 

[Bill Robb]

It's not really possible to satisfy everyone with a test. If you go to great lengths to control as many variables as possible, then people will view your test as too scientific and not relevant to real world shooting.

Tests don't have to be relevant to real world shooting, they do have to be accurate, repeatable and well controlled.

If you compare two products based purely on real world shooting, people will complain that you have too many uncontrolled variables and your comparison is invalid.

For myself, real world shooting is a better test, but some people get off on counting angels on the head of a pin.

 

[jvkelley]

It's not really possible to satisfy everyone with a test. If you go to great lengths to control as many variables as possible, then people will view your test as too scientific and not relevant to real world shooting.

Tests don't have to be relevant to real world shooting, they do have to be accurate, repeatable and well controlled.

There are a lot of people on dpreview who hate DXOmark. The major complaint seems to be that their results don't always correlate with real world results. I've never heard anyone accuse them of not being accurate, repeatable and well controlled.

If you compare two products based purely on real world shooting, people will complain that you have too many uncontrolled variables and your comparison is invalid.

For myself, real world shooting is a better test, but some people get off on counting angels on the head of a pin.

 

[Bill Robb]

It's not really possible to satisfy everyone with a test. If you go to great lengths to control as many variables as possible, then people will view your test as too scientific and not relevant to real world shooting.

Tests don't have to be relevant to real world shooting, they do have to be accurate, repeatable and well controlled.

There are a lot of people on dpreview who hate DXOmark. The major complaint seems to be that their results don't always correlate with real world results. I've never heard anyone accuse them of not being accurate, repeatable and well controlled.

One could equally complain that people's real world shooting isn't relevant to DXO's results. One could also ask if the people complaining really understand exactly what DXO tests for?

If you compare two products based purely on real world shooting, people will complain that you have too many uncontrolled variables and your comparison is invalid.

For myself, real world shooting is a better test, but some people get off on counting angels on the head of a pin.

--
J.V.

 

[fotolopithecus]

It seems to me that no matter what forum I'm on, when someone, out of the goodness of their heart and taking up of their time, post some test - camera to camera, sensor to sensor, lens to lens, iso to iso, whatever - someone posts, "That's Not a Good Test!"

Then what is?

Can you all help those who put in their heart and time with some guidelines to create valid tests that will be accepted by the majority of DP members so you don't get 3 posts in and see the dreaded "That's Not a Good Test!" post.

Even when using DP's own test charts there is a post that will say "That's Not a Good Test!"

I'm interested because I'd like to see what the majority accepts as a valid test so when I'm reading through a post I can feel confident in the results.

Thanks

chk

 

[fotolopithecus]

A good test is a test showing the gear I own is better than the rest!

Even the best tests are dismissed because they did or did not count in ergonomics, economy, child labor, weight, sensor size, high ISO performance, low ISO noise, dynamic range, flare, resolution, jpeg-engine, size, file size, RAW results, distortion, distortion correction ... ad nauseam ad infinitum.

Just tell me my gear is allright so I can get on with my life.

Have a nice weekend. :-D

Lol, absolutely right, I wouldn't have bothered posting my answer if I had read this first. :-D

 

[Richard]

It seems to me that no matter what forum I'm on, when someone, out of the goodness of their heart and taking up of their time, post some test - camera to camera, sensor to sensor, lens to lens, iso to iso, whatever - someone posts, "That's Not a Good Test!"

Then what is?

A good test is repeatable, and is only testing for one variable.
Most ad hoc tests fail in both of these parameters.

What you are asking is for a scientific test. Most tests are not repeatable, Sure, black room, controlled flash system, you might be able to get it, but even then I see difference. If you do lenses on two different tripods, there is a difference, if you use only one camera/tripod, you make changes when removing one lens to replace it with another.

I think all tests are good in that you can learn something from them.

Here is a bokeh test I did. It is not repeatable at all, I do not have the original light, location, there are even differences in exposure between shots. It is not scientific, it is not repeatable and probably not valid, but you can learn something from it. It was in response to someone saying the camera and you cannot see the difference in bokeh from a f1.2 lens

I encourage people to do tests. Even comparisons of dissimilar format and lens you can learn from.

http://www.mxphotos.net/dof/lensDOF.gif

click on link above or on original size link below the image.

 

[fishy wishy]

Don't worry, if you can't criticise unfavourable test results because they're replicated, you can just curse forum protagonists as useless pixel-peepers. Then you can tell them to go out and take some pictures. You can abuse them as "measurebaters", conveniently forgetting the fact that the only reason you, too, were in the forum was because you like to waffle about cameras as well, but you just lost.