Time for a Standard Test Procedure?

Started Oct 6, 2008 | Discussions thread
ForumParentFirstPreviousNextNext unread
Flat view
Decoboy
Regular MemberPosts: 171Gear list
Like?
Time for a Standard Test Procedure?
Oct 6, 2008

Hi All;

Every time a new review test is published posted we inevitably get into a death spiral of dispute on how the tests were performed. The opinions range from those based on lots of experience and knowledge - to some based on .... reading their horoscope

There are lots of standards available for all sorts of things even the ISO term we use here relates to a method for evaluating sensitivity to light (I wonder how many posters know what ISO stands for?)

How come there is not a 'Standard' for this type of testing? (Or is there?) It seems every man and his dog have a different theory about how to do it. Even people who have only the vaguest concept of the basics of the photographic process.

I'm prompted in this by discovering that Imaging-Resource used, in their 40D and 50D tests - wait for it - different lens' (100-2.8 vs 50 2.5 with a 1.4 converter= 70mm LOL), and allowed the shutter speed to change. Even though in their own words:

"If we can't control the test conditions from one camera to another, any results we obtained will be meaningless and less than worthless: Images shot under different conditions could actually mislead people as to the cameras' actual performance relative to one another"

IMO the rules would not be difficult - here's a very rough starting point- naturally any rules would need to be defined by QUALIFIED people and not by popularity contests.

Test Purpose

1- These tests are to evaluate and compare the image capturing capabilities of digital cameras

2- The object of these devices is to produce images for human use e.g. I want to take a photo of this subject - framed/lit a certain way - with a certain DOF - and/or certain effect on moving objects

3 - These are not tests to see how a user could modify the environment to take the same image - they are tests to show how each device responds to the same EXACT environment

3- Therefore you do not modify any of the test conditions to allow for limitations of the device - you simply show what it produces from the same conditions. If it is not able to do a test you say so - you don't 'fudge' it
3- Therefore all tests (by a reviewer) must be conducted as follows:
CaptureConditions

  • Subject(s) are identical

  • Framing is identical

  • Camera position is identical (this shows what can be achieved from the same position)

  • Light source and intensity is identical

  • Shutter/Aperture/Lens should be identical where possible - or closest equivalent

  • Shutter/Aperture should be set by a calibrated device i.e. the same calibrated external light meter for each ISO setting - to show variations between the devices metering systems (these can and should be reported by the reviewer)

Process Conditions

  • Highest quality capture setting used on camera and lossless conversion to presentation format. ( you never dumb down one to allow for a lesser type) If required transfer to lossless format for comparisons i.e. Raw to Tiff, jpeg to tiff - not Raw to jpeg, jpeg to jpeg

  • Manufacturers recommended 'Standard' settings for NR , same for same for WB (as this cannot be set to the same standard e.g. when is off off, unless you know exactly what is done in-camera) - of course variations to show non standard performance are fine

  • Manufacturers processing software used (where available) as default - alternates used for comparison to manufacturers output and readers information

  • Never ever resize/downsize images for the (lame brained) idea that this helps comparison. This is for the user to do. -- 100% = 100% don't screw with it for mere 'convenience'.

Others -

Subject matter is up to the reviewer - as long as it is always exactly the same. Readers will judge on if they have chosen wisely

Never let manufacturers influence the test conditions or procedures

Never - ever - ever sacrifice accuracy for convenience - its OK for a user - but should be a sacking offence for a tester-reviewer.

Let the carnage begin!!
--
Brian

 Decoboy's gear list:Decoboy's gear list
Canon EOS 50D Canon EOS-1D X Canon EF 50mm f/1.4 USM Canon EF-S 10-22mm f/3.5-4.5 USM Canon EF 70-200mm f/2.8L IS USM +3 more
ForumParentFirstPreviousNextNext unread
Flat view
Post (hide subjects)Posted by
ForumParentFirstPreviousNextNext unread
Keyboard shortcuts:
FForum PPrevious NNext WNext unread UUpvote SSubscribe RReply QQuote BBookmark post MMy threads
Color scheme? Blue / Yellow