Time for a Standard Test Procedure?
Time for a Standard Test Procedure?
Oct 6, 2008
Every time a new review test is published posted we inevitably get into a death spiral of dispute on how the tests were performed. The opinions range from those based on lots of experience and knowledge - to some based on .... reading their horoscope
There are lots of standards available for all sorts of things even the ISO term we use here relates to a method for evaluating sensitivity to light (I wonder how many posters know what ISO stands for?)
How come there is not a 'Standard' for this type of testing? (Or is there?) It seems every man and his dog have a different theory about how to do it. Even people who have only the vaguest concept of the basics of the photographic process.
I'm prompted in this by discovering that Imaging-Resource used, in their 40D and 50D tests - wait for it - different lens' (100-2.8 vs 50 2.5 with a 1.4 converter= 70mm LOL), and allowed the shutter speed to change. Even though in their own words:
"If we can't control the test conditions from one camera to another, any results we obtained will be meaningless and less than worthless: Images shot under different conditions could actually mislead people as to the cameras' actual performance relative to one another"
IMO the rules would not be difficult - here's a very rough starting point- naturally any rules would need to be defined by QUALIFIED people and not by popularity contests.
1- These tests are to evaluate and compare the image capturing capabilities of digital cameras
2- The object of these devices is to produce images for human use e.g. I want to take a photo of this subject - framed/lit a certain way - with a certain DOF - and/or certain effect on moving objects
3 - These are not tests to see how a user could modify the environment to take the same image - they are tests to show how each device responds to the same EXACT environment
3- Therefore you do not modify any of the test conditions to allow for limitations of the device - you simply show what it produces from the same conditions. If it is not able to do a test you say so - you don't 'fudge' it
3- Therefore all tests (by a reviewer) must be conducted as follows:
Subject(s) are identical
Framing is identical
Camera position is identical (this shows what can be achieved from the same position)
Light source and intensity is identical
Shutter/Aperture/Lens should be identical where possible - or closest equivalent
Shutter/Aperture should be set by a calibrated device i.e. the same calibrated external light meter for each ISO setting - to show variations between the devices metering systems (these can and should be reported by the reviewer)
Highest quality capture setting used on camera and lossless conversion to presentation format. ( you never dumb down one to allow for a lesser type) If required transfer to lossless format for comparisons i.e. Raw to Tiff, jpeg to tiff - not Raw to jpeg, jpeg to jpeg
Manufacturers recommended 'Standard' settings for NR , same for same for WB (as this cannot be set to the same standard e.g. when is off off, unless you know exactly what is done in-camera) - of course variations to show non standard performance are fine
Manufacturers processing software used (where available) as default - alternates used for comparison to manufacturers output and readers information
Never ever resize/downsize images for the (lame brained) idea that this helps comparison. This is for the user to do. -- 100% = 100% don't screw with it for mere 'convenience'.
Subject matter is up to the reviewer - as long as it is always exactly the same. Readers will judge on if they have chosen wisely
Never let manufacturers influence the test conditions or procedures
Never - ever - ever sacrifice accuracy for convenience - its OK for a user - but should be a sacking offence for a tester-reviewer.
Let the carnage begin!!