Previous news story    Next news story

A sneak peek at our forthcoming camera test scene

By dpreview staff on Oct 9, 2012 at 23:59 GMT

As part of the development of connect.dpreview.com we've created a more advanced, more detailed test scene, which will be launched on dpreview.com soon. The new scene is roughly eight times larger (in area) than the existing test, allowing us to test to even higher resolutions than before. A larger scene also increases working distances - avoiding the inconsistent performances we've seen from some recent fixed-lens cameras, and the new design effectively overcomes the depth of field limitations experienced with our current studio scene.

We're also working on a low-light mode that will give a truer impression of how the cameras will perform in poor lighting in the real world. The low-light mode will not only use lower luminance but also a less color-balanced light source - giving a clearer impression of what happens under real-world lighting, when cameras struggle to capture and represent color accurately.

The test scene can be seen in our phone reviews, over at connect.dpreview.com, but we'll be using it for many of the cameras launched at Photokina in the very near future, and working to re-test key cameras from the last year or two.

Comments

Total comments: 324
1234
choochoo22
By choochoo22 (Oct 10, 2012)

Everyone has their favorite elements in the existing test screen. I often use the detail in the globe someone mentioned earlier. Also, it's amazing how many cameras, including some pro and semi-pro gear, have trouble resolving the double tic marks on the hours in the watch at the lower right.

3 upvotes
ovatab
By ovatab (Oct 10, 2012)

I will miss my favorite "BY APPOINTMENT TO HER MAGESTY THE QUEEN" :(

2 upvotes
Combatmedic870
By Combatmedic870 (Oct 10, 2012)

Something to show shadow detail is needed. Like in the current one with the thread and puffs.

3 upvotes
Barney Britton
By Barney Britton (Oct 10, 2012)

It'll be in there.

3 upvotes
Combatmedic870
By Combatmedic870 (Oct 10, 2012)

Very good then. All is well! I sure will miss the watch!

0 upvotes
PicOne
By PicOne (Oct 10, 2012)

"A larger scene also increases working distances - avoiding the inconsistent performances we've seen from some recent fixed-lens cameras".. uhmm don't we want to know about and see this in a camera review?

1 upvote
Barney Britton
By Barney Britton (Oct 10, 2012)

Sony RX100. It's mentioned in the review.

0 upvotes
Todd Ka
By Todd Ka (Oct 10, 2012)

Looks good to me.

0 upvotes
John King
By John King (Oct 10, 2012)

Good to see the Kodak grey scale chart with its C-Y-M header is still there.

IMHO, this is one of the most important tests of any camera, whether the contrast and brightness between the respective colour circle and the embedded character in white is relatively even over the three (reciprocal) primary colours; as well as the grey scale wedge reproduction.

IME, many cameras fall down on this simple test, and the final results from them also show this systemic bias towards (usually) two of the three colours, with the white character bordering on being indistinct in the other colour. This success or failure gives clues to the adequacy and appropriateness of the CFA and attendant processing IMO.

Not good to see the shadow box with its fluffy balls removed. IMHO, this was an excellent part of the target for evaluating shadow detail and noise. Looking at the full size image just now, I realised that there is nothing similar to it, or to the toys with their very sharp printed detail.

br, j

Comment edited 5 minutes after posting
3 upvotes
Douglas F Watt
By Douglas F Watt (Oct 10, 2012)

Well I'm not sure what anybody else thinks, but this is much more problematic than DP reviews appears to appreciate. First of all every new camera is going to shoot this test scene instead of the old test scene. How in the world are you going to compare newly reviewed cameras to older and previously reviewed cameras unless all those old cameras shoot the same test scene too. It seems to me one of the great advantages of your standard scene comparator tool was the huge database of images at different ISOs both JPEG and RAW that were shot with every camera. This means the you could compare easily a 100% view across four different cameras with little hassle. How are you going to keep that functionality with this new test scene unless every old camera re-shoots this new scene?

Comment edited 2 times, last edit 2 minutes after posting
4 upvotes
Richard Butler
By Richard Butler (Oct 10, 2012)

We appreciate that problem full-well, having been in a very similar situation when we created our current test scene and comparison tool about three and a half years ago.

We know there's a lot of work ahead of us to populate the tool.

Comment edited 29 seconds after posting
3 upvotes
Douglas F Watt
By Douglas F Watt (Oct 10, 2012)

Well, that's actually very reassuring and thanks for responding so promptly. You guys will make a lot of extra work for yourselves on this one!!

I assume that you have lots of older cameras around to re-shoot the standard scene with? I for one will miss the gold coin (from the Martini vermouth bottle?) Made some important (and correct) decisions about cameras from that gold coin.

0 upvotes
Barney Britton
By Barney Britton (Oct 10, 2012)

@ Douglas - thanks to a few very good relationships we're in the fortunate position now (not always true in the past) of being able to borrow pretty much any current and last-gen DSLR for additional photography whenever we need to. So we'll be doing as much 'backfilling' as possible.

Comment edited 16 seconds after posting
2 upvotes
jalywol
By jalywol (Oct 10, 2012)

I would suggest adding one item from your old display...there is a fabric swatch to the right and above the Pluto dog figure (and some threads from it to the left of the Mickey Mouse figure), that I found especially helpful when comparing different cameras abilities to resolve not only fine detail, but the ability of the camera's image processor to resolve and keep intact complex colors and shading in small areas.

Nothing in the new display is quite as good as this piece from the old one for that particular purpose, so if there was any way to put this or a similar swatch in the new display, I think it would really be useful.

Comment edited 2 minutes after posting
3 upvotes
Richard Butler
By Richard Butler (Oct 10, 2012)

We'll consider moving it across - thank you for the feedback.

Comment edited 18 seconds after posting
3 upvotes
jalywol
By jalywol (Oct 10, 2012)

That is great!

Thanks!

2 upvotes
ddolde
By ddolde (Oct 10, 2012)

Love to see the Sigma DP2 Merrill tested with it !

0 upvotes
PhotoBFA
By PhotoBFA (Oct 10, 2012)

One thing with these tests is that you're not only comparing one SLR's sensor to another's, but also the quality of the lens used in the test too. One camera's sensor might have exactly the same resolution as another brand's, but the differences in lens quality might skew the actual results, especially near the edges of the scene. The only way to eliminate that variable would be to use a 3rd party lens that comes in different mounts.

One thing I liked about the old studio scene was the little globe towards the top middle of the scene. It had white cross-hatch marks in the ocean around the islands on the globe. With a crop sensor camera like Canon T4i you couldn't see the hatch marks at all. With a full frame sensor camera like the 5D Mk3 you could just start to see the cross-hatches. With the Nikon D800 you could see them more clearly, and with the D800E you could really see the lines clearly.

4 upvotes
Underdog 3000
By Underdog 3000 (Oct 10, 2012)

I thought the globe and the watch were beneficial. But as an earlier post mentioned, a shadow box (the one with the colored puffs) is deffinitely needed.

On a side note, I believe we would all greatly appreciate it if you have a parallel test using the older scene or at least retest all current cameras to the new one.

Thanks for all of your work.

4 upvotes
Barney Britton
By Barney Britton (Oct 10, 2012)

@Underdog 3000 - yes.

1 upvote
chj
By chj (Oct 10, 2012)

Nice, I recently suggested adding low-light testing to the reviews. Keep up the great work.

0 upvotes
micahmedia
By micahmedia (Oct 10, 2012)

...and what happens to the ability to make comparisons to legacy cameras?

Just when you finally get around to adding things like the D700 to your standard test scene, you're doing away with it?

Or are you including this as an additional test? If so, my apologies.

0 upvotes
Richard Butler
By Richard Butler (Oct 10, 2012)

Sadly this is the challenge of updating a test, but we think this will a lot over the existing test scene.

We will try to re-test as many significant cameras as we can, but obviously it won't initially be as comprehensive as the current scene is.

1 upvote
micahmedia
By micahmedia (Oct 10, 2012)

I appreciate that it's time consuming to do these tests. But that's part of why it's important not to throw all that work (and good will!) away.

Do you plan to things like the D700/D3s in the database? The D3s is still a powerful benchmark in many ways.

0 upvotes
Ignace
By Ignace (Oct 10, 2012)

Please re-test the E-M5 and similar cameras.
Thanks.

0 upvotes
rootsup
By rootsup (Oct 10, 2012)

Flat lit scenes are great, but can we get a scene with different (and multiple) light sources, tonal gradations on flat colors via light sources, ect? I know this would be a hard thing to keep in your studio, and to keep consistant, but looking at a flat lit scene with a bunch of small objects can only tell us so much. Most photos do not exists in this environment. Understandably we want to see how the camera reproduces by default, but it'd be nice to have something else as well.

3 upvotes
Richard Butler
By Richard Butler (Oct 10, 2012)

But, as soon as you use a significantly 3D scene, you make consistent focus much harder to achieve (but much easier to nit-pick).

As explained in the story, we will be using two light sources.

2 upvotes
rootsup
By rootsup (Oct 10, 2012)

You don't necessarily need a 3d scene, just something with actual light gradation aposed to printed gradients. I want to see blown highlights/unreadable black, with multiple colors, surfaces, and light sources. I see you're going to do a dimly lit model, but i'm looking for more still haha. Thanks for all of you guy's hard work tho! Majorly appreciated.

3 upvotes
Mike Ronesia
By Mike Ronesia (Oct 10, 2012)

I liked the 3D scene, but I know there were a lot of battles about where the camera was focused when comparisons were made in the forums. With all the different size sensors it is not always easy to achieve consistent DOF so I do see the benefits of 2D. I just think you should have something out front to create some deep shadows. If you focus on the back it can also be a DOF indicator as well to compare the difference between the senor sizes, assuming you use the same fstop in all the tests.

Comment edited 1 minute after posting
0 upvotes
Mssimo
By Mssimo (Oct 10, 2012)

We are going to be looking at this for the next 10 years or so..i think we need to think ahead and add at least one more hot girl.

13 upvotes
Robert Morris
By Robert Morris (Oct 10, 2012)

Yes, one with some 3 DD's

2 upvotes
Mssimo
By Mssimo (Oct 10, 2012)

Ohh..a DD Mark III You have good taste my son.

1 upvote
SHood
By SHood (Oct 10, 2012)

We should get "Pocahontas" from Photokina to test skin tones.

2 upvotes
WillemB
By WillemB (Oct 10, 2012)

By doing some shifting there will be ample space for a beautyful doll wearing well detailled feathers on some critical spaces. Otherwise a skin detail with finely molded shadows...

2 upvotes
jcmarfilph
By jcmarfilph (Oct 10, 2012)

And when shoot this scene, please equalize the settings next time coz we are seeing inconsistencies in your studio shots in almost all cameras (within similar group).

2 upvotes
Richard Butler
By Richard Butler (Oct 10, 2012)

It's likely that our standard scene will remain brightness matched but the low-light mode might rely on camera metering, to give an understanding of what the camera chooses to do (we're still testing).

Unless you can explain exactly what you gain by arbitrarily fixing the settings, rather than standardizing output brightness, then we're unlikely to change. The required settings being the same is not, in itself, a solid reason.

Comment edited 11 minutes after posting
0 upvotes
Mssimo
By Mssimo (Oct 10, 2012)

Seems like if they are different brightness, its hard to compare DR, noise and color.

1 upvote
Richard Butler
By Richard Butler (Oct 10, 2012)

Exactly. The logic behind our current system is that most people expose their photos to achieve a certain brightness - not to have the satisfaction of having used a specific combination of shutter speed, ISO and aperture.

No matter what we do, this isn't going to become a proxy for ISO accuracy testing.

0 upvotes
micahmedia
By micahmedia (Oct 10, 2012)

Au contraire! If there's a stop difference in brightness at a given shutter/aperture/ISO then that can mean the difference between 1/30th and 1/60th in low light, which means a lot when you're shooting people. We can play with exposure to get a properly exposed picture, but standards of sensitivity give us a very clear measure of the quality of a particular tool.

...although, wow, I do have to applaud the idea of an testing low light ability with challenging light source colors. Excellent! That'll separate the wheat form the chaff right quick!

Comment edited 7 minutes after posting
1 upvote
Richard Butler
By Richard Butler (Oct 10, 2012)

Ultimately, there are choices:

Match brightness, so that you can see how properly exposed images from two cameras would compare, then report any ISO discrepancies.

Use an arbitrary set of exposure values at each ISO setting, in an attempt to do both at once, but losing the ability to compare noise, DR and colour (as Mssimo points out).

Let the cameras meter the scene and bring another variable into the mix.

Which gives you more useful information overall? Which gives the most visually comparable images and which most closely mimics the way most people use their cameras?

0 upvotes
PhotoByRichard
By PhotoByRichard (Oct 10, 2012)

will you be normalising the output to a standard resolution?
Comparing results at pixel level between a 36MP and a 12MP like the current tests doesn't really mean much.

0 upvotes
Richard Butler
By Richard Butler (Oct 10, 2012)

It's not part of the current plan (not least because we'd need to agree a resolution and re-sampling method), but it's not out-of-the-question.

0 upvotes
PicOne
By PicOne (Oct 10, 2012)

DPR used to always provide ISO equivalency between cameras, noting fudged attempts by manufacturers to get away with something.

1 upvote
Barney Britton
By Barney Britton (Oct 10, 2012)

@PicOne - we still do report true versus indicated ISO.

1 upvote
RPJG
By RPJG (Oct 10, 2012)

Would it be possible to take two exposures, one at the exposure setting suggested by the camera, at one at a fixed ISO/aperture/shutter speed setting? I realise that would complicate your database and review tool, but it would cover most bases.

2 upvotes
NektonFi
By NektonFi (Oct 10, 2012)

I think normalizing the resolution would make comparisons a lot easier. After all, most of the people in the end tend to look at the photos, not at pixels. 8 megapixels like DXO would be good. I doubt any future cameras will have less than that.

Of course full resolution images would be needed also, but normalized images would be the best for comparisons.

1 upvote
cesaregal
By cesaregal (Oct 10, 2012)

I agree with NektonFi. I would use normalized lens too as, for Full Frame, a Zeiss Planar 50 mm f/2 (that we have for Nikon, Pentax, EOS,... mount).

0 upvotes
bobn2
By bobn2 (Oct 10, 2012)

Regarding these two responses
It's likely that our standard scene will remain brightness matched but the low-light mode might rely on camera metering.
The logic behind our current system is that most people expose their photos to achieve a certain brightness - not to have the satisfaction of having used a specific combination of shutter speed, ISO and aperture.
The problem with your brightness matched methodology at the moment is that it gives neither the results someone would get using camera metering (as most would do) nor what they would get if they were advanced enough to set the f-number and shutter speed as constrained by DOF and motion blur, and then set brightness via ISO/processing, which is what most more advanced workers will do - so in a way it gives useful information neither to the 'get it right in camera' brigade nor the 'raw processor demons'.

2 upvotes
bobn2
By bobn2 (Oct 10, 2012)

(continued)

With respect to the low light camera metering, the problem with camera metering is that it leaves the path open for manufacturers to improve their low light performance simply by increasing the exposure above ISO nominal, which is no good to anyone trying to get some required shutter speed in low light.
What would be interesting would be a set of tests at ISO nominal exposure (i.e. the lux second rating according to the ISO SOS standard) - if your lighting is sufficiently controlled, you could have standard EV settings for each ISO and that would remove a number of variables. That information would be invaluable for people who concentrate on the processing. For the get it right in camera people, really you should be doing a series based on camera metering - the difference in exposure from the ISO nominal would give a good clue as to how much the observed performance came from metering differences.

4 upvotes
BrightTiger
By BrightTiger (Oct 10, 2012)

Some have mentioned the skin tone issue but with pictures, as alread done. Perhaps some out of the box thinking is needed here. While skinning people is clearly unacceptable - perhaps using a realistic substitute in terms of texture and color? Perhaps one of the universities in your area might provide some guidance on this. Pig skin is similar to human skin in terms of color, hair follicles, sweat glands... would a preserved dry piece work?
There are some opportunities here to create a better test. We understand one challange is sample degradation but this occurs even with the pictures being used. Dried leaves? A prism? Magnifying glass (useful it has a metallic rim for the shiny test as well as distortion test).

0 upvotes
BrightTiger
By BrightTiger (Oct 10, 2012)

<deleted by user>

Comment edited 3 minutes after posting
0 upvotes
krikman
By krikman (Oct 10, 2012)

Please read my message carefully.
1. Printed photo completely unacceptable for test purposes. Their colors very different from real objects beause of metamerism.

They look similar to real objects only for our imagination and in some lightning conditions. In reality colors in prints have completely different spectrum than original colored objects.
And that spectrum on prints have characteric spikes while real color's spectrum are continous in nature.

2. Colors on prints tend ageing very fast, especially from light emission. Also appereance change with air humidity.

Comment edited 2 times, last edit 6 minutes after posting
0 upvotes
krikman
By krikman (Oct 10, 2012)

So natural objects like feathers, ceramic barelief, hand made dolls, matte laquered woods, old metal coins covered with some oil, anodized metals etc. will serve very well for comparisons. They also have unlimited details.

Comment edited 32 seconds after posting
0 upvotes
Daniel Lamberger
By Daniel Lamberger (Oct 18, 2012)

That scene reflects the kind of output you'd get with a perfectly still camera and subjects. This is rarely the case, and is especially problematic for low-light photography.

When shooting that scene in low light conditions, it could be very useful to have 3 clock hands in the scene rotating at different speeds.

The slowest should correlate with normal hand shake when standing still (e.g. I can take crisp pictures without a tripod down to 1/30 sec, but below that I get some blur), the medium one should correlate with slow-moving objects in the scene (e.g. tree branches swaying in the wind, water flowing, people walking at some distance), and the fastest one with fast-moving objects such as a kid running around or cars in a street. The amount of blur behind each rotating hand would indicate how well the camera performs at the respective scenario.

0 upvotes
Total comments: 324
1234