Studio Test Scene

Welcome to the Studio Test Scene

We've worked hard over the last few months to develop a new test scene and a protocol for shooting it and processing the results. The underlying idea is to give a clearer idea of how a camera will perform in the real world - not just an idealized setting.

To achieve this, we've developed a scene that includes a variety of samples of fine detail, low-contrast textures and colored tones that should help you assess what a camera's output will look like. The new scene is shot in both good light and low light modes (explained below), which are designed to be visually distinct. As before, all images in the test scene are downloadable and shooting information (including the lens used) is available by clicking the [i] button under each sample.

In addition to a new lighting mode we have also included the option to 'normalize' the scene to a standard print size and social media size. 

On the top right of the new scene widget you can select from 'Daylight' and 'Low-light' modes as well as normalize the images to standard print and social media sizes.

The scene is shot so that the brightness of the scene is consistent, between JPEGs (since we believe most people aim for a particular brightness when shooting). Any difference in shutter speeds used is displayed in the settings information. Note that at the very highest ISOs, it is sometimes necessary to reduce the light level to prevent over-exposure.

What’s the deal with this low-light mode?

In addition to replacing our studio test scene we've also added a few new features that give a more complete view of what a camera is capable of in different lighting situations. Of course we still present our daylight scene that uses daylight-balance lighting (CRI 95) to represent a lighting condition that you might encounter outside on a typical day. In addition, we've also added a new ‘low light’ mode that replicates a lighting situation you are likely to encounter in a home or dimly lit bar. This ‘stress test’ is designed to push the camera to its limits and show off noise that would otherwise be masked by the abundant light in our ‘daylight’ scene.

The light source used in this low-light mode is a single standard household tungsten bulb placed just to the right of the scene providing 3EV of light at the center.

The extreme angle of the light source to the scene creates a distinct fall off of brightness that creates 2 separate areas of light, a highlight area and a shadowed area. As you may have noticed the scene is roughly diagonally symmetrical, providing similar objects in different light levels in a single image.

How are the low-light images shot?

In our comparison widget we present the low-light setting in JPEG as well as Raw where available. In JPEG mode, the images are presented straight from the camera. Unlike the daylight scene, the low-light scene is shot in AWB (Auto White Balance) and any option to preserve scene 'warmth' is left at its default setting, to give a realistic impression of the camera's output in the real world.

The Raw files are adjusted with a standardized processing method to reveal characteristics which would otherwise be hidden. The brightness of the Raw file is matched to that of the JPEG but here white balance is neutralized and noise reduction is minimized, to show blue-channel noise or banding. For the same reason, the black level is increased to 100 and the white level increased to 20, to brighten detail in the shadow regions while retaining a similar amount of tonal detail in the highlights. As with the daylight scene, sharpening is minimized and a standard amount applied in Photoshop.

What was wrong with the old scene?

The existing test scene, which dates back to October 2009 (and has only allowed user-selectable comparisons since June 2010), has served us well but has presented an increasing number of problems in recent years - most of which stem from it being too small.

The old test scene dates back to 2009 and has only been directly comparable since mid 2010. This scene had evolved bit-by-bit since 2000.

The small size of the scene meant that some cameras, particularly those with fixed normal and wide-angle lenses, needed to be shot very close to the scene (often requiring a mini tripod to be set up inside the box it's housed in). Additionally, some modern compacts with large sensors and complex lenses looked terrible when shot at these close-quarters - meaning the scene didn't reflect their real-world behavior.

The three-dimensional nature of the old scene also meant that the scene had to be shot at very small (diffraction-limited) apertures to ensure all the key targets were acceptably sharp, yet still didn't have any corner detail within depth-of-field. These drawbacks, plus the need to revise the lighting of the scene, prompted us to move to a new scene around 7 times larger than the existing one.

Comments

Total comments: 55
GeorgeP71
By GeorgeP71 (7 months ago)

I would like to see some REAL Scene photos of a scene more than 15meters away. I find that my indoor shots cannot be faulted with respect to resolution, but objects more than 15meters away leave a lot to be desired.

Comment edited 44 seconds after posting
1 upvote
Richard Butler
By Richard Butler (7 months ago)

Sadly it's impossible to control lighting in such a situation. That would risk counting badly against anything launched in the winter.

1 upvote
keeponkeepingon
By keeponkeepingon (7 months ago)

please add the 808 to the test scenes.

Why add a nitch phone?

The 808 has the largest sensor in a phone; arguably the best cell phone camera to-date.

If I still lived in seattle I'd loan you mine for a day or five but someone has to have one laying around?

Thanks!

1 upvote
Richard Butler
By Richard Butler (6 months ago)

So far we've added the Nokia 1020 - I'm not sure if we can get an 808 back in, but it's on our list of products we'd like to add.

A selection of phones is already available over on Connect, and we're working on a means of being able to access the cameras from Connect and the smartphones from dpr.

0 upvotes
JTO666
By JTO666 (7 months ago)

Am I the only one who thinks the new transparent overlay on the camera hides too much of the picture and so renders the app useless?

0 upvotes
logbi77
By logbi77 (7 months ago)

Any chance for you guys to put a 12 MP APS-C/MFT camera in the test scene as reference?

And maybe the Panasonic G3/GH2 and a Nikon 1 camera as well?

0 upvotes
KrisAK
By KrisAK (7 months ago)

Please consider a RAW workflow for non-Adobe supported cameras (yes, I'm thinking Foveon.)

I've read R Butler's comments about the need for "like-for-like", but I’m still puzzled: this tool allows me to make comparisons between a FF mirrored Canon, an APS-C Translucent-Mirror Sony, a mirrorless micro 4/3’s, and a Fuji X-Trans. Those very different hardware technologies are considered "like-for-like." But if a camera's RAW files don't pop open in Adobe, it's considered too exotic to shoot.

Why the emphasis on software? By all means, standardize methods where possible, but please don't ignore the outliers.

Comment edited 31 seconds after posting
3 upvotes
Richard Butler
By Richard Butler (7 months ago)

The point is that everything in the scene is tested and (for Raws) processed in as even-handed a manner as possible.

If we show one brand in a non-standard way (but without a clear way of showing that you're seeing something given advantageous processing), then why not others? Pretty soon you're processing everything with a motley collection of packages that only a tiny minority of users will ever use, and having constant arguments that 'DPR should have used...'

We've tried presenting just Sigma JPEGs, but been told that's even worse than not including them at all.

1 upvote
Claudio NC
By Claudio NC (7 months ago)

Try to use the Iridient Developer to develop the Sigma Foveon files, forget Jpg from Sigma cameras.
Iridient Developer can open also all, or almost all bayer sensor files.
At the present it can not compete for ease of use with converters made ​​by Adobe, but I think, in some respects, it's much better, functional, rich and faster than the SIGMA Photo Pro converter.

Inside DPReview test you could post a new experimental set to contain the convertions performed with alternative software, in which to put therefore Foveon along with some of the Bayer cameras and others that will be realized in future, without necessarily having to go crazy with grueling, rigorous methodologies.

An outside border test, without blinders, outside the usual constraining rails.

0 upvotes
KrisAK
By KrisAK (7 months ago)

I regret using the F-word.

I'm actually more concerned about the homogenizing effect of the studio tool when 'shopping' cameras: as in, if it's not in the drop-down, it doesn't exist.

Maybe current cameras awaiting ACR support COULD be included in the drop-down, but as disabled (greyed-out) entries. There probably aren't that many, and at least the oddballs and up-and-comers wouldn't be so easily overlooked.

0 upvotes
solsang
By solsang (7 months ago)

Now adobe is using bayer and missing foveon, then simply add those few sigma cameras with spp editing and explain that "all sigma camera raw are developed using spp, which is the default actions for foveon files".

There are currently only two different sensor groups, the foveons and bayers so this would be easy to do and adobe is making a decent job of all bayers, even xtrans.

You could use iridient instead of spp if you want indenpedence from company, and if someone makes a new techonology you should add them too, so you do what a normal photographer is doing instead of forcing an ideal.

Most professionals are using different developer programs and plugins, yet still adobe is the most widely used for bayer and spp is the most used for foveon, so presenting them in any different way or omitting them totally is illogical!

Comment edited 1 minute after posting
2 upvotes
Tristan Cope
By Tristan Cope (1 day ago)

I would support this suggestion. SPP is the default processor for Sigma raw files and, notwithstanding the caveats, it would be interesting to see the comparison with at least one of the Sigma 15MP Foveon cameras. It would also be nice to be able to compare with the Leica M. I appreciate that this would require some time and expense.

0 upvotes
baobob
By baobob (7 months ago)

The new studio scene brings definite improvement. I guess it can be further improved. IMHO in it's present stage, it lacks elements that can check fo very fine details such as the banknote and the feathers in the previous model. OTOH it brings new elements that was lacking before
I hope after readers have expressed their opinions, the studio scene could be upgraded according to remarks and suggestions
Thx for this comments space
Robert

0 upvotes
Richard Butler
By Richard Butler (7 months ago)

In fairness we've posted the new scene in various states of completeness for over a year, now, and tried to include elements addressing the concerns raised.

Having spent several months shooting the cameras you now see, we're not going to make major changes at this point, I'm afraid.

There are bank notes and feathers in the scene, though.

0 upvotes
PhilPreston3072
By PhilPreston3072 (7 months ago)

Select the PhaseOne and have a look at the woven textile threads in the mottled blue fabric. You won't see the textile threads with any other camera.

0 upvotes
Brandon birder
By Brandon birder (7 months ago)

I wish you had put some feathers in. As a bird photographer I used these to assess a cameras ability to render complex fine detail accurately and without moire. Can this be added into the new scene please.

0 upvotes
Richard Butler
By Richard Butler (7 months ago)

You should find some feathers at the bottom left and top right of the scene, next to the low-contrast green target.

0 upvotes
urix
By urix (7 months ago)

What did you mean with "3EV at the center" of low-lit scene? Is it correct to assume that the center illumination is 20 lx? Thanks.

0 upvotes
yabokkie
By yabokkie (7 months ago)

should be well below 1 lux.

0 upvotes
urix
By urix (7 months ago)

@yabokkie
Really? Illumination of 25-30 lux can be learned from EXIF data.

0 upvotes
yabokkie
By yabokkie (7 months ago)

@urix, couldn't find what data I used but I just calculated 39 lux lowlight D800 at ISO100 (4EV).

I'm still learning what DPReview's lowlight is designed for but for me, I may need two levels for low and high ISOs (with 3200, 6400, and 12800 overlapped if possible). the low-light high-ISO part is really to see the dynamic range (for digital gain).

Comment edited 2 times, last edit 11 minutes after posting
0 upvotes
ulfie
By ulfie (7 months ago)

I bet somebody sneaked in and drank all the booze in the bottles of the old scene, and then took them in to get the recycling money. That's really why you had to re-do the whole thing. Heh heh...

4 upvotes
utomo99
By utomo99 (7 months ago)

How about adding moving objects.
Slow moving and also faster moving object ?

Maybe can use some toys for this.
so we can see how good it handle moving object

0 upvotes
camcom12
By camcom12 (7 months ago)

The old scene was quite good, this new one even better. I see the advantages to be:

-improved relative skin tone evaluation
-off-center, identical, symmetrical objects, and corner charts; useful for evaluating lenses in fixed-lens cameras
-'green fuzzies' are great for evaluating jpeg renditions and/or cameras with small sensors
-true-black background, for tone & noise (is it true black?)

But truth be told, charts like this are now most useful for comparing lower cost & smaller cameras since high end camera sensors have become so good in just the last few years (notwithstanding the very subtle, pixel-peeping variations of high-end sensors at high isos).

0 upvotes
Gliderg
By Gliderg (7 months ago)

I downloaded the Olympus E-M5 3200 ISO Jpeg.
The info box reports m4/3 45 f1.8 lens while the EXIF lens data say 4/3 50 f2.
Which is the right lens?
And why didn't you use the 4/3 50 f2 also with E-M1 like you did in the last 5 or 6 years for all 4/3 and many m4/3 cameras?
Even if the scene now is different, at least we can compare more easily the files among different Olympus cameras if the lens is the same.
In addition, for the new m4/3 bodies we can consider all 4/3 lenses as native...

Comment edited 5 minutes after posting
0 upvotes
Richard Butler
By Richard Butler (7 months ago)

The correct lens is the 45mm F1.8. It's a native lens that is closer to the ideal angle of view and it performs really well. It will be the standard lens for Micro Four Thirds cameras.

I'll update the metadata.

2 upvotes
micksh6
By micksh6 (7 months ago)

Why did you close aperture so much? At F5.6 (E-M1 files) diffraction already makes image softer. The 45mm lens is best between F2.8 and F4.0. The EXIF for E-M5 shows more reasonable aperture - F4.5. So, was it 50mm macro for E-M5 or 45mm F1.8?

0 upvotes
guyfawkes
By guyfawkes (7 months ago)

It is irrelevant now as you're moving to a new test target, but I was somewhat taken aback at your comment that the scene had to be shot at very small (diffraction limited) apertures to ensure that all the key targets were acceptably sharp. I am assuming this was to allow for sufficient depth of field.

However, does this mean that the lenses used for the tests may not have been at their optimum apertures and that you may have shot at slightly smaller apertures but before noticeable image degrading diffraction played its part? I'm a little confused as could it be interpreted that not every camera/lens combination was at its best?

0 upvotes
Richard Butler
By Richard Butler (7 months ago)

The old scene was always shot at (slightly) smaller apertures than would be ideal.

2 upvotes
guyfawkes
By guyfawkes (7 months ago)

Thanks. So your new 2D test target could enable you to shoot at the lens' optimum aperture, or would this be too time consuming to always have to work out what it is?

Though I suppose you could keep one test lens for each brand of camera you test, but I can see it would take time to ascertain this where you were testing a fixed lens camera, especially a zoom where I can appreciate that the optimum can shift depending upon focal length and with the requirement to capture the test target.

I'm glad you test cameras and not me. :-D

Comment edited 31 seconds after posting
0 upvotes
DaytonR
By DaytonR (7 months ago)

I will really miss the Martini and Baileys from the old studio scene :)

2 upvotes
guyfawkes
By guyfawkes (7 months ago)

Aha, but don't overlook that the real things are available for purchase. lol.

0 upvotes
buttonz
By buttonz (7 months ago)

What exactly is your "daylight" illumination source?

0 upvotes
schufosi777
By schufosi777 (7 months ago)

Sad to see the old scene go. Sure was fun.

1 upvote
Photato
By Photato (7 months ago)

With Zoom lenses what Focal Length you use or will use?

0 upvotes
Richard Butler
By Richard Butler (7 months ago)

It'll depend on the camera - we'll shoot around the 80/90mm equivalent range unless we get really odd results.

1 upvote
alk_is
By alk_is (7 months ago)

ok, but what about the lenses used. Is it the kit lens for those with interchangeable? And which exactly is, for those with multiple "kit" options?

0 upvotes
Richard Butler
By Richard Butler (7 months ago)

Have you tried clicking on the image settings '[i]' button at the bottom of each image?

2 upvotes
Photato
By Photato (7 months ago)

Scene Dimensions? H.W.D. ?

0 upvotes
Richard Butler
By Richard Butler (7 months ago)

1 x 1.5m for the 3:2 chart.

1 upvote
Photato
By Photato (7 months ago)

Thanks, how deep is it?

0 upvotes
Richard Butler
By Richard Butler (7 months ago)

It's essentially planar, given how far back we're shooting from - the greatest depth is around 5.5cm, but most of it is less than 1cm deep.

3 upvotes
Photato
By Photato (7 months ago)

Thanks Richard.

0 upvotes
steras
By steras (7 months ago)

How is focus set, AF or manually, and at what spot in the target?

1 upvote
Simon Joinson
By Simon Joinson (7 months ago)

manually, bracketed, center

2 upvotes
Jack Hogan
By Jack Hogan (7 months ago)

Thanks. Is the lighting calibrated each time to the same illuminance?

1 upvote
Kelcey Smith
By Kelcey Smith (7 months ago)

Yes, the lights are tested with a light meter before each session.

2 upvotes
Franklin J Ellias
By Franklin J Ellias (7 months ago)

This is great.
I am always concerned about various tests since 90+% of my shoots are not taken in a studio with controlled lighting and the opportunity to "set up the shot".
If I read this notice correctly; you will be covering most of the situations that many of us find ourselves in.

0 upvotes
steelhead3
By steelhead3 (7 months ago)

Are you using the best lens available for each system? Tough to do comparisons of resolution since lens and cameras are a unit.

1 upvote
Kelcey Smith
By Kelcey Smith (7 months ago)

As with the previous test scene, we are using the best native prime lens (~85mm equiv.) for each system where possible. If you check the info button at the bottom of each image you can see which lens was used in the test.

2 upvotes
thornburg
By thornburg (7 months ago)

The Sony DT 50/1.8 is "the best native prime lens" ~85mm equiv. for APS-C Alpha mount cameras? Not the Sony 50mm/1.4 or the Zeiss 50mm? Or the 50mm Macro?

The Pentax K500 uses their 50mm Macro, and the Canons use the 50mm/1.4.

Is this an intentional statement that the DT 50/1.8 is better, or was it a lack of availability or something? Will the shots be redone with one of those other lenses?

0 upvotes
Richard Butler
By Richard Butler (7 months ago)

We have tried the 50mm F1.4 (several copies) and found the 50mm F1.8 performs better. Our intention is to move to the Zeiss 50mm F1.4 when it becomes available.

3 upvotes
yabokkie
By yabokkie (7 months ago)

would appreciate if a same copy of lens used for both Nikon and Canon.

1 upvote
I a n
By I a n (7 months ago)

Perhaps I am missing something obvious but now do we actually get to the place where we can examine and compare cameras using this tool?

0 upvotes
stevevelvia50
By stevevelvia50 (7 months ago)

I prefer the older version as it appears to present more depth of field challenges for the various cameras and formats. I think this is important when evaluating the strengths an weaknesses of various cameras aside from only noise, resolution etc.

0 upvotes
Total comments: 55