Scanner comparison benchmark (discussion)

[Overrank]

There are quite a few people come to this forum with questions like "can I scan 35mm films with a flatbed" or "how does camera copying compare to a film scanner", and nearly always they get lots of conflicting opinions (and normally by the 6th post they're thinking that watercolours would be a good new hobby ). So, inspired by things like "The travelling Yashica" project ( https://www.35mmc.com/18/09/2013/the-traveling-yashica-2/ - I doubt if anyone would be posting a Yashica T5 nowadays ) I thought what about building up a database of scans of the same image by lots of different scanners, cameras etc.

The idea would be to use a standard image (a slide would be best to avoid problems with inversion software ), scan it, post the results on here and then mail the slide onto someone else who would do the same. If initially there were three or four copies of the same scene taken then this could quickly grow into a useful database. "How does a Epson V600 with Silverfast compare to a V850 with EpsonScan", "How does a Kodak Scanza compare to a Sony A7iii + macro lens" etc.

What do people think - would anyone want to be involved ? The idea isn't just to look at "resolution" but also be able to compare the colours of the various methods, effects of IT8 calibration etc. To give you an idea I've attached three scans of the same slide with three different scanners (not IT8 corrected so you can see the difference). All were scanned with VueScan, with no sharpening.

Reflecta Proscan 10T @ 5000ppi

Canon FS4000 @ 4000ppi

Epson V550 @ 3400ppi

I can see a number of problems - defining the initial scene etc, losing the slide etc. but it could be a useful repository. I (for one) would be interested in seeing a drumscan of a 35mm image. Posting a slide (at least within one country) should be pretty cheap as it will in an envelope.

Comments ?

 

[sybersitizen]

The idea would be to use a standard image (a slide would be best to avoid problems with inversion software ), scan it, post the results on here and then mail the slide onto someone else who would do the same.

Nice idea.

I can see a number of problems - defining the initial scene etc, losing the slide etc.

Yep. The scene would have to be well thought out with a variety of details, colors, and tones, and carefully photographed with some good equipment.

Posting a slide (at least within one country) should be pretty cheap as it will in an envelope.

Starting in one country (or one region) would be smart. I think you're in the UK, so maybe see if posters local to you chime in first.

 

[Overrank]

The idea would be to use a standard image (a slide would be best to avoid problems with inversion software ), scan it, post the results on here and then mail the slide onto someone else who would do the same.

Nice idea.

I can see a number of problems - defining the initial scene etc, losing the slide etc.

Yep. The scene would have to be well thought out with a variety of details, colors, and tones, and carefully photographed with some good equipment.

I was thinking you’d want something pretty substantial, buildings rather than trees, with lots of colours and some decent shadows. Unfortunately most buildings where I live are black, white or stone, so it might involve a trip out somewhere. I think that getting lots of colours, textures and tones is more important than absolute fine detail - we can use USAF 1951 for that

Posting a slide (at least within one country) should be pretty cheap as it will in an envelope.

Starting in one country (or one region) would be smart. I think you're in the UK, so maybe see if posters local to you chime in first.

I can see this taking a couple of months to set up (deciding on what to take, shooting, developing etc) so there’s no rush.

 

[smithim]

I'm in the UK.

It's an interesting idea, but unless you standardise the procedure I'm not sure that you're actually comparing scanners - e.g. is the result down to the scanner itself, the software used or the particular settings used? Or are we looking at the best scan that each system can do? Do we use digital ICE, pixel polish etc? levels of default sharpening may be very different with various software. Agreed that a slow slide film would be best.

I agree about the type of subject. What about the file format - presumably raw or tiff?

As far as the results go, to get the best possible scans may produce files between 100-200Mb each. Can dpreview handle files that size?

This is the type of subject that I'd like to see, personally :-

 

[sybersitizen]

What about the file format - presumably raw or tiff?

Only needed if you plan on extensive PP, which probably shouldn't be part of the process.

As far as the results go, to get the best possible scans may produce files between 100-200Mb each. Can dpreview handle files that size?

Even if you did such extensive PP, you'd then generate JPEGs for posting here.

RAW and TIFF are not supported by DPR forums. You'd have to host those files elsewhere.

 

[Overrank]

I'm in the UK.

It's an interesting idea, but unless you standardise the procedure I'm not sure that you're actually comparing scanners - e.g. is the result down to the scanner itself, the software used or the particular settings used? Or are we looking at the best scan that each system can do? Do we use digital ICE, pixel polish etc? levels of default sharpening may be very different with various software. Agreed that a slow slide film would be best.

To discuss those types of things was one of the reasons I posted. I’m not sure that you can separate the performance of the scanner from the software used to drive it, so I’d sort of assumed the results would be “Plustek 8200i + SilverFast SE”,

I was assuming that the images would be the best, I.e. most faithful to the original, that could be obtained from the scanner / camera setup. So if ICE is part of the scanner or software that would be included - I’m not so sure about spotting out dust etc. Could maybe include additional details like the time taken to produce that image (which would include hand spotting time).

Theres a question about whether you want finished images or images that would be going into a editing chain, so things like flat or linear scans etc. That would be for discussion but I think the images have to be reasonably finished - my idea wasn’t to determine a “best”, as it’s always a trade off. It was more to have a resource to point someone too and say “if you want to spend £150 and do one scan every 5 seconds then the output will be like this (Scanza), if you want to spend £300 and take 4 mins per slide it will be like this (Plustek) and if you want to spend £10k it will be like this (Imacon)”

I agree about the type of subject. What about the file format - presumably raw or tiff?

Ideally tiff, but that would mean that it couldn’t be posted on dpreview. Camera raw wouldn’t (necessarily) allow the application of things like focus stacking or IT8 calibration for camera images, and not all scanning software will produce raw files either.

As far as the results go, to get the best possible scans may produce files between 100-200Mb each. Can dpreview handle files that size?

I wasn’t aware that dpreview couldn’t take raw/tiff until sybersitizen said it can’t. I suspect that jpegs won’t be too big

This is the type of subject that I'd like to see, personally :-

Yes, I was thinking building with some colours in there, either from flowers or artwork. Probably need to wait a few months for the flowers to come up, at the moment it’s all daffodils

 

[smithim]

RAW and TIFF are not supported by DPR forums. You'd have to host those files elsewhere.

So highest quality jpg would still be OK...

 

[smithim]

I was assuming that the images would be the best, I.e. most faithful to the original, that could be obtained from the scanner / camera setup.

So wouldn't that also be affected by the operator's experience?

e.g. I simply choose a few options and out pops the scan - no attempt at colour profiling, as long as it looks close to the original. Someone with more skill might be able to produce something much 'better'

 

[Overrank]

I was assuming that the images would be the best, I.e. most faithful to the original, that could be obtained from the scanner / camera setup.

So wouldn't that also be affected by the operator's experience?

e.g. I simply choose a few options and out pops the scan - no attempt at colour profiling, as long as it looks close to the original. Someone with more skill might be able to produce something much 'better'

Unfortunately it would, I’m not really sure what you can do about that. Possibly for some of the more popular scanners you might get a few people doing their own scans. Even familiarity with software can affect it - for example I’m much more familiar with SilverFast than VueScan but I wanted something that would work with three scanners for the demo so I used VueScan.

Theres also questions about calibration, as people might not have an Ektachrome target or a Velvia target or whatever is the final film chosen. But in the end we can only do what’s possible.

 

[smithim]

Even familiarity with software can affect it - for example I’m much more familiar with SilverFast than VueScan but I wanted something that would work with three scanners for the demo so I used VueScan.

I can't be sure, as I can no longer run it, but I'm sure that I used to have more detailed options with the old Minolta software. I'm now using VueScan as it runs on Win10, but I'm not convinced that the results are as good. Difficult to prove without running them side by side, though...

Theres also questions about calibration, as people might not have an Ektachrome target or a Velvia target or whatever is the final film chosen.

I certainly don't.

But in the end we can only do what’s possible.

Agreed... As long as the results are taken in context, it could still be worth doing.

 

[DrBormental]

TLDR: your idea is not going to work for evaluating colors.

You mentioned IT8 calibration yourself, and that's the key point. Every image capturing device has its own unique color response and should come with a customized color profile (DCP file for Adobe users) or provide a way to build one.

Adding digital cameras into the mix complicates situation even more, because their color profiles cannot be shared: you need a color profile for each camera+light source combination! And some camera users digitize with cyan/blue filters to extend the dynamic range of their sensors even further, which makes each camera-scanning rig fairly unique.

Without a color profile it is utterly pointless to discuss the color output of any device. The only three variables that can be objectively evaluated with your proposed database of samples are:

• Resolution
• Noise
• Dynamic range (closely related to noise anyway)

I will say that calibrating using transparent medium such as film is a lost art. Camera calibration kits all use reflective medium, and Kodak is no longer offering their transparent color targets. Even if you find a used one on eBay it's long expired and the colors have shifted.

I only found one vendor (Wolf Faust) who still makes IT87 targets. Too early to comment on quality, I'm in the beginning of my journey with them.

 

[Overrank]

TLDR: your idea is not going to work for evaluating colors.

You mentioned IT8 calibration yourself, and that's the key point. Every image capturing device has its own unique color response and should come with a customized color profile (DCP file for Adobe users) or provide a way to build one.

Adding digital cameras into the mix complicates situation even more, because their color profiles cannot be shared: you need a color profile for each camera+light source combination! And some camera users digitize with cyan/blue filters to extend the dynamic range of their sensors even further, which makes each camera-scanning rig fairly unique.

Without a color profile it is utterly pointless to discuss the color output of any device. The only three variables that can be objectively evaluated with your proposed database of samples are:

• Resolution
• Noise
• Dynamic range (closely related to noise anyway)

I will say that calibrating using transparent medium such as film is a lost art. Camera calibration kits all use reflective medium, and Kodak is no longer offering their transparent color targets. Even if you find a used one on eBay it's long expired and the colors have shifted.

I only found one vendor (Wolf Faust) who still makes IT87 targets. Too early to comment on quality, I'm in the beginning of my journey with them.

I would have thought that *where* people have IT8 calibration targets that they will be using them anyway for slides? I have a mix of the LaserSoft (Fuji) and Wolf Faust targets (more Fuji and Kodak) but I don’t have a target for the latest version of Ektachrome. These should work when using a camera - I did a very quick test a while ago with the Nikon ES-2 and VueScan. Obviously where people don’t have IT8 it would be straight out of the box, but there should be less variation than with negative film, which seems more dependent on peoples intent for the photograph.



I didn’t use an IT8 target with the examples I put up because Ive never created calibrations for the FS4000 or V550 as I don’t normally use them for slides. If I get chance I’ll redo them with IT8 calibration and post the results.

 

[sprocket-docket]

I'd be willing to scan something if it's sent to me in the US, and show the files from an Epson V550 flatbed and a (much better) Pacific Image XA Super.

 

[DrBormental]

Overrank wrote: I would have thought that *where* people have IT8 calibration targets that they will be using them anyway for slides? I have a mix of the LaserSoft (Fuji) and Wolf Faust targets

Prior to selling my Plustek 120 Pro last year, I played with the LaserSoft target it came with, but couldn't find a way to use it outside of SilverFast. And even there the calibration only worked for positive (slide) film scanning IIRC, so I deemed it useless. Have you found a way to build, say, a DCP profile with them?

 

[Overrank]

Overrank wrote: I would have thought that *where* people have IT8 calibration targets that they will be using them anyway for slides? I have a mix of the LaserSoft (Fuji) and Wolf Faust targets

Prior to selling my Plustek 120 Pro last year, I played with the LaserSoft target it came with, but couldn't find a way to use it outside of SilverFast. And even there the calibration only worked for positive (slide) film scanning IIRC, so I deemed it useless. Have you found a way to build, say, a DCP profile with them?

I mostly use SilverFast so they're OK for me (and obviously only for slide film). I think in SilverFast there is a way of having multiple calibrations for different slide films, but I've not managed to get that to work. But, as I'm only shooting Ektachrome for sides now I can just calibrate against that.

In VueScan I've done some playing with creating ICC profiles and then using them within VueScan to colour correct. They're not bad but I think they could probably be better. These are a couple from the image before, scanned on the V550 and the Reflecta - they're closer than they were, but oddly have swapped, so the V550 is now a little lighter. The Reflecta is much closer to the colour of the slide now - before it was too orange.

Reflecta ProScan 10T, VueScan 9.7, Wolf Faust Ektachrome IT-8 target

Epson V550, VueScan 9.7, Wolf Faust Ektachrome IT-8 target

I think there's still more I can do with them though. Silverfast pretty much automates the process, VueScan is more hands on.

 

[Overrank]

I'd be willing to scan something if it's sent to me in the US, and show the files from an Epson V550 flatbed and a (much better) Pacific Image XA Super.

Thanks. Hopefully we'll get some nice weather in the next month or two and I can get some source slides shot.

 

[Bob Janes]

It is an interesting idea - but there are problems.

I think a resolution test chart would be more useful than a scene - Scandig reviews use a basic chart to judge resolving power.

I don't think you need worry too much about colour because most scanners will not interpolate - instead they scan with three different filters. I've found a lot of variance between white balance settings even using the same software across scanners from the same company.

Single pass enlargements. Top row: Dimage Scan Dual II (2820dpi), Dimage Scan Dual IV (3200dpi), Dimage Scan Elite 5400 (5400dpi) Bottom row: Epson V350 flatbed (4800dpi), Sony a99 with 100 macro D (full-frame, 24MP)

You can compare colour on different scanners on how many bits they use per channel.

Then there is copy variance (which lots of samples from lots of sources would help with) - and I'm guessing that in non-autofocusing machines, this could be significant. The Scandig review of the Plustek 135 rated it very close to its stated resolution 3280 against 3600. However, when I looked at the results from my 135i for my review on 35mmc , I don't think it managed resolution that high.

Is there a standard reference test chart slide?

--
Save a life, become a stem-cell donor.
Hello to Jason Isaacs!

PoTB - Bob Janes

This gallery hosted by SmugMug; your photos look better here.

bobjanes.smugmug.com

Please respect a BY-NC-ND cc licence on all my public internet images

 

[Overrank]

It is an interesting idea - but there are problems.

I think a resolution test chart would be more useful than a scene - Scandig reviews use a basic chart to judge resolving power.

I don't think you need worry too much about colour because most scanners will not interpolate - instead they scan with three different filters. I've found a lot of variance between white balance settings even using the same software across scanners from the same company.

Single pass enlargements. Top row: Dimage Scan Dual II (2820dpi), Dimage Scan Dual IV (3200dpi), Dimage Scan Elite 5400 (5400dpi) Bottom row: Epson V350 flatbed (4800dpi), Sony a99 with 100 macro D (full-frame, 24MP)

You can compare colour on different scanners on how many bits they use per channel.

Then there is copy variance (which lots of samples from lots of sources would help with) - and I'm guessing that in non-autofocusing machines, this could be significant. The Scandig review of the Plustek 135 rated it very close to its stated resolution 3280 against 3600. However, when I looked at the results from my 135i for my review on 35mmc , I don't think it managed resolution that high.

Is there a standard reference test chart slide?

I have done some scanner resolution tests using a cheap IT8 target previously ( see https://www.dpreview.com/forums/thread/4547755 - long thread ). This was for a Plustek 8200i, Reflecta 10T, Epson V550, Canon FS4000, a Nikon D610 + ES-2.

The observed resolutions there were

Plustek 8200i
Expected 7200
Actual 3505

Plustek 8200i
Expected 3600
Actual 2794

Canon FS4000US
Expected 4000
Actual 3150

Epson V550
Expected 3600
Actual 1575

Nikon D610 + f/2.8 50mm D + ES2
Expected 4233 (based on 4000 pixels per 24mm)
Actual 2794

Reflecta Proscan 10T
Expected 5000

Actual 3960

Barry Twycross did some tests of a Minolta Scan Elite 5400. and that was 3960ppi

Based on recommendations from Bernard Delly I then looked at MTFs and got the following

Plustek 8200i @ 3600dpi - 3600dpi
Plustek 8200i @ 7200dpi - 3825dpi
Canon FS4000 @ 4000dpi - 4000dpi
Epson V550 @ 1200dpi - 1200dpi
Epson V550 @ 2400dpi - 1275dpi
Epson V550 @ 3200dpi - 1200dpi
Reflecta 10T @ 5000dpi - 5000dpi

But I’m no longer sure that the MTFs are representing the actual scanner resolution, I’m starting to wonder if they’re the optical system resolution.

Its worth adding that the USAF 1951 target is very jumpy, so 3150 ppi probably means you could read one more bar than 2794 ppi. And other USAF 1951 targets may have different boundaries.

One of the nice things with that target is that the pattern is repeated, so you can use it to test the resolution at the edges (for example I seem to remember that resolution at the edges of the D610 + 60mm + ES2 was less than at the centre, presumably because you’re using a big single lens)