Analysis of Rishi Sanyals claims on competence and scientific approach for the K-1 AF test

Started Jul 6, 2016 | Discussions
KKnipser Regular Member • Posts: 210
Analysis of Rishi Sanyals claims on competence and scientific approach for the K-1 AF test
47

So we read a lot of very defensive claims on the side of dpr's writers who put up a user review of the K-1 on this site.

The biggest one is that no user incompetence is part of the reason why failing to get an acceptable number of images in focus where many other users get far better results.

But what happens when we look at the minimal facts provided?

The user misfocussing issue is officially "proven" by one tiny image in the summary:

http://www.dpreview.com/files/p/articles/2896376384/AF-Bike-Test-Weeve-Hold-Off/IMGP2591_001.jpeg

Let us scientifically analyse it:

  1. We see a face not in focus.
  2. We see a thumbnail of the overall framing
  3. The text rambling around it explains that center AF is bad.
  4. The text further explains that AF tracking is bad
  5. AF tracking means handing over of focus from one AF point to another. So the subject obviously needs to be covered by one of the available focus points

Now you download the above image and crop the just keep the overall framing.

And then you use Barry Pearson's .psd file displaying the K-1 viewfinder overlay, which nicely tells you where the K-1s AF points are located.

And now you overlay the AF grid over the framing.

Scientific analysis result:

  1. No part of the cyclist is under the central AF sensor. So this clearly is a failed attempt at testing center AF point accuracy.
  2. The overall image frame also is not kept in a way that the face of the cyclist (which is magnified to the reader as "valid result") is under any one of the AF sensors which should track the subject.
  3. Since the user performing this shot has not been able to maintain the framing of the shot in a way that any of the available AF sensors could cover the subject (not any part, but the face) it is anything but surprising that the AF system did not focus on something (the face so nicely magnified) which is kept beyond its specification.
  4. The user has framed in such a way that all center 25 AF point are focussing on the pathway behind the cyclist in the distance.
  5. The framing does in no way follow the subject to even try to keep the subject under the AF grid. Seems like the camera was set on a tripod fully statically.
    All readers here may judge themselves if that is their real world way of shooting quickly moving subjects.

In short:

  1. The user was not competent enough to keep the subject "face" under the AF points
  2. The user obviously is not competent enough to understand that it is a prerequisite of autofocussing to keep the subject "face" under at least one of the available sensors all the time.
  3. No analysis of the camera AF system can be deduced from the user failing to work the camera within technical specifications.
  4. Do you think it is a very competent approach to completely fix the camera on a tripod and not follow the subject at all when you shoot say BIF or aeroplanes?

I sadly see no way to excuse this. Not for a beginner.

Please feel free to analyse the image presented by the "testing" user yourself and correct me if I am all wrong.

Go check how many images have the face of the cyclist under any of the available focus points and compare to where the majority of focus points are aimed at infinity.

Pentax K-1
If you believe there are incorrect tags, please send us this post using our feedback form.
soheil Veteran Member • Posts: 3,022
Re: Analysis of Rishi Sanyals claims on competence and scientific approach for the K-1 AF test
2

We all know K1 can do better. It's up to owners/users to start a thread and to post their own cyclist AF tracking results Like some did before in different threads, this time in a single thread.

-- hide signature --

Regards,
Soheil
------------------------------
It's the singer not the song.

 soheil's gear list:soheil's gear list
Pentax Optio 330GS Pentax MX-1 Pentax K-7 Pentax K-5 Pentax K-5 IIs +19 more
Donald B
Donald B Forum Pro • Posts: 12,015
Re: Analysis of Rishi Sanyals claims on competence and scientific approach for the K-1 AF test
3

Thank you for your evaluation, very enjoyable read . I can't wait for a response from DPR on this one.

Cheers don

-- hide signature --

Olympus xz1, e-pL5 , EM5 my toys.
http://www.dpreview.com/galleries/9412035244
past toys. k100d, k10d,k7,fz5,fz150,500uz,canon G9.

Zvonimir Tosic
Zvonimir Tosic Senior Member • Posts: 2,859
How on earth can AF-C test be standardised?
9

KKnipser wrote:

So we read a lot of very defensive claims on the side of dpr's writers who put up a user review of the K-1 on this site.

The biggest one is that no user incompetence is part of the reason why failing to get an acceptable number of images in focus where many other users get far better results.

But what happens when we look at the minimal facts provided?

It is even simpler than that what you wrote: there cannot be more 'facts'. In fact, there are no facts at all. While at least AF-S can be measured and standardised to an extent, even in different environments (AF-S environment can be controlled), and is possible to eliminate human factor to an extent, AF-C cannot be standardised nor the human factor can be excluded from it.

If it cannot be standardised, it cannot be analysed.
If it cannot be analysed, it cannot be properly measured.
If it cannot be properly measured it cannot be controlled.
If it cannot be controlled, the result will be different each time.

In other words, if the 'results' vary each time, and vary from person to person, which is exactly the case with K-1, you may throw your own conclusions in the bin. You can make a proper opinion only by a statistical median, based on the experience of several hundred different people performing very similar test repeatedly.

But even then it does not mean that one can't better the median result, or that one can't be much worse than the median value. Because AF-C strictly depends on human factor and the knowledge of the workings of an exact AF system.

-- hide signature --

Madamina, il catalogo è questo; Delle belle che amò il padron mio; un catalogo egli è che ho fatt'io; Osservate, leggete con me.

 Zvonimir Tosic's gear list:Zvonimir Tosic's gear list
Leica X2 Olympus PEN E-P5 Olympus M.Zuiko Digital 17mm 1:1.8 +2 more
klimbkat
klimbkat Senior Member • Posts: 2,336
Re: Analysis of Rishi Sanyals claims on competence and scientific approach for the K-1 AF test

Donald B wrote:

Thank you for your evaluation, very enjoyable read . I can't wait for a response from DPR on this one.

Cheers don

Whats the over/under on whether they respond at all?  Honestly, this is a pretty embarrassing fail, perhaps a form of confirmation bias.

OP KKnipser Regular Member • Posts: 210
I wonder if they redo the test the Nikon D610 underwent for the K-1 and the 6D
4

Now that it is 100% clear they screwed up at trying to run a "test" on the K-1, I wonder if they actually have the minimum professionalism and run both the 6D and K-1 through exactly the same bycicle test, the Nikon D610 had to run through.

This obviously would include

  • downscaling the 36 Mpx images to 24 MPx to compare and
  • also use the same apertures (The D610 was just tested at F4 with resulting more DoF sharpness, while the K-1 in the failed random test was tested at F2.8)
  • and showing similar distanced subject (the first Nikon test subject was shot farther away, making focussing easier)
  • and same light conditions (the Nikon was shot in brighter and contrasty direct sunlight; just check the two cyclists shadows).
kewlguy
kewlguy Senior Member • Posts: 1,952
Re: I wonder if they redo the test the Nikon D610 underwent for the K-1 and the 6D
2

Does it matter? Regardless of how DPR tries to make the K1 look bad - it is still a great camera. The review changes nothing, I still enjoy using it.

 kewlguy's gear list:kewlguy's gear list
Fujifilm X100F Canon EOS 6D Nikon 1 J5 Sony a7R III Sony a7 III +87 more
Mark Ransom
Mark Ransom Veteran Member • Posts: 5,651
Focus point

Rishi claims the focus point was on the jersey, not the face.

http://www.dpreview.com/forums/post/58016958

 Mark Ransom's gear list:Mark Ransom's gear list
Pentax K-7 Pentax K-01 Olympus E-M5 II Pentax smc DA 15mm F4 ED AL Limited Pentax smc DA 18-135mm F3.5-5.6ED AL [IF] DC WR +6 more
SBS
SBS Senior Member • Posts: 2,537
Re: Focus point
5

Mark Ransom wrote:

Rishi claims the focus point was on the jersey, not the face.

http://www.dpreview.com/forums/post/58016958

Wonder why they illustrated their point with an unfocused face?

Richard Murdey
Richard Murdey Senior Member • Posts: 2,879
Re: How on earth can AF-C test be standardised?
1

Zvonimir Tosic wrote:

In other words, if the 'results' vary each time, and vary from person to person, which is exactly the case with K-1, you may throw your own conclusions in the bin. You can make a proper opinion only by a statistical median, based on the experience of several hundred different people performing very similar test repeatedly.

There is a lapse of rigorous logic at "You can make a proper opinion only"

Statistics does not require hundreds of measurements. Technically it requires no more than two. In terms of making a conclusion, of forming an opinion, only a single data point or observation is required.

The more tests, the more measurements you perform, the more confidence you have in the conclusions, but that number could just as easily be 3 as 800. In fact three is very often used in science. Do the measurement three times, if the results cluster within what you imagine to be experimental error, you record the average and move on to the next test.

Evaluating the AF of a camera is the hardest possible part of a review, it's not surprising that dpreview basically ignored it for years and years. It is simply not possible to make a truly repeatable, quantitative, and real-world-applicable test. What they've come up with is a pretty good evaluation: asking "can the camera track focus on an approaching cyclist". There will be variations with lighting, luck, background, and so forth and so on, but a reviewer taking half an hour or so to do it over an over will get a very good relative idea of whether the camera basically does this task acceptably well or not.

That's good enough - high enough level of confidence - for me to accept the conclusion: Pentax, and the K-1 specifically, cannot do this task very well. It doesn't surprise me.

Of course I know from personal experience that Pentax AF-S performance is fine, was already fine in the K-3, and since I rarely use AF-C anyway I'm not fussed, but I would not dispute dpreview's conclusions. Just because one shot might happen to have the AF point off target cannot be used to imply that this was the case for all the out of focus results they obtained - it would be silly to imagine so - however much that might fit with the bunker mentality of some of the people on this forum.

 Richard Murdey's gear list:Richard Murdey's gear list
Nikon D750 Carl Zeiss Distagon T* 2,8/21 Carl Zeiss Distagon T* 2,8/25 Carl Zeiss Distagon T* 2/35 Carl Zeiss Planar T* 1,4/50 +8 more
Rishi Sanyal
Rishi Sanyal dpreview Admin • Posts: 817
Re: Focus point
9

SBS wrote:

Mark Ransom wrote:

Rishi claims the focus point was on the jersey, not the face.

http://www.dpreview.com/forums/post/58016958

Wonder why they illustrated their point with an unfocused face?

Because:

1. It falls within the DOF of the jersey.

2. If you look at the full resolution images which we provide you, you will see the jersey is also out of focus, with the AF system lagging behind the cyclist, typically falling behind the entire bike, catching up from time to time.

We chose the face, as we tend to do in all camera reviews despite focusing on shirt (which provides a higher contrast target) because it's easier to identify when something's out of focus.

If you don't like that, it shouldn't be too hard for you to click on the full-resolution click-throughs, which we *always* provide, to verify for yourself that the conclusions are no different if you pay attention to the jersey, which is also out-of-focus in many shots, except when the camera catches up momentarily.

We could re-make the entire rollover using the crop from his jersey. But do you think that'd stop the claims of incompetency and conspiracy theories?
----------------------
Rishi Sanyal, Ph.D
Technical Editor | Digital Photography Review
dpreview.com (work) | rishi.photography (personal)

 Rishi Sanyal's gear list:Rishi Sanyal's gear list
Sony RX100 IV Nikon D810 Canon MP-E 65mm f/2.5 1-5x Macro Nikon 85mm F1.8G Sigma 35mm F1.4 DG HSM Art +3 more
MightyMike Forum Pro • Posts: 38,915
Re: Focus point
2

Rishi Sanyal wrote:

Because:

1. It falls within the DOF of the jersey.

Not entirely convinced of this, but not disputing your test results. In my experience shooting at F2.8 and 200mm or less as the subject gets closer in my bike park series if the shirt was in focus the head was not, if the handle bars were in focus the head was, if the front tire was in focus the head was not, now the posture may have been different between the riders and i didn't double check yours to see if this was the case.

-- hide signature --

Mike from Canada
"I am not a great photographer! God is a great creator! All I do is capture His creation with the tools He has provided me."
'I like to think so far outside the box that it would require a telephoto lens just to see the box!' ~ 'My Quote :)'
http://www.michaelfastphotography.com/galleries/VP-BDI_3a.jpg
http://www.airliners.net/search/photo.search?sort_order=views%20DESC&first_this_page=0&page_limit=180&&emailsearch=mighty_mike88%40hotmail.com&thumbnails=

Ashurbanipal Forum Member • Posts: 71
Re: Focus point
4

Rishi Sanyal wrote:

We chose the face, as we tend to do in all camera reviews despite focusing on shirt (which provides a higher contrast target) because it's easier to identify when something's out of focus.

Is this mentioned in the review? I don't recall seeing this explained there but perhaps I missed it.

If you don't like that, it shouldn't be too hard for you to click on the full-resolution click-throughs, which we *always* provide, to verify for yourself that the conclusions are no different if you pay attention to the jersey, which is also out-of-focus in many shots, except when the camera catches up momentarily.

It was also not readily apparent (to me, at least) that I could click through to the full-res images, although when I first read the review the images weren't  displaying properly (some images were missing, clicking on anything after the 3rd image just took me back to the 3rd image, etc.), although it seems to be working now.

We could re-make the entire rollover using the crop from his jersey. But do you think that'd stop the claims of incompetency and conspiracy theories?

I don't think anything will stop the claims of incompetence or the conspiracy theorists. But this is the Internet, after all, so I can't imagine anything stopping those. That being said, it would be nice to know the methodology behind the AF tests without having to read through the comments on the review and several threads here in the brand forum. After reading these comments it's clear to me that the tests you performed are a bit different than what I'd thought they were when reading the review.

I'd like to see an article explaining the methodology of the AF tests; that way it's all in one place and I can educate myself on your methods before reading this, or any other, review. This should help reduce potential misunderstanding from readers. If you review a camera I own and your results don't match my experience, I'm likely to assume you don't know what you're doing. However, if you explain how you performed the test in some more detail, I can compare that with my usage and see where it differs from how you've tested the camera. I believe that in this case, at least, that would have alleviated a bit of the criticism and led to a more useful discussion of the issue, once you discount the usual fanboyism (both pro and con) that pervades this site.

I'd also add that it's perhaps a bit understandable that the Pentax users jumped on this one given the kerfuffle over the earlier pixel-shift article. Just more grist for the conspiracy mill...

One final question: in the past it's seemed to me that reviews generally compared cameras of similar levels (entry-level to entry-level, and so forth), but in this one it seems that you've moved to comparing a wider variety of cameras. There's certainly nothing wrong (and quite a few things right!) with that approach, but it's new to me--is it safe to assume this is the standard approach to reviews now? (It's entirely possible this has been standard practice for a while and I just haven't noticed; this is the first review that's actually interested me enough to read in quite some time. If that's case please accept my apologies for being so far behind the times!)

I do appreciate the hard work of you and the rest of the dpreview staff; hopefully this will turn into an opportunity to make further improvements to  the valuable resource that this site has always been for me.

-- hide signature --

Steve

-- hide signature --

Rishi Sanyal, Ph.D
Technical Editor | Digital Photography Review
dpreview.com (work) | rishi.photography (personal)

LightBug Senior Member • Posts: 2,396
Re: Focus point
1

Rishi Sanyal wrote:

SBS wrote:

Mark Ransom wrote:

Rishi claims the focus point was on the jersey, not the face.

http://www.dpreview.com/forums/post/58016958

Wonder why they illustrated their point with an unfocused face?

Because:

1. It falls within the DOF of the jersey.

2. If you look at the full resolution images which we provide you, you will see the jersey is also out of focus, with the AF system lagging behind the cyclist, typically falling behind the entire bike, catching up from time to time.

We chose the face, as we tend to do in all camera reviews despite focusing on shirt (which provides a higher contrast target) because it's easier to identify when something's out of focus.

If you don't like that, it shouldn't be too hard for you to click on the full-resolution click-throughs, which we *always* provide, to verify for yourself that the conclusions are no different if you pay attention to the jersey, which is also out-of-focus in many shots, except when the camera catches up momentarily.

We could re-make the entire rollover using the crop from his jersey. But do you think that'd stop the claims of incompetency and conspiracy theories?
----------------------
Rishi Sanyal, Ph.D
Technical Editor | Digital Photography Review
dpreview.com (work) | rishi.photography (personal)

Rishi,

I have few questions for you if you don't mind answering:

1.  Did you use the center focus point in your single-point AF-C test? I extracted following from IMGP2516.jpg's EXIF:

AF Point Selected : Unknown (17); 0

Which according to following site, equates to center (17), single point (0):

http://www.sno.phy.queensu.ca/~phil/exiftool/TagNames/Pentax.html

If so, the frame's center is not really the shirt, it's the bike's handle-bar or the black shorts of the cyclist.

2.  How far away from the camera was the cyclist's starting point in frame 1?

Thanks for taking time to address our questions.

Joey

 LightBug's gear list:LightBug's gear list
Sony a7R Pentax Q-S1 Pentax K-S2 Pentax smc DA 15mm F4 ED AL Limited Pentax smc DA 21mm F3.2 AL Limited +10 more
asahi man Contributing Member • Posts: 980
Re: Analysis of Rishi Sanyals claims on competence and scientific approach for the K-1 AF test
6

Sorry,but dpreview autofocus tests are the lowest quality autofocus tests I ever seen.

When I remember the low light test between Sony A7 R? Against Nikon 750?

What a rough nonsense test.

Best regards

BobORama
BobORama Senior Member • Posts: 2,633
Re: Focus point
7

Rishi Sanyal wrote:

If you don't like that, it shouldn't be too hard for you to click on the full-resolution click-throughs, which we *always* provide, to verify for yourself that the conclusions are no different if you pay attention to the jersey, which is also out-of-focus in many shots, except when the camera catches up momentarily.

We could re-make the entire rollover using the crop from his jersey. But do you think that'd stop the claims of incompetency and conspiracy theories?
----------------------
Rishi Sanyal, Ph.D
Technical Editor | Digital Photography Review
dpreview.com (work) | rishi.photography (personal)

You could start by constructing repeatable test for AF that can be universally applied to different camera makes and models

You ( I mean the collective "you" at DPR ) could also start by questioning your premises and take a more structured and disciplined approach to formulating your conclusions.

The botched  D810 and K-1 comparison images where the D810 images showed obvious sharpening artifacts, which was evident to the naked eye and easily confirmed mathematically had someone bothered to do a FTF on any one of the images.    Ugh.

What happened is someone looked at the images and said "well, of course the $3300 camera takes sharper pictures than the $1700 camera."   But this is an intellectually lazy, flawed conclusion that should never have escaped your office.

You need to get into the habit of reviewing EXIF and FTFs from your test images and using that as a basis for vetting your own test .

If you want to be a scientist - then may I suggest you take a scientific approach rather than a squishy, folksy approach that leaves the matter unsettled.

 BobORama's gear list:BobORama's gear list
Pentax K-5 Pentax K-1 Sigma 10mm F2.8 EX DC HSM Diagonal Fisheye Pentax smc DA 18-55mm F3.5-5.6 AL WR Samyang 14mm F2.8 IF ED MC Aspherical +9 more
OP KKnipser Regular Member • Posts: 210
numerous images where even the jersey is not under any focus point
7

Rishi Sanyal wrote:

SBS wrote:

Mark Ransom wrote:

Rishi claims the focus point was on the jersey, not the face.

http://www.dpreview.com/forums/post/58016958

Wonder why they illustrated their point with an unfocused face?

Because:

1. It falls within the DOF of the jersey.

2. If you look at the full resolution images which we provide you,

Instead of weaving conspiracy theories of sinister readers who all are against poor pdr writers, we should focus on facts.

Not only the face, but even the jersey was let slip by the testers beyond the scope of any AF point in images IMGP2590, IMGP2591 and IMGP2592.

And that is not a screwed up test? Nobody had the wits to recognize? Not even tried to keep the camera frame on the target?

It is also quite interesting that there has not been any response on the question, why this test is obviously so much different in half a dozen parameters (including aperture) from the one shown for 6D and D610.

BobORama
BobORama Senior Member • Posts: 2,633
Re: How on earth can AF-C test be standardised?

Richard Murdey wrote:

Evaluating the AF of a camera is the hardest possible part of a review, it's not surprising that dpreview basically ignored it for years and years. It is simply not possible to make a truly repeatable, quantitative, and real-world-applicable test. What they've come up with is a pretty good evaluation: asking "can the camera track focus on an approaching cyclist". There will be variations with lighting, luck, background, and so forth and so on, but a reviewer taking half an hour or so to do it over an over will get a very good relative idea of whether the camera basically does this task acceptably well or not.

Its a lot easier than you might think.   The camera does not care if you are a cyclist, or a fast approaching piece of plywood on a dolly.    Also there are very easy to construct optical systems which can simulate different distances to subject.   This could be automated and test both the speed of the AF servo to keep up, as well as get a much larger sample size than "guy on bike" its also something that can be applied to any make of camera.

That's good enough - high enough level of confidence - for me to accept the conclusion: Pentax, and the K-1 specifically, cannot do this task very well. It doesn't surprise me.

It is fine for you to believe such a statement, but not for a reviewer to say "Its crappy, that is what we expected."  It is their job to question their results, not create a self fulfilling prophecy.  Perhaps Nikon likes bicycles and Pentax likes biplanes.   We will never know because we are too busy nodding our heads in agreement than Pentax sucks at AF.

If you gave me a budget of $10,000 I am fairly confident I could construct a rigorous, verifiable AF test that is highly correlated with observed AF behaviour.

 BobORama's gear list:BobORama's gear list
Pentax K-5 Pentax K-1 Sigma 10mm F2.8 EX DC HSM Diagonal Fisheye Pentax smc DA 18-55mm F3.5-5.6 AL WR Samyang 14mm F2.8 IF ED MC Aspherical +9 more
Rishi Sanyal
Rishi Sanyal dpreview Admin • Posts: 817
Re: Analysis of Rishi Sanyals claims on competence and scientific approach for the K-1 AF test
13

asahi man wrote:

Sorry,but dpreview autofocus tests are the lowest quality autofocus tests I ever seen.

When I remember the low light test between Sony A7 R? Against Nikon 750?

What a rough nonsense test.

Best regards

Let me guess: you like the other ones other sites or users provide, where they either don't explain much, don't do controlled tests, or don't tax the AF system by mostly shooting lateral movement, or provide you with downsized 0.7MP images to evaluate where everything's so small and so far away that everything's in focus.

What these sources do do of course, that makes you like them, is praise the camera you own or like, right?

Funny how that works.

 Rishi Sanyal's gear list:Rishi Sanyal's gear list
Sony RX100 IV Nikon D810 Canon MP-E 65mm f/2.5 1-5x Macro Nikon 85mm F1.8G Sigma 35mm F1.4 DG HSM Art +3 more
Rosember Senior Member • Posts: 1,971
Re: Analysis of Rishi Sanyals claims on competence and scientific approach for the K-1 AF test
3

Actually, I like dpreviews test. It has a lot to do with what I am shooting regularly, and where I am constantly frustrated with my results (on a K-3). In a way the results are even reaffirming to me as it might not simply be my poor shooting technique that causes this frustration.

 Rosember's gear list:Rosember's gear list
Pentax K20D Pentax Q Pentax K-3 Pentax smc DA* 55mm F1.4 SDM Pentax smc DA 35mm F2.8 Macro Limited +4 more
Keyboard shortcuts:
FForum MMy threads