Comparing low light performance of different cameras

[Physics1]

I’m thinking about buying a new camera to replace my 2008 Canon Powershot G10. Clearly then I am not a professional photographer, or even a serious enthusiast. But I do like to take reasonable quality photos, and in particular, I care about low light performance. I shoot JPEG and I don’t shoot videos. I have had a physically bulky SLR in the past, but never looked back after buying the much more compact Canon G10. I have no interest in carrying and changing multiple lenses. The G10 has a 5:1 zoom range which I find to be about right for my needs. Less zoom is not enough, while more unduly compromises the speed of the lens and thus the low light performance. So for my needs, I’m looking at a high-end point-and shoot camera, or maybe a mirrorless one fitted with a zoom lens. I have no idea if others on a photography forum would have the slightest interest in low-performance cameras of this type.

OK. So I happened to be walking past a camera shop with my G10 in my pocket and, on a whim, I stepped inside with the intention of actually trying a number of cameras to see how their low light performance compared. And then I realized that deciding on the methodology for such a comparison is not entirely straight forward.

For 90% of the time I simply point and shoot. Shock and horror! But with the likely exception of most on this forum, I suggest that most people overall do exactly the same. So for those that normally point and shoot, arguably the most relevant test for comparing low light performance is to simply point and shoot at some particular scene in low light, and let the camera decide on the best settings for shutter speed, f-number and ISO. Of course, all cameras must be zoomed so that the same scene fills the frame, which is another way of saying that the angular field of view must be same for each camera being compared. So the camera shop kindly turned off the lights for me, and I performed comparison testing as described, with the scene being one wall of the shop. The pictures are then compared on the basis of image noise, with a particular part of the image zoomed in on the display device.

However, being the scientific type, I was not entirely happy with this approach where the camera was choosing the shutter speed and f-number. Surely it is not a fair (or at least scientific) comparison unless all cameras being compared are set to the same shutter speed and f-number?

OK. So let’s talk about shutter speed. The practical reality (at least for the stationary scenes that I shoot) is that unless you use a tripod, the lowest usable shutter speed is set by the effectiveness of the image stabilization. So all else equal, a camera with better IS also has better low light performance, because a longer expose is possible. Well that throws the cat among the pigeons. However, being the scientific type that likes to tie down the parameters, I took the view that all modern cameras have similar IS, so best to ensure that all camera under comparison are set to the same shutter speed. I presume others would agree.

So what about f-number? Perhaps for a fair comparison the f-number should be set identically for all cameras? I suspect that is what most people do when comparing cameras using the DPR Studio Scene here :-

https://www.dpreview.com/reviews/im...1&x=-0.8793179317931795&y=0.16869931261429288

where in this case (actually provided by Bill Ferris in my previous thread) you can see that the 4 cameras under comparison are indeed all set to the same shutter speed of 1/20 sec and f/5.6. That ensures identical exposure (light/area on the sensor) so in that case it is obvious that the amount of captured light will be proportional to the area of the sensor, so image noise is less for the camera with larger sensors, and this is indeed observed in these DPR images.

Hmmm. But wait a minute. What I am doing is comparing the low light performance of different cameras when used in the real world, and for that propose, surely every camera should be set such that it is “doing the best that it can” in low light, optimally using whatever hardware comprises that camera. And that means that every camera being compared must be set to the lowest available f-number. The noise performance will (and always does) scale according to which cameras have the maximum effective aperture diameter, as that is what determines how much light is actually captured and sent to the sensor, and this may or may not correlate with sensor size. Or one can use “equivalent f-numbers”. Anyway, the real point I am making is that for meaningful comparison of low light performance, all cameras need to be set with their lens apertures fully open, and not to the same f-number. Of course, all images need to be made equally bright by appropriate setting of ISO.

Does anyone agree or disagree with any of the above?

Anyway, using the procedure above of equal FOV, same shutter speed and lowest f-number, I did manage (rather roughly) to compare low light performance of my G10 with a few new cameras in the shop, and found no surprises. I said the comparison was a bit rough, well you know how it is, with the shop staff frantically turning the lights on and off, and pesky customers standing in the way of the scene etc :-D

FWIW, for my needs I quite like the Sony A6700 fitted with Sony f/3.5-5.6, 27-202 equiv focal length lens, except that it has no inbuilt flash. For me, the discontinued A6500 with flash is the better camera. I hate using flash BTW which is why low light performance matters to me, but there are times when you need a flash whether you like it or not, and have no intention of carrying an external flash everywhere. A second hand A6500 is probably the go. Or if I want something smaller, then the Canon G7X Mk3.

 

[MarkInSF]

Do remember that aperture affects depth of field, and is often an artistic choice, not just a way to get better low-light images. Lenses may also have compromises in their designs that come into play.

The Sony (either one) is a very reasonable choice. Many people dislike the handling, but if you're mainly shooting in auto mode that matters less. The newer a6700 has better autofocus abilities (about as good as it gets), including advanced subject recognition and tracking, so would be better for active kids, pets, and sports. I haven't looked at them, but there are often tiny, folding flashes available for these sort of systems that can be left permanently attached (if you treat your camera with reasonable care).

 

[jiberlin]

All cameras i know open the aperture as wide as possible in low-light before cranking up the ISO. What remains is the point at which the camera 'switches' from using a longer exposure to using a higher ISO. On more sophisticated cameras this can adjusted in the configuration (Minimum shutter speed for Auto-ISO).

But i think for your mentioned use case all this is not relevant. If you use the camera in simple point'n'shoot, it is only relevant what the camera choose.

To get an overview of low-light performance use the test-scene compare in the DPreview tests.

 

[Physics1]

All cameras i know open the aperture as wide as possible in low-light before cranking up the ISO. What remains is the point at which the camera 'switches' from using a longer exposure to using a higher ISO. On more sophisticated cameras this can adjusted in the configuration (Minimum shutter speed for Auto-ISO).

But i think for your mentioned use case all this is not relevant. If you use the camera in simple point'n'shoot, it is only relevant what the camera choose.

I normally use point and shoot except sometimes for challenging shots, or when I don't agree with the settings chosen by the camera. But for those that use point and shoot almost exclusively, do you agree that the sensible testing methodology for comparing low light performance of different cameras is to use point and shoot for taking the images that will be used to compare image noise?

To get an overview of low-light performance use the test-scene compare in the DPreview tests.

The DP Test Scene is certainly a valuable and wonderful resource. However, you may not have read my first posting carefully, where I point out that they generally compare cameras at the same f-number, shutter speed and ISO. And as I pointed out, this is not useful for comparing the low light performance of different cameras. If you are setting out to compare low light performance, then each camera should be set to the lowest available f-number, not the same f-number. Read the first posting again and tell us if you agree.

As a related question, if you do compare different cameras, all set to the same f-number, shutter speed and ISO, then exactly what are you measuring and comparing? You are not measuring or comparing the low noise performance of which each camera is capable, so just what are you measuring? It is worth thinking about.

 

[108]

Yes, the A6500 with 18-135 is an excellent choice , zoom versatile enough, iso good up to 3200, and 6400 with care

 

[Sonyshine]

A Pixel or iPhone has phenomenal computational low light ability.


Most have from wide to 5x optical zoom.

There is nothing they can’t do that meets your specifications above and plenty they can do that exceeds it.

 

[Horacecoker]

All cameras i know open the aperture as wide as possible in low-light before cranking up the ISO. What remains is the point at which the camera 'switches' from using a longer exposure to using a higher ISO. On more sophisticated cameras this can adjusted in the configuration (Minimum shutter speed for Auto-ISO).

But i think for your mentioned use case all this is not relevant. If you use the camera in simple point'n'shoot, it is only relevant what the camera choose.

I normally use point and shoot except sometimes for challenging shots, or when I don't agree with the settings chosen by the camera. But for those that use point and shoot almost exclusively, do you agree that the sensible testing methodology for comparing low light performance of different cameras is to use point and shoot for taking the images that will be used to compare image noise?

To get an overview of low-light performance use the test-scene compare in the DPreview tests.

The DP Test Scene is certainly a valuable and wonderful resource. However, you may not have read my first posting carefully, where I point out that they generally compare cameras at the same f-number, shutter speed and ISO. And as I pointed out, this is not useful for comparing the low light performance of different cameras. If you are setting out to compare low light performance, then each camera should be set to the lowest available f-number, not the same f-number. Read the first posting again and tell us if you agree.

As a related question, if you do compare different cameras, all set to the same f-number, shutter speed and ISO, then exactly what are you measuring and comparing? You are not measuring or comparing the low noise performance of which each camera is capable, so just what are you measuring? It is worth thinking about.

At the same f-number you are comparing the low-light performance for the same DOF if the sensor sizes are the same in the cameras you are comparing. That's an important consideration for a lot of photographers.

 

[Bill Ferris]

So, you're upgrading from a Canon G10 (1/1.7-inch CCD) and the current top contender is a used Sony A6500 (APS-C) system. Let's establish a baseline of equivalent performance to see how the A6500 compares and what other crop sensor systems might also be worth considering.

The G10's 1/1.7-inch sensor has a crop factor of 4.6x. Its 6.1-30.5mm, f/2.8-4.5 built-in zoom lens is equivalent to (puts as much light on its sensor at a given framing and shutter speed) as a 28-140mm, f/13-21 lens on a full-frame body.

A compact portable system is important to you so let's translate the above to APS-C. The G10's crop factor relative to most APS-C bodies (Canon APS-C will be slightly different) is 3.07x. This makes the G10's zoom equivalent to a 19-94mm, f/9-14 lens on a 1.5x APS-C body such as the A6500.

You mentioned the Sony 18-135mm, f/3.5-5.6 zoom as a lens you'd consider pairing with the A6500. At the wide end, you wouldn't be giving up anything to what the G10 currently provides in framing. You'd be gaining significant additional reach at the long end. From a light-gathering standpoint, the Sony zoom puts much more light on the A6500 sensor.

The A6500, being a newer body, is built around a CMOS sensor and better internal image processing. Combined with its greater light-gathering potential, the A6500 has far superior dynamic range compared with the ca. 2007 G10. Here's a link to the photonstophotos chart comparing the G11 with the A6500: https://photonstophotos.net/Charts/PDR.htm#Canon PowerShot G11,Sony ILCE-6500

The A6500 delivers 2.5-stops more dynamic range than the G11 and - to offer a conservative estimate - should blow the G10's doors off when it comes to low-light performance

The best aspect of the upgrade you're doing is that you've gotten so many years of good use from the G10 that any APS-C camera you might choose from the last 5-6 years should deliver obviously improved performance on multiple levels, including low-light situations.

Within the APS-C format universe, you might also consider Fujifilm X-series. A used X-T30II paired with Fujifilm's 18-135mm f/3.5-5.6 zoom is one option. Fujifilm makes a complete line of dedicated APS-C lenses and you may find their systems a bit more compact & portable than others. They offer retro styling which may or may not appeal. Fuji also makes rangefinder style bodies, if that's a preference.

Good luck with your upgrade.

 

[jrtrent]

. . . Hmmm. But wait a minute. What I am doing is comparing the low light performance of different cameras when used in the real world, and for that propose, surely every camera should be set such that it is “doing the best that it can” in low light, optimally using whatever hardware comprises that camera. And that means that every camera being compared must be set to the lowest available f-number. The noise performance will (and always does) scale according to which cameras have the maximum effective aperture diameter, as that is what determines how much light is actually captured and sent to the sensor, and this may or may not correlate with sensor size. Or one can use “equivalent f-numbers”. Anyway, the real point I am making is that for meaningful comparison of low light performance, all cameras need to be set with their lens apertures fully open, and not to the same f-number. Of course, all images need to be made equally bright by appropriate setting of ISO.

Does anyone agree or disagree with any of the above?

I might differ on a couple things. For instance, on a full-frame camera, f/1.4 might not provide enough depth of field to satisfy me, or it might have other issues like heavy vignetting or compromised across-the-frame resolution, so setting such a camera to "the lowest available f-number" might result in simply deleting a bunch of disappointing pictures. To get equal brightness, it's both the ISO and the shutter speed that need to be set in concert with the allowable aperture.

When I compare the low light performance of different cameras, I look at: 1) how high an ISO I can use and still get keepable image quality; I've found Imaging Resource a reliable guide, whatever they say is the highest ISO for a good 11 X 14 inch print has worked as a good maximum ISO for me; 2) how fast an aperture I can use and still get enough depth of field and absence of lens aberrations (lens test sites can help out here); and 3) how slow a shutter speed I can use and still get satisfyingly sharp hand-held pictures (this varies with focal length, but I shoot about 95% of my pictures at around a 50mm equivalent).

The best low-light camera I've owned has been the Panasonic FZ300. Lens performance is still quite good wide open at f/2.8, and f/2.8 gives the same depth of field that full-frame would give at nearly f/16; ISO 400 (Imaging Resource's max for an 11 X 14 inch print) provides image quality I'm happy with; it has image stabilization that reviews say is good for about 4 stops of slower shutter speeds for static subjects; for my aging hands, I find 3 stops (relative to the inverse of the full-frame equivalent focal length) to be very reliable.

I like -1/3 stop exposure compensation with this camera, so sunny f/16 settings for bright sunny, EV 15 conditions would be equivalent to f/16, 1/125 second, ISO 100. For low light using my preferred 50mm equivalent focal length, I can use f/2.8 (gaining 5 stops), 1/8 second (gaining 4 stops), and ISO 400 (gaining 2 stops). Add it up, and I'm good down to EV 4 light levels. The dim interiors I've experienced (museums, classrooms, conference halls) rarely go below EV 5, so my tiny-sensored bridge camera works well in low light for me.

 

[Bob A L]

The low light capabilities of any new APSC ILC camera is so far above that of your Canon that there is really no comparing the two. High ISO's are so much better as is stabilization and in camera processing that there is just a different world. I recently traded off my Canon like yours or I would have been glad to take a comparison shot or two.

 

[Physics1]

A Pixel or iPhone has phenomenal computational low light ability.

Most have from wide to 5x optical zoom.

There is nothing they can’t do that meets your specifications above and plenty they can do that exceeds it.

Yes, I was already aware that a modern high-end mobile phone has unbeatable low-light performance, especially for stationary scenes, and had considered that as a possibility. I didn't mention it for fear of being banned from the Forum for life for mentioning the word mobile phone on a camera-photography forum

I don't like mobile phones, but that does not alter the facts.

One problem is that I would never use an expensive, high end phone as my phone that I often carry with me and use for making phone calls, because they are too big to conveniently fit in my top pocket, and I know from experience that my mobile phone that I often carry and use as a phone will get dropped from time to time, or damaged or lost. My present (small, second hand) iphone lasted about 2 months before being dropped and smashing the screen.

So that means I would be buying a fairly expensive phone that would not have a SIM card installed, and would be used only as my camera. Now even that is not necessarily absurd as a good camera would cost as much or more and be bigger, except that from my experience with my iphone, it is utterly useless unless connected to the internet. For example, try transferring files (eg pictures) from your PC to your iphone without the internet. It is for all practical purposes impossible. Perhaps Android is better, I don't know. But what I do know is that I prefer the old fashioned approach where I simply connect my camera to my PC with a USB cable, and it comes up as a virtual drive, and you can easily transfer files and do whatever you like with no need for sim card or internet. Can you do that with a Pixel 8, for example?

 

[Physics1]

So, you're upgrading from a Canon G10 (1/1.7-inch CCD) and the current top contender is a used Sony A6500 (APS-C) system. Let's establish a baseline of equivalent performance to see how the A6500 compares and what other crop sensor systems might also be worth considering.

The G10's 1/1.7-inch sensor has a crop factor of 4.6x. Its 6.1-30.5mm, f/2.8-4.5 built-in zoom lens is equivalent to (puts as much light on its sensor at a given framing and shutter speed) as a 28-140mm, f/13-21 lens on a full-frame body.

A compact portable system is important to you so let's translate the above to APS-C. The G10's crop factor relative to most APS-C bodies (Canon APS-C will be slightly different) is 3.07x. This makes the G10's zoom equivalent to a 19-94mm, f/9-14 lens on a 1.5x APS-C body such as the A6500.

You mentioned the Sony 18-135mm, f/3.5-5.6 zoom as a lens you'd consider pairing with the A6500. At the wide end, you wouldn't be giving up anything to what the G10 currently provides in framing. You'd be gaining significant additional reach at the long end. From a light-gathering standpoint, the Sony zoom puts much more light on the A6500 sensor.

The A6500, being a newer body, is built around a CMOS sensor and better internal image processing. Combined with its greater light-gathering potential, the A6500 has far superior dynamic range compared with the ca. 2007 G10. Here's a link to the photonstophotos chart comparing the G11 with the A6500: https://photonstophotos.net/Charts/PDR.htm#Canon PowerShot G11,Sony ILCE-6500

The A6500 delivers 2.5-stops more dynamic range than the G11 and - to offer a conservative estimate - should blow the G10's doors off when it comes to low-light performance

The best aspect of the upgrade you're doing is that you've gotten so many years of good use from the G10 that any APS-C camera you might choose from the last 5-6 years should deliver obviously improved performance on multiple levels, including low-light situations.

Within the APS-C format universe, you might also consider Fujifilm X-series. A used X-T30II paired with Fujifilm's 18-135mm f/3.5-5.6 zoom is one option. Fujifilm makes a complete line of dedicated APS-C lenses and you may find their systems a bit more compact & portable than others. They offer retro styling which may or may not appeal. Fuji also makes rangefinder style bodies, if that's a preference.

Good luck with your upgrade.

Thank you for your analysis which I agree with 100%. I had already calculated equivalent f-numbers for the G10 and A7500, and got the same numbers, and drawn the same conclusions. I did likewise for the Sony RX100 and Canon G7X.

Thank you for alerting me to the Fujifilm XT30-11, which can be paired with their 18-135 lens. This camera does have an inbuilt flash so it is of interest. I don't care if the look is retro or not per se, but if anything I do like the analog control dials.

The XT-30 is 118x83x47 mm vs 120x67x53 mm for the Sony, so similar size, just a slightly different shape. However the Fuji 18-135 lens is bigger, Diam 76 x Length 98, versus a very compact D67 x L88 for the Sony. I really don't want to go back to carrying a camera with a lens that big, actually the same size as on my old 35 mm film SLR. The A6700 would be perfect if it had a flash ....

You mentioned buying 2nd hand, but if I felt it was exactly right then I would be prepared to buy new. It seems like the XT30 camera is cheaper than the Sony A6700, but the Fuji lens is more expensive. The Fuji lens is arguably better made with more metal and less plastic, though it is unclear if if it is better optically. Apparently both are good optically for an 18-135 zoom lens. Let me think about it.

The reality of course is that, all else equal, the X7.5 zoom would make it a slower lens than the X5 zoom lens on my old G10, and the main reason that these Sony or Fuji lenses still end up well in front for speed is because they are vastly bigger. Good low light performance still needs a big lens. Except for mobile phone cameras which cheat by combining multiple images using their vast computational power.

 

[jiberlin]

...

But i think for your mentioned use case all this is not relevant. If you use the camera in simple point'n'shoot, it is only relevant what the camera choose.

...

To get an overview of low-light performance use the test-scene compare in the DPreview tests.

...

As a related question, if you do compare different cameras, all set to the same f-number, shutter speed and ISO, then exactly what are you measuring and comparing? You are not measuring or comparing the low noise performance of which each camera is capable, so just what are you measuring? It is worth thinking about.

What the DPreview test scenes show is the noise at different ISO settings. It is simple to transfer this to your requested 'low-light performance'. If you have a camera A with a maximum aperture of 2.8 and a camera B with a maximum aperture of 2.0, camera A would need an ISO setting double that of camera B if everything else stays same.

Shutter speed is a very different thing. The old rule of thumb is to use a shutter speed 1/focal-length (FF equivalent) for hand holding. For most cameras the effectivity of the image stabilization is in the specs. If you use this, the slowest shutter speed for hand holding is IS-factor/focal-length, if you take pictures from static scenes.

You can include the image stabilization into the calculation of ISO settings to compare by simple multiply the ISO setting ratio to compare with the ratio of IS effectivities from camera A and B.

 

[Sonyshine]

I use a wallet case for my phone's. I have to as my phone's regularly get dropped, and wet. They protect phones very well indeed. They also give you an extra panel to hold when you are taking photos.

It is possible to transfer photos by cable but not straight forward. I move mine around by WiFi and internet , this is done automatically anyway so my photos appear on my computer and iPads almost instantly.

 

[Physics1]

To get an overview of low-light performance use the test-scene compare in the DPreview tests.

...

As a related question, if you do compare different cameras, all set to the same f-number, shutter speed and ISO, then exactly what are you measuring and comparing? You are not measuring or comparing the low noise performance of which each camera is capable, so just what are you measuring? It is worth thinking about.

What the DPreview test scenes show is the noise at different ISO settings. It is simple to transfer this to your requested 'low-light performance'. If you have a camera A with a maximum aperture of 2.8 and a camera B with a maximum aperture of 2.0, camera A would need an ISO setting double that of camera B if everything else stays same.

For me, what you wrote above does not really answer my question, perhaps because I did not make myself clear.

Imagine two cameras with identical sensors. For the sake of the example, assume the sensors are 13.2x8.8 mm, commonly known as a "one inch" sensor with a crop factor of x2.7. For simplicity, assume they are point-and-shoot style cameras with a fixed focal length of 18.5 mm, equivalent to 50 mm for a 35 mm sensor. One of them, Camera A, has a cheap-and-nasty lens that can go down to F/4.0, while camera B has a faster lens that can go down to F/2.0

No photographer would dispute that Camera B with the faster f/2.0 lens is the better choice for working under low light conditions - Camera B clearly has better low-light performance. Surely no one would disagree with that.

OK, so the question is, what methodology should be used for comparing the low light performance of different cameras. Clearly any methodology worth a pinch of salt must be able to determine that Camera B has the better low light performance. In a previous posting I described how such testing would be done. Both cameras are set to the "best that they can do" to optimize their low light performance, being F/2 and F/4 in this case. They are then both pointed at the same dimly lit scene, at the same distance. Because they both have the same focal length of lens, they will both capture the same angular Field of View, and the scene image will be the same size on both sensors. ISO is set to a highish value to ensure that there is visible noise in the images, and both are set to the same shutter speed. The captured images are then compared, and it is found that Camera B produced a less noisy image. Therefore the result of the test is that Camera B performs better in low light. I trust there is nothing controversial in anything so far.

OK, now lets turn out attention to the methodology employed using the DP Review Test Scene. Again let me say, if it is worth a cracker for comparing different cameras on the basis of low light performance, then it must show that Camera B has the better low light performance. Except that it fails. Totally. Under this methodology, both cameras would be tested at the same f-number and shutter speed, and the noise in both images would be identical, and it would be incorrectly concluded that both cameras had the same low light performance. As you said, this is repeated over a range of ISO, but this does not help to know which is the better camera. Both cameras will look equally worse as ISO is increased. Clearly this methodology cannot reliable rank cameras on the basis of their low light performance. Thus my question :-

... if you do compare different cameras, all set to the same f-number, shutter speed and ISO, as per the DP Review Test Scene webpage, then exactly what are you measuring and comparing? You are not measuring or comparing the low noise performance of which each camera is capable, so just what are you measuring?

Here is my answer. As all camera being compared are set to the same shutter speed and f-number, the exposure (photons/area) on the sensor is the same for all cameras. Therefore, the amount of light falling on the sensor (which sets shot noise) is simply proportional to the area of the sensor. So we would expect that the noise observed in the image simply depends on the area of the sensor in each camera being compared. A complicated way of indirectly measuring sensor area.

In same cases, notably when all cameras are fitted with the same speed (lowest available f-number) of lens, then this methodology will correctly rank the cameras in order of low light performance. But in general, cameras have different speed lenses, with the result that this methodology is not valid in general for ranking cameras on the basis of their low light performance.

So what does this methodology measure then? Basically, all it does is rank the cameras on the basis of their sensor size, which may or my not correlate with low light performance. And that is why I said very early on that the DP test scene results were not useful in ranking my candidate cameras on the basis of their low light performance.

The options are to perform my own testing according to my described methodology, which in general is not possible unless you own the cameras, or to make a reasonable theoretical estimate based on "equivalent f-numbers".

 

[Physics1]

. . . Hmmm. But wait a minute. What I am doing is comparing the low light performance of different cameras when used in the real world, and for that propose, surely every camera should be set such that it is “doing the best that it can” in low light, optimally using whatever hardware comprises that camera. And that means that every camera being compared must be set to the lowest available f-number. The noise performance will (and always does) scale according to which cameras have the maximum effective aperture diameter, as that is what determines how much light is actually captured and sent to the sensor, and this may or may not correlate with sensor size. Or one can use “equivalent f-numbers”. Anyway, the real point I am making is that for meaningful comparison of low light performance, all cameras need to be set with their lens apertures fully open, and not to the same f-number. Of course, all images need to be made equally bright by appropriate setting of ISO.

Does anyone agree or disagree with any of the above?

I might differ on a couple things. For instance, on a full-frame camera, f/1.4 might not provide enough depth of field to satisfy me, or it might have other issues like heavy vignetting or compromised across-the-frame resolution, so setting such a camera to "the lowest available f-number" might result in simply deleting a bunch of disappointing pictures. To get equal brightness, it's both the ISO and the shutter speed that need to be set in concert with the allowable aperture.

Totally agreed that there are often reasons why you may not want to use the lowest f-number, even if available. But if we assume a slowest acceptable shutter speed, and a maximum acceptable image noise, and still you don't have enough light, then you have no choice but to set the lens to a lower f-number.

But regardless, the topic was how to compare the potentially available low light capability of different cameras, and all else equal, a camera with a faster lens (lower f-number) has a greater low-light capability. Whether you choose to use it is another matter.

 

[robert1955]

. . . Hmmm. But wait a minute. What I am doing is comparing the low light performance of different cameras when used in the real world, and for that propose, surely every camera should be set such that it is “doing the best that it can” in low light, optimally using whatever hardware comprises that camera. And that means that every camera being compared must be set to the lowest available f-number. The noise performance will (and always does) scale according to which cameras have the maximum effective aperture diameter, as that is what determines how much light is actually captured and sent to the sensor, and this may or may not correlate with sensor size. Or one can use “equivalent f-numbers”. Anyway, the real point I am making is that for meaningful comparison of low light performance, all cameras need to be set with their lens apertures fully open, and not to the same f-number. Of course, all images need to be made equally bright by appropriate setting of ISO.

Does anyone agree or disagree with any of the above?

I might differ on a couple things. For instance, on a full-frame camera, f/1.4 might not provide enough depth of field to satisfy me, or it might have other issues like heavy vignetting or compromised across-the-frame resolution, so setting such a camera to "the lowest available f-number" might result in simply deleting a bunch of disappointing pictures. To get equal brightness, it's both the ISO and the shutter speed that need to be set in concert with the allowable aperture.

Totally agreed that there are often reasons why you may not want to use the lowest f-number, even if available. But if we assume a slowest acceptable shutter speed, and a maximum acceptable image noise, and still you don't have enough light, then you have no choice but to set the lens to a lower f-number.

But regardless, the topic was how to compare the potentially available low light capability of different cameras, and all else equal, a camera with a faster lens (lower f-number) has a greater low-light capability. Whether you choose to use it is another matter.

compromising on f-stop changes the image more fundamentally than compromising on noise

 

[Horacecoker]

To get an overview of low-light performance use the test-scene compare in the DPreview tests.

...

As a related question, if you do compare different cameras, all set to the same f-number, shutter speed and ISO, then exactly what are you measuring and comparing? You are not measuring or comparing the low noise performance of which each camera is capable, so just what are you measuring? It is worth thinking about.

What the DPreview test scenes show is the noise at different ISO settings. It is simple to transfer this to your requested 'low-light performance'. If you have a camera A with a maximum aperture of 2.8 and a camera B with a maximum aperture of 2.0, camera A would need an ISO setting double that of camera B if everything else stays same.

For me, what you wrote above does not really answer my question, perhaps because I did not make myself clear.

Imagine two cameras with identical sensors. For the sake of the example, assume the sensors are 13.2x8.8 mm, commonly known as a "one inch" sensor with a crop factor of x2.7. For simplicity, assume they are point-and-shoot style cameras with a fixed focal length of 18.5 mm, equivalent to 50 mm for a 35 mm sensor. One of them, Camera A, has a cheap-and-nasty lens that can go down to F/4.0, while camera B has a faster lens that can go down to F/2.0

No photographer would dispute that Camera B with the faster f/2.0 lens is the better choice for working under low light conditions - Camera B clearly has better low-light performance. Surely no one would disagree with that.

OK, so the question is, what methodology should be used for comparing the low light performance of different cameras. Clearly any methodology worth a pinch of salt must be able to determine that Camera B has the better low light performance.

In a previous posting I described how such testing would be done. Both cameras are set to the "best that they can do" to optimize their low light performance, being F/2 and F/4 in this case. They are then both pointed at the same dimly lit scene, at the same distance. Because they both have the same focal length of lens, they will both capture the same angular Field of View, and the scene image will be the same size on both sensors. ISO is set to a highish value to ensure that there is visible noise in the images, and both are set to the same shutter speed.

Both cameras can't be set to the same ISO in that scenario as the f/2 camera image will appear two stops overexposed.

The captured images are then compared, and it is found that Camera B produced a less noisy image. Therefore the result of the test is that Camera B performs better in low light. I trust there is nothing controversial in anything so far.

OK, now lets turn out attention to the methodology employed using the DP Review Test Scene. Again let me say, if it is worth a cracker for comparing different cameras on the basis of low light performance, then it must show that Camera B has the better low light performance. Except that it fails. Totally. Under this methodology, both cameras would be tested at the same f-number and shutter speed, and the noise in both images would be identical, and it would be incorrectly concluded that both cameras had the same low light performance. As you said, this is repeated over a range of ISO, but this does not help to know which is the better camera. Both cameras will look equally worse as ISO is increased. Clearly this methodology cannot reliable rank cameras on the basis of their low light performance. Thus my question :-

... if you do compare different cameras, all set to the same f-number, shutter speed and ISO, as per the DP Review Test Scene webpage, then exactly what are you measuring and comparing? You are not measuring or comparing the low noise performance of which each camera is capable, so just what are you measuring?

Here is my answer. As all camera being compared are set to the same shutter speed and f-number, the exposure (photons/area) on the sensor is the same for all cameras. Therefore, the amount of light falling on the sensor (which sets shot noise) is simply proportional to the area of the sensor. So we would expect that the noise observed in the image simply depends on the area of the sensor in each camera being compared. A complicated way of indirectly measuring sensor area.

In same cases, notably when all cameras are fitted with the same speed (lowest available f-number) of lens, then this methodology will correctly rank the cameras in order of low light performance. But in general, cameras have different speed lenses, with the result that this methodology is not valid in general for ranking cameras on the basis of their low light performance.

So what does this methodology measure then? Basically, all it does is rank the cameras on the basis of their sensor size, which may or my not correlate with low light performance. And that is why I said very early on that the DP test scene results were not useful in ranking my candidate cameras on the basis of their low light performance.

The options are to perform my own testing according to my described methodology, which in general is not possible unless you own the cameras, or to make a reasonable theoretical estimate based on "equivalent f-numbers".

 

[Mackiesback]

Buy something from that really patient shop.

 

[Bill Ferris]

. . . Hmmm. But wait a minute. What I am doing is comparing the low light performance of different cameras when used in the real world, and for that propose, surely every camera should be set such that it is “doing the best that it can” in low light, optimally using whatever hardware comprises that camera. And that means that every camera being compared must be set to the lowest available f-number. The noise performance will (and always does) scale according to which cameras have the maximum effective aperture diameter, as that is what determines how much light is actually captured and sent to the sensor, and this may or may not correlate with sensor size. Or one can use “equivalent f-numbers”. Anyway, the real point I am making is that for meaningful comparison of low light performance, all cameras need to be set with their lens apertures fully open, and not to the same f-number. Of course, all images need to be made equally bright by appropriate setting of ISO.

Does anyone agree or disagree with any of the above?

I might differ on a couple things. For instance, on a full-frame camera, f/1.4 might not provide enough depth of field to satisfy me, or it might have other issues like heavy vignetting or compromised across-the-frame resolution, so setting such a camera to "the lowest available f-number" might result in simply deleting a bunch of disappointing pictures. To get equal brightness, it's both the ISO and the shutter speed that need to be set in concert with the allowable aperture.

Totally agreed that there are often reasons why you may not want to use the lowest f-number, even if available. But if we assume a slowest acceptable shutter speed, and a maximum acceptable image noise, and still you don't have enough light, then you have no choice but to set the lens to a lower f-number.

But regardless, the topic was how to compare the potentially available low light capability of different cameras, and all else equal, a camera with a faster lens (lower f-number) has a greater low-light capability. Whether you choose to use it is another matter.

One can compare hypotheticals in isolation of real world needs and always land on the answer, the largest sensor with the fastest lens wins. However, if we're taking real world use and customer-centric factors into consideration, things get a bit more complicated...and, frankly, interesting.

As jrtrent mentioned, creative considerations such as depth of field and the image quality of the lens play a role.

You brought up the scenario of choosing a minimum acceptable shutter speed and depth of field, but not putting enough light on the sensor. Technically, one could open the lens to a wider f-stop or slow shutter speed even further. However, if those minimus carry any real weight, there is another option to consider: make a different photo.

I'm out a lot at sunrise & sunset to photograph wildlife. Generally, I like to use a shutter speed that will freeze movement. But in really low light, I'll scrap that plan to go even slower with the shutter shifted to intentionally blur movement.

There's no carved in-store rule declaring a photographer can only make the photo that was in their mind's eye when they left the house. We can adapt to unexpected opportunity or unsatisfactory light and make a different image that's more compatible with real-world conditions.

Budget, portability, and user interface are considerations that play a significant role in the gear we choose to buy and use. They don't directly impact light-gathering and low light performance but they're only a step or two removed.

If money, size & weight weren't factors, all wildlife photographers might be shooting with full-frame cameras and 600mm f/4 primes. But those considerations are factors. And when one looks at the work of Bence Mate, we're reminded that a photographer's ingenuity and creativity really do carry the day. Whole worlds of possibilities with "substandard" equipment are opened.

There are similar stories that can be told in most every genre about photographers who didn't have the best equipment but did find innovative & creative ways to do great work with the gear they did have.

So yeah, if we apply enough constraints and define the problem narrowly enough, there will always be just one answer to the question, which gear is best? But when we leave the theoretical for the real world of budgets and individual needs, things get complicated. Answers get more diverse, unexpected, and interesting.

--
Bill Ferris Photography
Flagstaff, AZ