TOF or LIDAR

Started 3 months ago | Discussions
eit412608 Junior Member • Posts: 34
TOF or LIDAR

I have seen a few threads about including TOF or LIDAR into a camera to increase focus speed. I think many times the TOF or LIDAR could be blocked by a lens or it's hood. There are not many places on the front of the camera that a LIDAR or TOF sensor would always work.  If the system was built into the lenses it would never be blocked and could be optimized for the specific lens. The lens would just have a hump on one side to house the senor. It would be too expensive to add it to every lens, but not all lenses need it. Is there an issue with this setup i am not considering?

Mark Scott Abeln
Mark Scott Abeln Forum Pro • Posts: 16,188
Re: TOF or LIDAR
2

eit412608 wrote:

Is there an issue with this setup i am not considering?

The Polaroid SX-70 Sonar, a folding medium format instant print SLR, and an engineering marvel of its day, used ultrasonic sound for focusing. The focus would be wrong when photographing through windows or screens, and so the camera also had manual focus. I it would also have problems when focusing through foreground matter, such as foliage, flowers, etc., as sometimes used in portraiture.

Modern optical focus is far more precise in selecting the required subject of focus. The Polaroid solution wasn't all that great, a gimmick in my opinion.

 Mark Scott Abeln's gear list:Mark Scott Abeln's gear list
Nikon D200 Nikon D7000 Nikon D750 Nikon AF-S DX Nikkor 35mm F1.8G Nikon AF Nikkor 50mm f/1.8D +2 more
OpticsEngineer Veteran Member • Posts: 6,842
Re: TOF or LIDAR
1

Time of flight sensing is somewhat new coming into the consumer market although the military and aerospace fields have been using it for decades.  The wikipedia article on it is a good introduction for people who are unfamiilair with it.  And a web search on "iphone time of flight" would probably we worthwhile for a lot of people.

But to your question, to put TOF through the lens, with a bumpout on the side of a camera like you described, the conventional solution would be to put a beam splitter in between the sensor and the lens.  Like the beam splitter in a Sony DSLT camera such as a Sony A99.   That beam splitter takes space and that goes against the trend to make lenses and short and light as possible.  That is one of the reasons mirrorless cameras are taking over from DSLRs (and DSLTs.)

 OpticsEngineer's gear list:OpticsEngineer's gear list
Fujifilm XF1 Olympus XZ-2 iHS Canon PowerShot G7 X Olympus Tough TG-4 Pentax *ist D +27 more
Sympa Senior Member • Posts: 2,492
Re: TOF or LIDAR

LG had laser autofocus in their G3 phone from 2014. I suppose that sensor based AF is cheaper and more flexible.

The laser AF would only focus where the laser was pointing, so no chace to prioritize a face or so.

 Sympa's gear list:Sympa's gear list
Sony RX100 Sony E 30mm F3.5 Macro Sony E 50mm F1.8 OSS Sony E 55-210mm F4.5-6.3 OSS Sony Vario-Tessar T* E 16-70mm F4 ZA OSS +4 more
Entropy512 Veteran Member • Posts: 4,862
Re: TOF or LIDAR

OpticsEngineer wrote:

Time of flight sensing is somewhat new coming into the consumer market although the military and aerospace fields have been using it for decades. The wikipedia article on it is a good introduction for people who are unfamiilair with it. And a web search on "iphone time of flight" would probably we worthwhile for a lot of people.

But to your question, to put TOF through the lens, with a bumpout on the side of a camera like you described, the conventional solution would be to put a beam splitter in between the sensor and the lens. Like the beam splitter in a Sony DSLT camera such as a Sony A99. That beam splitter takes space and that goes against the trend to make lenses and short and light as possible. That is one of the reasons mirrorless cameras are taking over from DSLRs (and DSLTs.)

Yup. You could technically replace the AF system with a TOF sensor and an illuminator in the hotshoe - but honestly that would probably not perform as well as a legacy PDAF system.

TOF sensors are great for initial hinting in the absence of PDAF, and may be beneficial for initial hinting in a case where current lens focus position is so far off that PDAF can only return an "out of range" result, but they're not going to be suitable to replace any final through-the-lens focusing system merely due to issues of lens/sensor assembly tolerances.

Most 3D TOF sensors on the market also have issues with mutual interference when multiple systems are looking at the same area.  Upcoming products from IFM make advances in mitigating that, but who knows how well they'll actually perform.  Scanning LiDARs are bulky but at least perform better with respect to mutual interference, since they'll only interfere when the scan pattern has them pointing at each other at the exact same time.

-- hide signature --

Context is key. If I have quoted someone else's post when replying, please do not reply to something I say without reading text that I have quoted, and understanding the reason the quote function exists.

 Entropy512's gear list:Entropy512's gear list
Sony a6000 Pentax K-5 Pentax K-01 Sony a6300 Canon EF 85mm F1.8 USM +5 more
Yunfan Junior Member • Posts: 27
Re: TOF or LIDAR
2

Entropy512 wrote:

Yup. You could technically replace the AF system with a TOF sensor and an illuminator in the hotshoe - but honestly that would probably not perform as well as a legacy PDAF system.

TOF sensors are great for initial hinting in the absence of PDAF, and may be beneficial for initial hinting in a case where current lens focus position is so far off that PDAF can only return an "out of range" result, but they're not going to be suitable to replace any final through-the-lens focusing system merely due to issues of lens/sensor assembly tolerances.

Most 3D TOF sensors on the market also have issues with mutual interference when multiple systems are looking at the same area. Upcoming products from IFM make advances in mitigating that, but who knows how well they'll actually perform. Scanning LiDARs are bulky but at least perform better with respect to mutual interference, since they'll only interfere when the scan pattern has them pointing at each other at the exact same time.

A lot of great points here. As a researcher working on ToF related computer vision problems, I would say ToF is far from perfect and reliable due to following reasons:

1. Dark objects, surfaces with a large incident angle to the IR illumination, and far away/tiny objects do not reflect enough active infrared illumination back to the sensor, so you cannot get readings reliably on these surfaces. I think the percentage of missing data can at least be comparable to the occlusion caused by lens/lens hoods.

2. You need to align the ToF image with the RGB sensor to know which part in the frame to focus on - this can be a really big issue especially for MILC/DSLRs as they can have lens with different FOV and distortion profiles. Correcting them perfectly might be much difficult than one would assume if we also consider the manufacturing tolerances.

3. ToF can have multi-path interferences if there are reflective surfaces in the scene, as the rays bounced back from surface A to the sensor might have hit surface B first and got reflected to surface A. This basically would make the sensor think a surface is further away than it actually is and can lead to terrible measurements.

4. I have not tested with this too much but I assume ToF (at least those on phones) can also get blinded by the sun if used outdoors under sunlight.

I think ToF is therefore useful only useful for distance hinting. For any accurate and reliable AF, use PDAF or CDAF.

Entropy512 Veteran Member • Posts: 4,862
Re: TOF or LIDAR

Yunfan wrote:

Entropy512 wrote:

Yup. You could technically replace the AF system with a TOF sensor and an illuminator in the hotshoe - but honestly that would probably not perform as well as a legacy PDAF system.

TOF sensors are great for initial hinting in the absence of PDAF, and may be beneficial for initial hinting in a case where current lens focus position is so far off that PDAF can only return an "out of range" result, but they're not going to be suitable to replace any final through-the-lens focusing system merely due to issues of lens/sensor assembly tolerances.

Most 3D TOF sensors on the market also have issues with mutual interference when multiple systems are looking at the same area. Upcoming products from IFM make advances in mitigating that, but who knows how well they'll actually perform. Scanning LiDARs are bulky but at least perform better with respect to mutual interference, since they'll only interfere when the scan pattern has them pointing at each other at the exact same time.

A lot of great points here. As a researcher working on ToF related computer vision problems, I would say ToF is far from perfect and reliable due to following reasons:

1. Dark objects, surfaces with a large incident angle to the IR illumination, and far away/tiny objects do not reflect enough active infrared illumination back to the sensor, so you cannot get readings reliably on these surfaces. I think the percentage of missing data can at least be comparable to the occlusion caused by lens/lens hoods.

Don't forget how poorly they usually perform on sharp edges.  You'll often get distance artifacting there.  Combatting distance aliasing requires frequency dithering, which negatively impacts framerate.

2. You need to align the ToF image with the RGB sensor to know which part in the frame to focus on - this can be a really big issue especially for MILC/DSLRs as they can have lens with different FOV and distortion profiles. Correcting them perfectly might be much difficult than one would assume if we also consider the manufacturing tolerances.

This will be a major PITA for interchangeable lens cameras.

3. ToF can have multi-path interferences if there are reflective surfaces in the scene, as the rays bounced back from surface A to the sensor might have hit surface B first and got reflected to surface A. This basically would make the sensor think a surface is further away than it actually is and can lead to terrible measurements.

IFM claims their next-gen sensor architecture will greatly improve performance in multipath scenarios, but TBD whether this will actually work as claimed.

4. I have not tested with this too much but I assume ToF (at least those on phones) can also get blinded by the sun if used outdoors under sunlight.

Even Leddar's 1D rangefinders have been documented by a customer (who installed an aftermarket system using one of them on a manually driven forklift) as false-alarming under skylights.

(My day job is industrial automation.)

I think ToF is therefore useful only useful for distance hinting. For any accurate and reliable AF, use PDAF or CDAF.

Yup.

-- hide signature --

Context is key. If I have quoted someone else's post when replying, please do not reply to something I say without reading text that I have quoted, and understanding the reason the quote function exists.

 Entropy512's gear list:Entropy512's gear list
Sony a6000 Pentax K-5 Pentax K-01 Sony a6300 Canon EF 85mm F1.8 USM +5 more
OP eit412608 Junior Member • Posts: 34
Re: TOF or LIDAR

Yunfan wrote:

Entropy512 wrote:

Yup. You could technically replace the AF system with a TOF sensor and an illuminator in the hotshoe - but honestly that would probably not perform as well as a legacy PDAF system.

TOF sensors are great for initial hinting in the absence of PDAF, and may be beneficial for initial hinting in a case where current lens focus position is so far off that PDAF can only return an "out of range" result, but they're not going to be suitable to replace any final through-the-lens focusing system merely due to issues of lens/sensor assembly tolerances.

Most 3D TOF sensors on the market also have issues with mutual interference when multiple systems are looking at the same area. Upcoming products from IFM make advances in mitigating that, but who knows how well they'll actually perform. Scanning LiDARs are bulky but at least perform better with respect to mutual interference, since they'll only interfere when the scan pattern has them pointing at each other at the exact same time.

A lot of great points here. As a researcher working on ToF related computer vision problems, I would say ToF is far from perfect and reliable due to following reasons:

1. Dark objects, surfaces with a large incident angle to the IR illumination, and far away/tiny objects do not reflect enough active infrared illumination back to the sensor, so you cannot get readings reliably on these surfaces. I think the percentage of missing data can at least be comparable to the occlusion caused by lens/lens hoods.

2. You need to align the ToF image with the RGB sensor to know which part in the frame to focus on - this can be a really big issue especially for MILC/DSLRs as they can have lens with different FOV and distortion profiles. Correcting them perfectly might be much difficult than one would assume if we also consider the manufacturing tolerances.

3. ToF can have multi-path interferences if there are reflective surfaces in the scene, as the rays bounced back from surface A to the sensor might have hit surface B first and got reflected to surface A. This basically would make the sensor think a surface is further away than it actually is and can lead to terrible measurements.

4. I have not tested with this too much but I assume ToF (at least those on phones) can also get blinded by the sun if used outdoors under sunlight.

I think ToF is therefore useful only useful for distance hinting. For any accurate and reliable AF, use PDAF or CDAF.

In regards to #2, that is why i suggested including the TOF inside the lens rather than the body.  this way the TOF is specific to the lens, and will not be occluded by the lens or a person's hand as it could be on the camera body.  By built into the lens, I do not mean in the optical path.  I just mean as an addition.  Kind of like the macro lenses that have LEDs built into the front of the lens for illumination of close objects.

I don't think TOF would completely replace the CDAF of PDAF, just speed AF up in dark situations and help with tracking fast objects.

Keyboard shortcuts:
FForum MMy threads