MIT algorithm aims to eradicate reflections from photos taken through windows
Researchers at the Massachusetts Institute of Technology claim to have developed a method for eliminating glass reflections in photos via digital processing. It is hoped that with further development the idea could see its way into cameras themselves, enabling reflections to be automatically removed when they interfere with the view through a window.
The new algorithm, due to be formally announced in June at the Boston Computer Vision and Pattern Recognition conference, is claimed to offer a way of distinguishing the image reflected by a window surface from the image of whatever is behind the window. Once the two images are clearly identified one can be subtracted from the whole to leave the other.
MIT researchers noted that objects reflected by glass surfaces tend to have two reflections, not one, and that the two images of reflected objects are slightly off-set. The multiple images in a reflection come about sometimes because windows are double glazed for insulation and thus have two panes of glass, and other times because light reflects from the rear surface of a pane of glass as well as the facing surface.
Further research showed that the interference of a reflection can often be identified by the way it interrupts the regular shape and color patterns of natural and man-made objects on the other side of the glass. Working with small clusters of 8x8 pixels, researchers analyzed color and tonal relationships within the group and came to a method by which certain data correlations indicated that the group included detail from a reflection.
The result is that when an image of, for example, a view shot through a bus window is analyzed, the team can separate the image of the view from the reflected image of the photographer that shot the view – and then can show either the view on its own, or the photographer on his or her own.
If successful the algorithm may have serious consequences for the flexible rubber lens hood market, though early indications suggest polarizing filter manufacturers will be safe for some time to come.
For more information see the Massachusetts Institute of Technology website.
Removing reflections from photos taken through windows
New algorithm exploits multiple reflections in individual images to distinguish reflection from transmission.
Larry Hardesty | MIT News Office
May 11, 2015
It's hard to take a photo through a window without picking up reflections of the objects behind you. To solve that problem, professional photographers sometimes wrap their camera lenses in dark cloths affixed to windows by tape or suction cups. But that's not a terribly attractive option for a traveler using a point-and-shoot camera to capture the view from a hotel room or a seat in a train.
At the Computer Vision and Pattern Recognition conference in June, MIT researchers will present a new algorithm that, in a broad range of cases, can automatically remove reflections from digital photos. The algorithm exploits the fact that photos taken through windows often feature two nearly identical reflections, slightly offset from each other.
“In Boston, the windows are usually double-paned windows for heat isolation during the winter,” says YiChang Shih, who completed his PhD in computer science at MIT this spring and is first author on the paper. “With that kind of window, there's one reflection coming from the inner pane and another reflection from the outer pane. But thick windows are usually enough to produce a double reflection, too. The inner side will give a reflection, and the outer side will give a reflection as well.”
Without the extra information provided by the duplicate reflection, the problem of reflection removal is virtually insoluble, Shih explains. “You have an image from outdoor and another image from indoor, and what you capture is the sum of these two pictures,” he says. “If A+B is equal to C, then how will you recover A and B from a single C? That's mathematically challenging. We just don't have enough constraints to reach a conclusion.”
Thinning the field
The second reflection imposes the required constraint. Now the problem becomes recovering A, B, and C from a single D. But the value of A for one pixel has to be the same as the value of B for a pixel a fixed distance away in a prescribed direction. That constraint drastically reduces the range of solutions that the algorithm has to consider.
Nonetheless, a host of solutions still remain. To home in on one of them, Shih and his coauthors — professors of computer science and engineering Frédo Durand and Bill Freeman, who were Shih's thesis advisors, and Dilip Krishnan, a former postdoc in Freeman's group who's now at Google Research — assume that both the reflected image and the image captured through the window have the statistical regularities of so-called natural images.
The basic intuition is that at the level of small clusters of pixels, in natural images — unaltered representations of natural and built environments — abrupt changes of color are rare. And when they do occur, they occur along clear boundaries. So if a small block of pixels happens to contain part of the edge between a blue object and a red object, everything on one side of the edge will be bluish, and everything on the other side will be reddish.
In computer vision, the standard way to try to capture this intuition is with the notion of image gradients, which characterize each block of pixels according to the chief direction of color change and the rate of change. But Shih and his colleagues found that this approach didn't work very well.
Playing the odds
Ultimately, they settled on a new technique co-developed by Daniel Zoran, a postdoc in Freeman's group. Zoran and Yair Weiss of the Hebrew University of Jerusalem created an algorithm that divides images into 8-by-8 blocks of pixels; for each block, it calculates the correlation between each pixel and each of the others. The aggregate statistics for all the 8-by-8 blocks in 50,000 training images proved a reliable way to distinguish reflections from images shot through glass.
In their paper, Shih and his colleagues report performing searches on Google and the Flickr photo database using terms like “window reflection photography problems.” After excluding results that weren't photos shot through glass, they had 197 images, 96 of which exhibited double reflections that were offset far enough for their algorithm to work.
“People have worked on methods for eliminating these reflections from photos, but there had been drawbacks in past approaches,” says Yoav Schechner, a professor of electrical engineering at Israel's Technion. “Some methods attempt using a single shot. This is very hard, so prior results had partial success, and there was no automated way of telling if the recovered scene is the one reflected by the window or the one behind the window. This work does a good job on several fronts.”
“The ideas here can progress into routine photography, if the algorithm is further robustified and becomes part of toolboxes used in digital photography,” he adds. “It may help robot vision in the presence of confusing glass reflection.”
|Brown Crown by Nilesh Trivedi|
from brown challenge
|D72_4852_DxO Smug by richpics|
from Aviation Legends: X-Planes
|Everyone look at the camera by cjf2|
from Looking down the lens.
|Ancient Bristlecone Pine by ed rader|
from My Best Picture of the Week
Sony's new 12-24mm F2.8 GM is the widest fast aperture zoom for full frame. Based on our tests it's a worthy recipient of Sony's 'GM' moniker.
Chris and Jordan took the new Sony 12-24mm F2.8 GM to Calgary's eclectic Ingelwood neighborhood. From record stores to spice shops, find out what got their attention when it was time to go wide.
The six prize-winning photographs and four honorable mentions were narrowed down from more than 6,000 entries captured across North America.
Though Thunderbolt 4 remains at 40Gb/s, its minimum requirements include dual 4K monitor support, faster external drive speeds and more.
You can now use compatible Fujifilm cameras with video conferencing software on macOS hardware without the need of a dedicated capture card.
The Epson V600 remains one of the most popular flatbed film scanners on the market. Revisit our review of this affordable and (mostly) easy-to-use option and see how its output compares to local lab scans.
Canon's mirrorless EOS R5 comes with a ton of features and capability stemming from its design inside and out. Come along with us on a guided tour of Canon's new high-end, high-megapixel camera and check it out for yourself.
Announced alongside the EOS R5, the R6 offers a lot of the same technology but in a more affordable, slightly more enthusiast-focused model. Take a closer look.
Alongside the EOS R5 and R6, Canon has announced a brace of lenses, all in the short to long telephoto range. Filling out the 'long' end are one L-series zoom, and two innovative primes.
Alongside a trio of telephoto lenses, Canon also announced a new 85mm this week. The RF 85mm F2 Macro IS STM is a compact, affordable alternative to the pro-oriented 85mm F1.2L.
The EOS R5 has been a long time coming – we knew it had 8K and we knew it had an AF joystick. But now that's it's here, what is it really like to use? Find out in our initial review based on hands-on time with the camera.
The R6 doesn't promise quite such headline-grabbing specs as its big brother, but it still packs a punch, whether you shoot stills, video or both.
Think you've read everything there is to know about the new Canon cameras? Chris and Jordan share eight important things you may have missed from today's Canon EOS R5 and R6 announcements.
We've been shooting around with the new Canon EOS R6. Initial impressions of image quality are positive, and out-of-camera JPEGs appear similar to that of the gold award-winning Canon EOS-1D X III. Have a look for yourself.
Canon has officially released the long-awaited EOS R5, the company's top-end full-frame mirrorless camera. Featuring a new 45MP CMOS sensor, Dual Pixel AF II system, 8K video capture and 20 fps bursts, this is the RF-mount camera we've been waiting for.
Although the Canon EOS R6 doesn't have the 45MP sensor and 8K video capture of the higher-end R5, it's still an incredibly capable camera with specs that outshine similarly priced peers.
The Canon RF 100-500mm F4.5-7.1L IS USM is the company's first super-zoom lens for RF-mount. Despite a relatively slow aperture range, it's very versatile, offering five stops of stabilization, weather-sealing and compatibility with Canon's new teleconverters.
Canon's RF 85mm F2 Macro IS STM is an inexpensive telephoto prime lens with a minimum focus distance of just 0.35m (14") and a 0.5x magnification. When attached to the new R5 and R6, it offers a whopping eight stops of shake reduction.
Canon has announced a pair of super-telephoto fixed-aperture primes. The 600mm and 800mm use diffractive optics to keep their size and weight down. They'll also be compatible with new 1.4x and 2x RF teleconverters.
Canon has announced a new small-footprint inkjet photo printer, the imageProGraf Pro-300. it will produce prints up to 13 x 19" and it goes on sale later this month for $900. A new textured photo paper will also arrive in July.
The new compression standard is set to reduce video file sizes by half to save space and speed-up transmission, paving the way for more portable 8K footage.
Sony recently confirmed plans to launch a successor to the video-centric a7S II. We don't even know the name of the camera, but Jordan already has a feature wish list for the new 'a7S III' – and it doesn't include 8K.
The Profot B10 is the first studio flash system that can be used when shooting with an iPhone camera.
The Pixii camera is an interesting little rangefinder camera that features a 12MP APS-C sensor and lacks a rear LCD display, opting instead to pair with your mobile device, which can be used to view and transfer images.
Sirui is launching an Indiegogo campaign for a wide-angle answer to its existing 50mm F1.8 anamorphic lens. The 35mm APS-C lens will come in a Micro Four Thirds mount with adapters for other systems.
Sony has added a 12-24mm F2.8 to its top-shelf 'G Master' series of lenses. It's the widest constant F2.8 zoom currently offered for full-frame, with a hefty price tag to match: it will sell for $3000 when it ships in mid-August.
Take a look at the view from Sony's new ultra-wide F2.8 zoom – we paired it with the a7R IV for some initial shooting.
Canon's EOS-1D X Mark III is one of the best DSLRs ever made. With fast burst speeds, great video quality and impressive autofocus, the 1D X III is equal parts cinema rig and sports shooter. Find out how it fares against steep competition in our full review.
Nikon Rumors is reporting that Nikon will announce successors to its Z6 and Z7 camera systems by the end of the calendar year.
Canon says the event, set to take place at 14:00 CEST in two days on July 9, will be its 'biggest product launch yet.'