MIT algorithm aims to eradicate reflections from photos taken through windows
Researchers at the Massachusetts Institute of Technology claim to have developed a method for eliminating glass reflections in photos via digital processing. It is hoped that with further development the idea could see its way into cameras themselves, enabling reflections to be automatically removed when they interfere with the view through a window.
The new algorithm, due to be formally announced in June at the Boston Computer Vision and Pattern Recognition conference, is claimed to offer a way of distinguishing the image reflected by a window surface from the image of whatever is behind the window. Once the two images are clearly identified one can be subtracted from the whole to leave the other.
MIT researchers noted that objects reflected by glass surfaces tend to have two reflections, not one, and that the two images of reflected objects are slightly off-set. The multiple images in a reflection come about sometimes because windows are double glazed for insulation and thus have two panes of glass, and other times because light reflects from the rear surface of a pane of glass as well as the facing surface.
Further research showed that the interference of a reflection can often be identified by the way it interrupts the regular shape and color patterns of natural and man-made objects on the other side of the glass. Working with small clusters of 8x8 pixels, researchers analyzed color and tonal relationships within the group and came to a method by which certain data correlations indicated that the group included detail from a reflection.
The result is that when an image of, for example, a view shot through a bus window is analyzed, the team can separate the image of the view from the reflected image of the photographer that shot the view – and then can show either the view on its own, or the photographer on his or her own.
If successful the algorithm may have serious consequences for the flexible rubber lens hood market, though early indications suggest polarizing filter manufacturers will be safe for some time to come.
For more information see the Massachusetts Institute of Technology website.
Removing reflections from photos taken through windows
New algorithm exploits multiple reflections in individual images to distinguish reflection from transmission.
Larry Hardesty | MIT News Office
May 11, 2015
It's hard to take a photo through a window without picking up reflections of the objects behind you. To solve that problem, professional photographers sometimes wrap their camera lenses in dark cloths affixed to windows by tape or suction cups. But that's not a terribly attractive option for a traveler using a point-and-shoot camera to capture the view from a hotel room or a seat in a train.
At the Computer Vision and Pattern Recognition conference in June, MIT researchers will present a new algorithm that, in a broad range of cases, can automatically remove reflections from digital photos. The algorithm exploits the fact that photos taken through windows often feature two nearly identical reflections, slightly offset from each other.
“In Boston, the windows are usually double-paned windows for heat isolation during the winter,” says YiChang Shih, who completed his PhD in computer science at MIT this spring and is first author on the paper. “With that kind of window, there's one reflection coming from the inner pane and another reflection from the outer pane. But thick windows are usually enough to produce a double reflection, too. The inner side will give a reflection, and the outer side will give a reflection as well.”
Without the extra information provided by the duplicate reflection, the problem of reflection removal is virtually insoluble, Shih explains. “You have an image from outdoor and another image from indoor, and what you capture is the sum of these two pictures,” he says. “If A+B is equal to C, then how will you recover A and B from a single C? That's mathematically challenging. We just don't have enough constraints to reach a conclusion.”
Thinning the field
The second reflection imposes the required constraint. Now the problem becomes recovering A, B, and C from a single D. But the value of A for one pixel has to be the same as the value of B for a pixel a fixed distance away in a prescribed direction. That constraint drastically reduces the range of solutions that the algorithm has to consider.
Nonetheless, a host of solutions still remain. To home in on one of them, Shih and his coauthors — professors of computer science and engineering Frédo Durand and Bill Freeman, who were Shih's thesis advisors, and Dilip Krishnan, a former postdoc in Freeman's group who's now at Google Research — assume that both the reflected image and the image captured through the window have the statistical regularities of so-called natural images.
The basic intuition is that at the level of small clusters of pixels, in natural images — unaltered representations of natural and built environments — abrupt changes of color are rare. And when they do occur, they occur along clear boundaries. So if a small block of pixels happens to contain part of the edge between a blue object and a red object, everything on one side of the edge will be bluish, and everything on the other side will be reddish.
In computer vision, the standard way to try to capture this intuition is with the notion of image gradients, which characterize each block of pixels according to the chief direction of color change and the rate of change. But Shih and his colleagues found that this approach didn't work very well.
Playing the odds
Ultimately, they settled on a new technique co-developed by Daniel Zoran, a postdoc in Freeman's group. Zoran and Yair Weiss of the Hebrew University of Jerusalem created an algorithm that divides images into 8-by-8 blocks of pixels; for each block, it calculates the correlation between each pixel and each of the others. The aggregate statistics for all the 8-by-8 blocks in 50,000 training images proved a reliable way to distinguish reflections from images shot through glass.
In their paper, Shih and his colleagues report performing searches on Google and the Flickr photo database using terms like “window reflection photography problems.” After excluding results that weren't photos shot through glass, they had 197 images, 96 of which exhibited double reflections that were offset far enough for their algorithm to work.
“People have worked on methods for eliminating these reflections from photos, but there had been drawbacks in past approaches,” says Yoav Schechner, a professor of electrical engineering at Israel's Technion. “Some methods attempt using a single shot. This is very hard, so prior results had partial success, and there was no automated way of telling if the recovered scene is the one reflected by the window or the one behind the window. This work does a good job on several fronts.”
“The ideas here can progress into routine photography, if the algorithm is further robustified and becomes part of toolboxes used in digital photography,” he adds. “It may help robot vision in the presence of confusing glass reflection.”
|Antares cloud complex in Scorpius by jeffbottman|
from Best Photo of the Week...
|26415640516_0644cfa2fc_o by Frederic Kelder|
|Straumur-Aurora by arnie1|
from Tripod needed
|last flight by drnikon850|
from Specialized and Specialty Aircraft
ColorChecker Camera Calibration software version 2.0 adds DNG profile support and support for X-Rite's new Digital ColorChecker SG target.
This is Lomography's first lens designed specifically for full-frame mirrorless systems.
The FAA has expanded LAANC to increase access for commercial drone pilots in controlled airspace. 100+ airports have been added.
We've updated our waterproof camera buying guide with the latest round of rugged compacts, and we've crowned a new winner as the best pick in the category: the Olympus TG-6. That is, unless you happen to find a good deal on the TG-5.
According to Sony's latest financial report, it's overtaken Nikon to have the second largest market share in the global digital camera market, based on revenue.
Lenny Kravitz and Leica team up to produce perhaps the most eye-catching M camera ever, in the snakeskin Drifter kit. Limited to just 125 units, you'll need to move fast to get one.
Researchers with the Samsung AI Center in Moscow and the Skolkovo Institute of Science and Technology have created a system that transforms still images into talking portraits with as little as a single image.
K&R Photographics, a camera store in Crescent Springs, Kentucky, was robbed by armed men, who not only took thousands of dollars worth of camera equipment, but also injured the 70-year-old co-owner of the store.
Canon's RF lenses can now have their Control Rings de-clicked to make for silent changes when shooting video.
The new Fujifilm GFX 100 boasts some impressive specifications, including 100MP, in-body stabilization and 4K video. But what's it like to shoot with? Senior Editor Barnaby Britton found out on a recent trip to Florence, Italy.
Apple has updated its MacBook Pro with 8-core processors for the first time ever, as well as an improved keyboard that should resolve the issues of past butterfly keyboards.
Back in early March, we had the opportunity to visit Fujifilm's factory in Sendai, Japan, where assembly line staff were manually assembling the earliest pre-production samples of the GFX 100.
Chris and Jordan are at Fujifilm's GFX 100 launch event in Japan (well, in a nearby hotel room) and have some initial thoughts about using this 100MP medium format powerhouse.
It's here! The long-awaited next-generation Fujifilm GFX has been officially launched. Click through to learn more about the camera that Fujifilm is hoping will shake up the pro photography market - the GFX100.
We recently had the opportunity to shoot in Florence, Italy with a pre-production sample of the Fujifilm GFX 100. Check out our gallery of JPEG samples to get an idea of its capabilities.
We've known about the Fujifilm GFX 100 since last fall, but now it's official: this 102MP medium-format monster will be available at the end of June for $10,000. In addition to its incredible resolution, the camera also has in-body IS, a hybrid AF system, 4K video and a removable EVF.
What must be the world's most camera-friendly airport has set out locations all around its perimeter fence specifically so photographers can get all the best angles unhindered by wires.
According to DJI, any drone model weighing over 250 grams will have AirSense Automatic Dependent Surveillance-Broadcast (ADS-B) receivers installed to help drone operators know when planes and helicopters are nearby.
Premiere Rush is targeted at Youtubers and vloggers who want to edit video on the go.
The U.S. Department of Homeland Security (USDHS) sent out an alert on Monday stating drones manufactured in China are a 'potential risk to an organization's information.'
The new module might be implemented in the upcoming Galaxy Note device generation.
Chris and Jordan are kicking off a new segment in which they make feature suggestions to manufacturers for the benefit of all photographer-kind. To start things off, they take a look at the humble USB-C port and everything it could be doing for us.
The Olympus TG-5 is one of our favorite waterproof cameras, and the company today introduced the TG-6, a relatively low-key update. New features include the addition of an anti-reflective coating on the sensor, a higher-res LCD, and more underwater and macro modes.
We've long held Olympus' Tough cameras in high regard and the TG-6 is no exception. It offers top notch image quality for its class and lots of useful features. It's also a blast to shoot with.
A meticulously curated video from YouTuber Guy Jones highlights the evolution of street photography from mid-19th century to present day.
A big design is in the pipeline for the popular Godox shoot-through flash trigger, with a new control layout and a collection of new features - including Bluetooth connectivity
The iconic statues of Easter Island are at risk of getting destroyed thanks to tourists climbing on them and picking their noses.
The Honor 20 Pro looks like an attractively priced alternative to some more established flagship competitors.
We've been playing around with a prototype of the new Peak Design Travel Tripod and are impressed so far: it's incredibly compact, fast to deploy and stable enough for the heaviest bodies. However, the price may turn some away.