MIT algorithm aims to eradicate reflections from photos taken through windows
Researchers at the Massachusetts Institute of Technology claim to have developed a method for eliminating glass reflections in photos via digital processing. It is hoped that with further development the idea could see its way into cameras themselves, enabling reflections to be automatically removed when they interfere with the view through a window.
The new algorithm, due to be formally announced in June at the Boston Computer Vision and Pattern Recognition conference, is claimed to offer a way of distinguishing the image reflected by a window surface from the image of whatever is behind the window. Once the two images are clearly identified one can be subtracted from the whole to leave the other.
MIT researchers noted that objects reflected by glass surfaces tend to have two reflections, not one, and that the two images of reflected objects are slightly off-set. The multiple images in a reflection come about sometimes because windows are double glazed for insulation and thus have two panes of glass, and other times because light reflects from the rear surface of a pane of glass as well as the facing surface.
Further research showed that the interference of a reflection can often be identified by the way it interrupts the regular shape and color patterns of natural and man-made objects on the other side of the glass. Working with small clusters of 8x8 pixels, researchers analyzed color and tonal relationships within the group and came to a method by which certain data correlations indicated that the group included detail from a reflection.
The result is that when an image of, for example, a view shot through a bus window is analyzed, the team can separate the image of the view from the reflected image of the photographer that shot the view – and then can show either the view on its own, or the photographer on his or her own.
If successful the algorithm may have serious consequences for the flexible rubber lens hood market, though early indications suggest polarizing filter manufacturers will be safe for some time to come.
For more information see the Massachusetts Institute of Technology website.
Removing reflections from photos taken through windows
New algorithm exploits multiple reflections in individual images to distinguish reflection from transmission.
Larry Hardesty | MIT News Office
May 11, 2015
It's hard to take a photo through a window without picking up reflections of the objects behind you. To solve that problem, professional photographers sometimes wrap their camera lenses in dark cloths affixed to windows by tape or suction cups. But that's not a terribly attractive option for a traveler using a point-and-shoot camera to capture the view from a hotel room or a seat in a train.
At the Computer Vision and Pattern Recognition conference in June, MIT researchers will present a new algorithm that, in a broad range of cases, can automatically remove reflections from digital photos. The algorithm exploits the fact that photos taken through windows often feature two nearly identical reflections, slightly offset from each other.
“In Boston, the windows are usually double-paned windows for heat isolation during the winter,” says YiChang Shih, who completed his PhD in computer science at MIT this spring and is first author on the paper. “With that kind of window, there's one reflection coming from the inner pane and another reflection from the outer pane. But thick windows are usually enough to produce a double reflection, too. The inner side will give a reflection, and the outer side will give a reflection as well.”
Without the extra information provided by the duplicate reflection, the problem of reflection removal is virtually insoluble, Shih explains. “You have an image from outdoor and another image from indoor, and what you capture is the sum of these two pictures,” he says. “If A+B is equal to C, then how will you recover A and B from a single C? That's mathematically challenging. We just don't have enough constraints to reach a conclusion.”
Thinning the field
The second reflection imposes the required constraint. Now the problem becomes recovering A, B, and C from a single D. But the value of A for one pixel has to be the same as the value of B for a pixel a fixed distance away in a prescribed direction. That constraint drastically reduces the range of solutions that the algorithm has to consider.
Nonetheless, a host of solutions still remain. To home in on one of them, Shih and his coauthors — professors of computer science and engineering Frédo Durand and Bill Freeman, who were Shih's thesis advisors, and Dilip Krishnan, a former postdoc in Freeman's group who's now at Google Research — assume that both the reflected image and the image captured through the window have the statistical regularities of so-called natural images.
The basic intuition is that at the level of small clusters of pixels, in natural images — unaltered representations of natural and built environments — abrupt changes of color are rare. And when they do occur, they occur along clear boundaries. So if a small block of pixels happens to contain part of the edge between a blue object and a red object, everything on one side of the edge will be bluish, and everything on the other side will be reddish.
In computer vision, the standard way to try to capture this intuition is with the notion of image gradients, which characterize each block of pixels according to the chief direction of color change and the rate of change. But Shih and his colleagues found that this approach didn't work very well.
Playing the odds
Ultimately, they settled on a new technique co-developed by Daniel Zoran, a postdoc in Freeman's group. Zoran and Yair Weiss of the Hebrew University of Jerusalem created an algorithm that divides images into 8-by-8 blocks of pixels; for each block, it calculates the correlation between each pixel and each of the others. The aggregate statistics for all the 8-by-8 blocks in 50,000 training images proved a reliable way to distinguish reflections from images shot through glass.
In their paper, Shih and his colleagues report performing searches on Google and the Flickr photo database using terms like “window reflection photography problems.” After excluding results that weren't photos shot through glass, they had 197 images, 96 of which exhibited double reflections that were offset far enough for their algorithm to work.
“People have worked on methods for eliminating these reflections from photos, but there had been drawbacks in past approaches,” says Yoav Schechner, a professor of electrical engineering at Israel's Technion. “Some methods attempt using a single shot. This is very hard, so prior results had partial success, and there was no automated way of telling if the recovered scene is the one reflected by the window or the one behind the window. This work does a good job on several fronts.”
“The ideas here can progress into routine photography, if the algorithm is further robustified and becomes part of toolboxes used in digital photography,” he adds. “It may help robot vision in the presence of confusing glass reflection.”
We were in Japan earlier this month for the annual CP+ show in Yokohama, where we sat down with senior executives from several camera and lens manufacturers, among them Nikon.
Sony has released firmware version 5.0 for its flagship mirrorless camera, the a9. The update brings AI-driven autofocus modes, an improved menu structure and other updates.
Night Sight, Portrait Mode and (surprisingly) wide-angle selfie mode are features that we're currently loving about the Pixel 3's camera.
The Auschwitz Museum has asked visitors to be more respectful after an upsurge of pictures posted on social media showing people posing on the train tracks that lead to the main gate.
This week Chris and Jordan take the new Leica Q2 for a spin, and while most of us in the Northern Hemisphere are welcoming spring, they head even farther north than usual to visit ice castles. Because #Canada.
Harvard is facing a lawsuit over profiting from 19th century daguerreotypes that captured the portrait of a slave and his daughter on a South Carolina plantation.
From the detailed textures in rural landscapes to the incredible lighting inside futuristic buildings, the photorealism of Unreal Engine 4 is blurring the lines between fiction and reality...you know...aside from the spaceship.
According to a report from The Informant, a number of Instagram users' passwords were shared as plaintext in URLs used to download their data.
We've added Panasonic's new Lumix S1 and S1R full-frame mirrorless cameras to three of our buying guides. If you're looking for a quick summary of each model, then have a read.
YouTube channel Photoshop Cafe has shared a video detailing ten tips and tricks you can do to both fix and speed up Photoshop when it's running slow and sluggish.
It's not going to be the banger of the year, but it'll get a few laughs.
DJI has confirmed its drones won't be affected by the GPS 2019 week rollover.
Dogfish Head Craft Brewery has teamed up with Kodak to release a beer that's capable of doubling as a film developer.
The Diana Instant Square is a retro-inspired camera with manual controls that's fun to shoot in good light, but largely unpredictable in its operation.
Residents of a Paris street plagued by Instagrammers, selfie takers and music video crews are asking the city government for a weekend and evening ban to give them some peace.
The adapter plugs into the Osmo Pocket's USB Type-C port and features a 3.5mm TRS jack to plug in various external microphones.
Checkout allows Instagram users to select products for purchase and make payments directly in the app.
GauGAN as it's known, can create photorealistic images from basic drawings using the power of artificial intelligence.
The EOS RP is Canon's latest full-frame mirrorless camera, with diminutive dimensions and a diminutive price. Find out how it stacks up and get our thoughts in our early review.
Montana judge Dana L. Christensen has ruled the Republican National Committee did not infringe upon the copyright of photographer Erika Peterman after they took a photo from a Democratic candidate's Facebook page without permission and altered it to use in a derogatory promotional mailer.
Nikon has launched updates for three of its programs to address various bugs and glitches that could cause crashes and unwanted results.
LEE Filters has launched the LEE100, its next-generation filter holder that improves the design and looks in all the right places.
With the arrival of some much-needed sunshine and final production firmware for the Panasonic S1, we've been able to get outside and really start putting the camera through its paces.
Importing, culling and tagging photos is about to get a whole lot faster and look a whole lot better with the impending arrival of Photo Mechanic 6.
On its own, the FTZ adapter retails for $250 and when bundled it dropped the cost to just $150. Now, Nikon is offering it for free with all Z6, Z7 purchases in the United States.
Profoto said it spoke with Godox back at Photokina 2018 and continues to contact Godox in an effort to stop it from marketing its V1 light.
Product renders in Italian publication Notebook Italia show an unusual design that conceals all cameras with the help of a slider mechanism.
Canon says its new EF 400mm F2.8L IS III and EF 600mm F4L IS III lenses can suffer from an intermittent flickering when shooting video in M or Av modes with certain cameras.
Leica recently announced the Q2, a digital rangefinder with a fixed 28mm F1.7 lens. It's a heck of a lot of fun to shoot with, but is it right for you? Based on our time with the camera, and its specifications, we've examined how well-suited it is for common photography use-cases.
Now that our Panasonic Lumix S1R has final firmware, we couldn't wait to get out shooting with it - and we also tried the high-res mode, which combines files to get 187 megapixel images. Because sometimes, 47 megapixels just isn't enough.