During its I/O 2017 conference yesterday, Google demonstrated a new algorithm-based technology that can remove unwanted objects from existing photographs. The demonstration showed the technology removing a chainlink fence from the foreground of an image, with the final result offering no discernible indications that the fence had ever existed (around 10:45 in the video below).
The technology was demonstrated on stage by Google's CEO Sundar Pichai during a conversation about the company's expanding visual technology. 'Coming very soon,' Pichai explained, 'if you take a picture of your daughter at a baseball game and there's something obstructing it, we can do the hard work and remove that structure and have the picture of what matters to you in front of you.'
It looks to be an evolution of the research Google and MIT have been collaborating on for some time – in fact, their demonstration from 2015 includes a very similar chain-link fence demo. This method takes advantage of the parallax effect to identify and remove obstructions from photos.
Unfortunately, Pichai didn't elaborate on when this technology will be made available aside from 'very soon,' nor did he specify where the technology will be available. Given the company's Google Photos announcements, however, it seems likely the technology will be implemented within that product.
Oops - I missed that there was a reference to 10:45. And - it is too late to edit my post. Anyhow - ignore my post. If someone from DPR sees this, please remove it (and this).
I'm Sorry, but that was not a demonstration of the technology. It was an illustration of the technology, or perhaps a description of it. Those chain-link fences were simple overlays, that they then removed, at the precise moment he says "in the future".
What? You are sure? In that case, I am quite stunned. Hmmm ... that the second image has a perfectly sharp fence hints at something. Enormous DOF for that camera.
I listened at 10:45, and yes, he said "coming very soon" we can do this. So, it was fake. And the audience cheered. Very effectful presentation. And also very manipulative. Lets see when and if this really is released.
yes. I saw that--it's from 18 months ago and involves removing the foreground from a still frame of video image...but the "demo" at 10:45 doesn't demonstrate to us that technology in any way. It's an illustration that what that technology "could" do, "soon". The "demo" doesn't appear to be showing an actual removal...the foreground fence doesn't look like it's part of the original image. I certainly have no doubt that this will be, or is, possible, but the "demo" in the video doesn't appear to be an actual demonstration of the technology (which the article reports, incorrectly)...which is all I said.
As for the technology itself, I'd love to see what it does if that little girl decided to "not be a statue" and actually swing the bat. The results might vary a little. :-)
Two more thoughts. I think the isolation of the reflected image off glass is actually more interesting...the fact that you could pull that much detail from the reflection has numerous creative and "forensic" uses. The second thought is why would anyone shoot with their smart phone 6-12 inches from glass or a chainlink fence? The lens on a smartphone is small enough that it would have little trouble peering through 1/2 inch wire fencing much less 2-3" chainlink. And pressing a phone camera lens against a window generally leads to largely reflection free photos...
This is not an idea. Talk about this has been going on for ages and there are some (more or less successful) tries. Now it is about implementation. The idea stadium was in another millenium.
Interesting - but there is a problem. In the second example the camera is focused on the chain link fence. When the fence is removed that does not change, so then the entire picture is out of focus!!! Loren
I find that if something is in the way from my viewpoint and I can't get round it then it makes me think harder, can I actually make it an active part of my composition ? Fences too.
They removed the chain link fence. Why stop? Remove the ugly construction materials and ugly buildings, replace the asphalt with grass and put leaves on the leafless trees.
Usually, I take the shot through a hole in the chainlink fence. It is even easier... ;-) Or take the shot with a lens allowing a DOF shallow enough to blurr the fence away.
All the years I sat at Little League games staring through chain-link fences, I kidded about inventing holographic sunglasses that would filter out chain-link fences. Maybe Google can help me...
It is but a matter of time. Narrow AIs can achieve above human level easily for particular tasks already. Some aspects of image recognition being one of those tasks... this is just a logical step in that direction.
Not long from now, they will probably do almost everything better and faster than us.
Not very sure, but they might have expanded the U-point technology that Nik first developed and which Google purchased a couple of years back? Nikon was using the U-point in their Capture NX software.
I can imagine turning up to a place to take a picture at a particular spot because I saw it on the internet and want to take one of my own, only to say, "Ugh, there is an ugly chain fence here!!". This tech will be a bit of both good and bad, depending on who you are I guess.
I can see this technology being useful, but then also leading to laziness too, for some. I think it may cause some photographers (particularly newbies) to become "lazy" and say "Oh, I'll just remove it in post processing" and not try to move around the obstruction (if it's possible) to get a better shot. I hear this statement a lot from beginners: "I'll just remove that later." Some things can be fixed in post, some things cannot.
The parallax method, requiring at least 2 photos from slightly different PoV can be done using Hugin. Done it several times to remove inconvenient telephone poles, tripod legs, or some random person.
The description says "existing photos" which in most case would not have been shot with obstruction removal in mind.
So which is it? A well understood compositing method which requires a pair of special shot photos, or a new method that can take an arbitrary single image, as shot, and perform these operations?
And if there isn't a hole where you want it to get the shot you want? If Google want to help photographers like with the free NIK filters,let them do it.
Today it is quite common to put fences at high buildings and bridges in order to hinder suicide and throwing large objects. Those fences are usually a bit away from you. No possibility to take the image through any hole. Actually, one of my all favorite photo spots is destroyed this way. A high bridge with a view over Stockholm. Now there is a fence. OK - it is possible to hold the camera out on the other side of the fence, but taking panoramas and also using tripod is no longer possible. This technique (if and when it works) sounds like a good idea for me.
Lots of things that make sense do not make sense. The bridge I am talking about is 26 meter high. The probability to survive is almost 100%. For some reason it is almost impossible to get statistics on suicides from the bridge. But, if I dig deeper, it looks like it almost never happens. It seems to be a myth. But, all "know" it is the most common way to commit suicide in Stockholm, something that definitely is untrue. Jumping from high houses, jumping in front of a train and medicine are the most common.
but if they stop putting fencing on every bridge and making kids wear helmets to chess matches , what , pray tell will our nanny state do?
I know of no cannabis fatalities here in America , but legal booze cigarettes and big -pharma heroin kill untold numbers but are just too cash imbued for the nannys to care ... instead they will continue to care where they are not wanted
Removing something with a consistent pattern is fine. What about removing more complex, discreet objects from the image? I occasionally make use of Photoshop context sensitive fill, and that sometimes takes quite a bit of work with layers and masks to i) make sure only what I want removing is actually removed and ii) what it gets replaced with looks credible.
This does look incredible. If it uses the parallax effect, that would imply the need for two slightly offset photos (like stereo photography), however, it states it can work on existing photos.
I would love to see this technology integrated into Google's Nik software in the future. In the meantime, it's made me think about taking two offset photos where there are obstructions close to the camera, overlay them in PS, then use the mask tools to eliminate the obstruction from the top layer. Hmmmm.
That's probably because it is. I'd like to see how it would deal with text or humans in the background, or a much more out of focus chain link fence. Reminds me very much of when Adobe acquired the rights to the Context Aware / Fill technology. Yes, it can be a useful starting point, but nowhere near as good as their examples suggested.
There are things I cannot do which others can do well. So I need other people. But what if I can create technology that begins where my capabilities end? I will no longer need others to do things I cannot do or those things I wish to avoid doing. I will be able to direct my technology to do everything I want to do but don't do well myself. I will be an emperor in an empire of my own choosing. I will only see a world of my choosing. I will only interact with others I choose. There will be no uncertainty. There will be no inefficiency. There will be no accidental natural wonder. And I will never have to face my limitations, other than the limits of my own imagination, because I will have no limitations to know. Or will there?
My thoughts. I'm pretty sure the world is nowhere near ready for everyone to be gliding around carefree in a seamless, hassle-free existence in which we are assisted and cared for in almost every manner by AI.
looks good unfortunately, the Pros amongst us wont use it and the $200 husband and wife teams on facebook are too lazy to edit their photos . So not sure who the market is
Photography is becoming more and more than just about the lenses and sensors. That's why Nikon is falling behind (I think). They are not really into software things. Nikon (and other manufacturers for that matter) must offer cool tricks like this to attract casual users.
It's not this (or any) technology that would cause someone to be uncreative, but the way one utilizes it. One can use a tool in an uncreative way and someone else may utilize it in a highly creative manner. The example that they show of removing fence seems pretty basic - obviously to show, very simply and in a way everyone enderstands, what this technology can do. I'm sure it can actually be used to do some very interesting things.
Right now I am using my creativity and my brains to either find a way to make a photograph where the fence doesn't show up on the image. Or try to incorporate the fence in an interesting photo.
Because of the google technology there is no need for that. It's that simple.
Did you watch the whole video? Google photos will also select your best photos for you and print it in a hardcover book. No need to spend precious time choosing the photos you want, let the algorithm decide!
Sadly no, the only possibly humorous part of this tech is that Google is taking it seriously enough that they are giving everyone in the (very huge) audience a free trial of the technology (i.e. a free photo book, with images automatically selected from all their Android photos).
Although I admit the facial recognition technology with auto-album update is kinda interesting. Now, if anyone takes a photo of your kids with an Android phone, you can automatically get a copy of the photo compliments of Google sharing technology :p
For most people, creativity never had anything to do with it in the first place. They're trying to preserve a memory or communicate, not make a work of art. This is just a tool to get a job done.
healing brush is completely software based and just guesses what's behind the fence
this is hardware based, it uses parallax error like is said in the article....same way humans with two eyes can see everything that is happening behind a fence
you're combining overlapping images..you're not guessing what's behind the fence like a healing brush, you can see what's behind it due to the overlapping image...the parallax error shifts the viewing frustrum, combine both correctly and the fence is gone
Might be difficult as most moire are very tiny patterns. If anything, the process will be ADDING the lines back that the moire ruined in the first place.
That's the future of photography. With increasing computational power more and more thing will become possible. And smartphones will use that power via cloud computing. Software will even overcome many limits of technology and will create images even DSLRs can't keep up with
with smartphone dual sensors they can now cleanly remove any element in the foreground without any loss in detail in the background
really, the next evolution will be cameras that can recognise objects with the help of dual cameras giving them depth..in the video the camera can regonize a flower..but this is just the beginning
with the help of depth and AI allowing them to recognize patters and colors and looking them up in a database...the cobots will truly have AI..see things..recognise it..learn and evolve..we'll have cobots doing household chores as if it's normal
All those are already possible with todays smartphones. You point camera around and in real-time it speaks what it see, like chair, apple, orange, teacup, chairs and table, newspaper on table etc...
Really? I don't get that attitude. LR automatically adds them when uploading to Flickr so why wouldn't you use it? Maybe it will stop someone from using your photo without telling you someday, who knows. And it's fun to put it there anyway, just more artistic expression.
@Docno I don't think that speaks to a sense of entitlement at all - it sounds like a different problem entirely. But I don't know the eastern half of the world well (except through the eastern immigrants who are definitely not entitled here) so I could not speak for the east.
In a broad and very general sweeping statement, I find that the people who are most possessive are the ones that have no reason to be, to put it politely. Of course that isn't the case for everything
You are affecting greater knowledge than you possess by making sociological assumptions about which you know very little. Now you can know what it means also (here's another example). ;)
I'll explain in simple terms shall I. I don't mean watermarks on agency images, I don't mean that I want to steal images from hard working individuals.
I rarely see good work posted in this forum that has a water mark attached to it or a 'photography by JT26' label somewhere on it for example. All the good images seem to be water mark or name free...
Secondly when I say 'posess' I meant photography ability. Shame I have to spell this out for you as it's kinda takes the edge off it all
The EF-M 32mm F1.4 is a welcome addition to Canon's APS-C mirrorless lens lineup. It's a good performer all-around and enjoyable to use on the EOS M50, and we hope to see more like it introduced to the EF-M range.
We don't often get excited about $900 cameras, but the Fujifilm X-T30 has really impressed us thus far. Find out what's new, what it's like to use and how it compares to its peers in our review in progress.
The S1 and S1R are Panasonic's first full-frame mirrorless cameras so there's a plenty to talk about. We've taken a look at the design and features of both cameras and have some initial impressions, as well.
The Olympus OM-D E-M1X is a more powerful dual-grip evolution of the E-M1 II. Aimed at sports shooters it promises improved AF, including advanced subject recognition, along with the highest-ever rated image stabilization system.
Following testing of the Panasonic Lumix DC-LX100 II, we've added it to our Pocketable Enthusiast Compact Cameras buying guide as joint-winner, alongside Sony's Cyber-shot RX100 VA.
If you're looking for a high-quality camera, you don't need to spend a ton of cash, nor do you need to buy the latest and greatest new product on the market. In our latest buying guide we've selected some cameras that while they're a bit older, still offer a lot of bang for the buck.
A question frequently asked on the Internet is 'what's the best DSLR?' In this buying guide we've answered that question – but also whether it's the right question to be asking in the first place.
What's the best camera for under $500? These entry level cameras should be easy to use, offer good image quality and easily connect with a smartphone for sharing. In this buying guide we've rounded up all the current interchangeable lens cameras costing less than $500 and recommended the best.
Whether you've grown tired of what came with your DSLR, or want to start photographing different subjects, a new lens is probably in order. We've selected our favorite lenses for Sony mirrorlses cameras in several categories to make your decisions easier.
The EF-M 32mm F1.4 is a welcome addition to Canon's APS-C mirrorless lens lineup. It's a good performer all-around and enjoyable to use on the EOS M50, and we hope to see more like it introduced to the EF-M range.
The data breach we reported on last week did not only affect 500px but a total of 16 websites, including mobile image sharing platform EyeEm, Animoto, Artsy and Fotolog.
Camera Rescue, a Finnish organization determined to rescue more than 100K analog, has already saved 46,000 cameras and plans to more than double that number by 2020.
Independent lens manufacturer Sigma has announced that its new 28mm T1.5 cine lens for full frame sensor cameras will be available from the middle of March.
At Dubai's recent Gulf Photo Plus event, Fujifilm showed off several of its early concept mockups for GFX cameras that (sadly) never made it into production. We took a closer look.
Panasonic is well known for including impressive video features on its cameras. In this article, professional cinematographer Jack Lam explains one killer feature the company could add to its S series that would shake up the industry – and it all comes down to manual focus.
Full-frame cameras get a lot of attention lately, but Technical Editor Richard Butler thinks that APS-C makes the most sense for a lot of people – and there's just one company consistently giving the format the support it deserves.
The 12th International Garden Photographer of the Year winners have been announced. We've gathered the top photos from each category and rounded them up into a slideshow.
Uber software engineer Phillip Wang has created a website that shows a portrait of a person that doesn't actually exist by using AI to merge multiple faces together.
Want to know more about the Canon EOS RP? Dying to ask a question that hasn't been addressed anywhere else online? Join the editors of DPReview for a live Q&A about this new camera next Tuesday, Feb. 19 on our YouTube channel. Click through for details.
Got a couple of minutes? Then you have all the time you need to learn about Canon's second full-frame mirrorless camera body – and why it's a compelling option for someone stepping into full-frame for the first time.
A quick glance at the spec sheet doesn't make the Canon EOS RP look that exciting. But having shot with it, we've become oddly fond of this little full framer.
Pixelmator Pro has received an update with new and improved features, including support for Portrait Masks with images captured by the iPhone's Portrait Mode.
Alongside the EOS RP, Canon showed us mockups of the six lenses it says are in development for 2019. There's a distinct high-end flavor to the options in the works.
The new X-T30 may not be Fujifilm's flagship model, but it arrives with some very impressive features and specifications. Chris and Jordan have been shooting it for a few days and share their first impressions, along with a look at an iconic new building in their hometown of Calgary.
We don't often get excited about $900 cameras, but the Fujifilm X-T30 has really impressed us thus far. Find out what's new, what it's like to use and how it compares to its peers in our review in progress.
The Fujifilm X-T30 is equipped with the same 26.1MP X-Trans sensor and X-Processor 4 Quad Core CPU as the X-T3, along with some autofocus improvements. The new camera arrives in March for $900 body-only.
Fujifilm's XF 16mm F2.8 is one of the widest lenses in the company's lineup of compact primes for its X-series interchangeable lens cameras. We've been up and down the streets of snowy Seattle - a rare sight - to see just what our pre-production copy of this petite prime is capable of.
Canon has unveiled its second full-frame mirrorless camera: the entry-level EOS RP. Touting its compact size and approachability for beginners, the RP uses a 26.2MP sensor and will sell for $1300 body-only this March.
A pre-launch event gave us a chance to shoot a sample gallery to show what sort of image quality you can expect from the least-expensive digital full frame camera ever launched.
Comments