ENHANCE!
August 5, 2015 3:01 PM   Subscribe

A computational approach for obstruction-free photography takes out the chain link fence obscuring the target of your photo, removes reflections, and--this is the crazy TV show part--can even build a separate image from the reflection. It uses multiple frames and magic math to build up the two "clean" images.

You can read the abstract and download the paper itself at the project's site. Researchers: Tianfan Xue, Michael Rubinstein (previously), Ce Liu, and William T. Freeman working with MIT CSAIL and Google Research.

Goes great with previously for your spy thriller needs!

h/t NPR.
posted by wintersweet (28 comments total) 31 users marked this as a favorite
 
Give me a hardcopy right there.
posted by sparklemotion at 3:02 PM on August 5, 2015 [11 favorites]


I literally just watched this via NPR and was coming here to post it!

Such awesome technology, I wish this was available for a ton of photos I've already taken.
posted by numaner at 3:06 PM on August 5, 2015 [1 favorite]


I was about 95% sure someone would post it while I was posting it.

It would be great if this could straight-up work on stills someday. I wonder how long till the video-based version reaches consumers...
posted by wintersweet at 3:14 PM on August 5, 2015 [1 favorite]


I know some of the words there... like "Canny edge detection"... and after that it's total magic.
posted by floatboth at 3:15 PM on August 5, 2015 [3 favorites]


This is quite cool, although in some cases, the reflection isn't so much "removed" as "drastically reduced".

I feel like we're just on the verge right now of some serious leaps forward in automated image recognition and processing. I think within a few years, we'll have both consumer products like cameras and cellphones, as well as professional products like image compositing and VFX software, doing stuff that was previously nearly impossible or at least extremely labor intensive.
posted by Joakim Ziegler at 3:22 PM on August 5, 2015


MORE LIKE UNCANNY EDGE DETECTION, AMIRITE?
posted by Joakim Ziegler at 3:22 PM on August 5, 2015 [8 favorites]


John F. "Not Un-" Canny (hey, that must be Soda Hall!).
posted by wintersweet at 3:25 PM on August 5, 2015


It would be great if this could straight-up work on stills someday.

I don't think that's possible... it requires several frames from slightly different angles/perspectives in order to work out the part of the central frame that's occluded. There's information about the background/foreground that's missing in any one frame (i.e., still photo) that's basically being filled in by the other frames where that information is present. I don't think any amount of clever math can overcome the fact that in a single still photo with occlusion there's information literally missing.
posted by axiom at 3:25 PM on August 5, 2015 [4 favorites]


Yeah, this requires multiple images from slightly different angles, but if you imagine this combined with something like Microsoft Photosynth, maybe those multiple other images don't need to be ones you yourself take. You could take your own tourist shots, and then fix them up and enhance them using the tourist shots of thousands of other people, given of course that you're shooting a sufficiently popular view.
posted by Joakim Ziegler at 3:28 PM on August 5, 2015 [8 favorites]


Or perhaps some machine-learning algorithm trained on large numbers of examples, which infers which part of the image is most plausibly a reflection and processes it out (perhaps even filling in plausible detail to replace what has been lost). Of course, this goes beyond image enhancement and into straight confabulation so it's useless for, say, forensic evidence, but would make for a fun smartphone app.
posted by acb at 3:42 PM on August 5, 2015


acb, Google's DeepDream already does this. It works especially well if you take a picture of alien dogs through a chainlink fence.
posted by hellphish at 3:44 PM on August 5, 2015 [17 favorites]


I have wanted this app for years, to get pictures of buildings that now have large trees in front of them. I imagine flying a drone between the trees and the building to get the multiple samples and synthesizing the view from the street, or from across the street, with the trees magically erased.

My specific interest is 80+ year old multistory brick schools.
posted by Bruce H. at 4:07 PM on August 5, 2015 [1 favorite]


Won't somebody think of the cranky internet nerds who have been ranting about the unrealism of 'Enhance!'?
posted by jacquilynne at 4:09 PM on August 5, 2015 [1 favorite]


I've been doing something similary to this (by hand) for a while... here's my first attempt at getting a tree removed from in front of a building. That was back in 2008, inspired by the work of Marc Levoy at Stanford.

I've since gotten a bit better at it, here's a picture of the old Chicago Public Library building, with the obscuring trees nicely blurred away a bit.

Synthetic focus is pretty cool, it allows you to see through things that practically opaque.
posted by MikeWarot at 4:17 PM on August 5, 2015 [5 favorites]


This could have some serious forensic applications I think.
posted by Evstar at 4:48 PM on August 5, 2015 [1 favorite]


You'll notice that there are no pesky human beings or cats or cars or other things moving around in the field of view, just a static scene and a moving camera, which is all their model can cope with. This does not map well to the kinds of thing people actually shoot images of, e.g. each other or themselves.
posted by w0mbat at 4:57 PM on August 5, 2015 [2 favorites]




w0mbat, one of the final examples is of a tiger in a zoo. Granted, it isn't moving much.
posted by Evstar at 5:13 PM on August 5, 2015


I dunno, to me this presentation comes off as very defencive.
posted by dephlogisticated at 5:43 PM on August 5, 2015 [9 favorites]


Great for surveillance: with enough cheap cameras you can video a crowd and extract a continuous image of a suspect, with the surrounding people removed.
posted by Joe in Australia at 6:23 PM on August 5, 2015 [1 favorite]


This demo is of the algorithm doing things automatically without any guidance... give them (or someone else) time to streamline it, and you'll soon be able to do videos this way.

The idea of using camera arrays for surveillance goes back to at least the demos Levoy et al did with the Stanford Camera array prior to me reading about it from there in 2008.
posted by MikeWarot at 6:43 PM on August 5, 2015


I have wanted this app for years, to get pictures of buildings that now have large trees in front of them. I imagine flying a drone between the trees and the building to get the multiple samples and synthesizing the view from the street, or from across the street, with the trees magically erased.

My specific interest is 80+ year old multistory brick schools.


Bruce H., this has been possible for several years using a handheld camera. Move parallel to the façade, keeping a consistent height, and ideally with your camera in manual mode to reduce the need for exposure compensation, and take overlapping photographs as you move. Make sure that each part of the façade is visible (not obscured by a tree) in at least one photograph. Use Hugin to stitch the panorama, selecting a rectilinear projection. Here is an example using only two images but demonstrating the appropriate modes and the process of masking out foreground objects (trees, in your case). With real-world façades, you may need to invest additional manual effort in ensuring coplanar control points; this is not important in a typical panorama with a fixed perspective, but is crucial in these mosaics with linear camera motion.
posted by musicinmybrain at 7:39 PM on August 5, 2015 [9 favorites]


It would be great if this could straight-up work on stills someday.

I don't think that's possible... it requires several frames from slightly different angles/perspectives in order to work out the part of the central frame that's occluded.


True, but one possible implementation is where I press the button on my smart phone to take a photo (as it's presented to the user), but the device actually takes a very short video, splits it into frames, does the magic thing to the extent made possible by my hand's micro-jitters, then milliseconds later shows me a de-cluttered still image.
posted by univac at 8:03 PM on August 5, 2015 [1 favorite]


I've thought for a decade or so that the ability to remove obstacles and figures from moving images was the next thing. A generation from now, viewers will look at old movies and not realize the logistics of emptying out London for 28 Days Later or New York for Vanilla Sky, because this sort of trick will be done on your personal computer by about 2025. I recall how proud I was around 1999 when my self-taught Photoshop skills allowed me to manipulate a photo of my workplace and vanish a distracting bicycle from the image. That does not impress my teenager that much in 2015.
posted by ricochet biscuit at 2:06 AM on August 6, 2015 [3 favorites]


Also in this year's SIGGRAPH: Sampling Based Scene-Space Video Processing. "Video Inpaining" and "Action Shots" are especially interesting applications.
posted by ikalliom at 2:20 AM on August 6, 2015 [1 favorite]


univac: "True, but one possible implementation is where I press the button on my smart phone to take a photo (as it's presented to the user), but the device actually takes a very short video, splits it into frames, does the magic thing to the extent made possible by my hand's micro-jitters, then milliseconds later shows me a de-cluttered still image."

Possible as long as the micro-jitters are big enough that you can view all of the obstructed object at some point. So maybe workable for chain link fences, not so much for big trees, probably.
posted by Joakim Ziegler at 8:32 PM on August 7, 2015


ricochet biscuit: "the logistics of emptying out London for 28 Days Later"

Those scenes were mostly shot very early morning in summer, if I'm not mistaken, so that wasn't as hard as it seems.
posted by Joakim Ziegler at 8:33 PM on August 7, 2015


musicinmybrain: "Here is an example using only two images but demonstrating the appropriate modes and the process of masking out foreground objects (trees, in your case)."

That is bloody awesome. I bet it would work to compose a people free image of a slightly to moderately busy location to so you could be the only person posing in front of the leaning tower or something. I am so trying this this week.
posted by Mitheral at 5:07 PM on August 9, 2015


« Older Don't know much about history: New, New Framework...   |   How to do it Newer »


This thread has been archived and is closed to new comments