Many photos make light work
July 29, 2009 8:58 PM Subscribe
Assistant professor Noah Snavely at Cornell is into making models. Computer models of real scenes are assembled from photographs, and can then be used to create a better image, a better video, or a 3D representation of an entire city.
Snavely was on the team that created the jaw-dropping demo video in 2008. That project, titled “Using Photographs to Enhance Videos of a Static Scene”, was followed by “Enhancing and Experiencing Spacetime Resolution with Videos and Stills” (without Snavely and some of the original team).
Now Snavely is working with six people from his alma mater (the University of Washington) to re-create Rome, Italy. Pulling over 150 thousand photos from Flickr (tagged with "rome" or "roma"), the team used a cluster of computers working in parallel to compare photos and model the common buildings. This results in models of several sites, each with incredible precision.
Snavely was on the team that created the jaw-dropping demo video in 2008. That project, titled “Using Photographs to Enhance Videos of a Static Scene”, was followed by “Enhancing and Experiencing Spacetime Resolution with Videos and Stills” (without Snavely and some of the original team).
Now Snavely is working with six people from his alma mater (the University of Washington) to re-create Rome, Italy. Pulling over 150 thousand photos from Flickr (tagged with "rome" or "roma"), the team used a cluster of computers working in parallel to compare photos and model the common buildings. This results in models of several sites, each with incredible precision.
Sadly, I foresee a whole bunch of people in the future using this technology in the same ham-handed way they're currently using easy-bake HDR software to stuff Flickr with garish HDRs of muscle cars and sunsets. The Internet PervertsTM will probably find a way to create scenes that never happened in the same way they're currently using Poser and Second Life for their boytaur clown needs, and it'll be all the easier for incriminating evidence to be doctored out of war zone / protest videos and the like.
Seriously though, this is really neat stuff- if anything it should be a boon to low-budget filmmakers.
posted by dunkadunc at 10:00 PM on July 29, 2009 [1 favorite]
Seriously though, this is really neat stuff- if anything it should be a boon to low-budget filmmakers.
posted by dunkadunc at 10:00 PM on July 29, 2009 [1 favorite]
there was an earlier video (with the same narrator) describing the process they use to match the stills to the video (as in the scene with the 'patch' on the tree trunk). it was kind of even more amazing...they were able to create 3D imagery and video with nothing but stills using a program that auto-recognized features in a photograph and assigned them to locations in space. the examples used were a train and some buildings...does anyone know what i'm talking about...i thought it was posted on the blue, but i can't seem to find it now...beuller?
posted by sexyrobot at 10:16 PM on July 29, 2009
posted by sexyrobot at 10:16 PM on July 29, 2009
also, which sounds better:
MetaFilter: The Internet PervertsTM
or
MetaFilter: for your boytaur clown needs.
?
posted by sexyrobot at 10:25 PM on July 29, 2009 [1 favorite]
MetaFilter: The Internet PervertsTM
or
MetaFilter: for your boytaur clown needs.
?
posted by sexyrobot at 10:25 PM on July 29, 2009 [1 favorite]
Video evidence: now twice as useless
cool stuff though
posted by Salvor Hardin at 10:47 PM on July 29, 2009
cool stuff though
posted by Salvor Hardin at 10:47 PM on July 29, 2009
Didn't Snavely used to teach Defence Against the Dark Arts at Hogwarts?
posted by fatbird at 11:06 PM on July 29, 2009 [3 favorites]
posted by fatbird at 11:06 PM on July 29, 2009 [3 favorites]
You know, I thought I knew something about algorithms and imagery and mathematics. But this is fuckin' magic to me.
posted by phrontist at 11:09 PM on July 29, 2009
posted by phrontist at 11:09 PM on July 29, 2009
Is the input video stereo or mono? The OP's YT video @ 1:00 mentions stereo video as a source of depth info, but the diagram shows stereo as a post process of the "structure from motion" step. So is the stereo info synthesized from a previous step or is this jaw dropping video only applicable to dual camera recordings?
posted by ecco at 12:03 AM on July 30, 2009
posted by ecco at 12:03 AM on July 30, 2009
So is the stereo info synthesized from a previous step or is this jaw dropping video only applicable to dual camera recordings?
it uses sequential frames (from the video source) for the stereo information...much the way u can take stereo pictures with a normal camera by taking two shots sequentially a few inches apart...there's more info in this video about the process of using a single camera
however i found the previous post i mentioned earlier...thought it was the same guys, but then remembered it was from carnegie mellon...you gotta check it out...3d extraction from a single still frame...automatically! its pretty rad.
posted by sexyrobot at 2:17 AM on July 30, 2009
it uses sequential frames (from the video source) for the stereo information...much the way u can take stereo pictures with a normal camera by taking two shots sequentially a few inches apart...there's more info in this video about the process of using a single camera
however i found the previous post i mentioned earlier...thought it was the same guys, but then remembered it was from carnegie mellon...you gotta check it out...3d extraction from a single still frame...automatically! its pretty rad.
posted by sexyrobot at 2:17 AM on July 30, 2009
I want to see it done for Dealey Plaza, Nov 22nd 1963
posted by A189Nut at 2:17 AM on July 30, 2009 [1 favorite]
posted by A189Nut at 2:17 AM on July 30, 2009 [1 favorite]
Reminds me of Microsoft's Photosynth.
smackfu: PhotoSynth originated at UW with Snavely (see here), and I'm pretty sure Blaise name-checks him in the TED video.
posted by paulesque at 3:03 AM on July 30, 2009 [1 favorite]
smackfu: PhotoSynth originated at UW with Snavely (see here), and I'm pretty sure Blaise name-checks him in the TED video.
posted by paulesque at 3:03 AM on July 30, 2009 [1 favorite]
I went to grad school with Noah. Dude knows his shit.
posted by lucasks at 8:02 AM on July 30, 2009 [1 favorite]
posted by lucasks at 8:02 AM on July 30, 2009 [1 favorite]
The vast majority of the machine vision work for Photosynth came from Noah Snavely while at UW (which benefited from the guidance of Noah's UW and MSR advisors, Steve Seitz and Richard Szeliski). It was made a bit more scalable in our lab (taking the computation time down from weeks to minutes) and we also added some new navigational metaphors like the circular "donut" used to traverse around an object. However, I think the whole thing really came together when we married it to the multi-scale technology in Seadragon, which is the piece that Blaise Aguera y Arcas designed and built.
So, yes, this looks like Photosynth because Noah built it, and Noah is one of the primary inventors of Photosynth.
(BTW, I founded and run Live Labs, the team which put a lot of these things together.)
Noah is a supremely talented scientist, and I suspect that his work will profoundly change how many of us use and interact with visual mediums for decades to come.
posted by dr.flakenstein at 8:27 AM on July 30, 2009 [1 favorite]
So, yes, this looks like Photosynth because Noah built it, and Noah is one of the primary inventors of Photosynth.
(BTW, I founded and run Live Labs, the team which put a lot of these things together.)
Noah is a supremely talented scientist, and I suspect that his work will profoundly change how many of us use and interact with visual mediums for decades to come.
posted by dr.flakenstein at 8:27 AM on July 30, 2009 [1 favorite]
I want to see it done for Dealey Plaza, Nov 22nd 1963
done.
posted by sexyrobot at 10:44 AM on July 30, 2009
done.
posted by sexyrobot at 10:44 AM on July 30, 2009
This is one of the most exciting things I have seen in a long time.
posted by zzazazz at 5:12 PM on July 30, 2009
posted by zzazazz at 5:12 PM on July 30, 2009
« Older The Wonderful Details of... wait, reddit? Really? | Is there no problem the internet can't solve -... Newer »
This thread has been archived and is closed to new comments
Although the "better video" looked a bit CG to me, I do think this is the wave of the future.
While doing color correction on video lately, I was thinking about the fact that I was altering the image so much I was almost just drawing what I wanted, and at some point what's actually recorded from life will be more like a suggestion than something you're stuck with.
posted by drjimmy11 at 9:12 PM on July 29, 2009