Dreaming Reconstructions
January 20, 2014 1:04 AM   Subscribe

With the advent of Flickr, Picasa, Instagram, i.e. massive databases of images, researchers have devised tools like simple markers like SIFT,  in order to reduce image information to bits.  As it turns out, many people think this degraded information can only be used for simple operation on images such as comparison, etc... Hérvé Jégou and other researchers have shown in some vivid manner that one can reconstruct images from these "information degraded" markers. As an extension of the original  paper, Hérvé Jégou has produced a wide variety of reconstructions from photos taken while on vacation. The result: stunning dream-like, artistic-like rendering of the original scenes. And while it may raise all kinds of privacy issues, one cannot escape the similarity between these renderings and the scenes reconstructed from MRI readings.
posted by IgorCarron (18 comments total) 31 users marked this as a favorite
the scenes reconstructed from MRI readings on human brain activity are quite impressive. thanks for the video.
posted by gbenard at 3:51 AM on January 20, 2014 [1 favorite]

The privacy implications were not obvious to me so I delved into the paper.

The example they use is someone trying to locate illegal copies of an image by submitting SIFT data to a search engine which indexes publicly available images. You may not want to disclose the image you have while looking for copies, but the SIFT data allows a third party to reconstruct it to a certain extent.
posted by sweet mister at 4:49 AM on January 20, 2014 [1 favorite]

My favourite part of the paper was when they exhumed Monet and connected the remains of his brain to a digital projector from 1996.
posted by oulipian at 5:14 AM on January 20, 2014 [6 favorites]

Now I wish I was in Verona.
posted by Segundus at 5:23 AM on January 20, 2014

Is it just me or is anyone else having difficulty figuring out what any of this means.
posted by bleep at 6:49 AM on January 20, 2014 [7 favorites]

Can you do it twice? Build a SIFT reconstruction, then SIFT it, then reconstruct again?

What does an MRI of person looking at an image reconstructed from an MRI reveal?

Basically, I demand recursion.
posted by anotherpanacea at 7:11 AM on January 20, 2014 [2 favorites]

I remember being skeptical of the MRI video, and I think I remember going to the source article previously but don't have time to debunk this.
posted by saber_taylor at 7:40 AM on January 20, 2014

Go away impressionist computers!
posted by Mister_A at 7:41 AM on January 20, 2014 [1 favorite]


Imagine you have 20GB of family photos on your hard drive. Searching through these 20GB might take time especially if you are making a request such as "find all images with buildings in them". or "find images with faces in them". For this searching tasks, computer vision scientists have devised simple ways to find interesting elements in an image through what they call keypoint descriptors. An image has many of those low level descriptors for instance. For a long time, people thought that these descriptors did not contain that much information, i.e. they were good for comparing images to each other but that was pretty much the end of that. The work of the authors show that using these descriptors and a database that does not contain any of the images being reconstructed one can reconstruct the images as shown in the webpage. Does that make sense ?
posted by IgorCarron at 8:06 AM on January 20, 2014 [4 favorites]

Thanks, IgorCarron. Any links (Wikipedia?) to short descriptions of keypoint descriptors, what they contain (easy) and how they are found (probably complicated)?
posted by benito.strauss at 8:56 AM on January 20, 2014

The reconstructions are too pixellated to be properly Impressionist, but I could see someone championing that style in physical paintings after seeing a bunch of these (and the MRIs).
posted by immlass at 9:08 AM on January 20, 2014 [1 favorite]

Hey, computer-generated impressionism is not static! Get with the times, man!
posted by Mister_A at 9:09 AM on January 20, 2014 [1 favorite]


For a wikipedia description of SIFT: http://en.wikipedia.org/wiki/Scale-invariant_feature_transform
Here is the original paper: http://www.cs.ubc.ca/~lowe/papers/iccv99.pdf

There are other keypoint descriptors like FREAK ( http://infoscience.epfl.ch/record/175537/files/2069.pdf ), BRISK (http://www.robots.ox.ac.uk/~vgg/rg/papers/brisk.pdf ). See figure 8 of this last reference to see how these descriptors are used to match one image with another.
posted by IgorCarron at 9:41 AM on January 20, 2014 [2 favorites]

Thanks, Igor. There's a lot of interesting stuff there.
posted by benito.strauss at 10:39 AM on January 20, 2014 [1 favorite]

This is still not making any sense. You construct data with descriptors... how? Why does it come out like those photos?
posted by crapmatic at 11:06 AM on January 20, 2014 [1 favorite]

You construct data with descriptors... how?

You use a computer program to identify parts of the picture it thinks are interesting, and create a brief description of them. If you got a person to do it they'd say something like "It's a cat, hanging by its paws from a clothesline, and it looks worried." The computer program isn't that smart so its description is "There's a pointy triangle here, another one there [i.e., the cat's ears], a horizontal line at this height [the clothesline], two circles [its eyes] ..."

Anyway, you now have a brief description of the picture. You can run the program over other pictures, and find ones with similar descriptions. This is much faster than comparing the pictures dot-for-dot, and it often finds matches even where one picture has been cropped or otherwise altered. People thought that these descriptions were too brief and too computer-ish to actually reconstruct the original picture from the description, but it turns out that no, you can use the description and some other information to make something that a person would recognise.

Why does it come out like those photos?

Because the compressed description is very brief. It throws away color information and the actual details of the picture, so the program grabs the color and details from what it thinks must be similar pictures, and it fills in gaps by making guessses. If the photos are actually similar, this works well. Otherwise it can match the details of someone's face (for instance) with a chunk of another picture that just happened to have some shadows that looked like that face - to a computer program. It turns out that buildings are easy to reconstruct this way, plants are hard, sky (which doesn't have many things the computer finds "interesting") is very hard.
posted by Joe in Australia at 12:23 PM on January 20, 2014 [3 favorites]

I want all of these framed on my wall. Especially this one.
posted by feckless fecal fear mongering at 2:17 PM on January 20, 2014 [3 favorites]

Reminds me of glitch art a little bit, although the science behind it is a lot more fascinating.
posted by gucci mane at 2:34 PM on January 20, 2014 [1 favorite]

« Older Am I being detained? Am I free to go?   |   "She had become a local martyr for the vegan... Newer »

This thread has been archived and is closed to new comments