Escher Circuits - Software for your Wetware

March 29, 2010 2:57 PM

Escher Circuits. What if you could compute the output of complex algorithms just by viewing an image?

Our everyday visual perceptions rely upon unfathomably complex computations carried out by tens of billions of neurons across over half our cortex. In spite of this, it does not “feel” like work to see. Our cognitive powers are, in stark contrast, “slow and painful,” and we have great trouble with embarrassingly simple logic tasks. Might it be possible to harness our visual computational powers for other tasks, perhaps for tasks cognition finds difficult? I have recently begun such a research program with the goal of devising ways of converting digital logic circuits into visual stimuli – “visual circuits” – which, when presented to the eye, “tricks” the visual system into carrying out the digital logic computation and generating a perception that amounts to the “output” of the computation. That is, the technique amounts to turning our visual system into a programmable computer.

posted by scalefree (57 comments total)

Drawing circles in the sand.
posted by TwelveTwo at 3:01 PM on March 29, 2010

uuuhhhh... seems like a lot of setup to save a little computation cost. And even the XOR was confusing to look at and it's not really that complex. But neat idea, sure.
posted by GuyZero at 3:05 PM on March 29, 2010

I wrote papers like this as an undergrad in my AI class -- only I mused with Salvador Dali pictures way back in '94 -- it was lots of fun...
posted by Alexandra Kitty at 3:05 PM on March 29, 2010

I don't really see how this is different from having to know how to look at a regular circuit diagram.
posted by cmoj at 3:05 PM on March 29, 2010

It's all fun and games until someone codes the nam-shub of Enki.
posted by five fresh fish at 3:06 PM on March 29, 2010 [31 favorites]

Or finds a basilisk...
posted by zztzed at 3:08 PM on March 29, 2010 [6 favorites]

I don't know how this makes things easier. I thought this would be like how you can solve problems by drawing them out without ever resorting to equations, but this is deeply different, and ... I am having trouble making heads or tails of it. It reminds me of the research on using facial expressions to convey mathematical relationships but a lot less intuitive.
posted by TwelveTwo at 3:10 PM on March 29, 2010

Should I be disturbed or relieved that I'm not the first one who immediately thinks "Optic Nerve Hack"?

How soon till I my contact lenses must incorporate a firewall...
posted by theDTs at 3:23 PM on March 29, 2010 [5 favorites]

This idea is very interesting, but the prototype diagrams in the article don't work: they still use the part of my brain that solves logic puzzles, rather than the part of my brain that looks out the window and says "there's a goose!"
posted by fantabulous timewaster at 3:26 PM on March 29, 2010 [1 favorite]

Hints of the Intersect from Chuck and/or the momerath process from Greg Bear's Anvil of Stars...
posted by weston at 3:27 PM on March 29, 2010

I thought this would be all Chernoff Faces and Tufte infoporn. The concept is neat, but I don't see how this is really taking advantage of the visual processing centers in a new way.
posted by echo target at 3:32 PM on March 29, 2010 [2 favorites]

Isn't this how Picard almost killed the Borg?
posted by rbs at 3:33 PM on March 29, 2010 [1 favorite]

I have two ways of thinking. One of them is linear, verbal, logical. The other is visual, wholistic (I really hate that word). I can switch between them when I need to, and some kinds of problems are better solved one way and some the other way.

When I used to play the game of Go, I tended to use visual thinking rather than logical thinking, for the most part.
posted by Chocolate Pickle at 3:38 PM on March 29, 2010

You could argue that this is how Kekule solved the structure of benzene, and that dreams in general, which are mainly visual, are using the visual cortex to solve some kind of problems, even if we can't easily articulate what those problems are.
posted by jamjam at 3:44 PM on March 29, 2010 [1 favorite]

I don't see it.....
posted by warbaby at 3:44 PM on March 29, 2010

It's not clear that at problem sizes at which computers become impractical that human visual processing wouldn't also be totally overwhelmed. Even assuming that isn't a concern, it's a lot more interesting to turn these sorts of questions around: "What is the assignment of inputs that causes the output to be 1?" for example...
posted by Zach! at 3:44 PM on March 29, 2010

This is kind of cool, but it doesn't quite work for me — I don’t always see the figures the way they’re supposed to go, and it’s not terribly hard to convince my brain to see it the “wrong” way.
posted by spitefulcrow at 3:59 PM on March 29, 2010

It's all fun and games until somebody encodes one of those 3D magic eye pictures into a diagram. Two weeks after viewing one you'll wake up in jail after having stolen all your coworkers' credit cards and used them to ship v!agRa and ROIEX watches to Nigeria.
posted by ardgedee at 4:07 PM on March 29, 2010

The leap from conceptual--thinking about why the brain works the way it does--to the actual "optic hack" seems to be the work of several lifetimes. Perhaps the half-life of scientific inquiry will shorten this, but I don't see it happening any time soon (no pun intended).

And I'm still trying to figure out Shea Zellweger's X-stem Logic Alphabet, which at least seems plausible and achievable. This optic hack--in the way the author contends? Not so much.
posted by beelzbubba at 4:12 PM on March 29, 2010

Forgot the link to X-Stem Logic Alphabet.
posted by beelzbubba at 4:13 PM on March 29, 2010

I find the idea of it fascinating- the idea of using things our brain does exceedingly well on a subconscious level and using an input 'language' that can then make us process problems near-instantly in parallel that we'd otherwise struggle to comprehend. It's like using the powerful circuitry of a computer's GPU to solve problems more befitting that type of processing.

That's the idea- but this doesn't seem to do that. It still seems to want us to follow a linear process with our eyes (and to be fair, he calls this out near the end as an issue) which then becomes subjective to the viewer as well requires a concentration to remember which flip/flop the visual is currently in. After all, the trick of an Escher drawing is that as we move our eyes around it, we can't quite find the exact point where it suddenly changed orientation, so any computation that relies on our "correct" processing of the orientation of ambiguous graphical elements is inherently a crapshoot, and thus useless. Not to mention, recreating linear circuits seem pointless, since almost no problem you'd design that way would be faster with visual processing compared to the same circuit as a an actual electronic circuit.

Still, the idea of it is intriguing, very "Diamond Age". The only value would be if the visual image was processed holistically in a way that a computer working in serial or mildly parallel could never emulate. Something like a picture that effectively sends a signal to the different parts of our visual cortex, so that the part for shapes would process the shape part and return one element subconsciously, the part of light/dark would process those things, etc, etc, and effectively your visual cortex would become an unconscious parallel computer. However, there'd have to be a feedback element so that the contents of your visual cortex 'register' would feed back to the image, altering it and triggering a new set of "instructions"; if the entire "circuit" was printed ahead of time, it kind of implies the answer was already available. But if this visual language could be a way of encoding a problem (such as some NP problem) that the brain would parallelize and solve, then... that would kind of be stupendously, face-meltingly awesome.
posted by hincandenza at 4:36 PM on March 29, 2010 [3 favorites]

Our everyday visual perceptions rely upon unfathomably complex computations carried out by tens of billions of neurons across over half our cortex.

The other half is eating a donut.

Threads like this make me twitch.
posted by cortex at 4:39 PM on March 29, 2010 [1 favorite]

ardgedee: It's all fun and games until somebody encodes one of those 3D magic eye pictures into a diagram. Two weeks after viewing one you'll wake up in jail after having stolen all your coworkers' credit cards and used them to ship v!agRa and ROIEX watches to Nigeria.

I was thinking more like the samizdat film of James Incandenza in Infinite Jest, but yeah- it's not a long walk from "co-opt the processing power of the brain" to "use the visual cortex to introduce the same kind of reprogramming currently reserved for entheogenic chemicals". I'm guessing that as powerful as it is, the human brain was not designed with good protection to buffer overrun exploits.

beelzbubba's comment that this would be the product of several lifetimes of neuroscience and computing seems to capture it. But the scientific journey of several lifetimes starts with simple beginnings like these.
posted by hincandenza at 4:40 PM on March 29, 2010

Not to mention, recreating linear circuits seem pointless, since almost no problem you'd design that way would be faster with visual processing compared to the same circuit as a an actual electronic circuit.

Exactly. By the time you've drawn the damn thing, it's solved without even looking at it. It's clever, but the encoding step swamps the process.
posted by GuyZero at 4:42 PM on March 29, 2010

Yeah, Infinite Jest or Snow Crash.
posted by logicpunk at 4:42 PM on March 29, 2010

I do think the general approach- looking at what optical illusions tell us about how we process information visually/separately- is a good tack to take. If there was some way to feed back the reaction to an optical illusion quickly and unconsciously, then you'd really have something.

For now, we'll have to settle for a very, very large donut.
posted by hincandenza at 4:42 PM on March 29, 2010

logicpunk: Yeah, Infinite Jest or Snow Crash.

Oh I'll have to re-read "Snow Crash"; I'd forgotten it for the most part, but having read the Wiki page on the plot, it's pretty much exactly what we're talking about here. Tres cool...
posted by hincandenza at 4:50 PM on March 29, 2010

...generating a perception that amounts to the output of the computation.

Imagine an Escher circuit encoding a mathematical computation in the ink-and-paper equivalent of visual cortex assembly language. But because it's acted upon immediately by our visual subsystems, the viewer never comprehends the parameters of the question and can only perceive the answer. The viewer then has to relay his or her perceptions to the asker in a precise and accurate manner; otherwise the answer to the computational problem will be erroneous. So the Escher circuit has to include methods for encoding the answer into unambiguous symbols and actions.

In the real world, here's how it would most elegantly work; the asker shows the viewer a sheet of paper or similar substrate with the problem encoded, and instead of boxes, shading and dots, the viewer "sees" the answer—42, 7i + 8, 6.02 x 10²³ or what have you.

Now imagine a word problem encoded as an Escher circuit—answer? "Three apples." Now imagine a real-world situation encoded as an Escher circuit, with general methods and specific inputs. Answer? "Pay off the credit cards before you buy the car."

Now imagine a real-world situation where the solution set includes violence. Now imagine that Escher coding of unambiguous actions can exploit a buffer overrun of the conscious control of body functions (can you pat your head and rub your belly at the same time?).

Before you know it, you're killing your best friend because a piece of paper told you to do it. Sounds kind of like magic/schizophrenia to me. It's intriguing; I'd read a Peter Watts or Ted Chiang story about it.
posted by infinitewindow at 5:13 PM on March 29, 2010 [2 favorites]

Didn't the enterprise fight the Borg this way?
posted by sevenyearlurk at 5:34 PM on March 29, 2010

I wouldn't be surprised if this person also thought the spaghetti sort was a practical and efficient algorithm.
posted by edd at 5:41 PM on March 29, 2010

Well, imagine if you could run a spaghetti sort with your mouth.

It's intriguing; I'd read a Peter Watts or Ted Chiang story about it.

I was gonna say: crucifix glitch!
posted by cortex at 5:43 PM on March 29, 2010 [1 favorite]

Didn't the enterprise fight the Borg this way?

No.

They decided it would be genocide and, instead, released Hue with his notions of individuality.
posted by The Great Big Mulp at 5:51 PM on March 29, 2010

I ran a spaghetti sort in my mouth the other day, but it all went to shit in the end.
posted by five fresh fish at 6:58 PM on March 29, 2010 [5 favorites]

Probably everyone's brain is so differently laid out that 'buffer overrun' exploits are completely infeasible, leading at best to seizures.
posted by Pyry at 7:00 PM on March 29, 2010

People have alluded to this point, but I just want to highlight it:

One does not yet immediately and holistically perceive the output.

Without this, you're just replacing one set of symbols with another. And isn't the logic diagram for an XOR just four strokes?
posted by eddydamascene at 8:18 PM on March 29, 2010

Is this like how a complicated logic problem can turn simple when you draw out a Karnaugh map?
posted by miyabo at 8:34 PM on March 29, 2010 [2 favorites]

If only there were a way to use the perception of the image in conjunction with a stretched or skewed text such that it becomes readable if you are perceiving in the correct way.

Alternatively, the perception of flow through the diagram could perhaps be used to force a branch if you design forked shapes where the path ahead only makes sense in one perceptual mode and not the other.

In any case, once the building blocks are developed, eye tracking software in combination with this technique (and perhaps some thrown in erotic imagery to compel focus) could speed up the technique dramatically.

Still, I thought instead of the Escher like flow, he'd be exploiting our pattern perception abilities in conjunction with eye tracking to make us focus in on a specific area of an image, with feedback that transitions through zooming, focus in on the next patterned area, and repeat. I'm not sure how the patterns would express the inputs, but I'm confident someone could figure it out.
posted by BrotherCaine at 9:21 PM on March 29, 2010 [1 favorite]

Maybe a grid of sets, with the starting assumptions presented as a group of images beforehand? Shape and color? This ties in oddly well with last weeks post about using gamers to solve real world problems, except not as a brainstorming session, but more as a computational engine.
posted by BrotherCaine at 9:26 PM on March 29, 2010

And now that I read the Wikipedia entry on Karnaugh maps, I realize I should have just said "what miyabo said".
posted by BrotherCaine at 9:38 PM on March 29, 2010

This is some serious crack pottery.
posted by delmoi at 11:40 PM on March 29, 2010

Is this like how a complicated logic problem can turn simple when you draw out a Karnaugh map?

Aren't Karnaugh maps limited to 4 variables? I suppose you could do 6 if you extended into 3 dimensions, but I think something that requires n/2 dimensions is a little impractical.
posted by delmoi at 11:43 PM on March 29, 2010

I thought from reading the wikipedia entry that Karnaugh maps could be extended to any number of variables, with the number of cells in the map being equal to 2ⁿ where n=number of variables. So 5 variables equals a 4x8 grid, 6 variables= an 8x8 grid, etc...
posted by BrotherCaine at 12:27 AM on March 30, 2010

BrotherCaine: I thought from reading the wikipedia entry that Karnaugh maps could be extended to any number of variables, with the number of cells in the map being equal to 2ⁿ where n=number of variables. So 5 variables equals a 4x8 grid, 6 variables= an 8x8 grid, etc...

Delmoi has this one. The problem is in adjacency - in a 4x4, you have two dimensions with two variables each - which works fine.
    | 00 | 01 | 11 | 10
00 |     |     |     |
01 |     |     |     |
11 |     |     |     |
10 |     |     |     |

As you go from one row/column to the next, only one bit changes at a time - i.e. 01 -> 11 -> 10. And here, in both row and column, 01 is next to 11 is next to 10.

The problem is if you have eight possibilities (three bits), it's impossible to arrange them all such that every adjacent combination is also physically adjacent (which is the magic of the Karnaugh). 010 is "adjacent" to 011, 000 and 110, yet a column only has two sides. As Delmoi said, while Karnaugh maps are brilliant for simple circuits, they're impractical anywhere above ~4-6 variables.
posted by Arandia at 12:57 AM on March 30, 2010 [1 favorite]

Argh now whenever I look at anything my visual processing tries to flip/flop it.. including this sentence and the textbox!

Really cool idea though.
posted by Joe Chip at 1:16 AM on March 30, 2010

Anyway - brilliant post, scalefree, thanks.

While a fascinating idea, I think the guy in the post still has a few problems to deal with before there'd be something useful here. For one, to accurately arrive at an "answer" in this scheme you have to hold in your mind all the various configurations, and then trace down the wire path for all branches simultaneously. Holding many things in perfect memory is, unfortunately, another one of those areas where computers beat humans handily.

Perhaps if there was a way to encode the logic in pattern identification it would work, rather than geometric interpretation (where humans are decent but not great)? Sort of like those illusions where you're not sure what exactly is being shown. I admit I have no clue as to how you'd expand on this - maybe dot densities forming shapes or some such??
posted by Arandia at 1:17 AM on March 30, 2010

Oh I get it. The guy spewed a lot of text before explaining how these things worked, and I just looked at the pictures. I kinda go it, but the 'not' gates didn't work, and I didn't realize he was using 'looking away/looking forward' as 0 and 1.

But the problem for me is that the not gates don't really work, either type. I see 'em, and it still looks to me like 'wire' is facing the same way as the wire before the thing.

Maybe with shading or something. This guy is actually a terrible artist, which is Ironic for someone trying to do what he's doing.
posted by delmoi at 1:36 AM on March 30, 2010

Delmoi - I find that the latter ones, with the cones, work better (figure 6 I think). Not sure why he didn't just start with those.

BrotherCaine - reading your earlier post, I just realized I mostly echoed what you had said. And while not strictly circuitry, I guess the "branch that only makes sense in one direction" could be an if-type statement? But then we get into flow-control and a whole new ballgame (plus these diagrams have crazy cross-talk - at least for me).
posted by Arandia at 1:59 AM on March 30, 2010

Now imagine a word problem encoded as an Escher circuit—answer? "Three apples." Now imagine a real-world situation encoded as an Escher circuit, with general methods and specific inputs. Answer? "Pay off the credit cards before you buy the car."

Now imagine a real-world situation where the solution set includes violence. Now imagine that Escher coding of unambiguous actions can exploit a buffer overrun of the conscious control of body functions (can you pat your head and rub your belly at the same time?).

Why is this any different than the scenario:

Now imagine a verbal utterance encoded as a set of lines and spaces - Answer? You hear in your head the words as if you spoke them!!! Spooky!

Our brains already decode symbolic information and send it to any variety of processing units. Watching a turkey blink its eye or ruffle a feather triggers an entire agent-action-goal-movement subroutine in the brain automatically, one that doesn't get interrogated at all by a cooked turkey on a platter. Nothing is spooky about the fact that we can't "turn off" the agent subroutine.
posted by odinsdream at 4:37 AM on March 30, 2010 [1 favorite]

I just remembered a trick to get your visual system to perform a parallel XOR. If you take one of those "Spot the Difference" games (for instance), line the two images up side by side, and then cross your eyes so that the images overlap, any areas in the image which don't match will pop out. Not that this seems more efficient than having a computer compare each image pixel by pixel, but still interesting...
posted by logicpunk at 6:54 AM on March 30, 2010 [1 favorite]

Can't Chuck already do this?
posted by Grundlebug at 8:37 AM on March 30, 2010

delmoi : This is some serious crack pottery.

I agree with you in that this seems almost pointless in its crudeness - Although I thought his AND and OR gates worked amazingly well, I have to agree with those criticizing NOT (and that XOR, probably the single most important logic gate in digital computing, just looks like a tangled mess). But his AND and OR do illustrate his intent nicely - You don't need to know their truth-tables to see, at a glance, the output of a given pair of inputs. Arrange the two inputs and bam, the bottom line magically faces the "right" way.

As for a useful purpose - Just pretend for a minute that, with years more work on this, he comes up with a truly generalizable (and easy to interpret) means of encoding fairly complex problems. If nothing else, we could use this to answer the holy grail of consciousness-vs-computation questions, whether or not humans have a way to "cheat" at NP-complete problems. Just with what he has now, he could already encode the 3-SAT problem (though that NOT really ruins it, for now). If we can, in P-time solve for a consistent set of inputs, then we have a way to cheat¹, some "quantum² spark" behind our intelligence. If not, then we just have a massively parallel but ultimately "only" Turing-complete processor on our shoulders.


^{1: Or, alternately, that P == NP, which while not so philosophically mind-blowing, would mean as much to CS as the Bohr model did for chemisty
2: Or "divine", for those so inclined}
posted by pla at 9:37 AM on March 30, 2010

I've had this thought. Really cool to see it attempted.

A variation: a shape-shifting surface you walk on that interprets your steps as general computation. Treadmill computing farms.

Of course, the channel may never be big enough to be useful.
posted by grobstein at 10:07 AM on March 30, 2010

Pfft - all I had to do was drop acid, and my circuitry spawned a whole virtual reality system from emergent properties!
posted by symbioid at 10:18 AM on March 30, 2010

I'm glad people enjoyed it. I have no more idea than anyone else how useful or practical it may end up becoming, but it was just so unusual I had to share it. That plus the Snow Crash bit. :)
posted by scalefree at 10:40 AM on March 30, 2010

That was really interesting, albeit not quite practical yet.
posted by DU at 11:46 AM on March 30, 2010

Sorry for the Karnaugh map digression, but maybe the third dimension is implied here?
posted by BrotherCaine at 2:30 PM on March 30, 2010

Oh, should have read before posting, now I see it.
posted by BrotherCaine at 2:32 PM on March 30, 2010