Word Lens

December 16, 2010 7:37 PM

Word Lens REPLACES text viewable in your iPhone camera with its translation, in real time, with formatting intact. Be sure to watch the demo video. Pretty much straight up magic. The app itself is free, but Spanish->English or English->Spanish dictionaries are $5 each, via in-app purchase. It's been a while since my jaw has dropped like this from any piece of software.
posted by 3rdparty (95 comments total)

Incredibly cool, I was impressed by the way the text formatting remains the same. I would definitely be curious to see how it works in a real-world environment.
posted by Solon and Thanks at 7:41 PM on December 16, 2010

The free app includes a demo which will either flip text backwards, or remove it completely - just to show you how it works in realtime. wow.
posted by 3rdparty at 7:43 PM on December 16, 2010 [1 favorite]

Now I am regretting getting my Samsung Mythic instead of an iPhone.
posted by Samizdata at 7:45 PM on December 16, 2010

Wow.
posted by TheShadowKnows at 7:46 PM on December 16, 2010

Smells like a self-link.
posted by ZenMasterThis at 7:46 PM on December 16, 2010 [1 favorite]

Add a text to speech option, and you've got a universal translator. Add phasors to that, and you have a sudden need for dilithium crystal.
posted by Panjandrum at 7:47 PM on December 16, 2010 [13 favorites]

Stunning app. Congrats to the development team that made this happen. Best use of augmented reality I've seen yet.
posted by fake at 7:48 PM on December 16, 2010

@ZenMasterThis > nope, just an incredibly cool app and example of technology. I actually came across it from this tweet
posted by 3rdparty at 7:50 PM on December 16, 2010

Once you can attach these things to your wrist it will be The Future.
posted by The Whelk at 7:51 PM on December 16, 2010

Incredibly cool, I was impressed by the way the text formatting remains the same.

Well, I'm sure they carefully picked their card styles/phrases to give the best demo results. Still, damn impressive. Now all I need is the money to travel somewhere where this would be useful...
posted by sbutler at 7:53 PM on December 16, 2010

This opens up some kick ass ARG possibilities.
posted by ignignokt at 7:55 PM on December 16, 2010 [2 favorites]

Aber es funktioniert nur mit Spanisch?
posted by Flashman at 7:56 PM on December 16, 2010 [2 favorites]

Insanely cool app! My husband just bought it and we tried it out with the homepage of El Pais with mixed, but readable, results. One problem: it tried to translate the word "Assange" in one of the headlines.
posted by Asparagirl at 7:59 PM on December 16, 2010 [1 favorite]

Any testimonials from the wild?
posted by goodnewsfortheinsane at 7:59 PM on December 16, 2010

You don't even have to leave the U.S. to have World Lens be helpful! there are huge numbers if people who only speak Spanish where I live and for that matter in any agriculturesl place on the West Coast.
posted by Katjusa Roquette at 8:00 PM on December 16, 2010

I haven't tried the translation, but playing with the text detection (in the reversal demo) is pretty impressive... it does seem to need pretty good contrast, and you can certainly confuse it if you try, but still I'm well impressed...
posted by russm at 8:02 PM on December 16, 2010

I'm sure this will make some high school kids spanish homework go much more quickly...
posted by blaneyphoto at 8:02 PM on December 16, 2010 [4 favorites]

I'm curious as to how big the text needs to be.
posted by BurN_ at 8:04 PM on December 16, 2010

Let me know when it goes verbal, so I know what the Latina lady on the bus is screaming at me.

She has it in for me, I swear. LADY, I'm just STANDIN' HERE.
posted by Capt. Renault at 8:05 PM on December 16, 2010 [2 favorites]

And they didn't call it Babelfish-Eye Lens?
posted by goodnewsfortheinsane at 8:10 PM on December 16, 2010 [9 favorites]

I just tried it (the free, word reversing functionality, not the translation). Works pretty well with only a handful of words in the field of view, as in the video. But can not at all keep up with a page full of text.

Still though, pretty cool idea...
posted by jpdoane at 8:11 PM on December 16, 2010

Seconding the Wow and adding a holy shit that's cool.
posted by eyeballkid at 8:13 PM on December 16, 2010

Trying it out now. It works surprisingly well!

But it's really fun to make the detection algorithm go wacky. I pointed it at a magazine cover, and the girl's dress on the cover has patterns that reveal some crazy hidden messages like 'MANHOOD' and 'BIO WIMP'. I'm not even joking. I'd share but I have no idea how to save these pictures. I feel like this app might be a real-life version of the glasses in They Live.
posted by naju at 8:15 PM on December 16, 2010 [41 favorites]

I'd share but I have no idea how to save these pictures.

You can take a screenshot at any time in an iPhone app by pressing the Home button and the Power button simultaneously. It's a little awkward, but fun.
posted by verb at 8:18 PM on December 16, 2010 [3 favorites]

Thirding the wow and adding a "that's amazing!"
posted by stp123 at 8:18 PM on December 16, 2010

Holy crap. Welcome to the 21st century. I'd gladly take this over living in an anarchistic microcommune, any day of the week.
posted by mark242 at 8:28 PM on December 16, 2010 [4 favorites]

That is how you do a demo video. Show, not tell. Fantastic.
posted by Nelson at 8:29 PM on December 16, 2010 [3 favorites]

One day I will miss the zany culinary adventures I have had at Asian grocery stores with products that completely lack English. Thankfully, that day is not yet here my sweet, sweet, cuttlefish balls.
posted by munchingzombie at 8:30 PM on December 16, 2010 [4 favorites]

Hoax.
posted by charlie don't surf at 8:30 PM on December 16, 2010

I like that the demo didn't use contrived examples that translated perfectly. Sad that honesty would be so shocking, though perhaps they (probably correctly) assumed that a deceptive sales pitch would get them a lot of bad reviews.
posted by jedicus at 8:31 PM on December 16, 2010 [1 favorite]

Jibbigo is an app that does something similar but for spoken language.

It currently has versions for both Spanish/English as well as Japanese/English translation.

(Man, the future rocks!)
posted by Jaybo at 8:32 PM on December 16, 2010 [4 favorites]

Trying out the Spanish->English. It translates individual words, but not sentences, something I imagine would be far more complex. Trying it in a Spanish paperback produces useless gibberish.

I can easily see using this just like the example video to translate signs or other brief snippets of text. It also tries to translate English to English with ridiculous results. It needs a lot of work, but it's still one of the neatest mobile apps I've seen if only for the demo filters.
posted by eyeballkid at 8:35 PM on December 16, 2010

OK, here's the They Live app in action (remember, it's in Reverse Words mode, so hidden messages will appear backwards):

IMBIBE
BIMBO
MOJITO
BABOON
MINION, WIDOWER
posted by naju at 8:49 PM on December 16, 2010 [17 favorites]

I'm using it right now. It can't restructure the grammar (and thus perfectly duplicates my spanish speaking abilities) and it chokes on large chunks of text, but it's as mind-blowing as it is in the video.

Man, I love the future.
posted by unsupervised at 8:54 PM on December 16, 2010

Hoax.
I can vouch for its legitimacy, though I am not involved. FYI price will go to $10 at the end of the month (many people telling the developers it is too cheap, including me). So if you would use this, you can save a few $$.
posted by anigbrowl at 8:54 PM on December 16, 2010

OK, here's the They Live app in action

I don't want an iPhone. Does this mean we have to fight in an alley?
posted by kid ichorous at 8:54 PM on December 16, 2010 [13 favorites]

It also can't force me to preview my comments.
posted by unsupervised at 8:54 PM on December 16, 2010

Here's a screenshot from the only Spanish paperback I own.
posted by eyeballkid at 8:59 PM on December 16, 2010

Bring me Japanese, and I bring the money! :)
posted by lundman at 9:03 PM on December 16, 2010 [5 favorites]

So excited for my new Facebook Pro.
posted by naju at 9:11 PM on December 16, 2010

Thai please!!!
posted by pompomtom at 9:12 PM on December 16, 2010

naju, that is the greatest thing ever. Your They Live reference has me dying over here.
posted by King Bee at 9:13 PM on December 16, 2010

The image recognition is quite impressive, but the translation is crap. Even in their demo, where they are using simple phrases, it's pretty obvious that it's basically just word for word dictionary lookups. Handy to get the gist of something, but way behind the state of the art for machine translation.
posted by zachlipton at 9:15 PM on December 16, 2010

There's a little robot called "Moore" who's been very slowly climbing Mt Everest. It's hard to say when it started climbing, but by 1989 the robot had only made it 7/1000ths of an inch above sea level

Strangely though, the robot climbed faster as time went on, each 18 months doubling it's previous height. By 1996 it had made it over 1/10th an inch above sea level

By 2007 it finally made it past the first foot

By 2011 it was seven whole feet up the mountain

2017: 113 feet
2020: 453 feet
...
2029: 29,000 feet, the summit

-------

We've just about made it to seven feet, and people are starting to say "wow, look at the view". Well...
posted by crayz at 9:15 PM on December 16, 2010 [37 favorites]

Pleco does this for Chinese, although it only gives translations for individual words. Works fairly well!
posted by Aiwen at 9:23 PM on December 16, 2010 [3 favorites]

It seems like they have carefully avoided calling it a translator anywhere in their ad copy and are instead pitching it as a dictionary.
posted by Rhomboid at 9:25 PM on December 16, 2010

While English to Spanish and vice-versa is quite useless for me, it'll be awesome if they can have for English <> {Zh-CN | Zh-HK }

As in, not that it isn't awesome already with the demo, but if they want me to plonk down further cash, I need a Chinese - English dictionary.
posted by the cydonian at 9:27 PM on December 16, 2010

Pleco does this for Chinese, although it only gives translations for individual words. Works fairly well!

Metafilter delivers again, this time in -15 minutes. Thanks Aiwen! :)
posted by the cydonian at 9:29 PM on December 16, 2010

Wow. That's super cool. Totally gonna show it off to my coworkers tomorrow... gonna make that windows 7 phone jerk cry. (I work with nerds.)
posted by ph00dz at 9:55 PM on December 16, 2010

It doesn't do it in realtime, and requires an internet connection to actually do the OCR, but this app for Android does do Japanese->English dictionary lookup from the camera.
WWWJDIC
posted by netd at 10:00 PM on December 16, 2010

I'd gladly take this over living in an anarchistic microcommune, any day of the week.

Now at the cheap price of a 5 billion person underclass!
posted by threeants at 10:04 PM on December 16, 2010 [14 favorites]

I NOTICE THAT IT'S ONLY CAPABLE OF OUTPUTTING IN ALL CAPS. THIS IS BAD NEWS, BECAUSE I'M GOING TO ASSUME THAT ALL FOREIGN SIGNS ARE SHOUTING AT ME, AND THIS WILL INCREASE MY STRESS LEVELS, POTENTIALLY CAUSING LAPSES OF JUDGEMENT.
posted by The Discredited Ape at 11:23 PM on December 16, 2010 [5 favorites]

Looks like the site is overloaded; here's a direct link to the video
posted by Deathalicious at 12:40 AM on December 17, 2010

Naju -- I pointed it at a magazine cover, and the girl's dress on the cover has patterns that reveal some crazy hidden messages like 'MANHOOD' and 'BIO WIMP'. I'm not even joking. I'd share but I have no idea how to save these pictures.

Please figure out how to post the pics'. I really really really want to see that.
Amazing app'.
posted by Lukenlogs at 12:43 AM on December 17, 2010

Can you imagine how great this will be once it works with Asian languages? Tokyo would be so much less confusing!
posted by DanCall at 1:09 AM on December 17, 2010

Weird, all it did for me was translate "questvisual" into "Forbidden 403".
posted by maxwelton at 1:24 AM on December 17, 2010

That is extremely cool. Just IM'd a friend about it and it turns out that friend's brother actually worked on this; I hear it was a hard core dev project, two years of fancy assembler code. (Apparently).

We are now officially living in the future!
posted by Hartham's Hugging Robots at 1:40 AM on December 17, 2010

I've just tried it (& shown everyone in my office) & I can confirm that it's very, very good.

It doesn't recognize names and places - it tries to translate them too - but apart from that...
posted by DanCall at 1:46 AM on December 17, 2010

(hi, I'm John DeWeese, one of the creators)
Thanks for all the linkage and excitement, this is quite a wave we've found ourselves on! We're gathering reactions here, and have seen some stuff pop up pretty often so far:

naju - so cool! i don't even want to call this a bug. thanks for posting :)

as some have pointed out, we're translating word-to-word -- well, small phrases like 'por favor' work too, because they're so common. some of this will get better as time goes by and we see where it fails more clearly. however, it's a much harder problem to respect grammatical transformations and local idioms. the goal for us, then, is just to get you most of the way there!

katjusa - we're in california, and it was really fun walking around to mexican restaurants when we just got it working on the phone, and seeing it /actually work/. whoa.

BurN - the bigger the text, maybe the better. but one nice thing about it is that if it looks weird one way, you can move it around until it get solid. that's what i do, at least. it really helps to have an iPhone 4, that camera is really nice (we always want better, though)

jedicus - thank you, it's hard to set expectations and please everyone, but we're getting this out there because we think it's /useful/ pretty often. :)

Definitely check out Pleco for Chinese translation -- we are impressed with the quality of their learning material, and their real-time OCR is pretty nice (i tried it on a chinese wordbook i have, and it worked pretty well)

ok, i really have to get some sleep, been an amazing day :)
posted by johndeweese at 2:18 AM on December 17, 2010 [41 favorites]

Wow, this is awesome.
posted by Blazecock Pileon at 3:01 AM on December 17, 2010

This is astonishing.
posted by ~ at 4:51 AM on December 17, 2010

Hoping they'll bring out an Android version soon!
posted by leslies at 5:53 AM on December 17, 2010 [1 favorite]

the video suggests that it's not just a translation app but also a very capable image editor that instantly replaces text with blank space from the surrounding areas (think photoshop's stamp tool). that being done live and in real-time is very impressive and the main reason I was convinced this just had to be utter bullshit. I don't have my iphone on me at the moment and can't judge the app for myself but I trust you all would have already screamed bloody murder were it in fact a hoax. which is why I would like to add a resounding whoa.
posted by krautland at 6:39 AM on December 17, 2010

Fun thing to do: point the camera at your keyboard and watch it erase batches of keys at a time (works well with those metal chiclet apple keyboards).
posted by fungible at 6:40 AM on December 17, 2010

Ran some Spanish-to-English tests this morning. Seems to work best on short phrases, rather than long blocks of text. Also seems to do better with sans serif fonts. Still: totally amazing.
posted by Ike_Arumba at 7:13 AM on December 17, 2010

Got it today. It's pretty jittery and doesn't work nearly as well horizontally as it does vertically.

Regardless, it's very cool, handy and accurate enough. My Spanish is pretty solid but it's quite handy to fill in some blanks in my vocabulary (living in Spain allows for a lot testing).
posted by slimepuppy at 7:30 AM on December 17, 2010

This is a great consolation prize for the lack of jet packs and flying cars!
posted by Scoo at 7:59 AM on December 17, 2010 [2 favorites]

Hats off John. Fun with bugs aside, it really did give my brain a shock to see this thing in operation. Spanish-to-English dictionary bought. And if it's true you've been toiling away at the assembler level to make this, then you deserve a few beers on top of that!
posted by naju at 8:13 AM on December 17, 2010

Once you can attach these things to your wrist it will be The Future.

We seem to be getting there. Now, if the future would just stop for a moment to become the present, then we'd be all set!
posted by CaffeineFree at 8:36 AM on December 17, 2010

This seems relevant: Speakularity.
posted by psyche7 at 10:05 AM on December 17, 2010

I spent the night in my kitchen trying it against appliances and cereal boxes. I'm in awe. I downloaded both dictionaries because I'm sure I'll find a ton of use for it here in the San Fernando Valley.

Ten years from now, when this tech is translating all languages instantly and making remedial attempts at parsing syntax while rendering the results in the original font, I'll make sure I have it installed as a windshield HUD option on my flying car, that is if it's not part of the standard package.

Seriously, bravo johndeweese. This is amazing.
posted by eyeballkid at 10:14 AM on December 17, 2010

blaneyphoto: I'm sure this will make some high school kids spanish homework go much more quickly...

If they're smart (er, lazy), they've been using Google's translation feature for a while. Now, if they're sly, they can translate text in class, and be even lazier.

But because it only translates the words and not the phrase, it could actually help kids who flub the vocabulary memorization, but do well with the syntax and grammar.

Any and all ways: Awesome. And welcome to MetaFilter, John!
posted by filthy light thief at 10:30 AM on December 17, 2010

I'm impressed. It's not perfect, but for a 1.0 release, it's stunning. SPANISH SHALL NOT KEEP ITS SECRETS FROM ME ANY LONGER!
posted by grubi at 11:07 AM on December 17, 2010

I've been resistant to the iPhone and its friends due to crappy service out here, but...this is becoming very difficult to resist.
posted by maxwelton at 11:10 AM on December 17, 2010

I'm not a tech geek, really. And I have no need for this whatsoever. Yet, for some reason, I want it so badly. (Is this what it's like all the time for all of you with the freshest of doodads and gewgaws? Well, I understand. Now, I finally understand.)
posted by .kobayashi. at 11:20 AM on December 17, 2010 [1 favorite]

I got it and bought both dictionaries despite there being very little chance of me seeing anything written in spanish in my day to day life. It's just so awesome.

Clothing optional on this beach.
posted by doublehappy at 11:43 AM on December 17, 2010

I learned all the Spanish that I know on Sesame Street.

Please please please do Russian next! Or at least soon!
posted by newdaddy at 12:02 PM on December 17, 2010

Please please please do Russian next! Or at least soon!

I want Old English! And Gothic! And Talossan! And Quenya!

...

"Russian." Ha. Sounds made-up, if you ask me.
posted by grubi at 12:06 PM on December 17, 2010

Ok wow I bought it and tried the demo mode. I showed it the cover of the book on my desk, "The Designer's Guide to VHDL". Interestingly, it reversed everything but the VHDL - maybe because it's all acaps or maybe because it's not recognized as a word(?). It's close enough to be called 'real-time'. No fooling.
posted by newdaddy at 12:13 PM on December 17, 2010

Once you can attach these things to your wrist it will be The Future.

It might not be all that long ...
posted by Twang at 12:57 PM on December 17, 2010

This looks pretty cool, thanks!

Has anyone tried this on tattoos yet?
posted by ODiV at 1:53 PM on December 17, 2010

Re the They Live connection - my first comment on seeing this was

My only question: if I purchase the "Skull-faced aliens' lingo -> English" module and point it at some billboards, will it read "Obey" & "Consume"? If I point it at some paper money, will it say "This is your God"?
posted by Philofacts at 2:06 PM on December 17, 2010

According to this Quora response, an English/Russian 'dictionary' is in the works...
posted by 3rdparty at 2:06 PM on December 17, 2010

(I did buy the Spanish modules, patiently awaiting French & German.)
posted by Philofacts at 2:07 PM on December 17, 2010

Next week comes the upgraded version that you point at someone and it types out what they're thinking. Default fonts are Comic Sans and Papyrus.
posted by asfuller at 2:37 PM on December 17, 2010 [1 favorite]

Here's another official video; one of the developers explaining and demo'ing it. Nicely done.
posted by 3rdparty at 4:38 PM on December 17, 2010

Please please hook this OCR tech up to a text-to-speech library. I'll kickstarter the fuck out of that idea for a decently fast implementation. There's a couple of apps taht try to do this but they're painfully slow and don't really aim to be purely about saying the words out lou.
posted by Space Coyote at 6:38 PM on December 17, 2010

Sorry I typed that too quickly out of excitement at seeing the developer in this thread.

Ahem.
posted by Space Coyote at 6:41 PM on December 17, 2010

Japanese, please!
posted by zardoz at 8:58 PM on December 17, 2010 [1 favorite]

I want a Futurama alien language module.
posted by Scoo at 9:25 PM on December 17, 2010

Is this correct that it is only for English/Spanish? I could seriously be persuaded to go for a new iPhone, if they had German, French, and Dutch. Better still, port the app to Android, as iOS has become boring.
posted by Goofyy at 10:15 PM on December 17, 2010

Eagerly awaiting Adblock Plus for Reality.
posted by benzenedream at 12:58 AM on December 20, 2010 [3 favorites]

slime puppy: works great in landscape mode... once you click the unlock button! try it out!
grubi: lol
asfuller: make that tomorrow. always tomorrow.
Space Coyote: yes sir or madame
Goofyy: workin' on it!

meanwhile, i sure wish i could have a helpful, rational conversation with those who are rating Word Lens on the app store at this time. alas!

cheers, etc :)
posted by johndeweese at 7:09 PM on December 20, 2010

Just to clarify: for Japanese, they shouldn't bother with full sentence syntax. A Rikai-chan-like kanji dictionary is what we need. So you can look at kanji and get the English (and hiragana) for the individual characters as well as multiple.
posted by zardoz at 7:57 PM on December 20, 2010

Amazing
posted by jossan111 at 12:03 PM on December 25, 2010