Bots, spiders and crawlers, oh my!

March 16, 2006 7:08 AM

"Imagine this: a digital butler that roams the Internet, intuitively knowing your likes and dislikes, retrieving perfect strands of news and information that you never would have discovered through old-fashioned surfing." There's RumorBot and the Iliad fetchbot (perfect for bot newbies), or you can try your hand at writing your own in php using this as a tutorial, or if you prefer, C++ or Perl.
posted by sluglicker (15 comments total)

Ithought this was the purpose of Metafilter?
posted by Yer-Ol-Pal at 7:14 AM on March 16, 2006

Y-O-P beat me to it. All of you guys are my content-aggregation bots.
posted by Faint of Butt at 7:17 AM on March 16, 2006

@Yer-Ol-Pal
Sure, if you only want to leech. If you want to contribute, try a bot!

Correction: the Perl link should be Perl.
posted by sluglicker at 7:23 AM on March 16, 2006

Haven't personalized content-aggregation tools such as this one been the Next Big Thing for the last, like, 15 years?
posted by Brian James at 7:44 AM on March 16, 2006

Speaking as a php, c++, etc. illiterate, it seems that this page might be a little more helpful as a tutorial for building your own bot. It has lots of other php beginner tutorials as well.
posted by mekanic at 7:46 AM on March 16, 2006

There is also The Ultra Gleeper
posted by mikepop at 7:55 AM on March 16, 2006

I begin to imagine a digital butler, thinking back to 1997 or so when such concepts were aired regularly, but then I start getting angry.
posted by killdevil at 8:21 AM on March 16, 2006

The main link, the one that the quote is taken from, reads as though it was written in 1996, in a breathless issue of Wired ("Push! The next big thing!"). There's a reason that nothing has really happened with this idea: nobody knows quite how to do it yet. At least, nobody knows what to build that will be:

• broadly useful
• simple to instruct and direct
• more than just a script that periodically performs searches that the user tells it to

How do you make this "butler" universally available to the user? From where does it execute its actions? On the user's home computer? That's not very useful, not unless the user is home all the time. On the user's phone service, and talking to them through their mobile? This idea is not ready for prime time yet. Somebody will make a lot of money if it does come true, though.
posted by dammitjim at 8:26 AM on March 16, 2006

Just ask Jeeves
posted by ernie at 8:45 AM on March 16, 2006

Didn't I have something just like this back in like 1997 or 1998? And didn't it completely NOT work? And wasn't it pretty well decided that web viewing is an ACTIVE activity, not a passive one?

Am I the only one that thinks push is unnecessary?

I just fail to see much difference between having a story from CNN "pushed" to me so I click the link to view it, or viewing the link directly because I navigated to CNN.com.

Agents have been highly overrated for a very long time.
posted by Ynoxas at 9:46 AM on March 16, 2006

does anyone else remember knowledge navigator?
posted by Dr. Twist at 12:05 PM on March 16, 2006

Imagine a digital paperclip, appearing as you compose a document to give you help and advice...
posted by Artw at 1:33 PM on March 16, 2006

ernie: Jeeves is gone.
posted by reklaw at 2:27 PM on March 16, 2006

Haven't personalized content-aggregation tools such as this one been the Next Big Thing for the last, like, 15 years?

Yes. And, if it ever comes to pass, it'll be like all those people who listen religiously to Alan Jones, John Laws, or Bill O'Reilly - a self-selected audience who only see what they want to see, who never have their beliefs challenged or opinions confronted (except to foment outrage), and who believe everything they're told.

Imagine a million individual LGFs - left wing, right wing, and every shade in between - consisting of one person plus their "intelligent agent" / "digital butler". Just as the porn industry can provide for your every fantasy no matter how twisted, so will the content industry come to supply information/opinion tailor-made for every prejudice and belief.

The only winners will be those who provide this content. And those who choose not to play.
posted by Pinback at 12:17 AM on March 17, 2006

With the exception of dammitjim, did any of you actually read the linked texts? Your comments have nothing to do with the content of this post. And Pinback: WTF are you talking about? Bah!...Perl before swine.
posted by sluglicker at 8:48 AM on March 17, 2006