How (not) to write an online-dating message, based on a sample of 500,000 "first contact" messages. [more inside]
posted by Kadin2048 on Sep 14, 2009 - 79 comments

The National Security Agency is building a data center in San Antonio that’s the size of the Alamodome. Microsoft has opened an 11-acre data center a few miles away. Coincidence? Not according to author James Bamford, who probably knows more about the NSA than any outsider. Bamford's new book reports that the biggest U.S. spy agency wanted assurances that Microsoft would be in San Antonio before it moved ahead with the Texas Cryptology Center. Bamford notes that under current law, the NSA could legally tap into Microsoft’s data without a court order. Whatever you do, don't take pictures of it the spy building unless you want to be taken in for questioning.
posted by up in the old hotel on Dec 8, 2008 - 42 comments

Worried about social-network data mining? Facebook hires Ted Ullyot, former right-hand man to former Attorney General Alberto Gonzales, as its general counsel. Tapping Ullyot, who worked on the infamous torture memo and other illustrious projects, is a sign that the burgeoning Scrabble platform "is a little more grown-up," says Facebook public-policy VP Elliot Schrage.
posted by digaman on Sep 30, 2008 - 40 comments

Pluribo is a way-cool Firefox extension that automagically summarises Amazon product reviews.
posted by matthewr on Jul 1, 2008 - 25 comments

The idea was that a spike in, say, falafel sales, combined with other data, would lead to Iranian secret agents in the south San Francisco-San Jose area. I've read this article twice now because I was laughing too hard the first time. If I were more paranoid I might actually seriously ask what sort of data mining the FBI is doing, but... falafel sales! via. [more inside]
posted by tarheelcoxn on Nov 6, 2007 - 75 comments

Arguing Against Datamining MySpace in search of Pedophiles. In certain circles, MySpace has become the villain de jour for all sorts of debauchery (threatening the President, phishing , dismembered women , etc.), as well as being fertile hunting grounds for the pedophile. Given the huge size of MySpace, reported as 100 million accounts (although estimates of active accounts are far lower, at approximately 43 million ), and an hypothetical and absurdly low natural incidence of pedophiles and pedarasts (let's say just 1%), one could assume that there could be as many as 430,000 to 1,000,000 of them out there. Wired contributor and reformed hacker (Kevin Poulson) has developed a script to weed out the bad seeds [via]. His script was effective, although it took several months of sifting and refining, as well as numerous false positives - 744 registered sex offenders, 497 with convictions for crimes against children. While such an experiment has merit, how much time, resources, and law enforcement manpower will be wasted chasing down the ""high-cost "false positives", and what will be neglected and sacrificed for that effort?
posted by rzklkng on Oct 16, 2006 - 38 comments

AOL releases 3-months of queries from 500k users. AOL, either fairly or unfairly, is sometimes considered the internet with training wheels. So while parsing this data, keep that in mind. Some of these queries seem like spam email subjects, don't they? Don't forget, this is the same demographic that brought you the September that didn't end. AOL tried to retract the data, but it's of no use - it's out there, on the web.
posted by rzklkng on Aug 7, 2006 - 89 comments

The Secret History of Able Danger The WP may have have the goods on Able Danger. The Pentagon and Intel officials are mum on the data mining project because it could have been illegal.
posted by raaka on Sep 29, 2005 - 16 comments

Exploring enron -- A breathtaking web of conspiratorial email messages. How often did Jeff Skilling email Ken Lay? How often were those emails about company business? Internal alliances? The company's allegiance? The California energy crisis? Who else was talking about it? Who wasn't? Temptingly complete with software download and MySQL tables for your own tinfoil hat explorations.
posted by boo_radley on Jun 13, 2005 - 10 comments

Docusearch settles claim for 75K with family whose daughter was killed by a stalker who purchased her personal information from them -- a killer whose intentions were described on a Googleable website. The NH Supreme Court determined last year that Docusearch, the company who sold Amy Boyer's work address and SSN to her killer could be held liable for her death, even though some of that information was publicly available. An "Amy Boyer's Law" intended to increase privacy by restricting the display, sale or use of SSNs received negative reviews by privacy organizations and ultimately was removed from an appropriations bill. In a statement, Amy's parents encourage others to use the Internet to keep track of who may be keeping track of their kids. "If only we had typed our daughter's name into any search engine, the Amy Boyer Web site that was posted by her killer would have come up, and we could have called the police...This may never have happened."
posted by jessamyn on Mar 11, 2004 - 6 comments

That U.S. intelligence agencies confuse terrorists with children on passenger jets is a reminder that data collection is easy, but data analysis is hard. That must be why the six-year-old daughter of one of Boing Boing's co-founders is on the CAPPS list as a security risk. All this is also a reminder that we need privacy safeguards for these data mining programs.
posted by homunculus on Jan 11, 2004 - 34 comments

The Patriots didn't win; Britney did. TiVo analyzed their viewers behavior during the Superbowl and they came up with some pretty interesting results. How soon till TV programming adapts to viewer behavior?
posted by costas on Feb 5, 2002 - 36 comments