Yotta vote against this

November 1, 2009 10:13 AM

Roughly equal to about a septillion (1,000,000,000,000,000,000,000,000) pages of text, numbers beyond Yottabytes haven't yet been named. Once vacuumed up and stored in these near-infinite "libraries," the data are then analyzed by powerful infoweapons, supercomputers running complex algorithmic programs, to determine who among us may be—or may one day become—a terrorist.

James Bamford talks to Nathan Thrall about the politics and technology behind the National Security Agency's warrantless wiretapping program, and about the reforms he'd like to see in the Agency's operations. (mp3)

posted by acro (62 comments total)

Sorry, I meant to add that the second link is a PDF.
posted by acro at 10:15 AM on November 1

We're still waiting - perhaps forever - to find out what domestic spying initiative was so heinous that even John Ashcroft, that stalwart champion of police-state surveillance, couldn't stomach it.

There are few political realities so disturbing that I could honestly say that I may not even want to know about them. This is one.
posted by Joe Beese at 10:22 AM on November 1 [6 favorites]

It occurs to me that any right to privacy we might retain in the information age comes more from being lost in the crowd that explicitly denying information to any gov't body. Our hope, in other words, is that the NSA collects so much data that any individual's records are a tiny speck, lost in the blizzard.

There's this much to hope for: at the end of the day, the NSA is still trying to say "it's this guy, not this one, that we have to watch." Surveilling everyone is, in a sense, surveilling no one.
posted by fatbird at 10:24 AM on November 1 [5 favorites]

Now that's how you write a lede.
posted by andromache at 10:29 AM on November 1 [3 favorites]

The system goes on-line August 4th, 1997. Human decisions are removed from strategic defense. Skynet begins to learn at a geometric rate. It becomes self-aware at 2:14 a.m. Eastern time, August 29th. In a panic, they try to pull the plug.
posted by fet at 10:34 AM on November 1 [3 favorites]

Surveilling everyone is, in a sense, surveilling no one.

All they need to do is establish a precedent that googling the wrong things and calling the wrong people gets you disappeared and we'll all police ourselves.
posted by phrontist at 10:39 AM on November 1 [3 favorites]

Just hope those machines don't get the Buttles and Tuttles mixed up.
posted by bonobothegreat at 10:52 AM on November 1 [5 favorites]

Stored data is one thing. Sorting out who might be of interest a different issue and technique. And then the real problem: how that info gets interpreted and carried out into action taken by the White House et al. If you read that book--and the author is using declassified materials from Georgetown archives--and the two Bamford books, you will see how NSA and other intel operations often got misused or ignored for political reasons.

ps: it has become fashionable among those on the Left to blame Bush for domestic spying, something he initiated. this is hardly the truth. And it was NSA that suggested to Bush that a lot more Domestic spying needed, and he approved their recommendation.
posted by Postroad at 10:54 AM on November 1 [2 favorites]

it has become fashionable among those on the Left to blame Bush for domestic spying, something he initiated. this is hardly the truth

The truth is that domestic spying really got rolling in the 1960s, under the aegis of Edgar J. Hoover and Richard Nixon, two of America's most infamous Leftists.
posted by Blazecock Pileon at 11:02 AM on November 1 [11 favorites]

Sorry, J. Edgar Hoover.
posted by Blazecock Pileon at 11:03 AM on November 1

One thing is for sure: Given the things I've written on the Internet, I now have a snowball's chance in hell at becoming President.
posted by dunkadunc at 11:09 AM on November 1

As long as they're just looking for stuff that's already happened, happens very rarely and almost never arises out of the same set of circumstances it seems like a big boondoggle to me.
posted by fshgrl at 11:09 AM on November 1 [1 favorite]

From the guys who brought you the $5000 toilet and the $300 screwdriver, we introduce you to the wonder of massive parallel computing for $30 bazillions. Worried about those nosy government accountants and pro bono lawyers looking for publicity while screwing up your lucrative contract? Worry no more, not even the most advanced crack team of mathematicians, physicist and science fiction writers, not even the Great Computer of Douglas Adams can make sense of all that junk.

And it's all against teh terrorist, double plus good double full of win, or you hate 'Merica!

Guaranteed paying your vacation to Dubai or your money and defending lawyers expenses back, no shit.
posted by elpapacito at 11:11 AM on November 1 [2 favorites]

i think im losing my mind Dave......would you like me to sing you a song?.....Its called Daisy....Daisy, Daisy giveee meee yourr answer doooooooo immmmmm halffffff crazyyyyyyyyyy
posted by Seekerofsplendor at 11:11 AM on November 1 [3 favorites]

Surveilling everyone is, in a sense, surveilling no one.

Not exactly. If you're tracking everyone, and storing the information, you can look into a person retrospectively once you become interested. That's a big difference from a world where you don't have years of data on everyone, and have to do actual work to find out about them today and about them in the past.

Convenience changes everything. For instance, many people use Google and Wikipedia to answer questions that, two decades ago, they would have gone to the library for. When you can answer a question in 5 minutes, instead of waiting a week for inter-library loan, without having to ask a librarian how to find something, without worrying about late return fees, and without having to carry around a stack of books, you start to behave and act differently. You can ask questions more quickly and learn what questions to ask more quickly.

Today, one become a near-expert on a random topic without additional cost, without travel, without anything extra.

Convenience changes everything.
posted by zippy at 11:23 AM on November 1 [13 favorites]

And it was NSA that suggested to Bush that a lot more Domestic spying needed, and he approved their recommendation.

He could have said no, so the choice was still his to make.
posted by krinklyfig at 11:35 AM on November 1 [3 favorites]

MetaFilter: Sorry, J. Edgar Hoover.
posted by oaf at 11:48 AM on November 1 [6 favorites]

Blazecock Pileon: actually, J. Edgar Hoover was compiling his infamous files a long way further back than 1960 -- make it the 1930s and you'd be closer to the mark. More to the point, Hoover got his start under Palmer in 1919-21 (before the FBI acquired its subsequent name and identity) and, arguably, goes right back to the post-Civil War period; consider the activities of the Pinkertons against "anarchists", for example, as a privatised/outsourced forerunner of the FBI's subsequent state remit.
posted by cstross at 11:49 AM on November 1 [3 favorites]

Ultimately this system is just trying to predict behavior, so its usefulness is probably less important when it comes to terrorism and more important in running simulations on what people might do in response to certain situations. Presidential elections, wars, natural disasters... that information seems more valuable compared to who and who isn't a terrorist. In any case, the people most likely to become terrorists these days are not even going to register in that database, since they're probably off the grid entirely.
posted by infinitefloatingbrains at 11:50 AM on November 1 [1 favorite]

Yes, it's about convince. This is something that's been bugging me in general for a while. It seems things like background checks, data collection, surveillance cameras, etc have been more and more prevalent in the world lately, and while it seems like everyone is just becoming more paranoid the real issue is just cost. This stuff is just getting cheaper. Think about how large, low res and expensive an (analog) electronic video camera was in the early 90s. Now, a tiny digital camera circuit module cost just $19 retail and that's just the first thing I came across on google, I'm sure you can find devices like that for even less.

The price of doing background checks has gotten far cheaper. And the other day a the University of Akron announced it was going to start requiring DNA samples from it's employees, no doubt inspired by the new low cost of doing DNA analysis. Of course, congress has the good sense to make this illegal in 2008.

That doesn't matter, says Jeremy Gruber, president of the Council for Responsive Genetics in New York City. "GINA specifically prohibits employers from requesting or requiring genetic information," Gruber says. "It does not draw a distinction about how the DNA sample could be or should be used. There is no exception under GINA for employers in this context at all."

That said, it's going to be a real challenge to sort through all this data. Even if you could pinpoint a terrorist with 99% accuracy, since there are so few terrorists the 1% of false positives will be overwhelmingly non-terrorists if 1% of the population is flagged, and there are 10 terrorists in the US, then (3 million - 10) = 2,999,990 people will be false positives.

Of course, since there would be some superficial similarities to terrorists the government could easily argue that they were. Or if they were sent off to gitmo and waterboarded then what? You might argue that that wouldn't happen now that bush is out of office, but no one can predict who the president is going to be 8, 12, or 16 years down the road.

People will say they're doing this to "keep us safe" Of course that's bullshit. It's just going to suck down resources to companies with the right lobbyist connections. These projects are always about funneling money to lobbyists who can manage to tell a scary story to congress. But those same lobbyists tell the same scary stories to the corporate media, who loves scaring viewers because it keeps 'em tuning in.

There is really a fear industry in Washington. Crazy idiots at think tanks spend all day trying to think up scary things to sell to congress, which of course requires huge outlays of money to the very companies that fund those think-tanks. Then, of course, those same idiots sell those stories to TV reporters who need something "juicy" that will bring in eyeballs on the 24/7 news cycle, regardless of how big of a threat it is.

Meanwhile, 40,000 Americans die each year due to a lack of healthcare, but we spend more in Afghanistan each year then the entire 10 year projected budget of health reform.
posted by delmoi at 11:57 AM on November 1 [13 favorites]

There is really a fear industry in Washington. Crazy idiots at think tanks spend all day trying to think up scary things to sell to congress, which of course requires huge outlays of money to the very companies that fund those think-tanks.

I know Naomi Klein doesn't get much love on Metafilter, but this is the very thing she writes about in The Shock Doctrine.
posted by Blazecock Pileon at 12:03 PM on November 1 [3 favorites]

numbers beyond Yottabytes haven't yet been named

(ahem) Googolplex.
posted by erniepan at 12:21 PM on November 1 [1 favorite]

US spy agencies' spending rises to $49.8 billion
posted by DreamerFi at 12:27 PM on November 1 [1 favorite]

I'm sure this system will just tell me I might be interested in Pixies albums, even though I've owned all of them for years.
posted by RobotVoodooPower at 12:36 PM on November 1 [2 favorites]

numbers beyond Yottabytes haven't yet been named

I always thought they were harpobytes, chicobytes, grouchobytes, and zeppobytes
posted by Spock Puppet at 12:38 PM on November 1 [12 favorites]

Given that a yottabyte would require about one trillion 1-terabyte hard drives I'm finding that number kind of hard to believe. It would take a cube of hard drives measuring 10,000 in each dimension to get you a yottabyte of data (about 250 meters tall - over half as tall as the sears tower, 760 wide and I guess about 1km long),

And it would cost $73 trillion dollars at today's prices. And of course that's with no additional hardware whatsoever, just the disks stacked up on top of each other with no electricity or connectivity.
posted by delmoi at 12:44 PM on November 1 [7 favorites]

In 2015, with my poor Moore's law math via wikipedia, that $73 trillion worth of equipment would only cost $200 billion.
posted by acro at 1:06 PM on November 1

It was General Eisenhower, leaving his office as President who warned the nation against the growing military/industrial complex. It was Gen Powell (yes, that guy) who gave us another and later waring (I had this at my blog):

a video seldom seen as Colin Powell warns of a nation so caught up with fighting terrorism that that issue becomes an excuse to go astray from our better nature--the sort of warning that Ike gave (as he retired from his presidency) about the Military/Industrial complex

posted by Postroad at 1:17 PM on November 1 [1 favorite]

to determine who among us may be—or may one day become—a terrorist.

Let me save you some time there. It's ALL OF US.
posted by fungible at 1:23 PM on November 1 [3 favorites]

Delmoi wins. Spock Puppet comes in a close second though. I really hope the Marxobyte numbering system wins out!
posted by JHarris at 1:48 PM on November 1

I don't think that the construction of these databases have much to do with terrorism. My gut tells me they're much more closely related to the FBI's blackmailing of Martin Luther King in response to his anti-war activities.

Finding a terrorist this way has the needle-in-a-haystack problem of false positives that people have pointed out. These databases are an excellent tool for blackmail. That is also why they are so pernicious. Even if you yourself are so virtuous that no information could come to light to harm you, political or philosophical leaders you support could be silenced this way, and that does harm you.
posted by Humanzee at 1:52 PM on November 1 [4 favorites]

In 2015, with my poor Moore's law math via wikipedia, that $73 trillion worth of equipment would only cost $200 billion.

The problem is that magnetic hard drives have pretty much hit their limit. And the platters are not made from transistors that the law applies too. There would need to be some kind of technological jump. If you wanted a yottabyte of data on solid state drives today, the cost would be $2 quadrillion dollars today. (at a price of 500 megabytes/$ vs. 13 gigabytes/$ with magnetic hard drives)
posted by delmoi at 2:00 PM on November 1 [2 favorites]

>>It occurs to me that any right to privacy we might retain in the information age comes more from being lost in the crowd<<

This is naive.

The whole point is to examine someone once they become of interest (for example, you present your passport at the border). This first-pass winnowing makes for a huge reduction in the burden.

I had an interesting experience flying back from overseas this week. The last time my wife traveled with me overseas, she brought back an apple and a carrot she'd intended to eat on the plane. We disclosed this on entry and they were confiscated (no issue with that at all). What bothers me is that a year later, on returning from overseas, we were first explicitly asked if we had carrots or apples with us, and then were flagged for hand-check by the agricultural inspectors of all baggage. This just seems silly, but it does sort of get at the level of data retention.
posted by rr at 2:03 PM on November 1 [4 favorites]

Joe Beese We're still waiting - perhaps forever - to find out what domestic spying initiative was so heinous that even John Ashcroft, that stalwart champion of police-state surveillance, couldn't stomach it.

My suspicion is the recruitment of civilians to secretly spy on each other in the manner of a Stasi, perhaps recruited from within Neighbourhood Watches, Home Owners Associations, school Parents & Citizens groups, the Scouts, etc. The idea is simple: just recruit a few key people who recruit those they trust, financially reward successful tipoffs, and let it be well-known among the citizenry that anyone (at least, any of the "good people") could be reporting your actions to the government.

Perhaps I'm giving too little credit to Ashcroft here but I suspect that the reason he objected wasn't the idea itself, but that the briefing itself made so many comparisons to secret police, Stasi, etc that it penetrated through his thick sense of self-righteous exceptionalism to the point where even he realized that there was nothing to exceptionalize - unlike other American things, like sabotage and fomenting political upheaval in foreign nations, an American Stasi would not become "good" just by Americans doing it.

Gonzales, on the other hand, seems too unimaginative to be an exceptionalist. To him, it seems, there is "us" and "them", and any question of "good" or "evil" is irrelevant.
posted by aeschenkarnos at 2:13 PM on November 1 [3 favorites]

Obligatory terrorism tread Schneier link.

Data mining works best when you're searching for a well-defined profile, a reasonable number of attacks per year and a low cost of false alarms. Credit-card fraud is one of data mining's success stories: all credit-card companies mine their transaction databases for data for spending patterns that indicate a stolen card.

Terrorist plots are different. There is no well-defined profile and attacks are very rare. Taken together, these facts mean that data-mining systems won't uncover any terrorist plots until they are very accurate, and that even very accurate systems will be so flooded with false alarms that they will be useless....

Let's look at some numbers. We'll be optimistic -- we'll assume the system has a one in 100 false-positive rate (99 percent accurate), and a one in 1,000 false-negative rate (99.9 percent accurate). Assume 1 trillion possible indicators to sift through: that's about 10 events -- e-mails, phone calls, purchases, web destinations, whatever -- per person in the United States per day. Also assume that 10 of them are actually terrorists plotting.

This unrealistically accurate system will generate 1 billion false alarms for every real terrorist plot it uncovers.
posted by Jakey at 2:19 PM on November 1

My suspicion is the recruitment of civilians to secretly spy on each other in the manner of a Stasi, perhaps recruited from within Neighbourhood Watches, Home Owners Associations, school Parents & Citizens groups, the Scouts, etc. The idea is simple: just recruit a few key people who recruit those they trust, financially reward successful tipoffs

I doubt it. First of all, they actually did it once Gonzo got into power, and nothing like that became "widely known" and second of all, it sounds like a lot of work and these guys were nothing if not lazy. It was obviously some kind of technological thing.
posted by delmoi at 2:20 PM on November 1

As long as this prevents repeat suicide bombers, it's all good, right?
posted by sebastienbailard at 3:01 PM on November 1 [2 favorites]

Why, do you think, no one, and I repeat no one ever touches the defense department's budget? They have all the dirt, on all the politicians, ever.

The DOD is the single greatest transfer of wealth from the many to the few in the history of the world. $150 billion is personnel, and about 50 billion that is classified Intelligence budget, and we don't see where that goes, which some significant portion pays the salaries of government workers, I assume. The bulk of the rest of the 800 billion dollars spent on the military, a full 68% of the federal budget, goes straight into the pockets of the military industrial complex.

The right talk a big game about "wealth transfers", about entitlement, about welfare queens and a whole stack of bullshit. It's a drop in the bucket of the amount of money we spend on the military, and will continue to spend on the military because our politicians are scared shitless of them.
posted by Freen at 3:32 PM on November 1 [7 favorites]

Convenience Expedience changes everything.

FTFY, and remember kids, when government is involved, expedience is ALWAYS the polar opposite of justice.
posted by ZenMasterThis at 3:33 PM on November 1 [1 favorite]

Obligatory Giant budget Picture. sorry about not linking to the original, it's hidden behind some horrible zoomy thing
posted by Freen at 3:37 PM on November 1

and we don't see where that goes,

Any body remember the big news of Sept 10th, 2001? How the military had a 2 trillion dollar accounting issue?

How about the 25ish million to ship under 10,000 in lp gas to Iraq?

spend on the military because our politicians are scared shitless of them.

Chalmers Johnson in Nemesis points out that if the military was stopped dead cold you'd see 25% unemployment.

But that was back a few years ago. Imagine the rate now.
posted by rough ashlar at 3:57 PM on November 1 [1 favorite]

It's a scaling problem.

Anyone who works in payday loans can tell you how easy it is to follow someone in near real time today, to do pattern recognition in real time. Anyone who works in knowledge management or competitive intelligence can tell you how easy it is to automate that process (even using something as rudimentary as yahoo pipes, let alone the beauties Autonomy software makes).

The government is not the problem as the old "If you give me six lines written by the most honest man, I will find something in them to hang him..." has always applied.

Now, imagine, you go to rent SCUBA equipment on your holiday to Cozumel and Amex declines the charge because you haven't been on the treadmill enough the last two months to keep your insurance company satisfied and Amex would be on the hook if you stroked out. That's how this data, and its real time applications, will play out.
posted by digitalprimate at 4:13 PM on November 1 [1 favorite]

Didn't Ashcroft say that he would guarantee that every person would have their day in court?
Terrorists obviously aren't persons.
posted by Balisong at 5:41 PM on November 1

Very interesting read..... up it goes on TechCrunch.
posted by BlackLeotardFront at 6:31 PM on November 1

This would make an awesome SETI@home-style distributed project .

It might even overheat all the freepers computers and burn their houses down.
posted by Rumple at 6:39 PM on November 1

The problem is that magnetic hard drives have pretty much hit their limit

I don't think this is in evidence.

We'll see, but the size of disks today is such that consumers really do not need larger drives, so all density pressure is coming from server vendors and cloud usage. It's not that they don't care about density (they do), it's that they care more about power, heat (power) and I/O operations per second. Tiered storage with flash hybrid drives and in-RAM caching [hence extremely large server memories] helps a lot with the last. Flash drives also do not have good failure behavior which matters a lot.

I'd rather deal with a disk that's gone marginal and can tell me than a disk that simply runs for awhile and, after generating a ton of interrupts, simply disconnects from the bus because the built-in controller doesn't know what do in response to a hard media error during certain operations. There are several other issues.

I'm not saying disks are going to go up by another few orders of magnitude, or that all-flash storage doesn't have applications, it's that all-flash storage applications are a very special case right now (mostly for thousands-of-video-streams servers) that can deal with the much lower density.
posted by rr at 7:54 PM on November 1

Real Enemies: Conspiracy Theories and American Democracy, World War I to 9/11 by Kathryn Olmsted gives a balanced overview of the history of domestic surveillance in the US.
posted by lukemeister at 8:06 PM on November 1

Meh. So now they have a lot of storage and their problem becomes CPU-limited and/or IO-bandwidth-limited instead of storage limited. If they're so smart they'd all quit and put Google out of business.
posted by GuyZero at 8:47 PM on November 1

Now, imagine, you go to rent SCUBA equipment on your holiday to Cozumel and Amex declines the charge because you haven't been on the treadmill enough the last two months to keep your insurance company satisfied and Amex would be on the hook if you stroked out. That's how this data, and its real time applications, will play out."

An old take on this by the ACLU.
posted by DreamerFi at 10:19 PM on November 1

In 2015, with my poor Moore's law math via wikipedia, that $73 trillion worth of equipment would only cost $200 billion.

Moore's law kinda sorta says that hardware capabilities per dollar double every two years. That would mean that 6 years from now, $73 trillion of stuff would cost about $9 trillion.

Anyone who works in payday loans can tell you how easy it is to follow someone in near real time today, to do pattern recognition in real time. Anyone who works in knowledge management or competitive intelligence can tell you how easy it is to automate that process (even using something as rudimentary as yahoo pipes, let alone the beauties Autonomy software makes).

Ironically, this comment unintentionally gets closest to the truth of the situation. American intelligence agencies would like to have the kind of authority to monitor their citizens that Britain, France, Germany, etc. give theirs. They'd also like collect the kind of data that American corporations can. Because of Hollywood, no one believes this, of course, but IC agencies are still government bureaucracies with long lists of regulations and rules, including many governing how and when information can be collected on U.S. persons, the implementation details of which are hashed out by lawyers in laboriously long meetings and verified methodically. I've implemented some of them myself, with exciting names like DoD 5240.1-R (my favorite, because of the awesomely cheesy dialogue in its training video) and can speak from experience that changes in the implementation specs or bugs are a Big Deal that would immediately take priority over anything else I was working on.

Once vacuumed up and stored in these near-infinite "libraries," the data are then analyzed by powerful infoweapons, supercomputers running complex algorithmic programs, to determine who among us may be—or may one day become—a terrorist.

Bullshit. As a friend of a friend once said when his son asked if he could read people's emails, "I can't even get spreadsheets to open." People vastly overestimate the extent to which analysts trust non-existent magical algorithms to do their work for them.

Meanwhile, 40,000 Americans die each year due to a lack of healthcare, but we spend more in Afghanistan each year then the entire 10 year projected budget of health reform.

I'm not sure under what math this is true, unless you're considering the overall 10 year budgetary effect of health care reform to be negative, in which case I spend more at Panera each week than the entire 10 year projected budget of health reform.

My suspicion is the recruitment of civilians to secretly spy on each other in the manner of a Stasi, perhaps recruited from within Neighbourhood Watches, Home Owners Associations, school Parents & Citizens groups, the Scouts, etc. The idea is simple: just recruit a few key people who recruit those they trust, financially reward successful tipoffs, and let it be well-known among the citizenry that anyone (at least, any of the "good people") could be reporting your actions to the government.

You are a loon.

As for the actual post, you'll notice in the first link that Bamford, who follows the NSA as closely as anyone, doesn't actually say that they're monitoring American communications, just that they have physical access, based on stuff that's been reported in the news for years now. Even though no one believes them, IC agencies take domestic surveillance laws very seriously, and although there are gray areas reported in the news from time to time that could legitimately upset some people, the idea that any IC agencies keep even a fraction of the data on U.S. persons available to them is seriously at odds with reality.
posted by stubblemaker at 11:01 PM on November 1

Bruce Sterling predicted how this software will be used in Distraction in 1998.
posted by Zed at 11:13 PM on November 1

As for the actual post, you'll notice in the first link that Bamford, who follows the NSA as closely as anyone, doesn't actually say that they're monitoring American communications, just that they have physical access, based on stuff that's been reported in the news for years now.

Actually, he does say they are analyzed by the NSA, from the first article linked:

"For example, the agency built secret rooms in AT&T's major switching facilities where duplicate copies of all data are diverted, screened for key names and words by computers, and then transmitted on to the agency for analysis."

Based on the news reports from the AT&T whistle-blower, they have installed Narus STA 6400 boxes in switching rooms throughout the country. These machines don't just reroute traffic, they actively analyze that traffic, at a fairly high level.

We have seen many many stories over the years about data collection on US citizens. From FBI infiltration of peace activism groups, to surveillance of civil rights leaders in the 50s and 60s, to the NSA spying on American citizens through it's arrangements with telegraph companies before the Church Committee.

While your story about your friend may indicate a distrust among human analysts about computer-based analysis of data (which seems obvious, considering they're competing for the same job), it doesn't mean the upper management doesn't love them selves some fancy privacy-abusing tech. And these technologies are only going to get better.

From the sounds of it you're in the defense/intelligence sector, and you're proud of your job. I understand this, and I think that we need a strong, efficient and flexible intelligence capability. But that doesn't I can't be skeptical and critical of it at the same time. That's the role of a good citizen, to make sure that those we have elected are really upholding the principles of the Constitution.

Finally, to tackle this statement:

the idea that any IC agencies keep even a fraction of the data on U.S. persons available to them is seriously at odds with reality.

The reality is that the NSA is rapidly expanding, building data centers throughout the country to store vast quantities of data. The reality is that the Patriot Act has broken down the walls put in place after the NSA was last caught spying on US citizens. The reality is that the we have had several whistleblowers come forward who have claimed the the agencies are not only intercepting domestic communications, but also personal financial transactions.
posted by formless at 12:22 AM on November 2

The actual report states that yottabytes are an unrealistic estimate. Hundreds of petabytes is the more realistic number — still a considerable amount. Adjust tinfoil hats accordingly.
posted by BlackLeotardFront at 12:38 AM on November 2

American intelligence agencies would like to have the kind of authority to monitor their citizens that Britain, France, Germany, etc. give theirs.

Echelon seems to have gotten around that Gordian Knot.
posted by Blazecock Pileon at 1:43 AM on November 2

fungible: "to determine who among us may be—or may one day become—a terrorist.

Let me save you some time there. It's ALL OF US."

Osama Bin Laden Found Inside Each Of Us

Since toppling the Taliban regime in 2001, U.S. forces in Afghanistan had searched for bin Laden primarily along the rugged Afghan-Pakistani border, but overlooked that place inside every one of us that has ever raised his voice in anger or turned away from someone in need.

"We were so busy tracking the remaining members of the Taliban regime and freezing al-Qaeda assets that we missed what was right in front of us all along," Rumsfeld said. "Osama bin Laden wasn't hidden in a cave in the mountainous Pakistani province of Waziristan or huddled in the back of a Chitral meat-market stall. He was lurking in the blackness within us all, right there with the laziness and the jealousy."

"It just goes to show that sometimes it's easier to look for the man in the FBI dossier than it is to look at the man in the mirror," Rumsfeld added.

posted by Rhaomi at 3:40 AM on November 2 [3 favorites]

Yottabytes

This sounds like an appetizer.
posted by jonmc at 5:19 AM on November 2

aeschenkarnos writes: My suspicion is the recruitment of civilians to secretly spy on each other in the manner of a Stasi, perhaps recruited from within Neighbourhood Watches, Home Owners Associations, school Parents & Citizens groups, the Scouts, etc. The idea is simple: just recruit a few key people who recruit those they trust, financially reward successful tipoffs, and let it be well-known among the citizenry that anyone (at least, any of the "good people") could be reporting your actions to the government.

stubblemaker responds: You are a loon.

He is no such thing. In fact, they tried that very tact early in the Bush administration, albeit only for a targeted list of 'high-risk' professional groups: Operation TIPS. If it had passed, it wouldn't have been much of a stretch to increase the size of the list of affected groups by bumping up the funding to a parent organization that doesn't need to line-item its budget. It narrowly died in the Senate, but not for a lack of effort by Lieberman and a handful of Republican stooges: the media got wind of it just in time to make a messy spectacle.

It wouldn't surprise me for a minute if Ashcroft tried to rebuild it a few years later through back channels, where the Times wouldn't have been able to dig up the text of the bill and air his dirty laundry. And it was such a crazy-repugnant prospect that people refused to believe it was being enacted, even as the text of the legislation was being distributed. Seven years later, we've got right-wing hacks telling us that documented actions of the previous administration are crazy-talk.
posted by Mayor West at 7:33 AM on November 2 [4 favorites]

While we're discussing this; a few years back I read a longish article about one of the companies that worked with the US security agencies on text search, computers, antiprivacy, etc.

They actually had a vice president and computer systems who looked at their corpus of previous business proposals to funding agencies and analyzed the proposals to come up with new 'attractive' proposals that would get greenlighted by the agencies.

Does anyone know what I'm talking about? (This, boys and girls, is why you should bookmark things.)
posted by sebastienbailard at 11:14 AM on November 2

In-Q-tel, the VC arm of the CIA?
posted by GuyZero at 11:32 AM on November 2

Oh, wait, that's not what you're talking about. Sounds like a great idea though.
posted by GuyZero at 11:32 AM on November 2

Here's the crux of the problem and why any such attempt at identifying terrorists is doomed to failure. Let's say there are, out of 300,000,000 or so US residents, 2000 that are terrorists. Now let's develop a really, really, really good algorithm that is 99% sensitive and 99% specific. (We can only hope for such precision because we have a lot of data.) Okay, apply that algorithm to the 300,000,000 people and you have approximately 2,800,000 false positives and 1980 true positives. That's about 1400 peaceful residents you'd have to interrogate for every dangerous terrorist. It's still like finding a needle in a haystack. So they have that rather huge challenge to get over right there.

The real problem is that to even know what the sensitivity and specificity are, you need to have a pretty large sample of known terrorists to validate the algorithm.

Which they don't have.
posted by Mental Wimp at 12:17 PM on November 2

You don't search 10 million random records for one piece of information. You take one piece of information and compare that to 10 million records looking for connections.

So a Mr. Suspicious leaves a tiny piece of data - called a known terrorist sympathizer, gave to a suspicious charity, etc. They take that info and dig. Pull his address, his friends and family, known associates. See who lived where, when. You'd be stunned at the info that names, dates and address can bring up. But you can go deeper. Where he went to school and who with. Does he still live near or talk to any of those people? Where does he work, and who are all his co-workers? Did he give to other charities? Take any trips?

And instead of flat text list of facts, you use visual analytics to show the data in easy to digest format. Heat maps or color codes of probabilities, showing where to look first. Here's the blog of a company doing this stuff now. It's fascinating stuff.
http://blog.palantirtech.com/

It occurs to me that any right to privacy we might retain in the information age comes more from being lost in the crowd that explicitly denying information to any gov't body. Our hope, in other words, is that the NSA collects so much data that any individual's records are a tiny speck, lost in the blizzard.

You have the illusion of privacy since the great eye of CIA-ron hasn't fixed it's gaze upon you.
posted by anti social order at 2:04 PM on November 2