Join 3,512 readers in helping fund MetaFilter (Hide)


Welcome to America. All of that stuff is being captured as we speak.
May 5, 2013 6:23 AM   Subscribe

Are all telephone calls recorded and accessible to the US government? This week, CNN interviewed Tim Clemente, a former FBI counterterrorism agent, about whether the FBI would be able to discover the contents of past telephone calls between Tamerlan Tsarnaev and his wife. Clemente stated that the FBI had ways of accessing those calls, and that all calls are recorded.


BURNETT: Tim, is there any way, obviously, there is a voice mail they can try to get the phone companies to give that up at this point. It's not a voice mail. It's just a conversation. There's no way they actually can find out what happened, right, unless she tells them?

CLEMENTE: "No, there is a way. We certainly have ways in national security investigations to find out exactly what was said in that conversation. It's not necessarily something that the FBI is going to want to present in court, but it may help lead the investigation and/or lead to questioning of her. We certainly can find that out.

BURNETT: "So they can actually get that? People are saying, look, that is incredible.

CLEMENTE: "No, welcome to America. All of that stuff is being captured as we speak whether we know it or like it or not."


Previously, we discussed AT&T's involvement in the NSA wiretapping program, capturing the entire stream of data travelling through AT&T's system
posted by benbenson (180 comments total) 21 users marked this as a favorite

 
I always wonder about this kind of thing though. It's always these "former agents" who "tell all" but half the time I wonder how much is just bullshit to get some attention and feed a line. It's not that I necessarily doubt it, but first, it would, I think, be the NSA doing that kind of thing, not the FBI, second, the technical requirements to store that amount of data would be massive (though of course, there's always those rumors about that NSA site in Nevada or wherever, from a couple years ago, from Wired Magazine)...

It's one thing to capture a stream of data for a period of time, it's another to retain every single bit ever that's going through all of the communication systems (or at the very least POTS + Cell -- how they retain VOIP that's direct IP to IP (vs IP to POTS), I have no idea if that's technically feasible without some form of packet scanning which is more burden on servers than providers want to have to deal with, but, never doubt AT&T and Verizon to cozy up to the fucking assholes in the police state).
posted by symbioid at 6:30 AM on May 5, 2013 [11 favorites]


How is this not overstepping a person's right to privacy? I get the "I have nothing to hide, and hence have nothing to fear" approach but I think everyone has conversations and a reasonable expectation of privacy.
posted by arcticseal at 6:30 AM on May 5, 2013 [1 favorite]


I've always assumed that was the case. Honestly I'm more interested in the technical aspects of this: how much data is stored? Are we talking about a file per call and if so, what file format is used? What kinds of automated analysis are they running on the data, e.g. can they automatically detect conversations about certain topics? I'm assuming it's possible to search the data but how powerful is the search?
posted by Foci for Analysis at 6:33 AM on May 5, 2013 [3 favorites]


Weren't there theories that intelligence agencies were swapping eavesdropping data between countries at one point? The U.S. would spy on the British and the British would spy on the U.S. and swap data as needed. That way no domestic laws were broken.
posted by srboisvert at 6:34 AM on May 5, 2013 [3 favorites]


"The government has a secret system: a Machine..."
posted by the man of twists and turns at 6:35 AM on May 5, 2013 [9 favorites]


It's always these "former agents" who "tell all" but half the time I wonder how much is just bullshit to get some attention and feed a line.

Yeah, it rings a bit "Former Air Force General tell all about UFOs".
posted by Lentrohamsanin at 6:35 AM on May 5, 2013 [8 favorites]


If this isn't happening now, it'll be happening soon. It's just too easy to record and store this sort of data. In the future, everything will be stored. There will be abuses, yes, but mostly it'll probably be ignored until Skynet achieves sentience.
posted by Brandon Blatcher at 6:39 AM on May 5, 2013 [2 favorites]


I get the "I have nothing to hide, and hence have nothing to fear" approach

Actually, I have never understood that idea. It always struck me as dangerously naive. I don't get to decide if I have nothing to hide or fear. It is the people with the guns and the power who decide that, and that is precisely the point here.
posted by Seymour Zamboni at 6:42 AM on May 5, 2013 [69 favorites]


Back of the envelope time:

Average American cell phone usage: 13 hours / month

# of Americans with cell phones: 200 million

Rough data rate for voice calls: 0.5 megabytes/minute

Amount of time Tsarnaev was on the FBI's radar: 2 years

Amount of data the gubment would need to store to have Tsarnaev's calls on record:

1.7 exabytes

If we limit that to the 3 million person terrorist watchlist that Tsarnaev was put on, that's

26 petabytes of voice data.


That's a lot, but not impossible.
posted by Salvor Hardin at 6:44 AM on May 5, 2013 [21 favorites]


how they retain VOIP that's direct IP to IP

In the City of San Francisco there is a door that only people with an NSA clearance can open. Through that door there is a room and in that room there is a major internet noden through which all internet traffic in a given region must pass. Near that internet node there is a device which captures every single packet which passes through that node. There are similar doors near virtually every major ATT and Verizon internet node in the country.

The reality of 21st century America is that we have no privacy and our freedom of speech only extends as far as the NSA, or their masters, care to allow.
posted by Podkayne of Pasadena at 6:44 AM on May 5, 2013 [15 favorites]


Why do police bother with wiretap warrants, then, if everything's already been recorded?
posted by desjardins at 6:46 AM on May 5, 2013 [1 favorite]


Amount of data the gubment would need to store to have Tsarnaev's calls on record: 1.7 exabytes

You are not taking data compression techniques into account into your calculations.
Also:

The Utah Data Center, also known as the Intelligence Community Comprehensive National Cybersecurity Initiative Data Center,[1] is a data storage facility for the United States Intelligence Community that is designed to be a primary storage resource capable of storing data on the scale of yottabytes (1 yottabyte = 1 trillion terabytes, or 1 quadrillion gigabytes).
posted by Podkayne of Pasadena at 6:48 AM on May 5, 2013 [16 favorites]


Well, every fuckwit with a podium has spent the last 10 years berating the government for not "connecting the dots" in time to prevent 9/11. As Schneier recently pointed out, connecting the dots is only possible in hindsight, so maybe the NSA has decided to develop the ability to do exactly that. Still insane overkill if all we get from it is the ability to confirm the flat out obvious regarding two losers who placed satchel bombs at the Boston Marathon.
posted by ocschwar at 6:54 AM on May 5, 2013 [6 favorites]


desjardins: "Why do police bother with wiretap warrants, then, if everything's already been recorded?"
They need them if they want to present the evidence in court.
posted by brokkr at 6:54 AM on May 5, 2013 [17 favorites]


arcticseal: "How is this not overstepping a person's right to privacy? I get the "I have nothing to hide, and hence have nothing to fear" approach but I think everyone has conversations and a reasonable expectation of privacy."

The National spooks exchange data so they technically never spy on their own citizens.

Y, it's a cop out.

Salvor Hardin: "1.7 exabytes"

And that assumes the NSA doesn't have automatic transcription. Get all those phone calls into text files and the data storage requirements drop through the floor.
posted by Mitheral at 6:57 AM on May 5, 2013 [5 favorites]


desjardins: "Why do police bother with wiretap warrants, then, if everything's already been recorded?"

To get stuff into court.
posted by Mitheral at 6:57 AM on May 5, 2013 [2 favorites]


Dittoing Podkayne. If they do this, they definitely compress. I have no idea what goes on in the three letter organizations, but if I were doing this, I wouldn't record everyone, that's just silly. I would, on the other hand, monitor everyone and then record those who are determined to be interesting based on social network analysis (i.e., people who have social links to criminal and terrorist organizations) or start recording someone when the content of their communications begins to match certain patterns deemed suspicious (simple keyword analysis is silly, I would use bayesian nets trained against known to be real criminal and terrorist communication patterns). That's what I would do, in any case, were I given panopticon like powers.
posted by dis_integration at 7:00 AM on May 5, 2013


And that assumes the NSA doesn't have automatic transcription.

I am deeply skeptical that the NSA is storing every single telephone conversation that takes place in the US. I am quite sure, however, that they have not perfected a magical "automatic transcription" machine that can accurately transcribe all conversations between all people of all accents in all languages with no individual "training" whatsoever.
posted by yoink at 7:03 AM on May 5, 2013 [18 favorites]


Easier to search with free existing tools.
posted by infini at 7:03 AM on May 5, 2013


I don't believe this (at least the gist of "every phone call in the US is recorded"), not because the data volume is too great, which it probably is but you can always just throw more hardware at it, but because the size and decentralization of the phone network would require a very extensive distribution of hardware to capture all this data. Local calls in particular can be connected within a single central office on a single switch, so if you really want to capture that traffic you have to have hardware in every single one of the thousands of central offices around the country. Even if you limit it to long distance calls, that's still hundreds of tandems that need to have recording hardware (and lots of it, and lots of backhaul). The sheer number of installations means that a lot of people would know about this. I'm ok with assuming NSA-omnipotence when it comes to data storage and computing power, but keeping widely-dispersed controversial secrets is much much harder.
posted by kiltedtaco at 7:05 AM on May 5, 2013 [11 favorites]


Actually, a good model for what Big Brother does, if it actually does this sort of thing, would probably be the way network intrusion detection systems like Snort operate.
posted by dis_integration at 7:08 AM on May 5, 2013


CIA's CTO: "Since you can't connect dots you don't have, it drives us into a mode of, we fundamentally try to collect everything and hang on to it forever... It is really very nearly within our grasp to be able to compute on all human generated information."

FBI: Our surveillance techniques are "private and confidential"

I don't have a hard time believing Clemente at all.
posted by RobotVoodooPower at 7:22 AM on May 5, 2013 [8 favorites]


I am no expert in these matters. But from all I have read, and a bit of first-hand experience, I think it is a fair guess to say that any and all things done, said, sent, via any electronic means can be accessed and thus also stored for retrieval if needed.
posted by Postroad at 7:24 AM on May 5, 2013 [1 favorite]


Even if the gov was doing this. The problem is not having data but having too much data creating statistical false positives. Beware the Big Errors of ‘Big Data’ by Nassim N. Taleb
posted by stbalbach at 7:27 AM on May 5, 2013 [2 favorites]


And recently: Senior Obama administration officials have secretly authorized the interception of communications carried on portions of networks operated by AT&T and other Internet service providers, a practice that might otherwise be illegal under federal wiretapping laws.
posted by RobotVoodooPower at 7:34 AM on May 5, 2013 [2 favorites]


In other news, the sale of tinfoil has skyrocketed.
posted by HuronBob at 7:37 AM on May 5, 2013 [5 favorites]


Anyone who read the Klein deposition in EFF v. AT&T isn't really surprised by this sort of news. One could opine, "RMS warned you all, but you didn't listen, and here we are." while throwing up their hands in frustration.

I wonder, if the gun-rights people are all fired up about 'needing guns to protect our rights and freedom', what they're planning on doing about this? And how they're planning on doing it when The Government is reading their email and listening in to their calls?
posted by mikelieman at 7:37 AM on May 5, 2013 [3 favorites]


This is the same government that didn't know the WMDs were fiction, couldn't organize a response to Katrina and, despite magical phone-recording tech, still has a ten most-wanted list.

So, which is it? Bumbling fools or nefarious super villains? Make up your mind, you conspiracy-minded dolts.
posted by Cool Papa Bell at 7:39 AM on May 5, 2013 [25 favorites]


It is really very nearly within our grasp to be able to compute on all human generated information.

The British claimed they had carrot-eating soldiers with night vision in WW2, to hide the development of radar. The Soviets spread rumors of ESP and telepathic capabilities to mess with the US. It's not the kind of subject where you believe everything people say they are capable of.
posted by Dr Dracator at 7:42 AM on May 5, 2013 [13 favorites]


kiltedtaco: not because the data volume is too great, which it probably is but you can always just throw more hardware at it, but because the size and decentralization of the phone network would require a very extensive distribution of hardware to capture all this data

It does, and that hardware *is* widely distributed, even in every dinky little class 5 end office:

     Lawful Interception

Now, whether that equipment is capable of recording and transmitting (for archival) all calls, or just selected calls, is something which is up for debate. But the widespread presence of the equipment is already a fact, I'm afraid.

As for backhaul, there's plenty of that to go around, and most of it conveniently terminates in central offices anyway.
posted by Juffo-Wup at 7:43 AM on May 5, 2013 [3 favorites]


It's always these "former agents" who "tell all"

Clemente wasn't telling all; he didn't go on CNN to blab about eavesdropping, he dropped that info as an aside to the interview about Tamerlan Tsarnaev and Katherine Russell. And, as to why it's always a "former agent," it's pretty obvious why a current agent never discusses this.
posted by benbenson at 7:47 AM on May 5, 2013 [2 favorites]


By "didn't' know WMD's are fiction", are you referring to Bush, Cheney, Rice, Rumsfeld, Powell, et. al.'s Conspiracy to Defraud the United States, in that by knowingly presenting information they knew or had reason to believe was false, they deprived Congress of their Constitutional role in oversight?

I knew WMD's were fiction, and so did everyone else who came out and said that they were real.

Look, at the end of the day, AT&T GETS PAID to spy on us for the government. I'm sure it works exactly as the contract specifies and meets all SLAs and uptime guarantees. AT&T isn't going to pass this revenue opportunity up.
posted by mikelieman at 7:47 AM on May 5, 2013 [3 favorites]


Salvor Hardin: Rough data rate for voice calls: 0.5 megabytes/minute

It's actually closer to 0.06 megabytes/minute (at 8 kbit/sec, which is a more-or-less average data rate for voice calls in modern cell networks). That's only 62 PB, for *all* Americans, not just ones on some terrorist watch list.

Although it sounds impressive, 62 PB is really not that much these days. A relatively small research group where I work has ~ 1.6 PB of RAID-6 spinning disk to support their research activities and, while that is big for us, is not considered particularly impressive in the overall scheme of things. And that system is run by one or two guys who have other sysadmin responsibilities as well. *And* this is just research computing, with no anti-terrorism/law-enforcement/NSA-type funding whatsoever.
posted by Juffo-Wup at 7:49 AM on May 5, 2013 [14 favorites]


So, which is it? Bumbling fools or nefarious super villains? Make up your mind, you conspiracy-minded dolts.


This is way too facile and reductive, both of conspiracy theorists and the government. The government isn't one entity, and even if it was, there are many sides.
posted by benbenson at 7:50 AM on May 5, 2013 [13 favorites]


Not to detract from the Orwellian hope that governments have of monitoring all digital communications, enjoy the parody website NSA Utah Data Center — Serving Our Nation's Intelligence Community:
Domestic Surveillance Directorate
Defending Our Nation. Securing The Citizens.


Utah Data CenterBackground

The Utah Data Center, code-named Bumblehive, is the first Intelligence Community Comprehensive National Cyber-security Initiative (IC CNCI) data center designed to support the Intelligence Community's efforts to monitor, strengthen and protect the nation. NSA is the executive agent for the Office of the Director of National Intelligence (ODNI) and will be the lead agency at the center.

The steady rise in available computer power and the development of novel computer platforms will enable us to easily turn the huge volume of incoming data into an asset to be exploited, for the good of the nation.
Don't miss the eavesdropping Federal eagle wearing headphones and grasping network cables in its talons, and the borrowed (but real) TITAN supercomputer.

As long as you cannot hide anything, Citizen, you have nothing to fear.
posted by cenoxo at 7:57 AM on May 5, 2013


It does, and that hardware *is* widely distributed,

My point is not that hardware cannot be widely distributed, nor that no intercept hardware exists at all. That's obvious. My point is that wholescale recording can't be widely distributed and secret.
posted by kiltedtaco at 8:04 AM on May 5, 2013 [1 favorite]


This doesn't look very secret to me.
posted by mikelieman at 8:06 AM on May 5, 2013 [7 favorites]


symbioid: there's always those rumors about that NSA site in Nevada or wherever, from a couple years ago, from Wired Magazine
The wired article in question, "Inside the Matrix," was just a year ago. (Previously [Not Horn Tootist])

cf. "Fault Lines - Controlling the Web"—Al Jazeera English
posted by ob1quixote at 8:07 AM on May 5, 2013 [4 favorites]


Local calls in particular can be connected within a single central office on a single switch

This was certainly true in the electro-mechanical days, but aren't US offices now computerised?

I know in the UK it's essentially all packet-switched data now, and that's easy enough to "cc" to the spooks without customised hardware, at least in principle.
posted by bonaldi at 8:10 AM on May 5, 2013


> As Schneier recently pointed out, connecting the dots is only possible in hindsight,

I'm always surprised that so few Americans actually bothered to read the 9/11 commission report.

The report makes it very clear that there was more than enough information to connect the dots before 9/11 happened - that is, if people had been doing their jobs and not engaging in political infighting.

For example, as we all know, the US government received word that "Al Qaeda determined to strike in US" by "attacking Federal buildings in lower Manhattan, perhaps by hijacking planes." This went to the highest levels of the government, who neither acted nor disseminated the information.

At the same time, FBI agents had discovered that suspects their were surveilling were taking lessons on how to fly jets - but seemingly uninterested in how to land. They passed this on - marked as urgent - to their bosses, who also sat on their data, because of whatever childish feud they were having.

The 9/11 report is full of howlers like this. Of course, we can't know for sure, but we can know that people weren't doing their jobs and it really does look like if people had simply being doing their jobs according to their job description, they should have been able to at least mitigate if not entirely prevent the attack.
posted by lupus_yonderboy at 8:11 AM on May 5, 2013 [35 favorites]


Call we call this "Hoovering"? Pretty please?
posted by srboisvert at 8:13 AM on May 5, 2013 [6 favorites]


So, which is it? Bumbling fools or nefarious super villains? Make up your mind, you conspiracy-minded dolts.

Is it just fashionable to dismiss everything as a conspiracy theory nowadays? This isn't much of a conspiracy; the government has all but explicity stated that this is what is happening. We know that telcos have been funneling data to the government for the last decade--hell, Congress passed a bill of retroactive immunity for them--so why is it some kind of tinfoil hat scenario to believe that they are actually storing all of this data, especially when someone with knowledge of the situation has just said so?

This is a far cry from believing that the FBI and the Mob consipred to whack Kennedy or that 9/11 was an inside job.
posted by Ickster at 8:16 AM on May 5, 2013 [28 favorites]


srboisvert: "Weren't there theories that intelligence agencies were swapping eavesdropping data between countries at one point? The U.S. would spy on the British and the British would spy on the U.S. and swap data as needed. That way no domestic laws were broken."

The NSA does something with the Radomes at Menwith Hill and GCHQ has shared intel with the NSA since WWII (and with the KGB, but that was unintentional).
posted by Auz at 8:18 AM on May 5, 2013


a data storage facility for the United States Intelligence Community that is designed to be a primary storage resource capable of storing data on the scale of yottabytes

We're now seriously speaking about yottabytes? We're going to need a bigger prefix.
posted by justsomebodythatyouusedtoknow at 8:18 AM on May 5, 2013 [3 favorites]


> Although it sounds impressive, 62 PB is really not that much these days.

As of right now, you can get a 3TB drive from Newegg for $130. 62PB is very roughly 20,000 of those drives, or about $3 million dollars in raw commodity drives.

Now, in some ways that's the tip of the iceberg - I'd (somewhat generously) triple that number for backups; double it on top of that to make sure that all the disks are housed somewhere and connected to a network; double it to account for graft and corruption; and then double it again for good luck.

I'd say that's a more than generous estimate - and that's still less than $100 million. There are many Pentagon programs in any year that overrun their costs by that much...
posted by lupus_yonderboy at 8:21 AM on May 5, 2013 [7 favorites]


This doesn't look very secret to me.

Some would say that the aliens at Area 51 aren't very secret either.

This was certainly true in the electro-mechanical days, but aren't US offices now computerised?

Why would having non-electromechanical switches require local calls to go outside the CO? "computerized" is not a very specific term, but switches have been non-electromechanical for the past 30 years. That's a different thing than packet switched. But still, the problem is that the network is decentralized and the hardware is generally old and fixed-purpose. I wouldn't be quite so quick to assume that the problems here are trivial.
posted by kiltedtaco at 8:21 AM on May 5, 2013 [1 favorite]


So, which is it? Bumbling fools or nefarious super villains? Make up your mind, you conspiracy-minded dolts.

Arguments based on a false dilemma commit fallacies of presumption. It is impossible to answer your question unless you know the all motives and agendas actually behind all actions, and not just professed. Government, almost by definition, is conspiracy in action.

The government isn't one entity, and even if it was, there are many sides.

Thank you for reminding everyone the "Gubmint" isn't an homogenized substance like Grade A milk.

Finally, on the original topic, the quote attributed to Cardinal Richelieu is the perfect comeback to the "Nothing to hide, nothing to worry about" apologists":

Give me six lines written by the most honorable of men, and I will find an excuse in them to hang him.
posted by Enron Hubbard at 8:22 AM on May 5, 2013 [5 favorites]


High time for end-to-end strong crytpo on your smart phone. Unless they get a bug into the software or physical phone, of course.
posted by Old'n'Busted at 8:23 AM on May 5, 2013


Eh, so if Dead Brother Bomber was on a watch list, and his comms were all sniffed and scooped by the Government's Secret Comm Scooper and available for analysis, why didn't they nab him before he and his brother blew up the marathon?
posted by notyou at 8:26 AM on May 5, 2013 [2 favorites]


Why would having non-electromechanical switches require local calls to go outside the CO?

It wouldn't -- but it would make it very much easier to have the calls copied to a recording facility via nothing more complicated than software, certainly without having to install custom hardware in every site.

By way of analogy, my home router is letting me FTP from my laptop to my server downstairs right now, and that doesn't need to cross the internet in any way. But there's nothing to stop my router having some firmware installed that would change that. And if I was co-operating, it would be trivial.
posted by bonaldi at 8:27 AM on May 5, 2013


Most civil liberties protections are enforced by (1) exposure of public officials to civil litigation by those whose rights are being violated and (2) exclusion of evidence from criminal trials of those whose liberties were violated.

#1 does very little. Terrorists still in the planning stages, or who have taken overt action but haven't been caught aren't going to sue themselves, and the courts have prevented suits on their behalf while they're on the run.

#2 is increasingly irrelevant in a world where the policy focus or expectation is that the military or police will finish their jobs with a dead, rather than living, terrorist. Drones with Hellfires and SEAL Team 6 shooting on site are Obama's solution to his lawyers' discomfort with Bush policies. The only reason there's even one Boston bomber to prosecute is because he shot himself in the throat and managed to live, anyway.

Civil libertarians need to rethink their approaches in this new paradigm.
posted by MattD at 8:35 AM on May 5, 2013 [2 favorites]


This is why I plan all my terrorist attacks using scrolls with wax seals.
posted by Damienmce at 8:39 AM on May 5, 2013 [4 favorites]


stbalbach: "The problem is not having data but having too much data creating statistical false positives."

This isn't a problem for post incident analysis as there isn't any chance of a false positive.

Cool Papa Bell: "This is the same government that didn't know the WMDs were fiction, couldn't organize a response to Katrina and, despite magical phone-recording tech, still has a ten most-wanted list.

So, which is it? Bumbling fools or nefarious super villains? Make up your mind, you conspiracy-minded dolts.
"

The US government has hundreds of thousands of employees; I'm willing to believe that collectively their competence spans a spectrum.
posted by Mitheral at 8:49 AM on May 5, 2013 [1 favorite]


I sort of thought this was common knowledge, it's certainly not been something that's a big secret across a lot of the phreaker/hacker groups that have been circulating over the last 10 years?

The news was always 614A, but from a practical perspective if you know the history of telecoms in this country, you can absolutely see how this happens. Read up on AT&T long lines, and the history of the industry.

The US government has hundreds of thousands of employees; I'm willing to believe that collectively their competence spans a spectrum.

This is absolutely true, and it's less likely that incompetencies are going to run rampant in less public agencies. Also, governments sub contract a TON of work out to large businesses, who in fact run whole separate divisions (IT, HR, ETC) handling tech for their government customers.
posted by iamabot at 8:56 AM on May 5, 2013


Read the wired article linked in the comments.
posted by Annika Cicada at 8:56 AM on May 5, 2013 [2 favorites]


Eh, so if Dead Brother Bomber was on a watch list, and his comms were all sniffed and scooped by the Government's Secret Comm Scooper and available for analysis, why didn't they nab him before he and his brother blew up the marathon?

Maybe because it's far easier to collect the data than to sort through it all. Once the attack happens you can focus all your resources on unwinding the relevant information from a known starting/end point.
posted by The Hamms Bear at 9:07 AM on May 5, 2013 [1 favorite]


Tamerlan was not just an average citizen, the FBI was alerted by a foreign government about possible links to terrorism which resulted in an in-person visit to the family. If there was a watch list of recorded numbers, he was on it. This is not surprising, what is also not surprising is that even with this level of surveillance the bombing was still missed.

But I would be surprised if this was really for preventative measures, stopping a terrorist attack by a very small group (of two!) individuals is incredibly hard and will always rely on someone close to the attackers rooting them out.

What's important is after the fact determining if there's more attacks planned or if this was part of a broader conspiracy.

While I don't agree with eavesdropping by government agencies, it is trivial to get around it. My Google and Facebook connections are all HTTPS, and while I have no reason to trust Google and Facebook, if I really wanted transmit secure information I'd encrypt it and be done with it. It is not that hard to defeat eavesdropping.
posted by geoff. at 9:12 AM on May 5, 2013


My point is not that hardware cannot be widely distributed, nor that no intercept hardware exists at all. That's obvious. My point is that wholescale recording can't be widely distributed and secret.

Yeah, I see what you mean, but... How 'secret' does it really need to be? I mean, if you're a technician working at a CO and an order comes in to set up a circuit and colocate a box for 'lawful intercept' purposes, how would you really know how the government is using that equipment? I'm pretty sure you'd get fired if you unplug their circuit and insert a sniffer on it.
posted by Juffo-Wup at 9:18 AM on May 5, 2013 [1 favorite]


justsomebodythatyouusedtoknow: "a data storage facility for the United States Intelligence Community that is designed to be a primary storage resource capable of storing data on the scale of yottabytes

We're now seriously speaking about yottabytes? We're going to need a bigger prefix.
"

That would be Hella.
posted by symbioid at 9:21 AM on May 5, 2013 [2 favorites]


notyou: "Eh, so if Dead Brother Bomber was on a watch list, and his comms were all sniffed and scooped by the Government's Secret Comm Scooper and available for analysis, why didn't they nab him before he and his brother blew up the marathon?"

Can you imagine how many people they have caught and they just aren't telling us? Every once in a while one slips through, but, there must be thousands, perhaps even millions of terror attacks stopped each year by this great network of databases and intelligence gathering we spend so much money on... Surely we wouldn't spend all these billions (trillions?) on it without a cost-benefit analysis that we've saved that much in damage, would we?
posted by symbioid at 9:23 AM on May 5, 2013 [1 favorite]


Eh, so if Dead Brother Bomber was on a watch list, and his comms were all sniffed and scooped by the Government's Secret Comm Scooper and available for analysis, why didn't they nab him before he and his brother blew up the marathon?

Is there any evidence he had previously discussed the plot on a phone call or in any other electronic communication? The call in question, to the widow, took place days *after* the bombing. There may have been no digital trail.

He was on a watch list with half a million other people, that's a lot of haystack.
posted by RobotVoodooPower at 9:31 AM on May 5, 2013 [1 favorite]


So, which is it? Bumbling fools or nefarious super villains?

It may be the distinction between, "I believe in Area 51" versus "I believe Area 51 is where they keep the alien spacecraft." (Some of you might be too young to remember when people didn't believe Area 51 existed.) It's one thing to believe that the government is capable of recording massive amounts of information, and quite another to believe it's capable of processing that information and doing much useful with it.
posted by cribcage at 9:35 AM on May 5, 2013 [3 favorites]


Is it just fashionable to dismiss everything as a conspiracy theory nowadays?

For some people, when confronted with the facts that parts of their freedom and privacy are as illusory as our former Bill of Rights, cognitive dissonance engenders disbelief.
posted by Podkayne of Pasadena at 9:37 AM on May 5, 2013 [3 favorites]


GSM 06.20 is a pretty good codec (2 channels @ 6.5kb/s). Assuming no framing overhead, at 5 hours per day per person and 330m people, that's 1.64M TB/year. Not believable even if you scale it down...
posted by rr at 9:38 AM on May 5, 2013


Surely we wouldn't spend all these billions (trillions?) on it without a cost-benefit analysis that we've saved that much in damage, would we?

Without being conspiratorial about it, and understanding that a lot of people on both the private and public sides of the deal think they are doing what is right and necessary for the good of the United States, this is the security-industrial (post-military-industrial) complex at work.
posted by immlass at 9:44 AM on May 5, 2013 [2 favorites]


Assuming no framing overhead, at 5 hours per day per person and 330m people, that's 1.64M TB/year. Not believable even if you scale it down...
I
I'm sure there are people that are on the phone 5 hours a day, but I dont think I use the phone 5 hours a month. On my cellphone I used 118 last month and even at the office I'm not a Chatty Cathy. So yeah this isn't a lot of data to store. It is a lot of data to do anything actionable with.
posted by birdherder at 9:46 AM on May 5, 2013 [3 favorites]


One nation, under surveillance . . .
posted by ahimsakid at 9:47 AM on May 5, 2013 [4 favorites]


The hoarding thread is still open, people.
posted by StickyCarpet at 10:07 AM on May 5, 2013 [1 favorite]


There was an FBI box at the last telecom I worked at, which was almost never used (I know, because we were monitoring traffic to it)
posted by empath at 10:13 AM on May 5, 2013 [2 favorites]


The news was always 614A, but from a practical perspective if you know the history of telecoms in this country, you can absolutely see how this happens. Read up on AT&T long lines, and the history of the industry.

I pay a good deal of attention to the history of long lines and have no idea what you are suggesting is so obvious. I guess if you are just willing to assume government omnipotence because the bell system was smart and sophisticated, then maybe none of this is news to you, but I think it's worth putting a little more critical thought into these claims instead of taking them at face value. There are ridiculous conspiracy theories and there are secret government operations we don't know about, and when all we have is a comment from one FBI agent it's damn hard to tell which side this falls into.
posted by kiltedtaco at 10:25 AM on May 5, 2013 [1 favorite]


As I understand it, the CIA and NSA pretty much do whatever they like, mostly ignoring the constitution. Yes, the NSA probably records all the phone calls. Just read about Bill Binney or watch him speak (29c3 previously).

Ain't ideal but the country remains livable so long as the CIA and NSA consider only foreign nationals interesting. There is a massive problem with ordinary law enforcement like the FBI receiving this information though, well the FBI will happily call your protest terrorism so they can find ways to arrest all the organizers.

At present, the FBI normally needs a FISA warrant to access the spook's data, including old telephone calls. I presume CISPA raison d'etre was granting the FBI a truck sized loophole to bypass FISA warrants. In this way, the FBI could do their own "big data" analysis domestic surveillance without depending upon what the NSA told them they needed to know.
posted by jeffburdges at 10:27 AM on May 5, 2013


Data storage is one thing - you can get a lot of off-the-shelf hard drives. But having those drives in a usable setup is another thing entirely - you're talking processors, racks, cables, cooling, physical space, manpower to keep setting up and error-checking and plugging in enough drives to store exabytes per month, and the electricity to power all of that. That's where it starts to get a bit much to believe.
posted by kafziel at 10:48 AM on May 5, 2013


For those concerned about how to tap a decentralized system of CO's:

http://en.wikipedia.org/wiki/Communications_Assistance_for_Law_Enforcement_Act

That problem was identified well before 1992, the solution was reliably implemented by 2007-ish.
posted by Annika Cicada at 10:53 AM on May 5, 2013


There are ridiculous conspiracy theories and there are secret government operations we don't know about, and when all we have is a comment from one FBI agent it's damn hard to tell which side this falls into.

The current technological head of the CIA has clearly stated in public "we want to do this, and we're getting close to doing it." We've been doing it for at least a decade and we've demonstrated that we're willing to commit many more dollars to it. Maybe we're only 20% of the way there today. Who cares?

Is the ridiculous conspiracy theory that the FBI is able to play in the CIA's sandbox under certain circumstances? I don't understand what's ridiculous.
posted by RobotVoodooPower at 11:13 AM on May 5, 2013 [2 favorites]


The problem is not having data but having too much data creating statistical false positives.

This isn't a problem for post incident analysis as there isn't any chance of a false positive.


This is exactly the sort of thing you say until it turns out that you're the guy who delivered a pizza to these guys three times in the last month. Oh, and look here in your credit card records.... Maybe you should just tell us everything you know. Things will go better for you if you do.
posted by Kid Charlemagne at 11:18 AM on May 5, 2013 [1 favorite]


Surely we wouldn't spend all these billions (trillions?) on it without a cost-benefit analysis that we've saved that much in damage, would we?

Those billions of dollars, is it possible that someone finds that revenue profitable enough to spend money lobbying and convincing people that their equipment and services are beneficial?
posted by spaceman_spiff at 11:20 AM on May 5, 2013 [1 favorite]


And recently: Senior Obama administration officials have secretly authorized the interception of communications carried on portions of networks operated by AT&T and other Internet service providers, a practice that might otherwise be illegal under federal wiretapping laws.

That's the carrot, here's the stick: Government Seeks to Fine Companies for Not Complying With Wiretap Orders
posted by homunculus at 11:20 AM on May 5, 2013


5 hours a day? 365 days a year? I don't think I've ever met anyone who spent so much time on the phone - except salespeople (a small portion of the world). I didn't find anything authoritative on this, but I found estimates online ranging from 20 minutes a day to 90 hours a year (which is about the same...)

It's also important to remember that you can a priori eliminate most of the big call centers, which have to occupy some vast chunk of the "telephone call space" - because these calls are already being recorded by their companies, and because there appears to be no way that you could actually use these numbers to foment rebellion or terrorism (not only is everything being recorded but the employees aren't even allowed to originate outgoing calls). It's likely that that eliminates over half the time - certainly looking at my personal calls, while there are a lot of calls to individuals, in terms of minutes most of the time goes to a few long calls, to order things, to get technical support, and that sort of thing.

If I were some crewcutted evil paranoid spy doing this, I might not totally drop even that huge stream on the floor but pass even that data through a matcher that flagged e.g. suspicious topics or conversations in Arabic before discarding it.

Non-business person-to-person only conversations are probably not even 10% of the whole voice stream data flow, but probably contain over 95% of the information that they are looking for.
posted by lupus_yonderboy at 11:20 AM on May 5, 2013 [1 favorite]


You know how much Nixon would have salivated over having access to something like this?
posted by JHarris at 11:25 AM on May 5, 2013 [3 favorites]


The amount of skepticism here is just hilariously naive. The NSA has poured billions of dollars into datacenters - how many weeks of MP3s fit on your god damn iPod?

There have now been a number of people who would know who've pretty explicitly said, they are recording everything:
After he left the NSA, Binney suggested a system for monitoring people’s communications according to how closely they are connected to an initial target. The further away from the target—say you’re just an acquaintance of a friend of the target—the less the surveillance. But the agency rejected the idea, and, given the massive new storage facility in Utah, Binney suspects that it now simply collects everything. “The whole idea was, how do you manage 20 terabytes of intercept a minute?” he says. “The way we proposed was to distinguish between things you want and things you don’t want.” Instead, he adds, “they’re storing everything they gather.” And the agency is gathering as much as it can.
posted by crayz at 11:27 AM on May 5, 2013 [4 favorites]


Data storage is one thing - you can get a lot of off-the-shelf hard drives. But having those drives in a usable setup is another thing entirely - you're talking processors, racks, cables, cooling, physical space, manpower to keep setting up and error-checking and plugging in enough drives to store exabytes per month, and the electricity to power all of that. That's where it starts to get a bit much to believe.

Right because distributed/fault-tolerant scalable storage is a totally untackled problem in the history of billion-dollar datacenters.

I mean, you realize *one small component* of Flame/Stuxnet made use of a novel MD5 collision - invented and deployed for that sole purpose? The idea that these people don't have the skills needed to store some fucking audio files on hard drives is just ... wow. Really, wow.
posted by crayz at 11:31 AM on May 5, 2013 [8 favorites]


As Schneier recently pointed out, connecting the dots is only possible in hindsight,

Schneier is a dunce. The moment we stop considering him an expert just because he happens to provide commentary that agrees with a certain worldview, the wiser we will all be.
posted by gjc at 11:31 AM on May 5, 2013 [1 favorite]


The problem is not having data but having too much data creating statistical false positives.

This isn't a problem for post incident analysis as there isn't any chance of a false positive.


Bruce Schneier: Why FBI and CIA didn't connect the dots
posted by homunculus at 11:32 AM on May 5, 2013 [2 favorites]


5 hours a day? 365 days a year? I don't think I've ever met anyone who spent so much time on the phone - except salespeople (a small portion of the world). I didn't find anything authoritative on this, but I found estimates online ranging from 20 minutes a day to 90 hours a year (which is about the same...)

I'll tell you who talks on the phone that much: a lot of attorneys. Who would have their own issues with all phone calls being recorded.

I'm skeptical.

And, yeah, the telcos that have lawful intercept boxes installed still have to do certain things on their end to ensure the traffic being intercepted is routed through the box. The FBI boxes, at least, are not passively capturing all traffic.

NB: Most call information funneled to law enforcement by telcos isn't real-time voice calls but is instead call records that don't require an intercept box.
posted by devinemissk at 11:46 AM on May 5, 2013 [1 favorite]


The small VOIP company that I used to work at captured 100% of all traffic to all of our servers for troubleshhoting purposes and held it for three days to a week. The data was searchable by time and the phone numbers involved, and it took just a few minutes to listen to a call after we found it (something we did if we were troubleshooting quality problems and had customer permission). Because of the way we captured it, we also captured a lot of our corporate traffic including emails and IMs, which were also indexed and searchable.

The box that did all this was about $20k, but you can do it with open source tools for significantly less. If the NSA was only storing data for a limited time and then saving only calls from people of interest for the long term, the storage costs for storing a big chunk of Internet traffic don't seem out of the realm of the possible.

There's no chance that the us government is capturing 100% of Internet traffic, just because I've worked at ISPs and I know what was connected to our gear, but I can imagine having some strategic port mirrors configured at various interconnects that can capture a significant percentage of it.
posted by empath at 11:55 AM on May 5, 2013 [2 favorites]


I pay a good deal of attention to the history of long lines and have no idea what you are suggesting is so obvious

AT&T (the original) and the Bell system have a long history of being friendly to the government at the expense of their customer's privacy.

For example, they used legal "diagnostic" call monitoring on the phone lines of mob bookies in the 60s, and forwarded that information on to the FBI.
They installed "greenstar" units behind locked doors at switching offices across the country where lots of trunk lines came together.
They also weren't above calling in a favor from law enforcement to harass or otherwise intimidate phone phreaks.

Which is not to say what this FBI agent claims is true, but if it is happening, I'd bet AT&T didn't fight all that hard.
posted by madajb at 12:12 PM on May 5, 2013


So, which is it? Bumbling fools or nefarious super villains?

I vote for bumbing super villains.

Exploding cigars, cats with listening devices, and paying for people to stare at goats.

The super villain part is taking your money under threat of force as an example.
posted by rough ashlar at 12:26 PM on May 5, 2013 [2 favorites]


The costs of storing Everything that passes through the choke points of the internet is small compared to the secret budgets that the NSA, CIA, et. al. are estimated to have.

Phone calls start out as 8k bytes/second before compression, etc. If half of the US called the other half, we're looking at about 160,000,000 calls * 8k --> 1280,000,000 k/second... or about 1300 Gigabytes/second... MAX. If those were store raw, spread across 200 Mb/second terabyte hard drives (0.2 Gb/sec), you'd need 6,400 drives to do it. At $100/drive (buying in bulk, and allowing for power supply, RAID controllers, networking, etc) you're looking at less than a million dollars... to record every single call all at once, in the USA.

Now, most internet traffic is going to be compressible, have redundancies beyond belief, and compress way better than voice ever could. Effectively that only things you actually type and pictures / video you send are the only real traffic that matters. All else is the output of a copy. Everything anyone reads, someone typed and uploaded once. All videos watched, same thing, etc.

I believe it is WELL within the capabilities of the governments of the world to record everything. It will get cheaper, and soon the ISPs will be able to do the same, if motivated.
posted by MikeWarot at 12:27 PM on May 5, 2013


Ya'll understand that AT&T has been selling as a service automated transcripts of conference calls from the late 1980's right?

Now if Google wants to be on the same level as AT&T WRT voice to text - google voice is a great way to get lots of real world voice samples.
posted by rough ashlar at 12:33 PM on May 5, 2013


r. a. your tinfoil hat needs another layer buddy

Crypto Anarchist Manifesto
posted by bukvich at 12:36 PM on May 5, 2013


It's like they're trying to make everything into the fruit of the poisonous tree.
posted by one more dead town's last parade at 12:37 PM on May 5, 2013


I'll tell you who talks on the phone that much: a lot of attorneys.

And as an attorney, I'll tell you who else talks on the phone that much: a lot of cabbies.
posted by cribcage at 12:41 PM on May 5, 2013 [3 favorites]


blah blah former blah blah I'm not believing

So what would it take for you to accept what is being claimed?

In the case of 'its being recorded' you had 'present at the time' employee who made a claim. Not good enough?

In the case of "Bunny" - the ex-military gal who blew the whistle on military procurement or Sibel Edmends - these were whistleblowers. How's whistle blowing working out for people in the present administration? How well has whistle blowing worked out in the past or do snitches still get stitches *bleet*?

If you refuse to accept a claim due to the person making the claim being an "ex" - how high a bar does a whistle blower need to reach for you to believe?
posted by rough ashlar at 12:42 PM on May 5, 2013


You know what would be a great conspiracy therory? That the internet/cellphones was designed tho take the place of landlines and the postal system to eventually make survalence easier.

Hell, I'm the furthest person to plan anything violent by myself or with others, but if I was such a person, cash, post office for letters, and driving oldest cars for transportation would be the thing... Better yet, only face tho face conversations.
posted by edgeways at 12:43 PM on May 5, 2013


Personally, I just want to know their naming convention for all that data...
posted by Nanukthedog at 12:43 PM on May 5, 2013 [4 favorites]


You know what would be a great conspiracy therory? That the internet/cellphones was designed tho take the place of landlines and the postal system to eventually make survalence easier.

The telephone companies have made pitches to the FCC to stop doing new Copper line POTS by the end of this decade and next decade force the remaining Copper POTS to VOIP.

Why?

To shake off the rules that govern voice on Copper. Tariffed lines are a compliance issue.

So one could say there is a reason. But its not hidden. Well, it may be hidden from you because you didn't know. Just because you didn't understand, does that make it a conspiracy?

(VOIP does seem to have less protection WRT admission into court than a POTS line conversation - but its hard to find records to point to to back your theory of a conspiracy.)
posted by rough ashlar at 1:01 PM on May 5, 2013 [1 favorite]


The largest storage systems built by companies like Google, Facebook, Microsoft, and Amazon only top out at about 300-1000 petabytes. US internet traffic is now probably close to 4,000 petabytes per month, so if the NSA were building a Google-size datacenter every month they could capture not even 25% of just the US internet traffic.

And storage is just the small problem: the bigger problem is processing this data.
posted by Pyry at 1:02 PM on May 5, 2013 [2 favorites]


the bigger problem is processing this data.


Soon, governments won't have to bother collecting personal data. We're willingly giving it to a vast network of for-profit data collectors, and they're more than happy to pass it on to the government without our knowledge or consent.

VS

Millions of people behave strangely enough to warrant the FBI's notice, and almost all of them are harmless. It is simply not possible to find every plot beforehand, especially when the perpetrators act alone and on impulse.
posted by rough ashlar at 1:24 PM on May 5, 2013 [1 favorite]


Obviously all internet traffic is a different order of problem than all phone traffic. But even there the vast majority of traffic on the internet is video from places like youtube and xvideo and there isn't any need to capture the contents of that traffic.
posted by Mitheral at 1:44 PM on May 5, 2013


The largest storage systems built by companies like Google, Facebook, Microsoft, and Amazon only top out at about 300-1000 petabytes

Why do you believe that the highest levels of national security use the same commercially available hardware as is available to the rest of us?

The military was using digitally enhanced photo technology able to recognize license plates on cars from orbit literally decades before such technology was available to any non-military entity.

If you do not believe that your government is listening in (via automated monitoring systems) to literally every telephone conversation out there then you haven't been really reading the news over the past 13 years. On the other hand, considering the deplorable state of American journalism these past 20 years, anyone's ignorance of the situation is likely understandable.
posted by Podkayne of Pasadena at 1:45 PM on May 5, 2013 [1 favorite]


But even there the vast majority of traffic on the internet is video from places like youtube and xvideo and there isn't any need to capture the contents of that traffic.

If you don't look at youtube videos, then terrorists will just use youtube videos to communicate. As soon as you say you don't need to analyze something, that something becomes your blind spot. Furthermore, computer vision is not nearly good or fast enough to actually analyze video on that scale. And if you think the government has some secret vault of computer vision algorithms, then it's a bit odd that DARPA keeps handing out computer vision grants.

Why do you believe that the highest levels of national security use the same commercially available hardware as is available to the rest of us?

I believe the government could hire a handful of the best masons in the world in secret; I don't believe the government could build a Great Wall in secret. This isn't a matter of driving down to office max and buying a few harddrives-- to capture just the US internet traffic would mean building a datacenter larger than Google's, Microsoft's, and Amazon's combined every month.
posted by Pyry at 2:33 PM on May 5, 2013


ctrl-F "Carnivore" .... nada

ctrl-F "Poindexter" .... nada

Wow. I'm sorry but I consider you all uninformed on the acknowledged government efforts to record our calls.
posted by surplus at 2:42 PM on May 5, 2013 [1 favorite]


Binary string de-duplication.
posted by Annika Cicada at 2:44 PM on May 5, 2013 [1 favorite]


There are so many opportunities for compression in the dataset of "all internet traffic the NSA might have direct access to" that it doesn't seem that unreasonable to assume that a large fraction of traffic is recorded. I don't necessarily believe that any useful analysis is being done with it yet, but I don't find it hard to believe that there is a database that can correlate a record of my browsing history with some sort of historical Google index type of structure to recreate my browsing session without having to actually record it directly. I'm not sure if this qualifies as "all internet traffic" or not, but a de-duplicated set of response data plus a record of everyone's HTTP requests gets you pretty close in a practical sense, and with the budgets involved, doesn't seem that unreasonable.
posted by feloniousmonk at 3:07 PM on May 5, 2013


Why do you believe that the highest levels of national security use the same commercially available hardware as is available to the rest of us?

Because it gives the most bang for the buck -- it takes the economic advantages of mass production only available from consumer adoption to lower the price of the technology to where a government organization can afford to buy lots of them. Contrary to popular belief, these organizations don't have unlimited resources.
posted by JHarris at 3:17 PM on May 5, 2013


The largest storage systems built by companies like Google, Facebook, Microsoft, and Amazon only top out at about 300-1000 petabytes.

You can buy up to 1.6 petabytes for storage/analysis, retail on Amazon Redshift.

The idea that Amazon is storing less than 1000 petabytes total is an absolute joke.
posted by crayz at 3:33 PM on May 5, 2013 [2 favorites]


Storage density increases faster than Moore's Law. And most of the data will likely be archived, as opposed to Google, which optimizes for latency.

A tape robot today can store exabytes of data using comparatively little power. Future storage voodoo could increase that even further.

Are we just arguing about whether the NSA can currently store 100% of internet traffic, or just 0.1% with a roadmap for the future? Kinda splitting hairs.
posted by RobotVoodooPower at 3:40 PM on May 5, 2013 [1 favorite]


My understanding of amazon web services is that they calculate the estimated global datacenter compute and storage capability and stay ahead of that.

I will deny knowing anything if you ask for a cite.
posted by Annika Cicada at 3:42 PM on May 5, 2013


There doesn't seem to be any reason to just talk about the cost of hard drives - you could do these things with a system that achieves a one-way dump to long term storage, buffering with hard drives and draw the buffer down overnight while there's less activity. They could have started recording everything long before they had the capacity to analyze and search it all. Here's a 1 PB tape backup system from early 2009.
posted by XMLicious at 3:47 PM on May 5, 2013


ALOT of ordinary harmless folk spend 5 hours a day and more yammering into their cell phones. After reading this, I have to reconsider my contempt towards all those fools endlessly nattering about what they bought on sale at Costco, and the boils on their arses -
It pleases me to contemplate gubmint hard drives filling up with all that endless useless blather, and black hatted spooks going through it all with fine toothed, virtual combs.
posted by Abinadab at 4:17 PM on May 5, 2013


surplus: what about Echelon?

or FinSpy? (More Schneier!)

Electronic Frontier Foundation: Report: Obama Officials Authorized New 'Cybersecurity' Warrantless Surveillance Program, Fresh Immunity Given to ISPs
posted by the man of twists and turns at 4:47 PM on May 5, 2013 [1 favorite]


It's funny, traditional IT types vastly overestimate the difficulty of storing petabytes of data, and novices only slightly underestimate the difficulty. For something like storing all telephone conversations, the feature requirements are extremely low, as most data is almost never accessed, and losing even 0.1% of your data isn't a huge problem. The indices on the data are minimal, there's no need for a unified file system namespace, and really it's just warehousing petabytes of data. An exabyte may sound like a lot, but it's not a blip on the radar of hard drive shipments, or on NSA budgets (when engineered to actual requirements rather than "enterprise" needs. I guarantee that the NSA isn't using EMC for this). I could personally engineer a solution to this, given a team of people to swap out hard drives and lift the tons of JBODs into racks, and I'm nowhere near as talented or smart as the only person I know to have been interested in working at the NSA (they didn't hire him).

The data transfer from telcos and the actual desire to store everything (as opposed to a subset of the population) are far better reasons to be skeptical of the total surveillance idea.
posted by Llama-Lime at 5:09 PM on May 5, 2013 [1 favorite]


Amount of data the gubment would need to store to have Tsarnaev's calls on record:

1.7 exabytes

That's a lot, but not impossible.
- Salvor Hardin
1.7 exabytes would take 1.7 million 1 terrabyte hard drives. If you could buy and install them for $100 each, it would cost $170 million dollars to build a data center with that capacity. Not cheap, but a drop in the bucket for the NSA. And they can probably get components wholesale for much lower costs. And that figure was sans-compression.
And that assumes the NSA doesn't have automatic transcription. Get all those phone calls into text files and the data storage requirements drop through the floor.-- Mitheral
It would actually be much smarter to simply record everything. A lot of times it's cheaper to throw hardware and data at a problem then come up with really good software that figures everything out.
My point is not that hardware cannot be widely distributed, nor that no intercept hardware exists at all. That's obvious. My point is that wholescale recording can't be widely distributed and secret. -- kiltedtaco
No one is saying it's secret.
Eh, so if Dead Brother Bomber was on a watch list, and his comms were all sniffed and scooped by the Government's Secret Comm Scooper and available for analysis, why didn't they nab him before he and his brother blew up the marathon?-- notyou
The FBI said he was on some list, but that he was removed after a few years because there wasn't any evidence against him.

But let's suppose he was on 'a list'. How many people do you think they have listening to these calls? Probably not very many. Simply having a lot of data doesn't tell you anything. And while these two were pretty dumb, they may have been smart enough to reference it only obliquely on the phone, or not at all.
GSM 06.20 is a pretty good codec (2 channels @ 6.5kb/s). Assuming no framing overhead, at 5 hours per day per person and 330m people, that's 1.64M TB/year. Not believable even if you scale it down... -- rr
Why is it not believable? Have you done the math on how much it would cost? It's in the hundreds of millions of dollars, and less then two one millionths of the data storage the NSA publicly claims to have.

Let me ask it this way: do you think the Chinese government is recording all it's citizens cellphone calls? How much do you think it would cost them to do that? If the Chinese government were capable of doing it, the US government probably can as well.
Schneier is a dunce. The moment we stop considering him an expert just because he happens to provide commentary that agrees with a certain worldview, the wiser we will all be. -- gjc
And I suppose you're fucking Einstein?
US internet traffic is now probably close to 4,000 petabytes per month, so if the NSA were building a Google-size datacenter every month they could capture not even 25% of just the US internet traffic. -- Pyry
How did "cellphone calls" become "all internet traffic". Obviously they are not recording all internet traffic. Why the fuck would they want a billion copies of "Gangnam Style"? The claim is that they are recording all cellphone calls
If you don't look at youtube videos, then terrorists will just use youtube videos to communicate. As soon as you say you don't need to analyze something, that something becomes your blind spot. -- Pyry
While that may or may not be true, it has no baring on whether or not the government can record all cellphone calls at this point in time. All terrorists have to do is encrypt their communications with open source tools, and they can communicate however they want. That has zero to do with cellphone calls
This isn't a matter of driving down to office max and buying a few harddrives-- to capture just the US internet traffic would mean building a datacenter larger than Google's, Microsoft's, and Amazon's combined every month. -- Pyry
AND AGAIN, NO ONE IS CLAIMING THEY ARE DOING THAT

Seriously what is going on here?

Someone from the FBI said they had the ability to go back and access any phone calls that took place in the past. How does "there's no way they're recording all internet traffic" in any way refute that?

It's so bizarre.
posted by delmoi at 5:11 PM on May 5, 2013 [3 favorites]


> There's no chance that the us government is capturing 100% of Internet traffic, just because I've worked at ISPs and I know what was connected to our gear,

They don't have to be in your piddling little ISP - they just need to be tapping the backbones...

But I agree they aren't recording all internet traffic... most email flows in the clear, though, I would imagine that there's no reason at all that they couldn't be intercepting and storing all your emails, at the very least the text part.
posted by lupus_yonderboy at 5:12 PM on May 5, 2013


As far as I can tell the arguments this isn't happening.

1) OMG PETABYTES! NO WAY
2) They can't be tapping all internet traffic (i.e. OMG YOTTABYTES NO WAY!)
3) If you can't capture all communications, there's no reason to bother trying to capture any of it.
4) They're not doing this because it can't be done in secret.

In the first case, it's just wrong. It wouldn't cost that much money to capture and record all phone conversations. Amazon sells data storage for 10 cents per gigabyte per month. If we assume an average of one hour of phone conversation per person per day at 8kB per second, that comes out to 365*3600*8 = 10,512,000 ~10GB per year per person, or $12 a person per year. Maybe around $4 billion for the whole population. And that's just if you use amazon's cloud services. If they were to use their own hardware, it would cost a lot less.

In case of #2, it's totally irrelevant. I don't even really understand why people brought it up. They don't need to record all internet traffic to record all cellphone calls.

In the case of #3, it's not an either or thing. The government isn't going to say "if we can't catch 100% of terrorists, we shouldn't even bother trying" Why would they?

#4: They aren't doing it in secret. There is a difference between "not publicly announced" and "actually secret"


If you want to say you think the government wouldn't do this because it would be unethical and that the FBI guy was just talking out of his ass, that would be fine. But arguing this is somehow technically infeasible is just bizarre.
posted by delmoi at 5:29 PM on May 5, 2013 [8 favorites]


I find it hard to imagine that all calls are being recorded, not so much because of the storage as because of the huge flow of data. Also, revealing a program like this is the sort of technique that can only be done once. Would it really be worth compromising it for the sake of prosecuting a bomber's co-conspirator?

On the other hand, I can totally believe that saying "we have all your phone calls anyway; we'll prosecute you for obstructing an investigation unless you tell us everything" is an effective way to pressure people into confessing.
posted by Joe in Australia at 5:34 PM on May 5, 2013


i hope they are listening. because i've just always wanted to say "fuck you, mr. cheney."
posted by quonsar II: smock fishpants and the temple of foon at 6:06 PM on May 5, 2013


Pyry: "If you don't look at youtube videos, then terrorists will just use youtube videos to communicate."

You don't have to capture it because you can just ask the hosters for it at anytime.
posted by Mitheral at 6:33 PM on May 5, 2013


"In the City of San Francisco there is a door that only people with an NSA clearance can open. Through that door there is a room and in that room there is a major internet noden through which all internet traffic in a given region must pass. Near that internet node there is a device which captures every single packet which passes through that node."

A whaaat?? I have no idea what you're talking about.
posted by Room 641-A at 6:35 PM on May 5, 2013 [8 favorites]


Someone from the FBI said they had the ability to go back and access any phone calls that took place in the past. How does "there's no way they're recording all internet traffic" in any way refute that? It's so bizarre.

There is a subset of humanity that firmly believe things like Government and Corporations work for the betterment of man in what they do. So no matter how many incidents one points out where such is not true, they will support Govenrment and Corporations.

Then you have the lickspittles and brown nosers who work for 'establishment causes' who will fear for their jobs and won't rock the boat.

El Jefe at an Ohio graduation today had a pull quote of "Reject Voices That Warn About Government Tyranny". In the last century the Citizens were told about the KGB, the East German watchers and what Saadam was doing recording what the citizens were talking about. But somehow what was "bad". Now with that out of context pull quote - you should reject warning about blanket recordings as an example of tyranny?

And finally you have the people scared and will make statements in an attempt to 'look good' to the watchers.

Sgt. Ed Mullins of the New York Police Department - The way law enforcement agencies approach online activity that appears sinister is this: “If you’re not a terrorist, if you’re not a threat, prove it,” he says. He also opened his yap with “This is the price you pay to live in free society right now. It’s just the way it is,”

Yup - how are YOU going to prove to the jury that you are not a terrorist when you spoke up about blanket recording?
posted by rough ashlar at 7:32 PM on May 5, 2013


These data numbers posted here really aren't confidence-inspiring.

According to the CTIA Semi-Annual Wireless Industry Survey, there were 2.3 trillion minutes of cell phone calls in the US last year. That's 38.3 billion hours.

Now, I can't find a data source on what percentage of those calls are domestic. I would guess it's something like 90%. But to be safe, let's say 50%. So 50% of those 38.3 billion cell phone calls were made to another domestic cell phone, which means that you can cut the number of recorded hours down. 2 people on 2 cell phones talking for 1 hour would be 1 hour of recording, not 2. So that's 28.8 billion hours you need to record.

The Adaptive Multi-Rate audio codec (AMR) spec encodes toll-quality voice at 7.4 kbit/s. In reality, you could go down to 5 kbit/s no problem, but let's say that the NSA or whoever want to preserve fidelity and go with 7.4 kbit/s.

28.8 billion hours at 7.4 kilobits per second is 87.2 PB. That's to record every cell phone call made in 2012 in the United States.

In August 2011, IBM built the largest-known storage array in the world, at 120 PB.

If you don't think the US Government has the technical capacity to store every phone call ever made, you're dreaming.
posted by Jairus at 8:18 PM on May 5, 2013


Jairus: they could store it, but how do they get it to the data center? It literally means doubling the amount of data transmitted, and then getting it to their data center securely. That a lot of physical infrastructure, and they'd need to do all the construction secretly. This is a much harder problem than tapping individual transmissions or recording the numbers that people call.

Once it's at the data center, though, I would agree with you. In fact it's better than that: they have a huge library of existing speech and they don't need to process it in real time, so their compression can probably be much better than any codec presently in use. Plus,. they're really smart.
posted by Joe in Australia at 8:32 PM on May 5, 2013


I find it hard to imagine that all calls are being recorded, not so much because of the storage as because of the huge flow of data.
"huge" is not a number. How much data do you think this is? How much would it cost to get the bandwidth in place?

Once again, we have another argument that purports to be about numbers, but includes none.

8 kilobytes per second, per call. If we assume the average person is on the phone 5% of the day (over an hour) with 300 million people you have 8*3*108*0.05 you have 120 gigabytes per second. That's 4 OC-768 lines. And actually you get two people per stream so you need just 60Gbps, so you'd need two OC-786 lines.

In comparison, you tube streams over 4 billion hours of video a month. At, I think around 300kbps at low res. Can you do the math yourself? Spoiler alert: it's 200 gigabytes per second. Over 3x what's needed to record everyone if they talked on the phone an hour a day (which they don’t).

According to this the live streaming of Will and Kate's marriage required 800Tbps, (bits) or 100 TB/sec (bytes). That's over a thousand times as much bandwidth as it would take.

I don't understand this at all. How can you say It's "too much" bandwidth or storage or whatever, when you don't even bother to figure out how much bandwidth it actually is? It's not like it takes that much effort, it's just basic multiplication.
Jairus: they could store it, but how do they get it to the data center?
With fiber optic cables? The same way youtube gets video out on the internet? How is this even a hard problem?
Would it really be worth compromising it for the sake of prosecuting a bomber's co-conspirator?
Probably not, but what does that have to do with anything? This is something some random ex-FBI guy said on TV, presumably that means it's not even classified. (And what he said was that the data was stored by the phone company, not by the government anyway. If that was the case, bandwidth would be even less of a problem.
posted by delmoi at 9:01 PM on May 5, 2013 [1 favorite]


That a lot of physical infrastructure, and they'd need to do all the construction secretly.
Or, they could just do it, you know, not secretly.
The $2 billion data center being built in Utah would have four 25,000 square-foot halls filled with servers, as well as another 900,000 square feet for administration.

It will use 65 megawatts of electricity a year , with an annual bill of $40 million, and incorporates a $10 million security system.

Since 2001, the NSA has intercepted and stored between 15 and 20 trillion messages, according to the estimate of ex-NSA scientist Bill Binney. It now aims to store yottabytes of data. A yottabyte is a million billions of gigabytes. According to one storage firm’s estimate in 2009, a yottabyte would cover the entire states of Rhode Island and Delaware with data centers.
I mean, what you're saying makes no sense. They clearly have the bandwidth and facilities. They're not even trying to hide it and why would they?

Phone calls don't take up a lot of bandwidth. This is something that would require less bandwidth then youtube or megaupload.
posted by delmoi at 9:09 PM on May 5, 2013 [1 favorite]


This kind of interception would be done at the telco central offices, just like described in that Wired article, anyway. These places have incredible amounts of bandwidth available. The same applies to whatever attempts to capture internet traffic are being made, although in those cases it's at the major peering facilities rather than telcos, in the cases where they aren't the same thing in the first place. Either type of facility is basically a well provisioned data center by definition. These are also already the types of places with very high levels of security. If the NSA has a locked-down server cage with crazy security it wouldn't even stand out.
posted by feloniousmonk at 9:17 PM on May 5, 2013


I grew up near the Yakima Training Center in Eastern Washingont. It's the place in the Northwest were the military practices shooting/blowing stuff up. A couple miles onto the range is the Yakima Research Station an NSA facility surrounded by satellite dishes. Back in 1994 I spent two weeks out at the range taking pictures with with a National Guard armored unit. Everyone I asked said that was where the NSA recorded everybody's phone calls. Ever time I go to visit my mom I see the dishes from the freeway and wonder what exactly goes on there. I've heard that the Yakima Research Station will be shut down soon and a facility in Colorado will be taking over.
posted by the_artificer at 10:05 PM on May 5, 2013


Oh come on guys, think of the massive physical infrastructure and long-distance fiber capacity and all that Skype has to have to route tens of millions of calls around the world per day. /🍔
posted by XMLicious at 12:25 AM on May 6, 2013


Skype's a good comparison. It handles some small fraction of (mostly) long distance and international calls by piggy-backing on top of existing infrastructure. Let's say that it suddenly had a lot more business. It would need to provide more server space and so forth. Now, let's say that we wanted it to handle all calls; local, long-distance, and international. Would existing infrastructure cope? I suspect not - ISPs would have to specifically factor Skype into their capacity requirements, Skype itself would have to create new data centers, it would all be very obvious.

The idea that "the government" could record all calls everywhere is like the idea of having all calls handled by Skype. There's probably a lot of slack in existing infrastructure, but you suddenly have twice as much information (the original call plus the clandestine copy) being transmitted from every exchange. Also, that information needs to be concatenated, which means you either build your own black exchanges or you suddenly have even more information being routed through existing ones. And then you ultimately need it to come to your data center, so you have fat pipes pumping all that to you, plus clever techniques to break the fire-hose rush back down into individual conversations.

All this is definitely doable, but at the very least it represents a whole lot of capacity upgrades around the country, and unexplained capacity losses wherever you're using public infrastructure. It's not the sort of thing you can do subtly, and once you're doing this publicly then you have a whole lot of problems with illegal searches and privileged conversations between attorneys and clients. So I would classify this as technically possible, but in fact it isn't happening because we would know about it.
posted by Joe in Australia at 2:41 AM on May 6, 2013


I shouldn't have said "records all the phone calls" quite so literally. I'd expect they record the source and destination of all calls, emails, etc. to make a "connection graph", which imho already violates the constitution. Worse, they'd probably record all cell tower changes, meaning they record movement. From this data, they'd find potentially interesting numbers or regular calls, which speech-to-text software like CMU Sphinx could transcribe in real time.

For me, the question is : What do they consider interesting? I'd expect the CIA and NSA interests cover islamic groups, communists, certain foreign nationals, etc., but the FBI's interests include activists involved with the occupy movement, environmentalists protesting the Keystone Pipeline, etc.
posted by jeffburdges at 3:34 AM on May 6, 2013


They don't need to store all the data in one place. They can keep it at the point of capture for the most part.
posted by empath at 4:03 AM on May 6, 2013


I've worked in "enterprise" (big business) data storage for over 20 years now. I think it's quite technologically possible for government agencies to be storing all phone conversations for some period of time. But it would need some co-operation on the part of the telcos. The clients I deal with have Petabytes of data on primary (disk) storage, then there's the whole archive to tape thing, where we are looking at exabytes, and up, merely constrained by budget and floorspace.

The latest trends I deal with are deduplication, then compression, and automatic tiering (automatically moving cold data to cheaper media, such as slow cheap disk, or tape). And simple monotone low bit-rate phone conversations are prime candidates for this type of data reduction.

As for the filesystem, I'd imagine this kind of stuff would go to some kind of content storage system, where filesystems are kind of irrelavent. A phone conversation might just be referred to as "source phone number-target phone number-date-time". Or just a string of digits, with the identifying tags attached in the metadata.

There's also data mining software, where the aim is to sort through all this data to find the parts that are interesting or useful. Google "big data" and "data analytics" and "global namespace".

These are the things I deal with day to day in my corporate life. Of course the government would be using these technologies. And, well, isn't it pretty much accepted that government agencies are using technology that is well ahead of what we see in the corporate world?

I dunno, I'm not a conspiracy theorist. I'm just giving my opinion based on what I see day to day.
posted by Diag at 4:05 AM on May 6, 2013 [2 favorites]


This is something some random ex-FBI guy said on TV, presumably that means it's not even classified.

And if it is classified, "so what". Any concerns can be 'debunked' with statements like 'he's ex-FBI ... why not say something when he was a working FBI agent' or even dissect the language with 'What does "record each phonecall" mean? History shows each phone call is recorded for billing purposes, that is what John Draper (Capt. Crunch) exposed to the FBI decades ago.'

The art of communication and deception is well studied and all around, so if any 'secrets' are 'leaked' the leak is just papered over with "Public Relations".
posted by rough ashlar at 4:17 AM on May 6, 2013


I'm not a conspiracy theorist.

Your framing of that statement makes it appear like a bad thing.

The NYPD, per the above quoted Sargent, has a 'theory' that there is a grand 'conspiracy' and that if only you'd not object to being recorded at all times then that theory about evildoers everywhere could be proven.

One can also read about what is presented as a historic use of 'public relations' and the words conspiracy theory.
posted by rough ashlar at 4:57 AM on May 6, 2013


It wasn't meant to be framed as a bad thing. I was just giving my technical opinion based on what I deal with in my work. No more, no less.
posted by Diag at 5:04 AM on May 6, 2013


The quantity of data being amassed every second beggars belief. I imagine a future where stored data will overtake us, leaving us helplessly thrashing about in an endless sea of useless information. In many regards we're already there.
posted by kinnakeet at 6:54 AM on May 6, 2013


All this is definitely doable, but at the very least it represents a whole lot of capacity upgrades around the country, and unexplained capacity losses wherever you're using public infrastructure.

You are assuming that this is something they decided to do in the last couple of years. If the government has always been monitoring and recording calls (we know this to be true for at least some calls including a large percentage of overseas traffic) then they've had decades to roll out the infrastructure. And there wouldn't be a spike in capacity losses because they'd have exsisted since day one of at least the digital switches.

The usefulness of listening to telephone conversations would be readily apparent to anyone who lived though the live operator and party line system. Barring strong oversight demanding adherence to the contitution by black budget agencies I'd be shocked if the government wasn't recording a lot of telephone conversations.
posted by Mitheral at 7:38 AM on May 6, 2013


I've seen large-scale capture setups, and I've designed medium to small capture setups specifically for voice.

Something that people who don't work in networking may not know exist: fiber optic splitter panels. These are pretty standard in data centers, and usually provide for a 80/20 split on the light - 80 percent continues down the path, 20 percent is redirected to a monitor port.

The storage itself isn't the problem, and is certainly inexpensive and widely available... It's getting everything TO the storage. Even capturing everything, you'd need to design for redundancy and the ability to split off archival from front-end.

It's very trivial to simply capture all network traffic and dump it to disk - with the right hardware. This was described upstream as being very similar to snort, and that's an absolutely accurate description.

What you need for a capture system:

-The above splitter, some other sort of network tap, or simply a switch configured to mirror all traffic to another destination. The advantage of the mirror based splitter is that it's undetectable at the network level.... I mean, you'll see a physical device, but if that's in a closet somewhere between switch A and router B, neither one will know that there's something in-between.

-A network interface capable of reading captured data at the line rate in promiscuous mode. When we are talking about intercept at major communication hubs, this won't be your off-the-shelf intel card -- although some of them can do very well at this job at the data center level.

-A capture system that's capable of actually reading this traffic and capturing it in pcap (or other format) - This is more of a function of bus speed and memory, this isn't nearly as CPU heavy as it sounds. Compression can do a number on that, so typically data is captured raw, and "interesting traffic" is compressed and brought to an "archive server", and "non-interesting traffic" is discarded after a rolling period measured in days at best. Again, this isn't so bad at the data center level - Probably only a few hundred TB if you are a small to medium service provider, and only keeping a limited amount of traffic short term.

-Enough IO capacity in your storage subsystem - controller and disks - to handle not only writing at that bandwidth, but to handle the number of writes per second. The front end capture servers are going to need a LOT of disk if you are only using standard 7.2k SATA drives, and you aren't going to use many advanced RAID levels and get these speeds. It's the long term archive servers that benefit most from the cheaper big drives at good RAID levels.

-A good design for redundancy. I duplicated all of my capture servers. Here's why - Even running RAID 10 wasn't enough. That was a good way to ensure that we didn't lose data, but as soon as you have to rebuild a disk, that server is no longer viable for front-end capture.

The best strategy was to have multiple servers running RAID-0, and to have two taps as opposed to just one... This way, if anything failed in the chain, you treated it as if that server failed. There was a cold-standby ready to go at any time that you could swap cables to, turn on, and it would just start going.

Now, for a datacenter capture system specifically recording voice traffic at a service provider level, I was looking at spending about 40K on frontend capture servers, and 60K on the backend capture, including encryption of the data itself. This is including a high degree of redundancy on the front end as well as back end, as well as backups.

That's 100K for a system built from commodity hardware, providing for up to seven years worth of archival of "interesting" voice traffic, and a rolling 1-2 days worth of ALL traffic that could be intercepted on demand.

Double or triple the redundancy, scale it up nationally, and you are still probably under the budget of many other government projects, even allowing for a higher grade of hardware capable of capturing at the rates that major hubs would provide.
posted by MysticMCJ at 8:02 AM on May 6, 2013 [5 favorites]



The report makes it very clear that there was more than enough information to connect the dots before 9/11 happened - that is, if people had been doing their jobs and not engaging in political infighting.


There was also a whole lot more information about a whole lot of people doing a whole lot of innocuous stuff. Yes, the authorities have lots of information which if they can distill properly they can use to avert many crimes, But that information is lost in a whole lot of noise. THAT is why "connecting the dots" is only feasible in hindsight. True for 9/11. True for the Marathon. True for a whole lot of stuff.
posted by ocschwar at 8:03 AM on May 6, 2013


Mild (but I hope interesting) digression: my late dad used to work for a major phone company from the late eighties to the mid '90s in a division whose responsibilities included cooperating with federal agencies on wire tapping, tracing hackers, etc. (basically any crime that took place over phone lines). Two things used to amuse him to no end in those days: the idea that there were phone records for local calls ("Why would we pay all that money to save information we couldn't bill from?") and the movie/tv trope of "keeping them on the line so we can trace them" for land lines. ("As soon as they've dialed the last digit, it's traced, even before it rings.")
posted by DirtyOldTown at 8:05 AM on May 6, 2013


[regarding Skype] Now, let's say that we wanted it to handle all calls; local, long-distance, and international. Would existing infrastructure cope? I suspect not - ISPs would have to specifically factor Skype into their capacity requirements, Skype itself would have to create new data centers, it would all be very obvious.

Wha? No. You know that Skype made its name doing video calls, right? Maximally compressed voice audio is a tiny fraction in size of the streaming video and all the other other traffic that goes over the internet today. The burden relative to other data traffic would maybe be similar if ISPs including cable companies could limit all of their customers to 4800 telephone modems, circa late 1980s technology speeds, an absolute drop in the bucket of today's capacity on the ISP side. I would not be surprised if the audio for all POTS calls or a significant portion of it could go over Skype's current infrastructure for video calls.

That's not even getting into the fact discussed above that in this thread we aren't talking about getting data from the end consumers anyways, just from some subset of telco locations.
posted by XMLicious at 8:43 AM on May 6, 2013


Iteresting to put numbers to the cost MysticMCJ. One way to keep things cheaper would be to not insist on 100% redundency. 95-98% capture as a goal would greatly reduce costs while probably having minimal impact on the usefulness of the data.
posted by Mitheral at 1:57 PM on May 6, 2013


Cool Papa Bell: This is the same government that didn't know the WMDs were fiction, couldn't organize a response to Katrina and, despite magical phone-recording tech, still has a ten most-wanted list.

So, which is it? Bumbling fools or nefarious super villains? Make up your mind, you conspiracy-minded dolts.
There's a world of difference between knowing what a foreign dictator is hiding in his most-secret military installations, and the US government knowing how to record data on "computer disks".

No conspiracy paranoia is necessary for what is easily possible, and as others have pointed out, some of us have been fairly sure of for a while now. Even in the 80s all long-distance telephone call connections were logged for a few days, according to a district manager I knew (just the connection info, not the conversations).
posted by IAmBroom at 2:17 PM on May 6, 2013 [2 favorites]


The idea that "the government" could record all calls everywhere is like the idea of having all calls handled by Skype. There's probably a lot of slack in existing infrastructure, but you suddenly have twice as much information (the original call plus the clandestine copy) being transmitted from every exchange.
Are you capable of doing any math at all? There isn't a single number anywhere here. Which says to me you have absolutely no idea what you're talking about. You're also completely ignoring the actual numbers that other people have provided, showing this is easily attainable.
but at the very least it represents a whole lot of capacity upgrades around the country,
No, it doesn't. I have no idea how I can make this any more clear. it does not require a lot of data, the bandwidth used for all voice calls in the U.S. is minuscule compared to what's available for the rest of the internet. If the U.S. government wanted too, they could set the entire thing up on Amazon EC2 tomorrow. for a few million dollars a month, without buying a single piece of physical infrastructure (that isn't to say Amazon wouldn't need to upgrade to accommodate that request, but they would be happy to do it, I'm sure. They even have a 'gov cloud' program for government clients already)
And then you ultimately need it to come to your data center, so you have fat pipes pumping all that to you, plus clever techniques to break the fire-hose rush back down into individual conversations.
Clever techniques? You can download open source packages today that will handle this kind of thing (probably require some tuning at this scale, I would guess). Yes, they would need "fat pipes" but so what? Since when is a fat pipe hard to get (and nevermind the fact they don't actually even need one data center, they could be spread out around the country, the way google does). This is an extremely parallelizable problem.
All this is definitely doable, but at the very least it represents a whole lot of capacity upgrades around the country,
No, it doesn't. The modern internet is more then capable of handling this. It's far less bandwidth then youtube or netflix would be using.

I'm not saying, oh this is happening for sure. I am saying the technical argument that it's somehow impossible or would even be difficult are completely absurd, and doesn't seem to be based on anything other then totally random speculation about how the telephone system works without doing any actual math.

(And of course, FBI guy didn't even say the government was doing recording, rather that it was being saved by telcos, and that the government could request it. That's actually a bit worse, since there is zero oversight in that case, and when combined with something like CISPA it means the government could potentially get all kinds of information easily.)
posted by delmoi at 2:35 PM on May 6, 2013 [3 favorites]


Delmoi, you're conflating the capacity of raw data transmission in the system as a whole with the abilities and capacity of every individual telephone exchange in the USA. Yes, transmitting the actual data isn't hard, and setting up a system to collect it wouldn't be technically difficult. But you can't do it secretly without a lot of cooperation from widely-separated people; and you'd still have a system that would be at risk of exposure if someone asked "Hey, what's that cabinet over there that doesn't appear on the wiring plan?" And unless you have all your maintenance and upgrades performed by a special team of technicians with security clearances (which would probably raise questions in itself), you'd have to rely on the sort of people that are most likely to ask awkward questions and keep on digging until they are satisfied with the answer.
posted by Joe in Australia at 6:54 PM on May 6, 2013 [2 favorites]


"Telco guys are simply too l33t to not notice anything is amiss, no matter what method is used to compromise the system" seems like a worse argument than the ones about hard disk capacity and data rates... I would think that making the same sorts of assumptions might have lead one to conclude that stuxnet was impossible, for example. But the nuclear scientists whose centrifuges had stopped working properly didn't notice anything and it was discovered by accident.

The Cuckoo's Egg is actually a pretty good illustration of how security is Swiss cheese even when you're talking about powerful, top-of-the-societal-food-chain organizations. I would expect that 99.99% of the hackers who did similar things during the eighties did not get caught.

I'm not saying that omnipresent recording of phone calls is a definite thing but the objections people have come up with so far as to why it should supposedly be impossible do not by any means sound like insurmountable issues for someone able to bring the resources of the U.S. government to bear here on U.S. soil.
posted by XMLicious at 8:27 PM on May 6, 2013 [1 favorite]


But you can't do it secretly without a lot of cooperation from widely-separated people;
No one is saying anything about doing it secretly? What does secrecy have anything to do with this?
and you'd still have a system that would be at risk of exposure if someone asked "Hey, what's that cabinet over there that doesn't appear on the wiring plan?"
And the answer is "it's the government's lawful intercept box". Or perhaps "it's something from the government". I don't understand why you think those answers can't be given. It's not a secret that the government has the ability to tap a large number of phone calls. They passed an entire law in congress various phone companies immunity from prosecution for mass wiretaps they did in the bush administration.
And unless you have all your maintenance and upgrades performed by a special team of technicians with security clearances (which would probably raise questions in itself), You'd have to rely on the sort of people that are most likely to ask awkward questions and keep on digging until they are satisfied with the answer.

Again you keep talking like this needs to be some huge secret. There are five million people who have security clearances in the U.S. Plenty of people to run a telephone exchange. And lots of people in private industry have clearances because they sell things to the government. Everyone knows that the government has boxes connected to the network. That isn't a secret at all. It's called lawful interception. Cisco has a page about how it works with their hardware.

I have no idea what you think is going to happen if someone "isn't satisfied with the answer" they get about what exactly the lawful intercept box is doing. What are they going to do, try to hack into it and risk going to jail? Or they might talk about it? Well, so what? they aren't secret. They might notice is has a lot more bandwidth then might seem necessary. Okay. So what? They tell people about it, so what? All you would be doing is confirming what most people who pay attention to this kind of thing already believe is the case anyway.

And, in fact people are talking about it. There's this ex-FBI guy, and doing a little googling I came across an EFF page about a lawsuit they were filing about what they say is illegal wiretapping. It includes (sworn) testimony from an AT&T guy (about the aforementioned Room 641-A) as well as three different people from the NSA. Some of these people were prosecuted for revealing classified material (but apparently not convicted)

To be clear we're not talking about what the NSA is doing, just what kind of capacity they have. In particularWilliam Binney, a former NSA intelligence official said that in 2001 his co-workers told him they were getting dialing data for everyone in the U.S, which is why he quit.

Now, your argument seems to be something like "If this were happening people would notice and talk about it, it couldn't be kept secret." The problem with that argument isn't that it actually is possible to do in secret. The problem is people have noticed, and have talked about it.

Now, again, I'm only talking about what kind of capacity the NSA has. We don't know what they're actually doing. It's entirely possible that the NSA/FBI/etc are being careful to follow the law, or they may not even want to tap all phone calls, and instead want to focus on more suspicious people. But that's based entirely their own ethical standards, not any kind of technical limitations.

The key point is that the hardware and infrastructure don't need to be secret.

This is something I've pointed out before, which you seem to have ignored.

___

Finally, you're missing a key aspect of what the FBI guy just said. He said that the phone companies were the ones recording the calls, not the government. This, potentially, gets around the warrant issue. The phone companies don't have any legal restrictions on what they can record (I guess?), The FBI can come back later and get a warrant if something happens, but unlike a traditional wiretap, they can get recordings of calls that happened before they got the warrant. And none of what you've said has anything to do with what the phone companies are capable of doing.

(I'm not really sure why the phone companies would be recording this data other then to give to the government, but either way it's certainly something they could do)
posted by delmoi at 11:35 PM on May 6, 2013 [1 favorite]


The year is 2017, I needed to contact my grandmother after my son broke the vase she gave us for our wedding anniversary, so I picked up my cellphone and told Siri to call her:

"Welcome to AT&T Verizon Corporation we're glad to have you as a customer. The privacy act of 2015 requires us to verify your agreement prior to using our services. You have the right to remain silent. Anything you say or do may be used against you in a court of law. You have the right to consult an attorney before speaking to the police and to have an attorney present during questioning. If you cannot afford an attorney, one will be appointed for you before any questioning, if you wish. Knowing and understanding your rights as I have explained them to you, are you willing to continue this call? It is required that the you be asked if you understands your rights. Firm answer of YOUR NAME and the phrase "I AGREE" are required to continue this call. If you do not agree, simply hang up."

"MY NAME IS MONKEYONCRACK AND I AGREE..."

"One moment, this may take a little while, the party on the other end of the line must also agree..."

"Yes dear?" says grandma, after a long pause that it took for her to agree to miranda after she picked up the line ...
posted by Monkey0nCrack at 4:53 PM on May 7, 2013


In other news: Dutch Law Would Authorize Police To Hack Into Foreign Computers And Phones: What Could Possibly Go Wrong?
posted by homunculus at 5:26 PM on May 7, 2013


Judge Allows Evidence Gathered From FBI’s Spoofed Cell Tower

Previously.
posted by homunculus at 5:25 PM on May 8, 2013


DOJ: We don't need warrants for e-mail, Facebook chats. An FBI investigation manual updated last year, obtained by the ACLU, says it's possible to warrantlessly obtain Americans' e-mail "without running afoul" of the Fourth Amendment.
posted by homunculus at 5:28 PM on May 8, 2013


US Secret Surveillance Court Approves All Domestic Spying Requests For A Second Year In A Row
posted by homunculus at 5:29 PM on May 8, 2013


In other civil liberties news: TSA hearing for "Naked American Hero" John Brennan
posted by homunculus at 5:19 PM on May 9, 2013


Bruce Schneier: Transparency and Accountability Don't Hurt Security—They're Crucial to It: After Boston, just like after 9/11, the nation is likely to adopt new anti-terror laws. But done wrong, law enforcement can undermine society.
posted by homunculus at 1:39 PM on May 10, 2013 [1 favorite]


This was an interesting comment from Schneier's blog.

The NSA has an estimated budget in the ballpark of $10 billion. (Estimated because the actual number is secret.) The budget of Google, Inc. is around $50 billion. Google is the world's most successful surveillance corporation, but their face and voice recognition systems fall apart at the slightest hint of noise or shadow, they won't allocate more than 5 gigs of storage to me unless I pay them, and their satellite surveillance of my house is a year old and 1/2 meter resolution.

If you expect the NSA to do better, they must be spending money at least 5 times more efficiently.

posted by bukvich at 3:56 PM on May 10, 2013 [1 favorite]


Or an advertising company and a Secret Service Agency might have different mission goals (and the secret service agency might be able to sponge some data off of other more public agencies). How much of Google's budget is consumed serving web pages?
posted by Mitheral at 8:03 PM on May 10, 2013


The NSA has an estimated budget in the ballpark of $10 billion. (Estimated because the actual number is secret.) The budget of Google, Inc. is around $50 billion. Google is the world's most successful surveillance corporation, but their face and voice recognition systems fall apart at the slightest hint of noise or shadow, they won't allocate more than 5 gigs of storage to me unless I pay them, and their satellite surveillance of my house is a year old and 1/2 meter resolution.

If you expect the NSA to do better, they must be spending money at least 5 times more efficiently.


Google's R&D budget is $6.7B, and they don't have access to classified technology or military satellites.
posted by Jairus at 9:16 PM on May 10, 2013


But, too, the NSA presumably doesn't spend all their money on one project.
posted by JHarris at 11:49 PM on May 10, 2013


The NSA has an estimated budget in the ballpark of $10 billion. (Estimated because the actual number is secret.) The budget of Google, Inc. is around $50 billion. Google is the world's most successful surveillance corporation, but their face and voice recognition systems fall apart at the slightest hint of noise or shadow, they won't allocate more than 5 gigs of storage to me unless I pay them, and their satellite surveillance of my house is a year old and 1/2 meter resolution.

If you expect the NSA to do better, they must be spending money at least 5 times more efficiently.
I don't think they have any way to actually analyze that data in an automated way. But they can have human analysts go through and look closely at what specific people were saying at various points in time, which could certainly be useful.

The other thing is that the voice recognition systems don't need to be perfect, and don't need to do automatic transcription. So for example, if someone says "mom ate a bun" it would be inaccurate if Google translated it as "bomb and a gun". But the NSA doesn't really need to distinguish different words that sound like gun and bomb, they need to assign a score to how much the word sounds like gun or bomb. The higher the probability each word being 'suspicious' the more likely it is the whole conversation can be flagged, then you can have a human go through and verify it.

So the kind of analysis is different and doesn't depend as much as getting each world completely correct.
posted by delmoi at 1:40 AM on May 11, 2013


One of the more recent methods of electronic communication amongst Al Qaeda cells was apparently sharing the username and password of a Gmail account and then writing messages in draft without sending them. Nothing was sent electronically and thus nothing got sucked up into the NSA/GCHQ machine. Obviously Google will have had this data stored and if that is available to snoopers it is not entirely secure but at the time it was certainly a good way to communicate electronically without the concern of someone reading over your shoulder.

Personally I fully expect to see a massive increase in actual tradecraft, a subject which I find very interesting and which is rather harder to learn IRL than you might think. Even more of a problem with geographically distributed cells.
posted by longbaugh at 2:53 AM on May 11, 2013


Incidentally - James Bamford wrote about ECHELON and the NSA/GCHQ data-sharing in the Body of Secrets for anyone interested in more reading around the subject. I would take some of it with a grain of salt but this and his other books around the NSA, The Puzzle Palace and The Shadow Factory are amongst the best books if you are looking for information about the NSA. The Puzzle Palace is perhaps a little long in the tooth now - it was written back in '83 but The Shadow Factory was published in 2008 and is about as up to date as you can get.
posted by longbaugh at 3:02 AM on May 11, 2013


Senator Mark Udall (D-CO) has demanded that the FBI and other Federal agencies obtain a warrant to read private emails.
posted by homunculus at 12:40 PM on May 11, 2013 [2 favorites]


Why The FBI Will Love Google Glass: As we enter the era of wearable computing, government requests for user data are at an all-time high.
posted by homunculus at 1:33 PM on May 11, 2013


Essentially the America is capturing the private information of anyone anywhere in the planet who communicates with an American on American soil.

Does that include interacting with Google, Yahoo, Microsoft, Apple and Facebook? Where'd Twitter go to?
posted by infini at 1:22 AM on May 12, 2013


Apple deluged by police demands to decrypt iPhones: ATF says no law enforcement agency could unlock a defendant's iPhone, but Apple can "bypass the security software" if it chooses. Apple has created a police waiting list because of high demand.
posted by homunculus at 10:47 AM on May 12, 2013


Under a different and short lived brand new day I FPP'd on the death of the letter (snail mail) ... now I'm ready with my brand new present to self Lamy fountain pen.
posted by infini at 1:18 PM on May 12, 2013


Personally I fully expect to see a massive increase in actual tradecraft, a subject which I find very interesting and which is rather harder to learn IRL than you might think. Even more of a problem with geographically distributed cells.
posted by infini at 3:24 PM on May 12, 2013


infini - is one of my ad blockers malfunctioning, or does that quote not appear in that link? Or maybe something was edited out?
posted by XMLicious at 6:07 PM on May 12, 2013


Personally I fully expect to see a massive increase in actual tradecraft, a subject which I find very interesting and which is rather harder to learn IRL than you might think. Even more of a problem with geographically distributed cells.
posted by longbaugh at 2:53 AM on May 11

posted by infini at 12:33 AM on May 13, 2013 [1 favorite]


The Justice Department Secretly Seized AP Phone Records — on a Terror Leak?
posted by homunculus at 3:10 PM on May 13, 2013


Former DHS Head On Google Glass: Intrusive Surveillance Is Bad -- If It's A Corporation Doing It
posted by homunculus at 3:44 PM on May 13, 2013


Justice Department's pursuit of AP's phone records is both extreme and dangerous: The claimed legal basis for these actions is unknown, but the threats they pose to a free press and the newsgathering process are clear
posted by homunculus at 4:06 PM on May 14, 2013 [1 favorite]


Wiretapping the Web? It would be a massive invasion of privacy by the world’s most powerful government—and the F.B.I. makes a persuasive argument to do it…
posted by homunculus at 7:15 PM on May 14, 2013 [1 favorite]


NSA Data Center the Focus of Tax Controversy
posted by homunculus at 7:48 PM on May 20, 2013


Your Word Against Ours: How The FBI's 'No Electronic Recording' Policy Rigs The Game... And Destroys Its Credibility
posted by homunculus at 7:49 PM on May 20, 2013 [1 favorite]


Bruce Schneier: Surveillance and the Internet of Things
posted by homunculus at 4:16 PM on May 21, 2013 [1 favorite]


How the Government Targeted Occupy: A new report reveals the U.S. spent millions spying on Occupiers and other anti-corporate activists.
posted by homunculus at 9:41 PM on May 25, 2013 [2 favorites]


Kafka, meet Orwell: peek behind the scenes of the modern surveillance state
posted by homunculus at 6:06 PM on May 27, 2013 [1 favorite]


The Associated Press
US intelligence embraces debate in security issues
prolly all favouriting homunculus
posted by infini at 9:37 AM on May 28, 2013


Glenn Greenwald: Obama's new FBI chief approved Bush's NSA warrantless wiretapping scheme. James Comey becomes just the latest symbol of the Obama legacy normalizing what was very recently viewed as radical
posted by homunculus at 10:58 AM on May 30, 2013 [1 favorite]


Google ordered to hand private customer data over to FBI investigators: Judge who earlier ruled National Security Letters unconstitutional orders Google to nonetheless comply with them
posted by homunculus at 1:12 PM on June 3, 2013


« Older "One can almost hear the anticipatory echoes of so...  |  What does the crowdfunding lan... Newer »


This thread has been archived and is closed to new comments