May 16, 2001

4:23 PM

Where (and how) does the European Communitythink they're going to store a million gigabytes of data per day -- for seven years?
posted by Steven Den Beste (20 comments total)

And what kind of infrastructure is going to be needed to transmit all the data which passes through every phone exchange in Europe to some central data facility? A standard landline voice call generates more than 3 megabytes of data per minute. A typical phone exchange is going to be generating hundreds of gigabytes of data per minute, and there are thousands of them in Europe.

Just how clueless can you get?
posted by Steven Den Beste at 4:27 PM on May 16, 2001

Actually, a standard landline voice call generates more like 300 kilobytes of data per minute. That's uncompressed, of course; compression can cut that down to about 75KB per minute pretty easily if voice quality is all that's needed.

It's still, obviously, completely impractical.
posted by kindall at 4:34 PM on May 16, 2001

I did my math wrong. The actual number is:

56 kbits/second/channel
* 2 channels
* 60 seconds per minute
/ 8 bits per byte
== 840 kilobytes per minute.

Of course, sometimes it will compress immensely and sometimes not at all. If it's carrying modem traffic full speed it won't compress even slightly.
posted by Steven Den Beste at 4:49 PM on May 16, 2001

This is the kinda proposal that'd lead to large scale riots in every European city if pushed through against what would surely be massive, overwhelming public disapproval. Doesn't matter how much tabloid propaganda is used to soften the blow and win favour. That data storage center will need to be like Fort Knox. Hope they don't keep this post for 7 years and blame me once its been blew up.
posted by Kino at 5:08 PM on May 16, 2001

Whoops, you're right -- I figured only one channel and I assumed 10 bits per byte, but phone lines are synchronous so that would in fact be only 8.
posted by kindall at 5:08 PM on May 16, 2001

Where do they think they're going to store it? Same place the FBI does, they designed the friggin scheme :) What I find funny is that the FBI designed it six years ago when storage facilities were much more meagre than now (but net traffic was too...)
posted by DiplomaticImmunity at 5:34 PM on May 16, 2001

Attention Hackers: please go to work on this one. The EC/EU Whatever is BAD for Europe, and this is but the last proof. At least the FBI/CIA?NSA is too incompetent to pull such a thing if.

Hackers: go. go. go.
posted by ParisParamus at 7:02 PM on May 16, 2001

"thing off". and "latest proof": it's late.
posted by ParisParamus at 7:04 PM on May 16, 2001

This particular European is wondering how "too many people whose cultures and interests are too wildly divergent" can simultaneously have some common "mindset".
posted by normy at 10:33 PM on May 16, 2001

This is transparently an attempt to get one back at the US for ECHELON. (And no, I don't know how much gets stored at Menwith Hill, but when you've got the fibre-optic capacity of a small city, I'd expect there's more than a few racks.) My guess is that the negotiations over individual states' rights to data will kill this one. Because the common interest -- essentially, the capacity to carry out the same industrial espionage on the US that was accused within the ECHELON report -- isn't enough to counter mutual suspicion across Europe.

As for European political union: expansion will make it intriguing, because a union that stretches to the Russian border simply can't be "run" from Brussels. There are two directions in which the power structure could develop, and it's my hope that we'll see greater powers to the elected parliament, combined with executive devolution. And that's certainly possible, if the will is there.
posted by holgate at 3:00 AM on May 17, 2001

We run Hawaii (and Guam!) from DC. Brussels to Poland is just a hop, skip and jump. Heck, it's not even as far as from DC to San Diego.
posted by Steven Den Beste at 6:38 AM on May 17, 2001

What a great excuse to encourage encryption in Europe. Not that it'll help much in the fax/phone world, but as long as the EU keeps dropping its Euros into data storage and not into incredible processing power. Then again Americans know they're constatly being spied on and encryption hasn't even reached the level of Crystal Pepsi.
posted by skallas at 7:13 AM on May 17, 2001

There is nothing I send that I feel particularly ashamed of.

But recently I did have an eerie experience. There is a man in China who was sending me a lot of letters asking questions about CDMA (something about which I know a great deal and about which I have a web page) and during the spy-plane crisis he wrote at one point and asked me how I felt about continuing the correspondence. I wrote back and said "No problem. Just because our governments are acting like asses doesn't mean we have to."

I received one more letter from him after that, and then silence. I haven't heard from him since. And I have to wonder whether I got him into trouble with my comment. You do not criticize the government of China if you live there, or you might vanish. You don't even do anything which suggests that you're thinking along those lines, if you want to stay safe. What I'm hoping is that he merely decided it was too dangerous to continue to the correspondence. What I fear is that he's dead. (I'm not going to try to find out, for fear of potentially making a bad situation worse.)

That brings home the fact that mail encryption might well be something I should be doing some of the time. After I get my server, I might well start.
posted by Steven Den Beste at 8:29 AM on May 17, 2001

I remember this issue a few months back when the UK government was reported to be considering a plan to store all IP going through Britain for 7 years. It sounds so impractical, but I started to wonder and I performed a few experiments.

The first experiment was that I ran a packet sniffer on my own computer and let it run for an hour. With web browsing, e-mail and chat going on I barely ran up a megabyte before compression. I then looked at hard drive prices and calculated that it would cost less than $400 in hard drives to store an entire year of my personal IP activity at that rate. (Then I looked at my previous year's tax filing - go figure).

I still think that my experiment gave me an inflated number because I was judging it on hard drive prices (a government would likely chose cheaper mediums, such as tape) and my sessions also consisted of large chunks of data that were redundant from the network's point of view - I downloaded web pages that have already been seen by millions of other people.

Any effort to archive my IP (plus phone, fax and other transmission) activity would undoubtedly make use of compression schemes that work on a data set larger than just my own. If two people download the same web page, then the archive only needs to store that page once along with the fact that it was seen by two people. If five people are conversing in a chat room, only one conversation needs to be stored - not five.

I am not altogether convinced that the EU will pull off a project of this scale, but I don't think that it's impossible, either. Governments do have government-scale resources, after all. If they can launch space programs, build interstate highways, and dig channel tunnels, then they might be up to the task of implementing another enormous engineering project.

Looking at the bright side I think that at the very least - even if their project fails - there may be some useful R&D fallout from it. They couldn't help but improve the state of mass storage and retrieval technology.
posted by wenham at 8:50 AM on May 17, 2001

Steven: I think (or maybe just hope) that your paranoia here is overdone--I wouldn't think even China (and I yield to no one in my dislike of the Chinese government) would actually *kill* anyone for receiving email like yours. Cut off email access, even send some people for a "visit" maybe, but kill? I don't think you should worry about that.

But anyway, if you do worry about it, don't you think that a Chinese citizen receiving encypted email from the US would be a major red flag? The thing about encryption is that it screams (to law enforcement) "I have something to hide!" Even in a society with more-or-less intact respect for privacy the government gets all spooky about it. What happens in an authoritarian, paranoid society with active secret police?
posted by rodii at 9:06 AM on May 17, 2001

Note to self, buy EMC stock tomorrow...
posted by machaus at 1:01 PM on May 17, 2001

Wenhem, you're absolutely right. Store the physical information once, link the viewers/participants to it logically. But I can tell you, even that logical information becomes absolutely ridiculously huge. Frankly, I personally hope they store it in triplicate.

Oh and by the way, Wenhem, spell check wants to change your name to enema. I might have a talk with Matt about that... :-)
posted by fooljay at 4:07 PM on May 17, 2001

Chris, the real problem here isn't storage but collection. It would be necessary to put a fat datapipe from every phone exchange and every ISP and every network switch back to the central office to carry all the traffic. That's a nontrivial problem in its own right.

It would also be necessary to modify all those things to make them feed all their traffic into said data pipe. That is non-trivial. (What happens if, when you ask for mods, the manufacturer of the equipment tells you to get stuffed?)

There's also the "drowning in a sea of data" problem. How the heck do you catalog all that stuff? (Can you imagine the database which permits you to search not merely for source and destination of the data, but also on its contents? Like "Show me every mail message sent by anyone in Europe during the month of May 2003 which included the words Waco and McVeigh and bomb and either revenge or vengeance?" The mind boggles.

Nothing here is impossible; the operative word is "impractical".
posted by Steven Den Beste at 4:44 PM on May 17, 2001

"Impractical" hasn't stopped the US Government from passing CALEA (Communications Assistance for Law Enforcement Act). In a nutshell, every phone exchange must install a T1 through which phone call records and even entire conversations can be piped to a law enforcement agency (selected conversations, of course. A T1 can only carry 24 phone conversations at once. But downloading 100% of the phone records is a piece of cake).

But impracticality is actually the motive for CALEA. The FBI wants to be the only ones in the wiretapping business. By making the requirements to do a wiretap so expensive (monthlies on a T1 can be $1,500 and higher) they make it prohibitive for "mom and pop" agencies. A big agency like the NYPD can afford it, but not - say - some small town like Binghamton. Maybe the EU is after the same thing.

And the technical problem could be solved already. CALEA - which was passed in 1994 and will finally take effect this September - required exactly the same kind of equipment modifications the EU will need. All the EU has to do is copy what Uncle Sam has already done. But instead of installing a T1 at each exchange, they'd install an Optical Carrier.

As for searching, I can't go into the problems involved with searching the content of the call, but I can say that much can be discovered by merely analyzing the source, destination, time and duration. You can figure out which calls are likely to be from - say - individual drug pushers to their regional distributor - just by studying a table with those four columns. Then you know which recorded calls to listen to (or have searched for patterns).

In other words, you don't have to search every call for keywords.
posted by wenham at 10:55 AM on May 18, 2001