FBI Raid curbs Curbed

June 22, 2011 8:49 AM

The F.B.I raided a data center in Reston, VA yesterday morning, seizing three racks of servers and disrupting service to the Curbed Network, Pinboard.in, Instapaper, altlabs.co.au, and took the physical servers of tens of other clients. Curbed and AltLabs are currently still down. The F.B.I was reported in pursuit of one individual user, and agents took entire server racks, perhaps because they mistakenly thought that “one enclosure is = to one server” according to DigitialOne, the Swiss hosting company.
posted by 2bucksplus (64 comments total)

Can the hosting company's clients sue the FBI for harm to their business?
posted by five fresh fish at 9:02 AM on June 22, 2011 [3 favorites]

Heckofa job, FBI. So elite with your cybersleuthing. My trust in the FBI's ability to understand The Information Superhighway has been increased.
Seriously, this is some epic "series of tubes" shit right here.
posted by Threeway Handshake at 9:06 AM on June 22, 2011 [4 favorites]

Curbed is actually back up as of this morning. But I had no idea until now about what caused them to go down. Wild.
posted by kimdog at 9:07 AM on June 22, 2011

A judge had to sign off on the warrant.
posted by smackfu at 9:08 AM on June 22, 2011

A judge had to sign off on rubber stamp the warrant.

FTFY
posted by Mister Fabulous at 9:09 AM on June 22, 2011 [12 favorites]

One of Glenn Reynolds' readers sent him an explanation for why taking the whole rack might make some sense:

In dense server environments, best practices for CPU/processor utilization dictate virtualization on the server environment. In layman’s terms, multiple virtual servers are created inside a physical server. In a cluster of physical servers in a single rack or across several racks, the virtual servers can move around from physical server to physical server to keep the load balanced and maximize cpu efficiencies. As such, it can be difficult to pin down the exact physical piece of hardware that a piece of evidence has “touched”. The only solution is to remove all the servers in the cluster.

Server virtualization has very widespread adoption so most environments it is very likely that the administrators were using it.

posted by Chocolate Pickle at 9:11 AM on June 22, 2011 [2 favorites]

Thanks alot FBI, now I get to add another fucking sub-paragraph to our disaster recovery procedures, 2.8.3c: Recovery of Physical Assets Unintentionally Seized.
posted by odinsdream at 9:13 AM on June 22, 2011 [24 favorites]

Pinboard's Twitter feed has been pretty helpful. (Pindroid gave me a error saving an item, for the first time ever.)
posted by epersonae at 9:13 AM on June 22, 2011

Per the New York Times post, the hosting company (DigitalOne) does not own the building in which the servers are kept: They lease space in the building from somebody else.

This is how the FBI were able to enter the building and take the servers without DigitalOne being aware of it.

Also interesting: The raid was apparently on LulzSec.
posted by ardgedee at 9:14 AM on June 22, 2011

Chocolate Pickle : Server virtualization has very widespread adoption so most environments it is very likely that the administrators were using it.

Very true!

Of course, also quite likely, the "server" contained nothing but the VM host, with the guest OSes and any actual data living on a SAN halfway across the room (or world, for that matter).
posted by pla at 9:15 AM on June 22, 2011 [3 favorites]

The status update from Pinboard also says: "I have no reason to believe it had anything to do with us, but unfortunately these blade servers pack many to a single box. "
posted by smackfu at 9:15 AM on June 22, 2011

I look forward to the day when some law enforcement agency does something stupid like this to the wrong corporation, and we get to watch the corporate power structure go to war with the police state.

It'll be like living in an early '90s cyberpunk novel.
posted by quin at 9:15 AM on June 22, 2011 [34 favorites]

The raid was apparently on LulzSec.

That's what I keep hearing but I haven't seen a source for it for it other than blowing smoke on the internet, not even a news report with "sources say".
posted by immlass at 9:16 AM on June 22, 2011

Thanks alot FBI, now I get to add another fucking sub-paragraph to our disaster recovery procedures, 2.8.3c: Recovery of Physical Assets Unintentionally Seized.

Wouldn't your disaster recovery procedure just generically cover the case "servers go away"?
posted by smackfu at 9:16 AM on June 22, 2011 [1 favorite]

I look forward to the day when some law enforcement agency does something stupid like this to the wrong corporation, and we get to watch the corporate power structure go to war with the police state.

Not gonna happen. It's only nickel-and-dime companies--relatively speaking--that are going to use third-party server space. Google? Amazon? Microsoft? Really, any Fortune 100 company? Those guys own their own server farms, they don't rent space from third-parties.
posted by valkyryn at 9:18 AM on June 22, 2011 [1 favorite]

Were they looking for Reston5?
posted by MrFTBN at 9:19 AM on June 22, 2011

From the NY Times blog post:

> A government official who declined to be named said earlier in the day that the F.B.I. was actively investigating the Lulz Security group and any affiliated hackers. The official said the F.B.I. had teamed up with other agencies in this effort, including the Central Intelligence Agency and cybercrime bureaus in Europe.

Correlation is not causation, even in statements from unnamed spokespeople. But the "I'm just sayin'" tone of the statement seems pretty directed to me. If it was a strike on a foreign terrorist org, the public statement would have gone down differently.
posted by ardgedee at 9:20 AM on June 22, 2011

Not gonna happen. It's only nickel-and-dime companies--relatively speaking--that are going to use third-party server space. Google? Amazon? Microsoft? Really, any Fortune 100 company? Those guys own their own server farms, they don't rent space from third-parties.

But they all do rent space TO third-parties which exposes them to similar problems. I look forward to distributed police raids seizing clouds.
posted by srboisvert at 9:22 AM on June 22, 2011 [1 favorite]

Chocolate Pickle and pla: DigitalOne only hosts dedicated Blade servers, they do no virtualization whatsoever.
posted by thebestsophist at 9:22 AM on June 22, 2011

I stopped in to make the "your clouds are really minerals" quip, but then this,

Per the New York Times post, the hosting company (DigitalOne) does not own the building in which the servers are kept: They lease space in the building from somebody else.

and this,

In a cluster of physical servers in a single rack or across several racks, the virtual servers can move around from physical server to physical server to keep the load balanced and maximize cpu efficiencies.

Turned it all over again. Your minerals are really clouds.
posted by notyou at 9:23 AM on June 22, 2011 [3 favorites]

But they all do rent space TO third-parties which exposes them to similar problems.

Only sorta. I've read contracts like those (used to be in-house counsel at an insurance company), and there are always provisions dealing with government seizure of equipment. Actually, those are pretty common in all corporate contracts. So if Amazon leases space to a third-party, and the government wants those servers, you bet your ass that Amazon is going to be amazingly cooperative if presented with a warrant.

I'm guessing that the reason the FBI just went and took everything is because whoever was running the place wasn't a big enough fish to be worth asking for help first.
posted by valkyryn at 9:28 AM on June 22, 2011

So, wait, all I have to do to disrupt ecommerce in the US is get a server at some hosting company that hosts other well known services, start doing bad things, and then wait for the FBI to come and fuck shit up?
posted by ennui.bz at 9:30 AM on June 22, 2011 [7 favorites]

This is why the closed the Google Page Creator.
posted by TwelveTwo at 9:32 AM on June 22, 2011

So, wait, all I have to do to disrupt ecommerce in the US is get a server at some hosting company that hosts other well known services, start doing bad things, and then wait for the FBI to come and fuck shit up?

DOS by FBI. I can't wait to read the scare article about it to show up in The Register. (but really there are cheaper, less risky ways to do the same thing.)
posted by papercrane at 9:34 AM on June 22, 2011

I would actually imagine Dropbox with a throw away email would be the go to for Anonymous sharing.
posted by TwelveTwo at 9:38 AM on June 22, 2011

So, wait, all I have to do to disrupt ecommerce in the US is get a server at some hosting company that hosts other well known services, start doing bad things, and then wait for the FBI to come and fuck shit up?

reverse the Players and you can see the clumsy game they will, shit have and done been doing.

They ain't playing no more, no more
Feds are pissed, so sore so sore
The boat sails and off they ride
booty from companies stuff inside
and it all looks crazy
when the feds come to town,
to town.
posted by clavdivs at 9:38 AM on June 22, 2011

Boardgame coming soon from Steve Jackson Games.
posted by Legomancer at 9:40 AM on June 22, 2011 [11 favorites]

DOS by FBI.

Why not.
posted by Skorgu at 9:40 AM on June 22, 2011

Boardgame coming soon from Steve Jackson Games.

Unless the hard drives get seized before it can go to print...
posted by yeloson at 9:45 AM on June 22, 2011 [4 favorites]

DOS by FBI.

Lulz by cop.
posted by dirigibleman at 10:00 AM on June 22, 2011 [5 favorites]

I look forward to the day when some law enforcement agency does something stupid like this to the wrong corporation, and we get to watch the corporate power structure go to war with the police statethe police state say "oops sorry, sir" and hand it all back right quick.
posted by DU at 10:06 AM on June 22, 2011 [1 favorite]

One of Glenn Reynolds' readers sent him an explanation for why taking the whole rack might make some sense

Ok, but it's still like seizing an entire shopping mall because you're investigating an alleged crime in one store. The shopping mall has a common loading dock, storage space, and building services for its tenants, so you'd better be safe and take it all.

Certainly the FBI could have made an effort to understand the hosting environment here and to image the relevant data in-place without grabbing everything belonging to completely unrelated businesses.

I'm curious whether the FBI even got what they are looking for given the complexity of some of these environments. To provide an example, I'm working on a system hosted on Amazon Web Services right now. When you access the site with your web browser, you're hitting a load balancer, which is shared with various other sites. The load balancer forwards your request to one of a number of web servers, which are also shared with other sites and can start and stop at will depending on demand. But the disks in the web servers are only temporary scratch space, as the application's files are stored in various Elastic Block Storage devices, which means the data on them could be basically anywhere and is almost certainly co-mingled on various disks with other customer's data. Except our users' sensitive data isn't even stored there (except ephemerally in log files and such); it's all in a database hosted by Amazon's Relational Database Service, which means all that data is stored on yet more servers and disks in arbitrary locations co-mingled with other sites' data. For even more fun, we don't store payment information like credit card numbers or even customer names and addresses ourselves. All that is outsourced to a payment processing service, and I suspect they only store the name/address/payment history data themselves and outsource the actual CC number storage to the payment gateway.

So suppose the FBI wanted to raid my site. What do they grab? The only part of the infrastructure that users actually interact with is the load balancer–everything else is hidden on private networks–, so it would seem to be a logical first choice, except that it's highly unlikely to be of any use and is shared with who knows how many other sites. The web servers are logical choices too, but they are shared virtual machines too, and there's no guarantee that the server that hosted our site an hour ago is still doing so now; our data isn't there either. The EBS disks could well be useful for log files and such, but the data could basically be anywhere "in the cloud," so you pretty much can't seize the physical disks without grabbing every storage array in the data center (or multiple data centers linked together). The most useful information is the user data stored in the database, but to get at that you'd have to know we used RDS, and there's no way of knowing that without understanding our internal infrastructure, because the database has no direct connection to the outside world. Customer identity and payment data would require yet another raid on the payment service's servers, except there's no way to know that we even use them until you've reviewed the internals of our app.

TL;DR: The standard model of "seize the servers and hold them for analysis" doesn't begin to work in the cloud. It's like living in a massive city where everyone shares houses and offices pretty much at random. Want to raid a drug lord's place? You'd have to raid every house in the city and try to sort out what belongs to who.
posted by zachlipton at 10:09 AM on June 22, 2011 [26 favorites]

It's good to know that the FBI is hard at work preventing LulzSec's goal of making government agencies look stupid.
posted by qvantamon at 10:16 AM on June 22, 2011 [12 favorites]

Is this one of those "in times of war, we all have to make sacrifices for our common security" things?
posted by acb at 10:46 AM on June 22, 2011

This reminds me of the time on Monday Night Raw where Edge got mad at the General Manager (who is anonymous and communicates through emails read from a laptop) and proceeded to get into a fight with a laptop. No really.

Only it really happened, and it was by law enforcement, and holy christ, I don't know whether to laugh or vomit.
posted by Uther Bentrazor at 10:51 AM on June 22, 2011 [2 favorites]

Wouldn't your disaster recovery procedure just generically cover the case "servers go away"?

Yes. It was mostly a flippant comment, but it has actually given me some things to think about, like the legal and procedural implications of having servers essentially stolen but otherwise perfectly operational.

With certain types of data it's the nightmare scenario; distinctly different from a natural disaster that makes the servers non-operational.
posted by odinsdream at 11:01 AM on June 22, 2011 [5 favorites]

Is it possible that maybe the absurd over reach and collateral damage to infrastructure is a feature, not a bug? The more that government agencies can inconvenience "normal people" in their campaign against cyber criminals, the more willing people (and business) will be to support measures against conditions that allow hackers to exist in the first place. Like privacy.
posted by crackingdes at 11:03 AM on June 22, 2011 [4 favorites]

Now, I understand the host isn't virtualized, but they obviously grabbed some data from some companies unrelated to the investigation.

Doesn't that fall under unreasonable search and seizure?
posted by lumpenprole at 11:04 AM on June 22, 2011 [1 favorite]

Given this event: Why would anybody want to host their sites in the US?
posted by DreamerFi at 11:13 AM on June 22, 2011 [2 favorites]

Want to raid a drug lord's place? You'd have to raid every house in the city and try to sort out what belongs to who.

DON'T WORRY! We are nowhere near the terminal velocity of this particular slippery slope.
posted by lalochezia at 11:13 AM on June 22, 2011

This indicates that FBI agents involved in this sort of cybercrime operation need to have better basic IT training, but little else. I'm sure it wasn't malicious, just misinformed. I strongly suspect something like this:

FBI: We need all systems pertaining to this, this, and this.

IT: Whoa, well, uh... yeah, okay. That's all over here, in these racks. I'll need to phone up Admin Joe to find out which servers they actually provisioned and Admin Jack to arrange a graceful shutdown, and... what are you doing?

FBI: These racks, right? Cool, thanks.
posted by gilrain at 11:15 AM on June 22, 2011 [6 favorites]

According to the New York Times piece: "DigitalOne had no employees on-site when the raid took place."

That may have something to do with FBI having trouble identifying a specific machine.
posted by Anything at 11:16 AM on June 22, 2011

Anything: "According to the New York Times piece: "DigitalOne had no employees on-site when the raid took place."

That may have something to do with FBI having trouble identifying a specific machine."

Wow. WOW. Now I understand why the Pinboard guy has been scrambling to find a new hosting provider rather than riding out the storm on his backup servers and getting a refund, or something.

That's unconscionable. Nobody onsite?
posted by gilrain at 11:17 AM on June 22, 2011 [3 favorites]

Nobody onsite?

A well-run modern data center is so fully automated, it has one employee and a dog on the premises.

The employee is there to replace broken servers with new ones.

The dog is there to bite him if he tries anything else.
posted by DreamerFi at 11:22 AM on June 22, 2011 [16 favorites]

The FBI really needs better IT training, then again it is a losing battle that cannot be one.
Here is a simple solution to keeping your internet activities from prying eyes of any kind (for the most part)
Step one: purchase vpn service to foreign located server with strong privacy laws
Step two: use vpn for private connection to rented storage space of suspected private information in another foreign country.
Step three: truecrypt any and all important/sensitive files as well as computer being used
Step four: stay away from any US services (google, hotmail,) use hushmail if anything.

And there you have it. Perhaps throw in some proxies, have multiple internet connections, renew ip addresses frequently as well as change mac address. Most of the above can be accomplished using timed scripts.
posted by handbanana at 11:27 AM on June 22, 2011

Given this event: Why would anybody want to host their sites in the US?

So, yeah, I'm looking at the locations provided by my hosting provider (my server is a VPS I use primarily for my personal email--but you know, that important to me and I really don't think the FBI would give a flying fuck if they were to inconvenience me and seize the physical box my VPS is on because someone else pissed them off) and the non-US choices are London, Sydney, or Auckland. Anyone familiar with the laws in the UK, AUS, or NZ?
posted by MikeKD at 11:31 AM on June 22, 2011

DreamerFi: "Nobody onsite?

A well-run modern data center is so fully automated, it has one employee and a dog on the premises.

The employee is there to replace broken servers with new ones.

The dog is there to bite him if he tries anything else."


I know that, but where was the one employee?

Not to mention the dog!
posted by gilrain at 11:31 AM on June 22, 2011

The one employee probably isn't an employee of the hosting provider. The hosting provider probably just rents space from the owner of the data center. Most hosts don't actually own DCs, because building a datacenter is very expensive and difficult to do correctly.
posted by wierdo at 11:35 AM on June 22, 2011

I was going to make a joke about 'screwing the pooch' but chose not to.
posted by Anything at 11:36 AM on June 22, 2011

> Want to raid a drug lord's place? You'd have to raid every house in the city and try to sort out what belongs to who.

I'm kind of liking this.

The further and further law enforcement gets away from keeping me safe from burglary, robbery or assault, and the closer they get to trying to make sure I don't or can't do things - self-medicate, view "objectionable" material, copy content, gamble online - the more difficult I hope it gets for them.
posted by mmrtnt at 12:04 PM on June 22, 2011 [2 favorites]

(but really there are cheaper, less risky ways to do the same thing.)

I'm trying to figure out a cheaper, less risky way, but spending $5 to rent a VPS and have the FBI come take down racks of servers seems pretty easy.
posted by ryoshu at 12:07 PM on June 22, 2011

Given this event: Why would anybody want to host their sites in the US?

The speed of light isn't quite fast enough. People only host elsewhere to get their servers physically closer to their customers (like the UK for Europe or Australia for Australia).
posted by smackfu at 12:07 PM on June 22, 2011 [1 favorite]

So for sites that need US clients and where the speed of light is an issue (a small subset, I'd say), you'd host in Canada?

The only servers where the speed of light is a huge showstopping issue is the trading computers at wall street, and I don't see the FBI knocking down doors there any time soon.
posted by DreamerFi at 12:22 PM on June 22, 2011

Is this one of those "in times of war, we all have to make sacrifices for our common security" things?

This is more like, "The criminal could be any one of us! So everyone punch yourself in the face! That'll show 'em!"
posted by Kid Charlemagne at 12:25 PM on June 22, 2011

MikeKD: "Anyone familiar with the laws in the UK, AUS, or NZ?"

From what I've seen, and I'm hardly an international lawyer, these are not the places one runs to to avoid the police state. UK, for example, has a far more punishing set of libel laws, and can jail you for refusing to provide a decryption key. And the AU will be rolling out internet filtering next month.
posted by pwnguin at 1:52 PM on June 22, 2011

So, wait, all I have to do to disrupt ecommerce in the US is [become the subject of a Federal investigation]?

Yup! Let us know how that works out for you.
posted by indubitable at 3:17 PM on June 22, 2011

FBI press release: http://1.usa.gov/lnKnCB For some reason doesn't mention cyber-disrupting cyber-business for no cyber-reason

(Pinboard)
posted by epersonae at 4:07 PM on June 22, 2011 [2 favorites]

"I'm trying to figure out a cheaper, less risky way, but spending $5 to rent a VPS and have the FBI come take down racks of servers seems pretty easy."

If you mean what I think you mean, that is positively evil. Brilliant.

It's the equivalent of mentioning to the school bully that the new kid said he was a stupid fatty.
posted by Xoebe at 7:13 PM on June 22, 2011

zachlipton writes "Ok, but it's still like seizing an entire shopping mall because you're investigating an alleged crime in one store. The shopping mall has a common loading dock, storage space, and building services for its tenants, so you'd better be safe and take it all.

"Certainly the FBI could have made an effort to understand the hosting environment here and to image the relevant data in-place without grabbing everything belonging to completely unrelated businesses. "

This has been SOP for federal computer crime warrants for decades. IIRC in the Steve Jackson case the Secrect Service even took the company printers (at a time when such devices didn't keep copies of everything they printed). And they weren't even investigating SJG but rather one of SJG's employees. It's probably incompetence but a little part of me believes it is deliberate incompetence as a form of punishment.
posted by Mitheral at 8:23 PM on June 22, 2011

gilrain: I know that, but where was the one employee?

From the NYT: DigitalOne had no employees on-site when the raid took place. The data center operator, from which DigitalOne leases space, passed along the information about the raid three hours after it started with the name of the agent and a phone number to call.

Before learning of the raid, Mr. Ostroumow, who is in Switzerland with the rest of his team, thought the problem was a technical glitch, he said.

Which kind of makes me wonder if part of the blame rests with the data center not giving a shit. "Eh, DigitalOne's stuff's on this rack. The IP directory is somewhere around here, if you want it."
posted by kagredon at 1:42 AM on June 23, 2011

The only servers where the speed of light is a huge showstopping issue is the trading computers at wall street

True, but even a normal site might notice it. Australia is 16,000 km from NY, so light takes 53 ms just to make a single one-way trip. Those add up pretty fast. 10 round trips is a second.
posted by smackfu at 5:04 AM on June 23, 2011

(gilrain, that should be "to say nothing of the dog"!)
posted by wenestvedt at 8:18 AM on June 23, 2011

Instapaper has a detailed blog post up now, and he is exploring legal options although he (very rationally) expects that nothing can be done. He is also very critical of how DigitalOne has handled everything.

He only had one server seized, but it's interesting to read what actual, sensitive data that included. (Like encrypted usernames and passwords for Instapaper's Pinboard users, oddly... including the encryption keys!) It's likely the FBI will never glance twice at anything aside from what they were looking for, but it's certainly a concern.
posted by gilrain at 11:27 AM on June 23, 2011

The only servers where the speed of light is a huge showstopping issue is the trading computers at wall street

True, but even a normal site might notice it. Australia is 16,000 km from NY, so light takes 53 ms just to make a single one-way trip. Those add up pretty fast. 10 round trips is a second.

I get it, I get it. We need to destroy the sun.
posted by TwelveTwo at 12:23 PM on June 23, 2011 [1 favorite]