Still going...

August 31, 2005 3:41 PM

Everyone is (probably) familiar with Something Awful. However, you may not be familiar with their hosting company - located in a New Orleans office building on Poydras in the CBD... but have you noticed that SA hasn't gone blank yet? It's because Zipa, and directNIC upstairs have the whole data center disaster contingency thing on lockdown. Blog and pictures from the directNIC guys are regularly updated. Color me impressed.
posted by kuperman (69 comments total)

God bless people like this who run major operations without being asked. Hats off to you.
posted by wheelieman at 3:51 PM on August 31, 2005

Okay, this part is pretty scary.
the people we've been talking to say they are not recognizing the NOPD [New Orleans Police Department] as a legitimate authority anymore, since cops have been seen looting in Walmarts and forcing people out of stores so they could back up SUVs and loot them..

So, if the cops are the looters and the cops are shooting looters then does this mean that the cops are shooting themselves?
posted by fenriq at 3:53 PM on August 31, 2005

I have a lot of people telling us to abandon ship and get out. Guys, that's not gonna happen. I'll eat roaches and drink the funky Quarter sludge in the gutters of Bourbon Street long before I abandon my city. I've got resources and will and so does my team, and we're here until this is over.

This should go directly on their front page, right above Hosting Plans & Prices.
posted by odinsdream at 4:00 PM on August 31, 2005

Fascinating. Great link.
posted by xmutex at 4:02 PM on August 31, 2005

We are pleased to announce that we haven't lost service once during this entire distaster, and we have three weeks of backup power secured, and more resources are on the way. Additionally, backup options outside of New Orleans are being implemented as an added precaution.

Impressive. Unsung data center heroes.
posted by stbalbach at 4:10 PM on August 31, 2005

While amazing, I wonder what they were doing in NO to begin with...
posted by MillMan at 4:20 PM on August 31, 2005

Uhm...excuse my ignorance, but seriously, can't that data just be put somewhere else? And can't they be using that generator with AC with enough juice for three weeks to save lives? Not trying to be an ass, I really want to know why uptime is so important.
posted by Stan Chin at 4:22 PM on August 31, 2005

Not to downplay these guys' impressive commitment, but this exact specific circumstance of having to patrol your shattered building in a LARP version of Day of the Dead is what offsite disaster recovery locations are for.

On 9/11, a sizable percentage of NY's IT infrastructure got pulverized, but nearly every midsize-and-up financial institution pressed their failover button and were ready to do business, in some form, from a non-Manhattan building within hours.

Maybe these guys should talk with e.g., Sungard.
posted by felix at 4:24 PM on August 31, 2005

My intro to psych 101 says that this is a hard-core data-center geek's way of coping with disaster. Servers must stay up at all costs!
posted by Mid at 4:26 PM on August 31, 2005

felix--

When it's your data, you can replicate it elsewhere.

When it's someone else's -- that's a much harder problem.
And it's aggressively nontrivial to replicate non-static content across multiple independent locations, especially when databases get involved.

Something Awful isn't built like a bank. Almost nothing is, really.
posted by effugas at 4:30 PM on August 31, 2005

Wow . . . a live running webcam from downtown is here . Go easy on it. They have a backup feed as well on the blog, which I won't post to keep the traffic from swamping it. Amazing.
posted by realcountrymusic at 4:30 PM on August 31, 2005

Stan, the generator is likely several miles horizontally, and several stories vertically, from people who need help, and is probably very specialized to the equipment it powers, and highly integrated. Your post reads like someone asking the rhetorical question, "Why are we trying to go to Mars when those scientists could be curing cancer?"

Well, because scientists have specialties, and Getting-To-Mars scientists are different than Curing-Cancer scientists. Similarly, these people have a job to do that they know how to do very well, i.e., keep a network running. Just because they have the equipment and the knowledge to do this doesn't mean they also have the equipment and knowledge (or even the responsibility) to go out into the streets and try to quell food riots (this despite the fact that the blog author clearly wants to do so, and probably will if it gets much worse and he is able to stabilize his own operation).
posted by odinsdream at 4:32 PM on August 31, 2005

what odinsdream said, and the fpp. i assume someone at NSA must be looking up these guys now for offers.
posted by realcountrymusic at 4:33 PM on August 31, 2005

Ooooooh.
posted by Stan Chin at 4:33 PM on August 31, 2005

As a sysadmin, I find this impressive.

At the same time, I would rather see one of the hospitals get that 3 weeks of emergency power rather than see SomethingAwful, but that's just me. Not to burst the bubble (as cool as it really is), but I think lives should come first.
posted by zerokey at 4:34 PM on August 31, 2005

This is great stuff. Truly the best of the web.
posted by armoured-ant at 4:35 PM on August 31, 2005

So, if the cops are the looters and the cops are shooting looters then does this mean that the cops are shooting themselves?

No no no , cops "find" while people "loot" ..so cops don't loot, they find, so there's no logical paradox.

Anyway, you know a lot of people can't swim and they still complain about cop looting do they ?
posted by elpapacito at 4:38 PM on August 31, 2005

On second read, I realize that I was harsh. I'm probably just more upset at the fact that the hospitals don't have the direct power feeds that this company does.
posted by zerokey at 4:38 PM on August 31, 2005

As far as "just move the data", they do colocation, so they would also have to physically move the systems elsewhere. I don't know any good solution to that, other than to foist it off on your customers.

Felix isn't wrong. Disaster recovery plans are a big deal in financial applications, and they usually start from a scenario of "the town with the server no longer exists". But that is very expensive, since you need backup offsite hardware, and very regular offsite tape backups of the data. It's also slow, usually the recovery time is measured in days. So if you can keep your systems live with just a generator, more power to you.

As far as using the generators to help people, I don't think that's an option unless they're ordered by the government. They have the generators because their customers pay them to have power backup, not to be altruistic.
posted by smackfu at 4:39 PM on August 31, 2005

Ops should have been on preview:

More details about the alleged officiers looting stores.
posted by elpapacito at 4:40 PM on August 31, 2005

If you read the blog, the guy went out and scrounged diesel fuel from a friend's dad's boat. Now on some strange planet, that fuel might have wound up in a hospital (I don't think you could actually move a generator across Poydras right now without getting assaulted by mobs, and I can't imagine you'd get far carrying a few cans of fuel). But any line of communications into the city has to be valuable. So I hope they can figure out a way to offer network access to law enforcement (prob. federal, since the NOLA cops appear to have lost this guy's faith).
posted by realcountrymusic at 4:41 PM on August 31, 2005

Keeping the data centering going is pretty impressive stuff, but my question is how are they keeping connectivity going.

I'm really really surprised that even redundant fiber connects are still going under a couple feet of water. And hitting that homepage, that ain't a sat connection...

3 weeks of diesel fuel for a big data center sure is alot. It's really hard to get a feel for what is happening down there. On one hand I look at nola.com and see chaos, but these guys are talking about returning from their safe harbors and seems to have a secured ongoing source of diesel fuel.

btw - real time replication of data is not all that hard. Just put it on a SAN and do a business continuity replication with it. We use a device from Kashya to queue up the SAN packets at the switch level and ship them off to a datacenter about 120 miles away over a VPN connection. Usually we're only a few seconds behind.
posted by PissOnYourParade at 4:48 PM on August 31, 2005

since the NOLA cops appear to have lost this guy's faith

But the cops did "find" some nice new big screen tv's and a few new SUVs.........
posted by fenriq at 4:50 PM on August 31, 2005

effugas writes:
When it's your data, you can replicate it elsewhere. When it's someone else's -- that's a much harder problem. And it's aggressively nontrivial to replicate non-static content across multiple independent locations, especially when databases get involved.

Aggressively nontrivial, but it's a solved problem -- especially when it's okay for the data to be a few minutes or hours old at the time of failover, as is the case with most web hosting. Even mysql has built-in and widely understood solutions.


Something Awful isn't built like a bank. Almost nothing is, really.

I happen to work at a large financial institution; you'd be surprised at how similar the infrastructure and systems are between any given asset manager and any given retail web operation. We trade operations guys with hosting companies and e-tailers all the time.

SA vs. Bank:

Enterprise-level DBs on one or more Unix boxes? Check.
Multiple network connections with routing redundancy? Check.
Critical uptime, with failover/recovery plan? Check.
Scheduled backups and archiving strategy? Check.
Transaction-based very high volume posting engine with validation, authentication, and audit trailing? Check.
Custom coded Java or similar three tier, possibly web based, front end with standard application server middle tier encapsulating business logic? Check.
posted by felix at 4:53 PM on August 31, 2005

I think the big difference is that the bank has money.
posted by smackfu at 4:54 PM on August 31, 2005

I'm really really surprised that even redundant fiber connects are still going under a couple feet of water. And hitting that homepage, that ain't a sat connection...

The transatlantic fibre cables operate under several miles of water at the bottom of the ocean. And fibre is typically buried fairly deeply in urban areas. You'll also notice from that webcam that there doesn't appear to be any flooding around the building, so as long as they have power and the other end of the fibre connection has power, they don't need to worry about connectivity.
posted by cmonkey at 5:04 PM on August 31, 2005

Aggressively nontrivial

Are you some horrible managementspeak drone on the loose or somethin'?
posted by xmutex at 5:08 PM on August 31, 2005

It's a given that SA isn't going to run a SAN replicator between coastal Netapps, but SA has paid ads and membership -- I bet they make enough to have a low-rent, 'one hour data is lost', scrape-by failover system on each coast and not even notice the cost.

And hosting companies have it easy -- they get to aggregate, so the cost of defending their thousands of customers with a serious failover solution is almost a no brainer.
posted by felix at 5:10 PM on August 31, 2005

The Interdictor blog guy seems to be living out some sort of Mad Max survivalist fantasy, only he's in a datacenter instead of the Outback.
posted by zsazsa at 5:16 PM on August 31, 2005

Cyberpunk, man.
posted by carter at 5:49 PM on August 31, 2005

I'm impressed.
There it is, the nightmare scenario: Society is falling to pieces, there's no law, now power, no food and innocent people are drowning in sewage.
But these guys, these guys have got their shit together. They know what to do, how to do it and when to do it. They've got a job to do. And (unlike the police) they're doin' it, hurricane be damned.


I tip my hat to you, Geeks.
posted by spazzm at 5:54 PM on August 31, 2005

To those people who are naysaying this guy's post-apocalyptic attention to making sure the data center stays up and running: what would you say if his servers were hosting redcross.org or some other important aid-relief type website?
posted by rlef98 at 5:56 PM on August 31, 2005

We probably wouldn't be as impressed. Half of the awe I get out of this is that this isn't the server farm of the Red Cross, or some big bank.
posted by TwelveTwo at 6:05 PM on August 31, 2005

DirectNIC is a somewhat major domain registrar, so they aren't just some el-cheapo hosting company.
posted by zsazsa at 6:23 PM on August 31, 2005

Yeah, bunch of jerks, those guys:


posted by ahughey at 6:25 PM on August 31, 2005

You know those AskMe threads where someone asks what would you do in a zombie attack, and everyone starts getting excited and recommends different strategies and guns and such? Well, they're living the dream, baby.

Fucking incredible job, guys (and gals).

My hope is that they're somehow archiving the video feeds to forward to real fucking cops that put each and every one of those looting blue bastards behind pound-me-in-the-ass prison for years.
posted by Civil_Disobedient at 6:51 PM on August 31, 2005 [1 favorite]

Aggressively nontrivial -- would you prefer the other way this can be said, which is "Metric Pain In The F***ing Ass"?

What I don't think some people are understanding is that the datacenter does not administer the servers inside. In other words -- it's just not part of the job description to set up some kind of large scale replication environment. Now, there are colos that cater to customers who by their nature adopt all the financial scale redundancy they can eat (and I was wrong about that -- there are indeed lots of e-commerce operations that do manage replication across distance). For instance, the whole point of this company (Netli) is that Replication Is Hard. That doesn't mean it hasn't been solved, but it's ... like I said, aggressively nontrivial. And when you're not the administrator of the underlying hardware -- impossible.

The coolest things start happening when these guys get their hands on gear from Vivato (or some sort of WiMax provider) and blanket New Orleans with wireless.
posted by effugas at 6:54 PM on August 31, 2005

I really wish people wouldn't use the word "monkeys" to describe the looters. Even if there's no racial intent behind the word (which, honestly, is a bit hard to believe given that this guy served in the army), the word has a long history of being used as a racial slur and every neonazi on the web will be linking to that entry as "proof" of their fucked up racist beliefs.
posted by longdaysjourney at 6:56 PM on August 31, 2005

Very impressive datacenter. The photo gallery on the other hand in the unfortunate result of someone with too much camera memory and too much bandwidth: a page of 900 thumbnails where each photo is repeated 7 times!
posted by StarForce5 at 7:13 PM on August 31, 2005

The latest message on that webcam - "Onan generators kick ass!"
posted by PY at 8:03 PM on August 31, 2005

It's impressive to me that the interdictor guy from LJ has a real job and is doing all this -- no offense macho dude but when I was reading your comments from insomnia_LJ's collection a few days ago I had you kinda figured as some ex-army nutjob itching to shoot something. That might still be the case, I guess, but apparently you can run a day job too! So um, cheers.
posted by cavalier at 8:17 PM on August 31, 2005

does this mean that the cops are shooting themselves?

You are attempting to apply logic to an arm of the US "justice" system.

Once you reject the rhetoric about what America offers, the cognitive dissonance will go away.
posted by rough ashlar at 8:37 PM on August 31, 2005

Weird, I read his lj when this first went down, now it's good to know people are defending the photoshop phridays.
posted by drezdn at 8:56 PM on August 31, 2005

longdaysjourney: Even if there's no racial intent behind the word (which, honestly, is a bit hard to believe given that this guy served in the army)

Wow. Just wow.
posted by event at 10:13 PM on August 31, 2005

What wow?
posted by Bugbread at 10:21 PM on August 31, 2005

He yells about the guy using the word monkey. Next sentence, he calls all soldiers racist.
posted by event at 10:44 PM on August 31, 2005

Ah, ok, thanks.
posted by Bugbread at 10:47 PM on August 31, 2005

I don't think longdaysjourney called all soldiers racist.

interdictor was in the Army. This was a unique formative experience. In the course of acquiring that experience, interdictor was exposed to the racist uses and implications of using the word monkey. In spite of that experience and that exposure, he used the word.

Maybe interdictor isn't a racist, but in any case he doesn't mind appearing to be a racist.
posted by tkb at 11:02 PM on August 31, 2005

tkb: that's how I interpreted it too (i.e. "he can't have used it unknowingly, because it's a common word in the military, and he was in the military"), but on rereading, it says "even if there's no racial intent behind the word (which, honestly, is a bit hard to believe given that this guy served in the army)" and not "even if he was unaware of the racial connotations of the word (which, honestly, is a bit hard to believe given that this guy served in the army)". I'm hoping it was just a flub of writing, but, as it's written, it does say that, because he was military, it's hard to believe it was used without racial intent.
posted by Bugbread at 12:25 AM on September 1, 2005

Interdictor's reports from within the city have been a bit insular. He and his are locked up tight in the data center, and although they have the cam as their eyes and ears, if all you paid attention to was his journal, you'd think that New Orleans was spared, even if you could walk a block or two and see that not everyone was so lucky.

I also do tend to think he's more than a bit sexist, and quite possibly racist. His crack about how women would panic under such circumstances was uncalled for. I know of one LJer who, the day before the storm, could've evacuated, but decided to stay in order to help out at the hospital. She's no doubt been too busy to post, and hasn't had the luxury of being safely inside a highly secure, locked down building.

There are a lot of others in New Orleans who are doing a lot more for the people, with a whole lot less bravado. I'm very proud of firemanjason, who is leading a medical / rescue team in New Orleans as we speak. His last public comment was appropriate...

"went friends only so CNN don't read my shit out loud."
posted by insomnia_lj at 1:05 AM on September 1, 2005

Interdictor was in the Army. This was a unique formative experience. In the course of acquiring that experience, interdictor was exposed to the racist uses and implications of using the word monkey. In spite of that experience and that exposure, he used the word.

What tkb said. Typing too fast. Sorry about that.
posted by longdaysjourney at 4:28 AM on September 1, 2005

See what I mean?
posted by longdaysjourney at 5:05 AM on September 1, 2005

Oh come on, I'm sure he's a sweet guy



The Interdictor
You are alive because I tolerate you.

Interests: 5: assassination, crisis management, rescue operations, target interdiction, termination.

I'm sure working with him is a pleasure.
posted by Ogre Lawless at 9:30 AM on September 1, 2005

odinsdream mostly said it, but regarding generators -- if theirs is anything like the generator that services my building (which hosts a small data center) it is essentially a structure unto itself and unmovable for all intents and purposes. This is not something you'd find at Home Depot -- it's built-in.
posted by Sangre Azul at 9:30 AM on September 1, 2005

I hope that when I find myself in a disaster area, locked into a fucking office building 24/7, that my efforts at providing information to the outside world aren't over-analyzed for half-obscure racial sensitivity by some jerkoff speculating on what I should and shouldn't know and have been exposed to.
posted by phearlez at 9:53 AM on September 1, 2005

Sorry phearlez, I've just seen too many references to monkeys, animals, apes, trash, lord of the flies, mad max over the past few hours. The English language has a perfectly good word for looter, which is looter. Sad that people have to resort to invective to communicate. Even if they're under stress.
posted by longdaysjourney at 9:57 AM on September 1, 2005

Yes, this Interdictor chap should use a different sort of language.

"Naughty fellows"

"Scurrilous gentlemen"

"Dastardly thieves"

would all be appropriate here.

And he shouldn't use violent speech in referring to how he would deal with them. Rather he should describe how he would take some tea and cookies down to the street and give the naughty fellows a stern lecture on the wrongness of their actions followed by an inspiring talk about how we are all indeed brothers.

This would make the situation much better and would be much more acceptable to the delicate sensibilities on display here.
posted by pandaharma at 10:55 AM on September 1, 2005

As a retired Police Officer from Philadelphia the "Niggers"= scum sucking assholes of the RACES are stealing from the working class and should be shot on sight. Somebody else said they were thief's be for the flood you were correct!!! The Gov. Should have had the National Guards on the scene and ready...LOCK AND LOAD!!!!

I rest my case. Nice to see you don't have a problem with crap like this though. (And for the clueless, no, I'm not suggesting he police his comments. Just don't be surprised when assholes get their cues from the words you use.)
posted by longdaysjourney at 10:59 AM on September 1, 2005

Well, you see Phearlez... In this flood, things get confused out there: power, ideals, the old morality, and practical military necessity. Out there with these natives it must be a temptation to be god. Because there's a conflict in every human heart between the rational and the irrational, between good and evil. The good does not always triumph. Sometimes the dark side overcomes what Lincoln called the better angels of our nature. Every man has got a breaking point. You and I have. Interdictor has reached his. And very obviously, he has gone insane. *
posted by Ogre Lawless at 11:16 AM on September 1, 2005

Looks like they're losing steam... updates on his blog.
posted by kuperman at 11:17 AM on September 1, 2005

I have a feeling Interdictor would use the same language if it were asians, caucasians, or chihuahuas looting and rampaging.

And I can't fault his use of the language nor can I fault his use of violent thoughts. When you see your city already at the breaking point pushed further by selfish and violent people, I would imagine even you would increase the severity of your tone.
posted by pandaharma at 11:18 AM on September 1, 2005

Rats, Something Awful is down now. Apparently not mission critical.
posted by smackfu at 12:07 PM on September 1, 2005

Nice to see you don't have a problem with crap like this though.

Are you fucking kidding me with that shit? I suggest that someone in an extreme situation shouldn't be expected to be as cognizent of situational subtleties of an otherwise perfectly acceptable word and you come back accusing me of being okay with violent racism?

Maybe it's appropriate to the subject that I'm thinking of a destination for your keyboard and it is often associated with the color brown. Should be easy enough for you to accomplish, just put it next to where your head is.
posted by phearlez at 1:20 PM on September 1, 2005

I accused you of nothing. I think you're being willfully blind to the impact of language though.

Everyone i have seen looting is a NIGGER!!! yes--- that is racist.... kill em all!

Have a nice day.
posted by longdaysjourney at 2:48 PM on September 1, 2005

A) The fuck you didn't. "You don't have a problem with crap like this" is quite clear. If you're going to be insulting you should have the courage of your convictions and own up to it.

B) You know perfectly well that there's a difference between suggesting that language has no impact and suggesting that someone sleeping in an office building on a concrete floor in a near war-zone has better things to worry about and maybe should be held to a different standard than you and I in our comfy chairs.
posted by phearlez at 3:51 PM on September 1, 2005

I only pray that stileproject and ogrish are similarly insulated against natural disasters.
posted by iron chef morimoto at 3:57 PM on September 1, 2005

longdaysjourney : "I rest my case."

Uh...what case is that? That the guy is racist? In which case, your evidence is a racist comment by someone else?
Or your case is that you wish he didn't use the word "monkey"? In which case, we believe you when you say you believe something. No evidence is needed to prove that you believe something.
posted by Bugbread at 3:58 PM on September 1, 2005

I started reading it yesterday and stopped for precisely that reason longdaysjourney--let him spout his shit all he wants--don't applaud him for his Bunker mentality (Archie and warzone).
posted by amberglow at 8:16 PM on September 1, 2005

Someone made an interesting point in the comments - that looted goods could later be traded for food and water. Or jewelry could be sold after they are out of town.

Considering that people are now homeless, with only the clothes on their backs (and some not that much of that), taking some jewelry would be a good idea.
posted by jb at 9:32 PM on September 1, 2005