Comments on: Still going...

Still going...

kuperman — Wed, 31 Aug 2005 15:41:36 -0800

Everyone is (probably) familiar with Something Awful. However, you may not be familiar with their hosting company - located in a New Orleans office building on Poydras in the CBD... but have you noticed that SA hasn't gone blank yet? It's because Zipa, and directNIC upstairs have the whole data center disaster contingency thing on lockdown. Blog and pictures from the directNIC guys are regularly updated. Color me impressed.

By: wheelieman

wheelieman — Wed, 31 Aug 2005 15:51:12 -0800

God bless people like this who run major operations without being asked. Hats off to you.

By: fenriq

fenriq — Wed, 31 Aug 2005 15:53:11 -0800

Okay, this part is pretty scary. the people we've been talking to say they are not recognizing the NOPD [New Orleans Police Department] as a legitimate authority anymore, since cops have been seen looting in Walmarts and forcing people out of stores so they could back up SUVs and loot them.. So, if the cops are the looters and the cops are shooting looters then does this mean that the cops are shooting themselves?

By: odinsdream

odinsdream — Wed, 31 Aug 2005 16:00:36 -0800

I have a lot of people telling us to abandon ship and get out. Guys, that's not gonna happen. I'll eat roaches and drink the funky Quarter sludge in the gutters of Bourbon Street long before I abandon my city. I've got resources and will and so does my team, and we're here until this is over. This should go directly on their front page, right above Hosting Plans & Prices.

By: xmutex

xmutex — Wed, 31 Aug 2005 16:02:23 -0800

Fascinating. Great link.

By: stbalbach

stbalbach — Wed, 31 Aug 2005 16:10:26 -0800

We are pleased to announce that we haven't lost service once during this entire distaster, and we have three weeks of backup power secured, and more resources are on the way. Additionally, backup options outside of New Orleans are being implemented as an added precaution. Impressive. Unsung data center heroes.

By: MillMan

MillMan — Wed, 31 Aug 2005 16:20:48 -0800

While amazing, I wonder what they were doing in NO to begin with...

By: Stan Chin

Stan Chin — Wed, 31 Aug 2005 16:22:09 -0800

Uhm...excuse my ignorance, but seriously, can't that data just be put somewhere else? And can't they be using that generator with AC with enough juice for three weeks to save lives? Not trying to be an ass, I really want to know why uptime is so important.

By: felix

felix — Wed, 31 Aug 2005 16:24:38 -0800

Not to downplay these guys' impressive commitment, but this exact specific circumstance of having to patrol your shattered building in a LARP version of Day of the Dead is what offsite disaster recovery locations are for. On 9/11, a sizable percentage of NY's IT infrastructure got pulverized, but nearly every midsize-and-up financial institution pressed their failover button and were ready to do business, in some form, from a non-Manhattan building within hours. Maybe these guys should talk with e.g., Sungard.

By: Mid

Mid — Wed, 31 Aug 2005 16:26:47 -0800

My intro to psych 101 says that this is a hard-core data-center geek's way of coping with disaster. Servers must stay up at all costs!

By: effugas

effugas — Wed, 31 Aug 2005 16:30:18 -0800

felix-- When it's your data, you can replicate it elsewhere. When it's someone else's -- that's a much harder problem. And it's aggressively nontrivial to replicate non-static content across multiple independent locations, especially when databases get involved. Something Awful isn't built like a bank. Almost nothing is, really.

By: realcountrymusic

realcountrymusic — Wed, 31 Aug 2005 16:30:22 -0800

Wow . . . a live running webcam from downtown is here . Go easy on it. They have a backup feed as well on the blog, which I won't post to keep the traffic from swamping it. Amazing.

By: odinsdream

odinsdream — Wed, 31 Aug 2005 16:32:04 -0800

Stan, the generator is likely several miles horizontally, and several stories vertically, from people who need help, and is probably very specialized to the equipment it powers, and highly integrated. Your post reads like someone asking the rhetorical question, "Why are we trying to go to Mars when those scientists could be curing cancer?" Well, because scientists have specialties, and Getting-To-Mars scientists are different than Curing-Cancer scientists. Similarly, these people have a job to do that they know how to do very well, i.e., keep a network running. Just because they have the equipment and the knowledge to do this doesn't mean they also have the equipment and knowledge (or even the responsibility) to go out into the streets and try to quell food riots (this despite the fact that the blog author clearly wants to do so, and probably will if it gets much worse and he is able to stabilize his own operation).

By: realcountrymusic

realcountrymusic — Wed, 31 Aug 2005 16:33:08 -0800

what odinsdream said, and the fpp. i assume someone at NSA must be looking up these guys now for offers.

By: Stan Chin

Stan Chin — Wed, 31 Aug 2005 16:33:22 -0800

Ooooooh.

By: zerokey

zerokey — Wed, 31 Aug 2005 16:34:45 -0800

As a sysadmin, I find this impressive. At the same time, I would rather see one of the hospitals get that 3 weeks of emergency power rather than see SomethingAwful, but that's just me. Not to burst the bubble (as cool as it really is), but I think lives should come first.

By: armoured-ant

armoured-ant — Wed, 31 Aug 2005 16:35:46 -0800

This is great stuff. Truly the best of the web.

By: elpapacito

elpapacito — Wed, 31 Aug 2005 16:38:08 -0800

So, if the cops are the looters and the cops are shooting looters then does this mean that the cops are shooting themselves? No no no , cops "find" while people "loot" ..so cops don't loot, they find, so there's no logical paradox. Anyway, you know a lot of people can't swim and they still complain about cop looting do they ?

By: zerokey

zerokey — Wed, 31 Aug 2005 16:38:21 -0800

On second read, I realize that I was harsh. I'm probably just more upset at the fact that the hospitals don't have the direct power feeds that this company does.

By: smackfu

smackfu — Wed, 31 Aug 2005 16:39:09 -0800

As far as "just move the data", they do colocation, so they would also have to physically move the systems elsewhere. I don't know any good solution to that, other than to foist it off on your customers. Felix isn't wrong. Disaster recovery plans are a big deal in financial applications, and they usually start from a scenario of "the town with the server no longer exists". But that is very expensive, since you need backup offsite hardware, and very regular offsite tape backups of the data. It's also slow, usually the recovery time is measured in days. So if you can keep your systems live with just a generator, more power to you. As far as using the generators to help people, I don't think that's an option unless they're ordered by the government. They have the generators because their customers pay them to have power backup, not to be altruistic.

By: elpapacito

elpapacito — Wed, 31 Aug 2005 16:40:58 -0800

Ops should have been on preview: More details about the alleged officiers looting stores.

By: realcountrymusic

realcountrymusic — Wed, 31 Aug 2005 16:41:31 -0800

If you read the blog, the guy went out and scrounged diesel fuel from a friend's dad's boat. Now on some strange planet, that fuel might have wound up in a hospital (I don't think you could actually move a generator across Poydras right now without getting assaulted by mobs, and I can't imagine you'd get far carrying a few cans of fuel). But any line of communications into the city has to be valuable. So I hope they can figure out a way to offer network access to law enforcement (prob. federal, since the NOLA cops appear to have lost this guy's faith).

By: PissOnYourParade

PissOnYourParade — Wed, 31 Aug 2005 16:48:36 -0800

Keeping the data centering going is pretty impressive stuff, but my question is how are they keeping connectivity going. I'm really really surprised that even redundant fiber connects are still going under a couple feet of water. And hitting that homepage, that ain't a sat connection... 3 weeks of diesel fuel for a big data center sure is alot. It's really hard to get a feel for what is happening down there. On one hand I look at nola.com and see chaos, but these guys are talking about returning from their safe harbors and seems to have a secured ongoing source of diesel fuel. btw - real time replication of data is not all that hard. Just put it on a SAN and do a business continuity replication with it. We use a device from Kashya to queue up the SAN packets at the switch level and ship them off to a datacenter about 120 miles away over a VPN connection. Usually we're only a few seconds behind.

By: fenriq

fenriq — Wed, 31 Aug 2005 16:50:51 -0800

since the NOLA cops appear to have lost this guy's faith But the cops did "find" some nice new big screen tv's and a few new SUVs.........

By: felix

felix — Wed, 31 Aug 2005 16:53:25 -0800

effugas writes: When it's your data, you can replicate it elsewhere. When it's someone else's -- that's a much harder problem. And it's aggressively nontrivial to replicate non-static content across multiple independent locations, especially when databases get involved. Aggressively nontrivial, but it's a solved problem -- especially when it's okay for the data to be a few minutes or hours old at the time of failover, as is the case with most web hosting. Even mysql has built-in and widely understood solutions. Something Awful isn't built like a bank. Almost nothing is, really. I happen to work at a large financial institution; you'd be surprised at how similar the infrastructure and systems are between any given asset manager and any given retail web operation. We trade operations guys with hosting companies and e-tailers all the time. SA vs. Bank: Enterprise-level DBs on one or more Unix boxes? Check. Multiple network connections with routing redundancy? Check. Critical uptime, with failover/recovery plan? Check. Scheduled backups and archiving strategy? Check. Transaction-based very high volume posting engine with validation, authentication, and audit trailing? Check. Custom coded Java or similar three tier, possibly web based, front end with standard application server middle tier encapsulating business logic? Check.

By: smackfu

smackfu — Wed, 31 Aug 2005 16:54:25 -0800

I think the big difference is that the bank has money.

By: cmonkey

cmonkey — Wed, 31 Aug 2005 17:04:29 -0800

I'm really really surprised that even redundant fiber connects are still going under a couple feet of water. And hitting that homepage, that ain't a sat connection... The transatlantic fibre cables operate under several miles of water at the bottom of the ocean. And fibre is typically buried fairly deeply in urban areas. You'll also notice from that webcam that there doesn't appear to be any flooding around the building, so as long as they have power and the other end of the fibre connection has power, they don't need to worry about connectivity.

By: xmutex

xmutex — Wed, 31 Aug 2005 17:08:29 -0800

Aggressively nontrivial Are you some horrible managementspeak drone on the loose or somethin'?

By: felix

felix — Wed, 31 Aug 2005 17:10:32 -0800

It's a given that SA isn't going to run a SAN replicator between coastal Netapps, but SA has paid ads and membership -- I bet they make enough to have a low-rent, 'one hour data is lost', scrape-by failover system on each coast and not even notice the cost. And hosting companies have it easy -- they get to aggregate, so the cost of defending their thousands of customers with a serious failover solution is almost a no brainer.

By: zsazsa

zsazsa — Wed, 31 Aug 2005 17:16:57 -0800

The Interdictor blog guy seems to be living out some sort of Mad Max survivalist fantasy, only he's in a datacenter instead of the Outback.

By: carter

carter — Wed, 31 Aug 2005 17:49:16 -0800

Cyberpunk, man.

By: spazzm

spazzm — Wed, 31 Aug 2005 17:54:11 -0800

I'm impressed. There it is, the nightmare scenario: Society is falling to pieces, there's no law, now power, no food and innocent people are drowning in sewage. But these guys, these guys have got their shit together. They know what to do, how to do it and when to do it. They've got a job to do. And (unlike the police) they're doin' it, hurricane be damned. I tip my hat to you, Geeks.

By: rlef98

rlef98 — Wed, 31 Aug 2005 17:56:05 -0800

To those people who are naysaying this guy's post-apocalyptic attention to making sure the data center stays up and running: what would you say if his servers were hosting redcross.org or some other important aid-relief type website?

By: TwelveTwo

TwelveTwo — Wed, 31 Aug 2005 18:05:31 -0800

We probably wouldn't be as impressed. Half of the awe I get out of this is that this isn't the server farm of the Red Cross, or some big bank.

By: zsazsa

zsazsa — Wed, 31 Aug 2005 18:23:52 -0800

DirectNIC is a somewhat major domain registrar, so they aren't just some el-cheapo hosting company.

By: ahughey

ahughey — Wed, 31 Aug 2005 18:25:29 -0800

Yeah, bunch of jerks, those guys:

By: Civil_Disobedient

Civil_Disobedient — Wed, 31 Aug 2005 18:51:41 -0800

You know those AskMe threads where someone asks what would you do in a zombie attack, and everyone starts getting excited and recommends different strategies and guns and such? Well, they're living the dream, baby. Fucking incredible job, guys (and gals). My hope is that they're somehow archiving the video feeds to forward to real fucking cops that put each and every one of those looting blue bastards behind pound-me-in-the-ass prison for years.

By: effugas

effugas — Wed, 31 Aug 2005 18:54:03 -0800

Aggressively nontrivial -- would you prefer the other way this can be said, which is "Metric Pain In The F***ing Ass"? What I don't think some people are understanding is that the datacenter does not administer the servers inside. In other words -- it's just not part of the job description to set up some kind of large scale replication environment. Now, there are colos that cater to customers who by their nature adopt all the financial scale redundancy they can eat (and I was wrong about that -- there are indeed lots of e-commerce operations that do manage replication across distance). For instance, the whole point of this company (Netli) is that Replication Is Hard. That doesn't mean it hasn't been solved, but it's ... like I said, aggressively nontrivial. And when you're not the administrator of the underlying hardware -- impossible. The coolest things start happening when these guys get their hands on gear from Vivato (or some sort of WiMax provider) and blanket New Orleans with wireless.

By: longdaysjourney

longdaysjourney — Wed, 31 Aug 2005 18:56:29 -0800

I really wish people wouldn't use the word "monkeys" to describe the looters. Even if there's no racial intent behind the word (which, honestly, is a bit hard to believe given that this guy served in the army), the word has a long history of being used as a racial slur and every neonazi on the web will be linking to that entry as "proof" of their fucked up racist beliefs.

By: StarForce5

StarForce5 — Wed, 31 Aug 2005 19:13:41 -0800

Very impressive datacenter. The photo gallery on the other hand in the unfortunate result of someone with too much camera memory and too much bandwidth: a page of 900 thumbnails where each photo is repeated 7 times!

By: PY

PY — Wed, 31 Aug 2005 20:03:18 -0800

The latest message on that webcam - "Onan generators kick ass!"

By: cavalier

cavalier — Wed, 31 Aug 2005 20:17:16 -0800

It's impressive to me that the interdictor guy from LJ has a real job and is doing all this -- no offense macho dude but when I was reading your comments from insomnia_LJ's collection a few days ago I had you kinda figured as some ex-army nutjob itching to shoot something. That might still be the case, I guess, but apparently you can run a day job too! So um, cheers.

By: rough ashlar

rough ashlar — Wed, 31 Aug 2005 20:37:29 -0800

does this mean that the cops are shooting themselves? You are attempting to apply logic to an arm of the US "justice" system. Once you reject the rhetoric about what America offers, the cognitive dissonance will go away.

By: drezdn

drezdn — Wed, 31 Aug 2005 20:56:41 -0800

Weird, I read his lj when this first went down, now it's good to know people are defending the photoshop phridays.

By: event

event — Wed, 31 Aug 2005 22:13:39 -0800

longdaysjourney: Even if there's no racial intent behind the word (which, honestly, is a bit hard to believe given that this guy served in the army) Wow. Just wow.

By: bugbread

bugbread — Wed, 31 Aug 2005 22:21:14 -0800

What wow?

By: event

event — Wed, 31 Aug 2005 22:44:52 -0800

He yells about the guy using the word monkey. Next sentence, he calls all soldiers racist.

By: bugbread

bugbread — Wed, 31 Aug 2005 22:47:20 -0800

Ah, ok, thanks.

By: tkb

tkb — Wed, 31 Aug 2005 23:02:53 -0800

I don't think longdaysjourney called all soldiers racist. interdictor was in the Army. This was a unique formative experience. In the course of acquiring that experience, interdictor was exposed to the racist uses and implications of using the word monkey. In spite of that experience and that exposure, he used the word. Maybe interdictor isn't a racist, but in any case he doesn't mind appearing to be a racist.

By: bugbread

bugbread — Thu, 01 Sep 2005 00:25:46 -0800

tkb: that's how I interpreted it too (i.e. "he can't have used it unknowingly, because it's a common word in the military, and he was in the military"), but on rereading, it says "even if there's no racial intent behind the word (which, honestly, is a bit hard to believe given that this guy served in the army)" and not "even if he was unaware of the racial connotations of the word (which, honestly, is a bit hard to believe given that this guy served in the army)". I'm hoping it was just a flub of writing, but, as it's written, it does say that, because he was military, it's hard to believe it was used without racial intent.

By: insomnia_lj

insomnia_lj — Thu, 01 Sep 2005 01:05:46 -0800

Interdictor's reports from within the city have been a bit insular. He and his are locked up tight in the data center, and although they have the cam as their eyes and ears, if all you paid attention to was his journal, you'd think that New Orleans was spared, even if you could walk a block or two and see that not everyone was so lucky. I also do tend to think he's more than a bit sexist, and quite possibly racist. His crack about how women would panic under such circumstances was uncalled for. I know of one LJer who, the day before the storm, could've evacuated, but decided to stay in order to help out at the hospital. She's no doubt been too busy to post, and hasn't had the luxury of being safely inside a highly secure, locked down building. There are a lot of others in New Orleans who are doing a lot more for the people, with a whole lot less bravado. I'm very proud of firemanjason, who is leading a medical / rescue team in New Orleans as we speak. His last public comment was appropriate... "went friends only so CNN don't read my shit out loud."

By: longdaysjourney

longdaysjourney — Thu, 01 Sep 2005 04:28:23 -0800

Interdictor was in the Army. This was a unique formative experience. In the course of acquiring that experience, interdictor was exposed to the racist uses and implications of using the word monkey. In spite of that experience and that exposure, he used the word. What tkb said. Typing too fast. Sorry about that.

By: longdaysjourney

longdaysjourney — Thu, 01 Sep 2005 05:05:34 -0800

See what I mean?

By: Ogre Lawless

Ogre Lawless — Thu, 01 Sep 2005 09:30:50 -0800

Oh come on, I'm sure he's a sweet guy The Interdictor You are alive because I tolerate you. Interests: 5: assassination, crisis management, rescue operations, target interdiction, termination. I'm sure working with him is a pleasure.

By: Sangre Azul

Sangre Azul — Thu, 01 Sep 2005 09:30:59 -0800

odinsdream mostly said it, but regarding generators -- if theirs is anything like the generator that services my building (which hosts a small data center) it is essentially a structure unto itself and unmovable for all intents and purposes. This is not something you'd find at Home Depot -- it's built-in.

By: phearlez

phearlez — Thu, 01 Sep 2005 09:53:01 -0800

I hope that when I find myself in a disaster area, locked into a fucking office building 24/7, that my efforts at providing information to the outside world aren't over-analyzed for half-obscure racial sensitivity by some jerkoff speculating on what I should and shouldn't know and have been exposed to.

By: longdaysjourney

longdaysjourney — Thu, 01 Sep 2005 09:57:01 -0800

Sorry phearlez, I've just seen too many references to monkeys, animals, apes, trash, lord of the flies, mad max over the past few hours. The English language has a perfectly good word for looter, which is looter. Sad that people have to resort to invective to communicate. Even if they're under stress.

By: pandaharma

pandaharma — Thu, 01 Sep 2005 10:55:12 -0800

Yes, this Interdictor chap should use a different sort of language. "Naughty fellows" "Scurrilous gentlemen" "Dastardly thieves" would all be appropriate here. And he shouldn't use violent speech in referring to how he would deal with them. Rather he should describe how he would take some tea and cookies down to the street and give the naughty fellows a stern lecture on the wrongness of their actions followed by an inspiring talk about how we are all indeed brothers. This would make the situation much better and would be much more acceptable to the delicate sensibilities on display here.

By: longdaysjourney

longdaysjourney — Thu, 01 Sep 2005 10:59:30 -0800

As a retired Police Officer from Philadelphia the "Niggers"= scum sucking assholes of the RACES are stealing from the working class and should be shot on sight. Somebody else said they were thief's be for the flood you were correct!!! The Gov. Should have had the National Guards on the scene and ready...LOCK AND LOAD!!!! I rest my case. Nice to see you don't have a problem with crap like this though. (And for the clueless, no, I'm not suggesting he police his comments. Just don't be surprised when assholes get their cues from the words you use.)

By: Ogre Lawless

Ogre Lawless — Thu, 01 Sep 2005 11:16:38 -0800

Well, you see Phearlez... In this flood, things get confused out there: power, ideals, the old morality, and practical military necessity. Out there with these natives it must be a temptation to be god. Because there's a conflict in every human heart between the rational and the irrational, between good and evil. The good does not always triumph. Sometimes the dark side overcomes what Lincoln called the better angels of our nature. Every man has got a breaking point. You and I have. Interdictor has reached his. And very obviously, he has gone insane. *

By: kuperman

kuperman — Thu, 01 Sep 2005 11:17:20 -0800

Looks like they're losing steam... updates on his blog.

By: pandaharma

pandaharma — Thu, 01 Sep 2005 11:18:39 -0800

I have a feeling Interdictor would use the same language if it were asians, caucasians, or chihuahuas looting and rampaging. And I can't fault his use of the language nor can I fault his use of violent thoughts. When you see your city already at the breaking point pushed further by selfish and violent people, I would imagine even you would increase the severity of your tone.

By: smackfu

smackfu — Thu, 01 Sep 2005 12:07:24 -0800

Rats, Something Awful is down now. Apparently not mission critical.

By: phearlez

phearlez — Thu, 01 Sep 2005 13:20:53 -0800

Nice to see you don't have a problem with crap like this though. Are you fucking kidding me with that shit? I suggest that someone in an extreme situation shouldn't be expected to be as cognizent of situational subtleties of an otherwise perfectly acceptable word and you come back accusing me of being okay with violent racism? Maybe it's appropriate to the subject that I'm thinking of a destination for your keyboard and it is often associated with the color brown. Should be easy enough for you to accomplish, just put it next to where your head is.

By: longdaysjourney

longdaysjourney — Thu, 01 Sep 2005 14:48:41 -0800

I accused you of nothing. I think you're being willfully blind to the impact of language though. Everyone i have seen looting is a NIGGER!!! yes--- that is racist.... kill em all! Have a nice day.

By: phearlez

phearlez — Thu, 01 Sep 2005 15:51:32 -0800

A) The fuck you didn't. "You don't have a problem with crap like this" is quite clear. If you're going to be insulting you should have the courage of your convictions and own up to it. B) You know perfectly well that there's a difference between suggesting that language has no impact and suggesting that someone sleeping in an office building on a concrete floor in a near war-zone has better things to worry about and maybe should be held to a different standard than you and I in our comfy chairs.

By: iron chef morimoto

iron chef morimoto — Thu, 01 Sep 2005 15:57:32 -0800

I only pray that stileproject and ogrish are similarly insulated against natural disasters.

By: bugbread

bugbread — Thu, 01 Sep 2005 15:58:53 -0800

longdaysjourney : "I rest my case." Uh...what case is that? That the guy is racist? In which case, your evidence is a racist comment by someone else? Or your case is that you wish he didn't use the word "monkey"? In which case, we believe you when you say you believe something. No evidence is needed to prove that you believe something.

By: amberglow

amberglow — Thu, 01 Sep 2005 20:16:15 -0800

I started reading it yesterday and stopped for precisely that reason longdaysjourney--let him spout his shit all he wants--don't applaud him for his Bunker mentality (Archie and warzone).

By: jb

jb — Thu, 01 Sep 2005 21:32:04 -0800

Someone made an interesting point in the comments - that looted goods could later be traded for food and water. Or jewelry could be sold after they are out of town. Considering that people are now homeless, with only the clothes on their backs (and some not that much of that), taking some jewelry would be a good idea.