Comments on: Singing in the Rain

Singing in the Rain

loquacious — Wed, 02 Apr 2008 03:28:40 -0800

Rain + server room in basement = nerd horror

By: pjern

pjern — Wed, 02 Apr 2008 03:38:50 -0800

heh. Reminds me of a time when I had an entire brokerage office flood over a weekend when a computer room installation on the floor above popped a 2 inch sprinkler line on Friday night, which was finally discovered on Sunday morning when water started flowing out of the elevator shafts on the first floor. 94 desktops and 4 servers and a couple hundred billable hours later....

By: pompomtom

pompomtom — Wed, 02 Apr 2008 03:42:38 -0800

But, on the bright side: Rain + server room in basement = admin department spends the arvo in the pub.

By: gen

gen — Wed, 02 Apr 2008 03:53:51 -0800

Jeebus. What a nightmare. At least they had the raised floor and only the cables were under water?

By: orthogonality

orthogonality — Wed, 02 Apr 2008 04:05:45 -0800

Rain + plushie Yoda collection in basement = nerd horror.

By: DU

DU — Wed, 02 Apr 2008 04:21:36 -0800

rain + living in parents' basement = wet nerd

By: bwg

bwg — Wed, 02 Apr 2008 04:36:53 -0800

rain + living in parents' basement + mom catching him fapping to stills of princess leia = nerd horror

By: soundofsuburbia

soundofsuburbia — Wed, 02 Apr 2008 04:54:49 -0800

HELP HELP SEA SNAKES UNDER THE FLOORBOARDS

By: mmcg

mmcg — Wed, 02 Apr 2008 05:03:47 -0800

Tragedy is when my mailserver has a hiccup. Comedy is when your server room floods and destroys all of your equipment in one fell swoop. But I digress. Hope they negotiated a decent disaster rate in their contract.

By: purephase

purephase — Wed, 02 Apr 2008 05:11:57 -0800

What a horribly designed server room. Feel bad for them, but I guarantee that this will most likely get them the necessary funding to do it properly.

By: smackfu

smackfu — Wed, 02 Apr 2008 05:32:59 -0800

At Rutgers back in the 70's, they thought they were clever. When they built a new complex of Math buildings, they built the server rooms underground, under the courtyard. This gave them a big footprint, but they had endless problems with leaks, because the courtyard basically had to be waterproof. I don't think they ever had a massive flood, but it was only a matter of time. So while I was attending, they ripped up every single brick, put a new waterproof membrane down, then built a new drainage system on top and redid the bricks. Must have cost a fortune. The best part was that they tested it by flooding the courtyard with six inches of water. It was fabulous and surreal. And I have a photo.

By: Horace Rumpole

Horace Rumpole — Wed, 02 Apr 2008 05:42:32 -0800

Rain + Underground special collections stacks + Librarian horror. Tell you what, my colleagues are like library ninjas that just swooped in and kicked that flood's ass.

By: smackfu

smackfu — Wed, 02 Apr 2008 05:51:26 -0800

At the New York Public library, they have stacks under Bryant Park (and are moving more there). In the winter they actually have an ice rink in Bryant Park. Talk about tempting fate.

By: jmd82

jmd82 — Wed, 02 Apr 2008 06:16:25 -0800

That doesn't look like just a rain "shower."

By: unixrat

unixrat — Wed, 02 Apr 2008 06:37:56 -0800

Now that's a great feature of raised floors that I hadn't considered. Well, I suppose it only buys you a little time to pull out the camera before the disaster really happens. :)

By: washburn

washburn — Wed, 02 Apr 2008 06:42:26 -0800

the Library Collections Emergency Team (LCET) arriving shortly after. Calling all screenwriters!

By: destrius

destrius — Wed, 02 Apr 2008 07:13:05 -0800

The Great Flood of 2005 Back when I was in CMU, a water mains located in one of the server rooms in Wean Hall (home of the School of Computer Science) broke. The resulting flood damaged tons of computers and I'm not sure how much data ended up being lost. I was an undergrad at the time so it didn't affect me much but imagine the poor grad students who had their entire academic lives stuck in a soaking wet hard drive. Not fun. Interestingly when I became a grad student I actually used one of the machines rescued from the flood for my research; I left it running for several months and it was still functioning perfectly.

By: Burhanistan

Burhanistan — Wed, 02 Apr 2008 07:23:10 -0800

During the great flood of 01 there were dozens of downtown and medical center underground data centers that were completely submerged before the respective managements relocated them to higher floors.

By: pineapple

pineapple — Wed, 02 Apr 2008 07:44:10 -0800

There is a state whose capital city is prone to sudden weather. The servers that run all three branches of government are located underground. One autumn, the Commander-in-Chief was coming to visit the capitol. Two weeks prior, his Secret Service advance team looked at the drain grates around the complex and said, "These grates are too wide, someone could slip bombs down them. We'll put temporary smaller grates of a fine mesh over them, or else the President won't visit." On the morning of the visit, there was a terrible storm. Leaves blew from trees by the bushel, and were thrown to the ground where they collected in rushing gutters. Ordinarily, the leaves would just wash away through the sewer grates, as designed. But thanks to the Secret Service's fine mesh toppers, the leaves piled up and up and up, blocking the drains completely. And the rainwater had nowhere to go and flooded into the servers' basement room. When everything was dried out and replaced, and the President had left the state, there was a well-intentioned suggestion from helpful state leadership to the IT team -- that they might build a wooden hut around the servers and cover it with a tarp, to prevent future server damage, in the case of future inclement weather, at a time when the city's drain and sewer system might be similarly compromised, creating possible flooding. The IT team smiled and nodded and dutifully installed the wooden hut over the servers. On places where no one can see, anonymous messages get scrawled on the wood, reflecting the IT team's colorful suggestions as to different ways the Secret Service and other various actors in the story might entertain themselves. Sadly, if understandably, there are no pictures to share.

By: Skorgu

Skorgu — Wed, 02 Apr 2008 08:08:52 -0800

Sump pumps anyone?

By: mr_crash_davis

mr_crash_davis — Wed, 02 Apr 2008 08:30:46 -0800

"Now that's a great feature of raised floors that I hadn't considered. Well, I suppose it only buys you a little time to pull out the camera before the disaster really happens." The server room at one of my former employers was raised - but the electrical outlets were under the floor, so this wouldn't have helped much.

By: Nelson

Nelson — Wed, 02 Apr 2008 08:53:51 -0800

I once visited archive.org and was treated to a tour of their datacenter in the basement in the Presidio. Rack after rack of nice inexpensive Linux boxes, smart design with commodity hardware. And all the racks about six inches off the concrete slab floor. "Why six inches off the floor?" I asked. "Because the biggest flood this room has ever seen was four inches". Perfect logic.

By: mrbill

mrbill — Wed, 02 Apr 2008 09:04:02 -0800

Luckily "my" datacenter isn't below "drain level", but then it's on the second floor of a building in hurricane country and two walls are floor-to-ceiling glass....

By: ericb

ericb — Wed, 02 Apr 2008 09:13:50 -0800

Rain + Underground special collections stacks + Librarian horror. Brings to mind the August 1998 flood at Boston Public Library.

By: jenkinsEar

jenkinsEar — Wed, 02 Apr 2008 10:06:49 -0800

barnes and noble dot com's server room (back in the day) was built with a moat, in case of flooding. Of course, this was a really high risk spot- they were on the 9th floor of the old port authority building in manhattan.

By: PuppyCat

PuppyCat — Wed, 02 Apr 2008 10:16:10 -0800

That's awesome. What a dillweed plan.

By: mrzarquon

mrzarquon — Wed, 02 Apr 2008 10:25:01 -0800

actually office buildings, old ones, are all sorts of nasty places for water damage. Random pipe bursts on the 11th floor, runs through the walls and then seeps across to your datacenter, etc. Or an improperly sealed window causes a slow leak that travels along a piece of conduit into your hvac system. If you can only control and inspect the floor your datacenter is on, build a moat, pitch a tent, and make it so you are protected from god knows what else happening. I still think a single story building in the middle of your office complex, raised off the ground, with fiber running to your network closets on each floor, is your best bet. No repurposing the basement where the boiler used to be, etc. If you don't want it to look ugly you can put a well designed green roof ontop (and save on the cooling a bit). Or just convert an old church.

By: Unicorn on the cob

Unicorn on the cob — Wed, 02 Apr 2008 10:52:20 -0800

Ugh, reminds me of the night I realized there was a cracked pipe in the wall of my year-old home after discovering two inches of water on the floor of the DJ room. 400 records sunk into three inches of water/wet carpet for 24 hours before I found it. There was much wailing and gnashing of teeth. Seems like these guys would've done well to have a colo!

By: salvia

salvia — Wed, 02 Apr 2008 11:19:14 -0800

This exact sort of thing took out my grad school department's servers -- they had to pay like $20,000 to a data recovery service to recover the data off of damaged hard drives (apparently, this sealed the deal on getting the department to start paying for offsite backup). We were on the ground floor, at the bottom of a large hill, so when the storm sewer backed up, water in the classrooms and labs started to rise. And thank goodness people were in the computer lab at 4 am -- they unplugged all the computers before the water got to those electric plugs. Too bad the server room was locked!

By: Mitheral

Mitheral — Wed, 02 Apr 2008 12:48:35 -0800

mr_crash_davis writes "but the electrical outlets were under the floor, so this wouldn't have helped much." A flood that takes out the electrical to the room is vastly less damaging than a flood that takes out the server hardware. Having experienced a flooding near miss I can't for the life of me figure out why server rooms still need to be in basements. But I see it in new commercial buildings all the time. Put meeting rooms or something else that doesn't get hurt much when your basement turns into a pool and keep an essential service like server hardware from being the first thing getting wet. Seems like a no brainer design feature to me.

By: IndigoJones

IndigoJones — Wed, 02 Apr 2008 17:15:47 -0800

I can't for the life of me figure out why server rooms still need to be in basements. It's a safety precaution. Computer criminals cannot live below ground level.

By: timsteil

timsteil — Wed, 02 Apr 2008 18:43:16 -0800

Could have been worse.

By: Civil_Disobedient

Civil_Disobedient — Wed, 02 Apr 2008 20:47:02 -0800

I can't for the life of me figure out why server rooms still need to be in basements. To best simulate their users' home environment.

By: Civil_Disobedient

Civil_Disobedient — Wed, 02 Apr 2008 20:48:40 -0800

/doesn't live in parent's basement, but misses the free snacks

By: blam

blam — Wed, 02 Apr 2008 23:23:32 -0800

the great richmond flood of '04

By: ctmf

ctmf — Thu, 03 Apr 2008 02:37:35 -0800

2 tips from someone who takes all kinds of computer gear underwater all the time: windows in the room = no good; pumps in the room = good.

By: wierdo

wierdo — Thu, 03 Apr 2008 03:02:22 -0800

Once, in an (at the time) small city in some backwater, Sprintlink had a point of presence, with an associated data center. They decided to put it in the basement of what would be a small office building in most cities, but was pretty large in that particular area at the time. Their space, through pure happenstance, was actually mostly above grade, since the building was built on a slope. They decided they needed a fresh air inlet for their dedicated Liebert cooling units. The only way to do this was to cut a hole in the side of the building, so with the building owner's permission, they had a contractor cut a hole in the side of the building, install a louver, and install a drop ceiling to use as an intake plenum. Because of the ceiling only being 12 or 14 feet high to begin with, they decided to forgo a raised floor and just bolted the racks to the floor. Sadly, they failed to take into account the fact that there was a big fat water pipe for the sprinkler system running over their shiny new POP. One night, about 8 years hence, it got very cold out, the air handlers continued drawing in outside air, the water pipe burst, and drowned most of their equipment. So what did they do? They moved out and sued the building owner. It goes to show that a) even if you're above grade, you might still have a bad day b) you shouldn't draw outside air through a plenum which also contains a large water pipe feeding the whole building's sprinkler system c) as a building owner, you shouldn't rely on your supposedly technically astute tenants to properly engineer modifications they make to your building, and d) you should expect to be sued for other people's mistakes.

By: flaterik

flaterik — Thu, 03 Apr 2008 16:54:57 -0800

Could have been worse. It can always be worse yes that's a building on fire during a massive flood. fun for the whole family.

By: Mitheral

Mitheral — Thu, 03 Apr 2008 17:59:49 -0800

I'm reminded of a story from the RISKS Digest: ... in Texas a number of years ago. Disaster planning was taken very seriously and the facility had an emergency diesel generator *and* backup battery supplies to hold the data center up in case the diesel was hard to start. Except for the dump truck that lost control while descending a rise, left the road and slammed into the adjacent power pole. The pole broke off at the base and fell onto the generator building, doing grievous damage to the generator. The broken engine cooling & fuel lines added to broken water mains to flood the battery room with a noxious mess (the engine bay had a fuel loss containment system but it was not designed to cope with a water main). Along the way, the fire control system triggered adding to the mayhem. Needless to say, the data center lost power rather suddenly.

By: TheSpot

TheSpot — Thu, 03 Apr 2008 19:11:39 -0800

Just saw this on Digg as well. Another reason why servers need to get off the basement.

By: azazello

azazello — Thu, 03 Apr 2008 19:45:40 -0800

Having experienced a flooding near miss I can't for the life of me figure out why server rooms still need to be in basements. But I see it in new commercial buildings all the time. Put meeting rooms or something else that doesn't get hurt much when your basement turns into a pool and keep an essential service like server hardware from being the first thing getting wet. Seems like a no brainer design feature to me. Classical sysadmin self-centered line of thought. The building is built for the people to work in. A basement is an awful place for a meeting room because it won't have any windows and will be less accessible. Floods are extremely infrequent in properly designed basements. If you're that concerned about critical services, push for multiple redundant locations. But complaining of a server room being stuck in the basement is a hallmark of sysadmin hubris. It's stuck in the basement because all the other places are better used by the more important asset... people.

By: Mitheral

Mitheral — Thu, 03 Apr 2008 23:49:55 -0800

A sysadmin concerned about the ongoing viability of the companies IT infrastructure is hardly self centred thinking, it's part of the job description because it effects the well being of those companies dependent on computers. I suggested meeting rooms because most of the modern meeting rooms I see either have no windows or have blackout curtains that are rarely pulled so that presenters can use cheap ass (and therefor dim) projectors with PowerPoint. However I also threw a "or something else" in there as well. So if your company needs natural light meeting rooms feel free to substitute something else. Help desk (I've never actually had a window to the outside in any of my offices), storage, gym, changing rooms, parking, security offices and secure rooms like auditing/cashier. Offices for the peons (I'd take an office with no window over a window equipped cube any day), shipping/receiving, building maintenance, motor pool, equipment servicing, boiler/mechanical services, etc. etc. Also I think you underestimate occurence of water contamination or flooding due to poor design, implementation or maintenance of buildings and their surroundings. Especially as a building ages and space within under goes renovation and re purposing over the years. I'd bet if not a majority then at least a significant minority of IT people (and apparently Librarians. I never noticed before that libraries tend to be in basements but thinking back I can see a trend) can tell the story of the "Flood/Broken Pipe of 'XX". I've got a couple, one in which the organization only avoided a catastrophic failure because we hadn't yet moved into the brand new facility when some plumber two stories up broke open a water main.

By: Skorgu

Skorgu — Fri, 04 Apr 2008 07:53:13 -0800

[Computers are] stuck in the basement because all the other places are better used by the more important asset... people Fun gedankenexperiment: take two identical large, global companies. Take away all the office workers from one and all the computers from the other, see which collapses into bankruptcy first. Hell in some well-sorted Japanese companies you could probably take away all the people and still have the system work for a month or two.