Singing in the Rain
April 2, 2008 3:28 AM Subscribe
But, on the bright side:
Rain + server room in basement = admin department spends the arvo in the pub.
posted by pompomtom at 3:42 AM on April 2, 2008
Rain + server room in basement = admin department spends the arvo in the pub.
posted by pompomtom at 3:42 AM on April 2, 2008
Jeebus. What a nightmare. At least they had the raised floor and only the cables were under water?
posted by gen at 3:53 AM on April 2, 2008
posted by gen at 3:53 AM on April 2, 2008
Rain + plushie Yoda collection in basement = nerd horror.
posted by orthogonality at 4:05 AM on April 2, 2008
posted by orthogonality at 4:05 AM on April 2, 2008
rain + living in parents' basement + mom catching him fapping to stills of princess leia = nerd horror
posted by bwg at 4:36 AM on April 2, 2008 [1 favorite]
posted by bwg at 4:36 AM on April 2, 2008 [1 favorite]
HELP HELP SEA SNAKES UNDER THE FLOORBOARDS
posted by soundofsuburbia at 4:54 AM on April 2, 2008 [6 favorites]
posted by soundofsuburbia at 4:54 AM on April 2, 2008 [6 favorites]
Tragedy is when my mailserver has a hiccup. Comedy is when your server room floods and destroys all of your equipment in one fell swoop.
But I digress. Hope they negotiated a decent disaster rate in their contract.
posted by mmcg at 5:03 AM on April 2, 2008
But I digress. Hope they negotiated a decent disaster rate in their contract.
posted by mmcg at 5:03 AM on April 2, 2008
What a horribly designed server room. Feel bad for them, but I guarantee that this will most likely get them the necessary funding to do it properly.
posted by purephase at 5:11 AM on April 2, 2008
posted by purephase at 5:11 AM on April 2, 2008
At Rutgers back in the 70's, they thought they were clever. When they built a new complex of Math buildings, they built the server rooms underground, under the courtyard. This gave them a big footprint, but they had endless problems with leaks, because the courtyard basically had to be waterproof. I don't think they ever had a massive flood, but it was only a matter of time.
So while I was attending, they ripped up every single brick, put a new waterproof membrane down, then built a new drainage system on top and redid the bricks. Must have cost a fortune. The best part was that they tested it by flooding the courtyard with six inches of water. It was fabulous and surreal. And I have a photo.
posted by smackfu at 5:32 AM on April 2, 2008 [10 favorites]
So while I was attending, they ripped up every single brick, put a new waterproof membrane down, then built a new drainage system on top and redid the bricks. Must have cost a fortune. The best part was that they tested it by flooding the courtyard with six inches of water. It was fabulous and surreal. And I have a photo.
posted by smackfu at 5:32 AM on April 2, 2008 [10 favorites]
Rain + Underground special collections stacks + Librarian horror.
Tell you what, my colleagues are like library ninjas that just swooped in and kicked that flood's ass.
posted by Horace Rumpole at 5:42 AM on April 2, 2008 [2 favorites]
Tell you what, my colleagues are like library ninjas that just swooped in and kicked that flood's ass.
posted by Horace Rumpole at 5:42 AM on April 2, 2008 [2 favorites]
At the New York Public library, they have stacks under Bryant Park (and are moving more there). In the winter they actually have an ice rink in Bryant Park. Talk about tempting fate.
posted by smackfu at 5:51 AM on April 2, 2008
posted by smackfu at 5:51 AM on April 2, 2008
Now that's a great feature of raised floors that I hadn't considered.
Well, I suppose it only buys you a little time to pull out the camera before the disaster really happens. :)
posted by unixrat at 6:37 AM on April 2, 2008
Well, I suppose it only buys you a little time to pull out the camera before the disaster really happens. :)
posted by unixrat at 6:37 AM on April 2, 2008
the Library Collections Emergency Team (LCET) arriving shortly after.
Calling all screenwriters!
posted by washburn at 6:42 AM on April 2, 2008 [2 favorites]
Calling all screenwriters!
posted by washburn at 6:42 AM on April 2, 2008 [2 favorites]
The Great Flood of 2005
Back when I was in CMU, a water mains located in one of the server rooms in Wean Hall (home of the School of Computer Science) broke. The resulting flood damaged tons of computers and I'm not sure how much data ended up being lost. I was an undergrad at the time so it didn't affect me much but imagine the poor grad students who had their entire academic lives stuck in a soaking wet hard drive. Not fun.
Interestingly when I became a grad student I actually used one of the machines rescued from the flood for my research; I left it running for several months and it was still functioning perfectly.
posted by destrius at 7:13 AM on April 2, 2008
Back when I was in CMU, a water mains located in one of the server rooms in Wean Hall (home of the School of Computer Science) broke. The resulting flood damaged tons of computers and I'm not sure how much data ended up being lost. I was an undergrad at the time so it didn't affect me much but imagine the poor grad students who had their entire academic lives stuck in a soaking wet hard drive. Not fun.
Interestingly when I became a grad student I actually used one of the machines rescued from the flood for my research; I left it running for several months and it was still functioning perfectly.
posted by destrius at 7:13 AM on April 2, 2008
There is a state whose capital city is prone to sudden weather. The servers that run all three branches of government are located underground.
One autumn, the Commander-in-Chief was coming to visit the capitol. Two weeks prior, his Secret Service advance team looked at the drain grates around the complex and said, "These grates are too wide, someone could slip bombs down them. We'll put temporary smaller grates of a fine mesh over them, or else the President won't visit."
On the morning of the visit, there was a terrible storm. Leaves blew from trees by the bushel, and were thrown to the ground where they collected in rushing gutters.
Ordinarily, the leaves would just wash away through the sewer grates, as designed. But thanks to the Secret Service's fine mesh toppers, the leaves piled up and up and up, blocking the drains completely. And the rainwater had nowhere to go and flooded into the servers' basement room.
When everything was dried out and replaced, and the President had left the state, there was a well-intentioned suggestion from helpful state leadership to the IT team -- that they might build a wooden hut around the servers and cover it with a tarp, to prevent future server damage, in the case of future inclement weather, at a time when the city's drain and sewer system might be similarly compromised, creating possible flooding. The IT team smiled and nodded and dutifully installed the wooden hut over the servers.
On places where no one can see, anonymous messages get scrawled on the wood, reflecting the IT team's colorful suggestions as to different ways the Secret Service and other various actors in the story might entertain themselves.
Sadly, if understandably, there are no pictures to share.
posted by pineapple at 7:44 AM on April 2, 2008 [3 favorites]
One autumn, the Commander-in-Chief was coming to visit the capitol. Two weeks prior, his Secret Service advance team looked at the drain grates around the complex and said, "These grates are too wide, someone could slip bombs down them. We'll put temporary smaller grates of a fine mesh over them, or else the President won't visit."
On the morning of the visit, there was a terrible storm. Leaves blew from trees by the bushel, and were thrown to the ground where they collected in rushing gutters.
Ordinarily, the leaves would just wash away through the sewer grates, as designed. But thanks to the Secret Service's fine mesh toppers, the leaves piled up and up and up, blocking the drains completely. And the rainwater had nowhere to go and flooded into the servers' basement room.
When everything was dried out and replaced, and the President had left the state, there was a well-intentioned suggestion from helpful state leadership to the IT team -- that they might build a wooden hut around the servers and cover it with a tarp, to prevent future server damage, in the case of future inclement weather, at a time when the city's drain and sewer system might be similarly compromised, creating possible flooding. The IT team smiled and nodded and dutifully installed the wooden hut over the servers.
On places where no one can see, anonymous messages get scrawled on the wood, reflecting the IT team's colorful suggestions as to different ways the Secret Service and other various actors in the story might entertain themselves.
Sadly, if understandably, there are no pictures to share.
posted by pineapple at 7:44 AM on April 2, 2008 [3 favorites]
"Now that's a great feature of raised floors that I hadn't considered.
Well, I suppose it only buys you a little time to pull out the camera before the disaster really happens."
The server room at one of my former employers was raised - but the electrical outlets were under the floor, so this wouldn't have helped much.
posted by mr_crash_davis at 8:30 AM on April 2, 2008
Well, I suppose it only buys you a little time to pull out the camera before the disaster really happens."
The server room at one of my former employers was raised - but the electrical outlets were under the floor, so this wouldn't have helped much.
posted by mr_crash_davis at 8:30 AM on April 2, 2008
I once visited archive.org and was treated to a tour of their datacenter in the basement in the Presidio. Rack after rack of nice inexpensive Linux boxes, smart design with commodity hardware. And all the racks about six inches off the concrete slab floor. "Why six inches off the floor?" I asked. "Because the biggest flood this room has ever seen was four inches". Perfect logic.
posted by Nelson at 8:53 AM on April 2, 2008 [1 favorite]
posted by Nelson at 8:53 AM on April 2, 2008 [1 favorite]
Luckily "my" datacenter isn't below "drain level", but then it's on the second floor of a building in hurricane country and two walls are floor-to-ceiling glass....
posted by mrbill at 9:04 AM on April 2, 2008
posted by mrbill at 9:04 AM on April 2, 2008
Rain + Underground special collections stacks + Librarian horror.
Brings to mind the August 1998 flood at Boston Public Library.
posted by ericb at 9:13 AM on April 2, 2008
Brings to mind the August 1998 flood at Boston Public Library.
posted by ericb at 9:13 AM on April 2, 2008
barnes and noble dot com's server room (back in the day) was built with a moat, in case of flooding. Of course, this was a really high risk spot- they were on the 9th floor of the old port authority building in manhattan.
posted by jenkinsEar at 10:06 AM on April 2, 2008
posted by jenkinsEar at 10:06 AM on April 2, 2008
actually office buildings, old ones, are all sorts of nasty places for water damage. Random pipe bursts on the 11th floor, runs through the walls and then seeps across to your datacenter, etc. Or an improperly sealed window causes a slow leak that travels along a piece of conduit into your hvac system. If you can only control and inspect the floor your datacenter is on, build a moat, pitch a tent, and make it so you are protected from god knows what else happening.
I still think a single story building in the middle of your office complex, raised off the ground, with fiber running to your network closets on each floor, is your best bet. No repurposing the basement where the boiler used to be, etc. If you don't want it to look ugly you can put a well designed green roof ontop (and save on the cooling a bit).
Or just convert an old church.
posted by mrzarquon at 10:25 AM on April 2, 2008
I still think a single story building in the middle of your office complex, raised off the ground, with fiber running to your network closets on each floor, is your best bet. No repurposing the basement where the boiler used to be, etc. If you don't want it to look ugly you can put a well designed green roof ontop (and save on the cooling a bit).
Or just convert an old church.
posted by mrzarquon at 10:25 AM on April 2, 2008
Ugh, reminds me of the night I realized there was a cracked pipe in the wall of my year-old home after discovering two inches of water on the floor of the DJ room.
400 records sunk into three inches of water/wet carpet for 24 hours before I found it. There was much wailing and gnashing of teeth.
Seems like these guys would've done well to have a colo!
posted by Unicorn on the cob at 10:52 AM on April 2, 2008
400 records sunk into three inches of water/wet carpet for 24 hours before I found it. There was much wailing and gnashing of teeth.
Seems like these guys would've done well to have a colo!
posted by Unicorn on the cob at 10:52 AM on April 2, 2008
This exact sort of thing took out my grad school department's servers -- they had to pay like $20,000 to a data recovery service to recover the data off of damaged hard drives (apparently, this sealed the deal on getting the department to start paying for offsite backup). We were on the ground floor, at the bottom of a large hill, so when the storm sewer backed up, water in the classrooms and labs started to rise. And thank goodness people were in the computer lab at 4 am -- they unplugged all the computers before the water got to those electric plugs. Too bad the server room was locked!
posted by salvia at 11:19 AM on April 2, 2008
posted by salvia at 11:19 AM on April 2, 2008
mr_crash_davis writes "but the electrical outlets were under the floor, so this wouldn't have helped much."
A flood that takes out the electrical to the room is vastly less damaging than a flood that takes out the server hardware.
Having experienced a flooding near miss I can't for the life of me figure out why server rooms still need to be in basements. But I see it in new commercial buildings all the time. Put meeting rooms or something else that doesn't get hurt much when your basement turns into a pool and keep an essential service like server hardware from being the first thing getting wet. Seems like a no brainer design feature to me.
posted by Mitheral at 12:48 PM on April 2, 2008
A flood that takes out the electrical to the room is vastly less damaging than a flood that takes out the server hardware.
Having experienced a flooding near miss I can't for the life of me figure out why server rooms still need to be in basements. But I see it in new commercial buildings all the time. Put meeting rooms or something else that doesn't get hurt much when your basement turns into a pool and keep an essential service like server hardware from being the first thing getting wet. Seems like a no brainer design feature to me.
posted by Mitheral at 12:48 PM on April 2, 2008
I can't for the life of me figure out why server rooms still need to be in basements.
It's a safety precaution. Computer criminals cannot live below ground level.
posted by IndigoJones at 5:15 PM on April 2, 2008
It's a safety precaution. Computer criminals cannot live below ground level.
posted by IndigoJones at 5:15 PM on April 2, 2008
I can't for the life of me figure out why server rooms still need to be in basements.
To best simulate their users' home environment.
posted by Civil_Disobedient at 8:47 PM on April 2, 2008 [1 favorite]
To best simulate their users' home environment.
posted by Civil_Disobedient at 8:47 PM on April 2, 2008 [1 favorite]
/doesn't live in parent's basement, but misses the free snacks
posted by Civil_Disobedient at 8:48 PM on April 2, 2008
posted by Civil_Disobedient at 8:48 PM on April 2, 2008
2 tips from someone who takes all kinds of computer gear underwater all the time: windows in the room = no good; pumps in the room = good.
posted by ctmf at 2:37 AM on April 3, 2008
posted by ctmf at 2:37 AM on April 3, 2008
Once, in an (at the time) small city in some backwater, Sprintlink had a point of presence, with an associated data center. They decided to put it in the basement of what would be a small office building in most cities, but was pretty large in that particular area at the time. Their space, through pure happenstance, was actually mostly above grade, since the building was built on a slope.
They decided they needed a fresh air inlet for their dedicated Liebert cooling units. The only way to do this was to cut a hole in the side of the building, so with the building owner's permission, they had a contractor cut a hole in the side of the building, install a louver, and install a drop ceiling to use as an intake plenum.
Because of the ceiling only being 12 or 14 feet high to begin with, they decided to forgo a raised floor and just bolted the racks to the floor.
Sadly, they failed to take into account the fact that there was a big fat water pipe for the sprinkler system running over their shiny new POP. One night, about 8 years hence, it got very cold out, the air handlers continued drawing in outside air, the water pipe burst, and drowned most of their equipment.
So what did they do? They moved out and sued the building owner.
It goes to show that a) even if you're above grade, you might still have a bad day b) you shouldn't draw outside air through a plenum which also contains a large water pipe feeding the whole building's sprinkler system c) as a building owner, you shouldn't rely on your supposedly technically astute tenants to properly engineer modifications they make to your building, and d) you should expect to be sued for other people's mistakes.
posted by wierdo at 3:02 AM on April 3, 2008
They decided they needed a fresh air inlet for their dedicated Liebert cooling units. The only way to do this was to cut a hole in the side of the building, so with the building owner's permission, they had a contractor cut a hole in the side of the building, install a louver, and install a drop ceiling to use as an intake plenum.
Because of the ceiling only being 12 or 14 feet high to begin with, they decided to forgo a raised floor and just bolted the racks to the floor.
Sadly, they failed to take into account the fact that there was a big fat water pipe for the sprinkler system running over their shiny new POP. One night, about 8 years hence, it got very cold out, the air handlers continued drawing in outside air, the water pipe burst, and drowned most of their equipment.
So what did they do? They moved out and sued the building owner.
It goes to show that a) even if you're above grade, you might still have a bad day b) you shouldn't draw outside air through a plenum which also contains a large water pipe feeding the whole building's sprinkler system c) as a building owner, you shouldn't rely on your supposedly technically astute tenants to properly engineer modifications they make to your building, and d) you should expect to be sued for other people's mistakes.
posted by wierdo at 3:02 AM on April 3, 2008
Could have been worse.
It can always be worse
yes that's a building on fire during a massive flood. fun for the whole family.
posted by flaterik at 4:54 PM on April 3, 2008
It can always be worse
yes that's a building on fire during a massive flood. fun for the whole family.
posted by flaterik at 4:54 PM on April 3, 2008
I'm reminded of a story from the RISKS Digest:
... in Texas a number of years ago. Disaster planning was taken very seriously and the facility had an emergency diesel generator
*and* backup battery supplies to hold the data center up in case the diesel was hard to start.
Except for the dump truck that lost control while descending a rise, left the road and slammed into the adjacent power pole.
The pole broke off at the base and fell onto the generator building, doing grievous damage to the generator. The broken engine cooling & fuel lines added to broken water mains to flood the battery room with a noxious mess (the engine bay had a fuel loss containment system but it was not designed to cope with a water main). Along the way, the fire control system triggered adding to the mayhem.
Needless to say, the data center lost power rather suddenly.
posted by Mitheral at 5:59 PM on April 3, 2008
... in Texas a number of years ago. Disaster planning was taken very seriously and the facility had an emergency diesel generator
*and* backup battery supplies to hold the data center up in case the diesel was hard to start.
Except for the dump truck that lost control while descending a rise, left the road and slammed into the adjacent power pole.
The pole broke off at the base and fell onto the generator building, doing grievous damage to the generator. The broken engine cooling & fuel lines added to broken water mains to flood the battery room with a noxious mess (the engine bay had a fuel loss containment system but it was not designed to cope with a water main). Along the way, the fire control system triggered adding to the mayhem.
Needless to say, the data center lost power rather suddenly.
posted by Mitheral at 5:59 PM on April 3, 2008
Just saw this on Digg as well. Another reason why servers need to get off the basement.
posted by TheSpot at 7:11 PM on April 3, 2008
posted by TheSpot at 7:11 PM on April 3, 2008
Having experienced a flooding near miss I can't for the life of me figure out why server rooms still need to be in basements. But I see it in new commercial buildings all the time. Put meeting rooms or something else that doesn't get hurt much when your basement turns into a pool and keep an essential service like server hardware from being the first thing getting wet. Seems like a no brainer design feature to me.
Classical sysadmin self-centered line of thought.
The building is built for the people to work in. A basement is an awful place for a meeting room because it won't have any windows and will be less accessible. Floods are extremely infrequent in properly designed basements. If you're that concerned about critical services, push for multiple redundant locations. But complaining of a server room being stuck in the basement is a hallmark of sysadmin hubris. It's stuck in the basement because all the other places are better used by the more important asset... people.
posted by azazello at 7:45 PM on April 3, 2008
Classical sysadmin self-centered line of thought.
The building is built for the people to work in. A basement is an awful place for a meeting room because it won't have any windows and will be less accessible. Floods are extremely infrequent in properly designed basements. If you're that concerned about critical services, push for multiple redundant locations. But complaining of a server room being stuck in the basement is a hallmark of sysadmin hubris. It's stuck in the basement because all the other places are better used by the more important asset... people.
posted by azazello at 7:45 PM on April 3, 2008
A sysadmin concerned about the ongoing viability of the companies IT infrastructure is hardly self centred thinking, it's part of the job description because it effects the well being of those companies dependent on computers.
I suggested meeting rooms because most of the modern meeting rooms I see either have no windows or have blackout curtains that are rarely pulled so that presenters can use cheap ass (and therefor dim) projectors with PowerPoint. However I also threw a "or something else" in there as well. So if your company needs natural light meeting rooms feel free to substitute something else. Help desk (I've never actually had a window to the outside in any of my offices), storage, gym, changing rooms, parking, security offices and secure rooms like auditing/cashier. Offices for the peons (I'd take an office with no window over a window equipped cube any day), shipping/receiving, building maintenance, motor pool, equipment servicing, boiler/mechanical services, etc. etc.
Also I think you underestimate occurence of water contamination or flooding due to poor design, implementation or maintenance of buildings and their surroundings. Especially as a building ages and space within under goes renovation and re purposing over the years. I'd bet if not a majority then at least a significant minority of IT people (and apparently Librarians. I never noticed before that libraries tend to be in basements but thinking back I can see a trend) can tell the story of the "Flood/Broken Pipe of 'XX". I've got a couple, one in which the organization only avoided a catastrophic failure because we hadn't yet moved into the brand new facility when some plumber two stories up broke open a water main.
posted by Mitheral at 11:49 PM on April 3, 2008
I suggested meeting rooms because most of the modern meeting rooms I see either have no windows or have blackout curtains that are rarely pulled so that presenters can use cheap ass (and therefor dim) projectors with PowerPoint. However I also threw a "or something else" in there as well. So if your company needs natural light meeting rooms feel free to substitute something else. Help desk (I've never actually had a window to the outside in any of my offices), storage, gym, changing rooms, parking, security offices and secure rooms like auditing/cashier. Offices for the peons (I'd take an office with no window over a window equipped cube any day), shipping/receiving, building maintenance, motor pool, equipment servicing, boiler/mechanical services, etc. etc.
Also I think you underestimate occurence of water contamination or flooding due to poor design, implementation or maintenance of buildings and their surroundings. Especially as a building ages and space within under goes renovation and re purposing over the years. I'd bet if not a majority then at least a significant minority of IT people (and apparently Librarians. I never noticed before that libraries tend to be in basements but thinking back I can see a trend) can tell the story of the "Flood/Broken Pipe of 'XX". I've got a couple, one in which the organization only avoided a catastrophic failure because we hadn't yet moved into the brand new facility when some plumber two stories up broke open a water main.
posted by Mitheral at 11:49 PM on April 3, 2008
[Computers are] stuck in the basement because all the other places are better used by the more important asset... people
Fun gedankenexperiment: take two identical large, global companies. Take away all the office workers from one and all the computers from the other, see which collapses into bankruptcy first. Hell in some well-sorted Japanese companies you could probably take away all the people and still have the system work for a month or two.
posted by Skorgu at 7:53 AM on April 4, 2008
Fun gedankenexperiment: take two identical large, global companies. Take away all the office workers from one and all the computers from the other, see which collapses into bankruptcy first. Hell in some well-sorted Japanese companies you could probably take away all the people and still have the system work for a month or two.
posted by Skorgu at 7:53 AM on April 4, 2008
« Older And I Feel Fine | Radiohead Re/mix Newer »
This thread has been archived and is closed to new comments
94 desktops and 4 servers and a couple hundred billable hours later....
posted by pjern at 3:38 AM on April 2, 2008 [1 favorite]