January 9, 2002

4:04 PM

The Morning News Gets Hosed. Due to a server meltdown (and probable incompetence by their webhosting provider) the guys at Morning News lost all kinds of data. Now on a new server, their old host is looking into the possibility of coughing up a decent backup. As a website designer who relies on the kindness of server farms, I know I've been hosed this way before. Since they can't be relied on to provide good backups, when was the last time you backed up your site yourself? Better make one today!
posted by crunchland (11 comments total)

The Wayback Archive has a ton of their content archived. Maybe they could whip up a Perl script to crawl Wayback links and re-archive everything they have.
posted by waxpancake at 4:14 PM on January 9, 2002

The good people at Storagereview.com recently suffered a similar catastrophe.

"As implied in the Discussion Forums as well as sites like arstechnica.com and theinquirer.com have suggested, StorageReview.com will cease operation by the end of the month. Though the database deletion described below was a huge setback, it mainly served as a catalyst that forced Davin and I to reexamine the time, effort, and rewards we received from continuing with the site...."

"We did have backups, albeit not off-site. Rather, they were conducted server-to-server, disk-to-disk. Unfortuantely, somehow the backups -and- the live databases were both deleted in the process of pruning and preparing for another backup." (more on their site)

:(
posted by lucien at 4:37 PM on January 9, 2002

" ... As a website designer who relies on the kindness of server farms, I know I've been hosed this way before ..."

Um, as a website designer, do you actually post new content directly to production servers? No development or staging? If it is content placed on production by users (like discussion boards), you don't have automated scripts running at 3AM every morning to back the DB's up? Goodness dude, don't say stuff like that publically if you want to actually get hired as a web designer ... it automatically classifies you as "beginner to intermediate".
posted by MidasMulligan at 5:04 PM on January 9, 2002

amen, midas.

my business is designing development environments, and i practice what i preach, for several reasons i won't bore you with. since change management is central to work, i use it at home, too.

i publish to both my business and personal sites through an automated script which also archives files in a local source repository. it's overkill for my personal use, but a server crash is no big deal for me now. i have all my data on disk, and on tape due to regular backups.

and i'm constantly amazed by the lack of formalized procedures in place at client sites upon my arrival. even at the fortune 500s.
posted by particle at 5:20 PM on January 9, 2002

Backing up data in your development environment is a great idea (I do it myself.)

I still do nightly backups on the production server. If you're away from the development environment, and it's your only backup, then you're screwed.

I want to go away on vacation without worrying about the server crashing...
posted by Chief Typist at 5:29 PM on January 9, 2002

Wow, a whole MeFi thread about us...Thankfully we backed up 99% of the site a week before the crash and our return should be swift, say, under a week.

But for everyone out there that has a personal site, may we advise with new wisdom: Back it up! Often!
posted by rosecrans at 5:46 PM on January 9, 2002

a moment of silence too for sensible erection, which btw promises to rise anew :)
posted by kliuless at 7:36 PM on January 9, 2002

i miss the morningnews already.
posted by panopticon at 8:17 PM on January 9, 2002

A good friend of mine who runs a small webhosting service claims to make more money restoring backups for his clients than he does setting up their sites.

Apparently, once Joe Shmoe gets a site, he figures that automatically makes him a Web God. Then when he (inevitably) screws it up, he's more than happy to pay a small fee to have it returned to him in a "known-good" state.

I find this fascinating.
posted by mr_crash_davis at 8:26 PM on January 9, 2002

" ... and i'm constantly amazed by the lack of formalized procedures in place at client sites upon my arrival. even at the fortune 500s ..."

Agree with this. I work for one, and while I don't do the back-ups myself, the buck still stops at my desk. We are quite rigorous about multiple, revolving, offsite backups. A couterpart at a competing firm, earlier this year, told me I was paranoid and probably overspending on that particular piece of IT operations. Funny thing, as of 9/11, suddenly those who were called "conservative and paranoid" started getting called "visionary and pragmatic".

You're even more right about the procedures ... thing is, these days, the process itself is largely automated. The most common point of failure is not that there aren't now many ways of easily accomplishing the task - it is that people themselves simply don't bother to get into the right habits (even, strangely enough, after a disaster or two).

For some reason, people treat servers (hell, not just desktops, but servers) as though they were irons, or toasters or something ... i.e., things one expects to be able depend upon to function flawlessly for at least a few years. Yeesh. Fact is, about 6 or 7 different fields of theoretical and applied science, operating using parts made by at least a couple of dozen different vendors and manufacturers (each with various degrees of quality control) all have to function almost perfectly, both alone and in conjunction with one another, to even boot a computer, let alone run big applications on it.

It simply isn't rational to believe anything on a computer is safe, or stable. If you have any material that is even mildly valuable, having it exist solely as a microscopic strip of magnetized disk on a single computer is pretty much the same as writing it on a sidewalk. In chalk. In Seattle. The question ain't whether or not you'll have a bad experience, it's when, and how bad it will wind up being.

The healthiest attitude to assume when approaching the question of whether it's worth the bother to back things up regularly ... even when you've gone months with no problem ... is to imagine yourself looking at Clint Eastwood, who is pointing a .44 Magnum at you that may or may not be empty, and saying, "Do ya feel lucky punk? Huh? Do ya?".
posted by MidasMulligan at 9:12 PM on January 9, 2002

Incidentally, Midas... just wanted to point out that you made a false assumption up there. When I was a beginner to intermdiate webmaster, my archiving practices were a bit shoddy. Once I got hosed, I learned. Now I'm pragmatic and visionary, too. I created this thread as a cautionary tale for all those who haven't yet learned the hard way.
posted by crunchland at 6:32 AM on January 10, 2002