Danger data loss

October 10, 2009 10:29 PM

T-Mobile and Microsoft/Danger probably lost all Sidekick users' data. They continue to advise customers to NOT reset their device by removing the battery or letting their battery drain completely, as any personal content that currently resides on your device will be lost.
posted by finite (125 comments total)

Chaos reigns.
posted by philip-random at 10:31 PM on October 10, 2009 [2 favorites]

Wow. I didn't think they meant Microsoft/Danger literally.
posted by happyroach at 10:35 PM on October 10, 2009 [11 favorites]

I wonder if T-Mobile USA will start talking to Apple about an iPhone now?
posted by infinitewindow at 10:38 PM on October 10, 2009

On the upside, at least Perez Hilton can't bother most of the outside world aside from people who read his blog.

Was forwarded that link, I would not personally subject myself to that.
posted by cgomez at 10:41 PM on October 10, 2009

Whoops.
posted by Blazecock Pileon at 10:42 PM on October 10, 2009 [1 favorite]

The Microsoft Outlook: Cloudy, with a chance of data loss.

(i know this has nothing to do with outlook but i absolutely could not resist this clusterbomb of a joke)
posted by Mikey-San at 10:42 PM on October 10, 2009 [24 favorites]

Perez Hilton is hilarious:

"T-Mobile/Sidekick, you are seriously effing up! STILL! WTF???? Major fail!!!!! If this problem persists further, I'm switching carriers!"

How can the problem persist further when you've already lost all your shit? How does this get worse, short of someone coming to your house to smash your device with a hammer?
posted by Mikey-San at 10:45 PM on October 10, 2009 [6 favorites]

Mikey-San: That's not quite fair. Microsoft acquired the company; I don't think they did anything to change the day to day operations and platform which caused some data redundancy failures. Also, Microsoft Outlook and Exchange are like two of the four software titles they make that are actually good.
posted by cgomez at 10:50 PM on October 10, 2009

How does this get worse?

When Perez Hilton is involved, it can always get worse.
posted by xedrik at 10:51 PM on October 10, 2009 [9 favorites]

Idiotic System Succumbs To Predictable Catastrophe
Man on internet "unsurprised"
posted by regicide is good for you at 10:53 PM on October 10, 2009 [15 favorites]

Haven't we learned this lesson hundreds of times before? If everything is dependent on a central operator, then eventually everything is going to fall apart. Despotism, absolute monarchies, Fascism, abusive relationships, mainframe computing, the buggers from Ender's Game, the Borg.... the list goes on, as far back and as far forward through human history as you wish to go.
posted by clorox at 10:59 PM on October 10, 2009 [26 favorites]

Really? In 2009? Computers have been everywhere for at least one human generation. Disk drives are dirt cheap. How does a real company lose data this late in the game?
posted by rdr at 11:03 PM on October 10, 2009 [9 favorites]

If T-Mobile was actually responsible for doing away with a bunch of Perez Hilton's data, I know a few people who might actually switch to them.
posted by JaredSeth at 11:04 PM on October 10, 2009 [7 favorites]

I own and support Blackberries, and resetting the device is common enough that backups are not optional, unless you just don't care about losing your data. Not sure if that's possible in this case, but it is sort of amusing that they're advising not to turn the phones off. I do depend on the BB a lot, but a major weakness is its dependence on RIM's server in the middle for email, which I can see is also the weakness here, but in a bigger way. No matter how much virtualization we can do, we do need to be able to use these devices "offline" without worrying about the data. This is not an unknown issue and goes back to early problems with software, so it is a bit surprising that nobody considered it.

Microsoft Outlook and Exchange are like two of the four software titles they make that are actually good.

I am afraid to ask what the others are. Speaking as someone who is paid to support these programs, let's just say that Outlook/Exchange has its uses, but it's a moving target, and I prefer some releases to others. Sometimes it's a major PITA, but I will admit that often it's the only option that does what it does which is so widely supported. But damn those Exchange CALs are getting expensive, and for small businesses making the leap to a full-fledged Windows domain with Exchange is getting more and more difficult to justify when most people just need email and calendar features.
posted by krinklyfig at 11:06 PM on October 10, 2009 [1 favorite]

Mikey-San: That's not quite fair. Microsoft acquired the company; I don't think they did anything to change the day to day operations and platform which caused some data redundancy failures. Also, Microsoft Outlook and Exchange are like two of the four software titles they make that are actually good.

How often do you get your prescription checked? You may be having difficulty seeing small things!
posted by Mikey-San at 11:07 PM on October 10, 2009

WAIT WAIT WAIT

SHIT I MADE A JOKE ON THE INTERNET THAT WASN'T QUITE FAIR

NOOOOOOOOOOOOOOooooooooooooooooooooooooooooooooooooooooooooooooooooooo
posted by Mikey-San at 11:08 PM on October 10, 2009 [8 favorites]

"Microsoft acquired the company; I don't think they did anything to change the day to day operations and platform which caused some data redundancy failures."

Um. Perhaps I'm an engineer and have a different outlook - but I read this as "MS bought the company; they didn't do anything to fix the technical issues; there was a lot of lost data because of it."

What, exactly, is a "data redundancy failure"? Does that really describe this case where all the data is not (not just redundant copies)?

"Also, Microsoft Outlook and Exchange are like two of the four software titles they make that are actually good."

I've known a bunch of Exchange administrators and they were run pretty ragged by poor design decisions and database corruption. Do you really have a good experience with this program in a real-world installation?
posted by lupus_yonderboy at 11:21 PM on October 10, 2009 [1 favorite]

...all the data is lost...
posted by lupus_yonderboy at 11:22 PM on October 10, 2009

Let me also add that gmail data, for example, is replicated 9 times over geographically separated areas... you'd need at least three nukes to wipe out any individual gmail account.
posted by lupus_yonderboy at 11:24 PM on October 10, 2009 [17 favorites]

Sometimes, Letting Data Die A Natural Death Is Ok
posted by zoinks at 11:25 PM on October 10, 2009 [1 favorite]

JaredSeth: Their coverage isn't good enough to be the carrier I rely on, but if there was, say -- some way I could auto-pay them $2.00 a month to make Perez Hilton's life harder, I'd do it.

krinklyfig: Those applications would be Microsoft Outlook, coupled with Exchange, Excel, RDP, and Windows Media Center (not Windows itself Media Center). I've actually had really good experiences with Exchange and Outlook and using it remotely through BlackBerry on BES. For all of its bloat and UI quirks, Outlook works well and collaboration across multiple users is easy for business. Personally, I use Mail and iCal -- but at work, I don't cringe at the thought of staring at it all day.

As for the weakness of BlackBerry, RIM has had a rather outstanding uptime performance record for push mail and PIM and I've never had issues with that and remotely accessing Exchange through it. Then again, I've never had to luxury of administering a BES and all the headaches I'm sure it brings. It's not perfect -- no product that requires somebody to administrate it using Internet Explorer on Windows can be.
posted by cgomez at 11:26 PM on October 10, 2009 [1 favorite]

Pfft! I scoff at Danger!
posted by aubilenon at 11:41 PM on October 10, 2009 [2 favorites]

fuck bush 666 420
posted by Damn That Television at 11:43 PM on October 10, 2009 [3 favorites]

Wait...Contact info on the sidekick was stored off the phone?

That just sounds really freaking absurd, I'm sorry. Contact info doesn't take up much space, memory is cheap, whatever. The only thing I can think of to store such a small amount of data that's accessed frequently off the phone would be to make the device use data transfers more, upping the monthly bill.

Can someone who is smart explain this to me?
posted by hellojed at 11:44 PM on October 10, 2009 [6 favorites]

Old man yells at cloud.
posted by Ratio at 11:48 PM on October 10, 2009 [10 favorites]

Hey, thanks for being kind of a dick, Mikey-San.

What I actually said was that Microsoft acquiring a company doesn't mean it's going to be Microsoft-ized and become instantly awful. I was under this weird impression that comments on Metafilter were to discuss the subject of the post.
posted by cgomez at 11:52 PM on October 10, 2009

Please lighten up. Did I stab your puppy or something? (And I was also pointing out that I actually, literally said that it had nothing to do with Outlook, but you were too busy penning your reply to read all of my post.)
posted by Mikey-San at 12:00 AM on October 11, 2009 [1 favorite]

I can't imagine how awful it must have been to be the person who first realized that the data was well and truly gone. I freak out when I lose a page of search results to a session timeout.
posted by Monsters at 12:00 AM on October 11, 2009 [3 favorites]

I can't imagine how awful it must have been to be the person who first realized that the data was well and truly gone.

It's probably a three-stage process:

1. Realize you just lost millions of people's data. A deep, crushing depression sets in when you imagine how many people just got hosed.
2. Realize you just hosed millions of customers. You begin drafting your resignation letter.
3. Realize there's no way you can put this job on your resume.
posted by Mikey-San at 12:07 AM on October 11, 2009 [7 favorites]

Can someone who is smart explain this to me?

Based on the people I know that use Sidekicks, device turnover is very high. Having the absolute latest model as soon as possible is a big concern, and new models come out pretty frequently. So if you're changing phones every year or so, maybe even more often if you can just switch SIM cards, having all your personal data stored off-phone would be pretty handy.

As for relying on off-phone storage as a primary storage location, instead of just a backup, I can think of a couple reasons:

-Minimizing costs by only having enough battery-independent memory to store the operating software on each phone. Even if you save just a dollar per phone in flash memory, you just saved a hundred thousand dollars per production run.
-Market research. Instead of just knowing who, what, when, where and for how long your subscribers text or browse or call, by having all their stuff on your servers you can also know how many contacts they have, how often they take pictures or video, etc. And, knowing that your subscribers have shown an eight percent increase in taking pictures between 2 and 3 AM while not roaming, for instance, you can now work that right into your next ad campaign.
-Theft/loss security. If their phone is stolen or lost, you can calmly tell your customers that some random person does not now have access to their text history, their phonebook, or those naughty pictures they took after they got home from the bar that one night.

But why they really chose to do it is beyond me.
posted by clorox at 12:27 AM on October 11, 2009 [1 favorite]

Danger really doesn't seem to be having a good time being part of MS, what with this article about MS's smartphone project going south and now this data loss thing.
posted by fairytale of los angeles at 12:47 AM on October 11, 2009

Can someone who uses one of these tell me what's likely to be lost? I mean, if my phone crashed, I'd lose about 20 phone book entries and a lot of "on my way home, please kick the high school football team out before I get there love m" text messages I've sent. In other words, not a big deal. But I just have a phone.

I would assume if I took photos, wrote emails, etc., on a device I would want them locally on my main desktop machine, so I would synch them constantly. I assume this is a big deal because people don't do that, and have all of their junk only at Danger?

Also, can someone explain to me how a single server going down (which is what they're claiming on that website) at an "enterprise level" company lead to the loss of all customer data? At worst, you'd think it might affect certain users, and even then only for a certain time-period (ie, "all Boston subscribers unfortunately will have lost the last three hours worth of data"). If that's not the case, that seems astonishingly unprofessional and sloppy.
posted by maxwelton at 1:05 AM on October 11, 2009

I wonder if T-Mobile USA will start talking to Apple about an iPhone now?

DUDE, BUYING AN APPLE DEVICE WILL SOLVE ALL YOUR PROBLEMS!!!

DID YOUR DATA CENTER BURN DOWN? SHOULD HAVE GOT AN MAC!
posted by delmoi at 1:37 AM on October 11, 2009 [17 favorites]

Danger really doesn't seem to be having a good time being part of MS, what with this article about MS's smartphone project going south and now this data loss thing.

With all the screw-ups in the last few years, I'm surprised Microsoft shareholders have not yet called for Ballmer to step down.
posted by Blazecock Pileon at 1:39 AM on October 11, 2009

(Also, I should point out that that apple Had some serious data loss problems in the past (2), (and more recent problems too))
posted by delmoi at 1:42 AM on October 11, 2009

Microsoft Outlook and Exchange are like two of the four software titles they make that are actually good.

I think you're out of your mind. Outlook? Exchange!?

2. Realize you just hosed millions of customers. You begin drafting your resignation letter.
3. Realize there's no way you can put this job on your resume.

Okay for failures this big there is always enough blame to go around. Just like the Bush DOJ, failing institutions tend to spread responsibility around.

The low-level grunts just put "Sys-admin at Danger, Inc" they don't have to specify precisely which sys they admin'd. Higher ups will just talk about their management experience, and at that level their next job is likely to come through personal networking, not regular job hunting.
posted by delmoi at 1:48 AM on October 11, 2009

Who said I was speaking from the perspective of a single, "low-level grunt"?
posted by Mikey-San at 1:57 AM on October 11, 2009

(Obviously, no one's going to put "dude who contributed to roasting Sidekick data" on his resume, and will come up with something else to say. I'm not that dumb, guys. I just look the part.)
posted by Mikey-San at 2:00 AM on October 11, 2009

Obviously, no one's going to put "dude who contributed to roasting Sidekick data" on his resume, and will come up with something else to say.

"Optimized data center efficiency by down-sizing storage requirements", perhaps?
posted by TedW at 2:54 AM on October 11, 2009 [15 favorites]

DUDE, BUYING AN APPLE DEVICE WILL SOLVE ALL YOUR PROBLEMS!!!

DID YOUR DATA CENTER BURN DOWN? SHOULD HAVE GOT AN MAC!

I realize you're just being facetious but if you were using an iPhone with MobileMe and MobileMe suffered a catastrophic failure SyncServices would not only not lose all your data but fix all the shit on the server for your MobileMe account automatically once it did come back without any data.

We now return you to the thread already in progress.
posted by Talez at 3:12 AM on October 11, 2009

Can someone who is smart explain this to me?

If you conceive your platform as an online service and your devices as dumb terminals for accessing it, then the idea of having a persistent store of anything locally is absurd. In fact, you're proposing that Danger's backup strategy should have been to push the users' data back to their terminals, just in case.

Granted, it would have been better than (apparently) having no backup strategy at all, but that still doesn't make it a good idea.
posted by cillit bang at 3:24 AM on October 11, 2009

Perhaps naming your company Danger is like naming your pet Lucky.

(That poor turtle. He lived through so many lawnmowers...)
posted by poe at 3:44 AM on October 11, 2009 [3 favorites]

Exchange is pretty damned rock solid in my experiences. I've got dozens of clients running Exchange 2003 and Exchange 2007 in SBS and standalone environments and over the last seven years I can count exactly zero corruption issues, zero downtime due to Exchange and zero support calls about Exchange.

Lest you think I'm a Windows fanboi, I'm more a Mac partisan, but I'm willing to give credit where credit is due: Exchange and Outlook are probably Microsoft's most solid products, and on their own merits work really well if you know what the hell you're doing when you set them up properly.

Now, don't get me started on Blackberry Enterprise Server - that has been a nightmare for us to support.
posted by tgrundke at 3:57 AM on October 11, 2009 [3 favorites]

I hate cellphones. That is all I have to say.
posted by GavinR at 4:31 AM on October 11, 2009

I'm bookmarking this thread for the next AskMe about "Why won't my boss approve us moving all our data to the cloud?"
posted by rokusan at 4:52 AM on October 11, 2009 [2 favorites]

I predict falling prices for Sidekick devices. And an acceleration of Zune phones.
posted by surplus at 5:50 AM on October 11, 2009 [1 favorite]

Microsoft/Danger has invited you to join the group HEY LOST MY SERVER NEED UR NUMBERS PLZ!!!!!1!
posted by Mr. Anthropomorphism at 6:07 AM on October 11, 2009 [5 favorites]

Rokusan, between this and all of the Google Apps issues ... I think the best thing to be said for keeping things local is that when an outage occurs, you don't have to say "The cloud did it" to anyone.

"The cloud had a storm but the outlook is sunny" will be the new "zombie children are floating in the pipe."
posted by adipocere at 6:10 AM on October 11, 2009

I wonder if this will turn out to be another "we've got so many replicas of this data all over the world that we don't need to back it up separately" kind of situation?

I used to see this same statement regarding hard drive mirroring and RAID, and occasionally still do. It works until you hit a software bug that corrupts your data, and the corrupted data gets replicated to all the copies. And then you're dead.

I work at a pretty small place with little money, and yet we're required to prove that we have backups of everything important that are:

• off-line, so that no software or hardware bug can wreck them.
• off-site, so that a fire at our head office doesn't destroy the servers AND all the backups simultaneously.
• made frequently and kept for a long time, so that we can go back to older backups if a problem isn't noticed for a while.
• tested regularly, to make sure the backups are actually good and contain everything we need. We do this by using the backups to get the system up and running again starting from an empty server, which also makes sure we know how to do that. We write down the procedure so that we're not scrambling trying to figure it out during an actual emergency.

Were they not doing all these things? Did they feel like they had a system architecture that was so bullet-proof that it wasn't necessary to do all these things, in some form or another? How common is that view with online services and cloud computing, anyway? Or maybe the system is so big and complicated that nobody can figure out how to back it up and restore it effectively.

Or did they really think they had all this covered, and they've been hit by some incredible bad luck? Because I've been there, too, where several things break all at the worst possible time, and you realize that you just lost a bunch of stuff that you now have no way of getting back. Just having to tell people that we'd lost a couple of weeks worth of e-mail once was bad enough.

Hmm... potential interview questions for technical staff: "Have you ever experienced a significant data loss incident?" and "Do you ever want to go through that again?", and don't hire them unless the answers are "yes" and "hell no", respectively.
posted by FishBike at 7:06 AM on October 11, 2009 [8 favorites]

Paranoid Linux neckbeards smugly celebrate never trusting their data to someone else!

(Or maybe it's just me.)
posted by Zed at 7:17 AM on October 11, 2009 [2 favorites]

Yo Will.I.Am, I'm really happy for you and I'ma let you finish, but the T-Mobile Sidekick personal data loss just created the best Perez Hilton hissy fit of all time. OF ALL TIME.
posted by fire&wings at 7:23 AM on October 11, 2009

I came here to say what FishBike just said. I've never worked for a company that provided services to a customer but everywhere I've worked we had data that needed to be backed up: mail, databases, source code, shared file systems, etc. We backed them up daily and offsite. You'd think that a company that is providing cloud type services would have a bullet-proof risk mitigation plan that would be able to survive multiple points of failure.
posted by octothorpe at 7:24 AM on October 11, 2009

> Microsoft acquired the company; I don't think they did anything to change the day to day operations and platform which caused some data redundancy failures.

Rumors are that since Microsoft's acquisition, most of the talent that made Danger worth acquiring has left. If that's true, Microsoft's acquisition and management did, in fact, change the company. But this may or may not be relevant to the quality of Danger's IT unit.

But IT companies responsible for other people's data are expensive specifically because they spend a lot of time and equipment on ensuring total data loss doesn't happen. So it makes me wonder whether the outage was, say, a consequence of moving Danger's data systems to Microsoft technology (This was what Microsoft did when they bought Hotmail - the full cutover took over two years) before the replacement backup system was fully deployed.
posted by ardgedee at 7:26 AM on October 11, 2009

DUDE, BUYING AN APPLE DEVICE WILL SOLVE ALL YOUR PROBLEMS!!!

DID YOUR DATA CENTER BURN DOWN? SHOULD HAVE GOT AN MAC!
posted by delmoi

In this case, yes, having an iPhone would have prevented this problem.

But feel free to continue the little OHMYGOD IHATE APPLE SO LET ME GET MY HATERZ ON AMIRITE?! schtick you got goin' on there.
posted by Dennis Murphy at 7:29 AM on October 11, 2009 [2 favorites]

"You'd think that a company that is providing cloud type services would have a bullet-proof risk mitigation plan that would be able to survive multiple points of failure."

Never underestimate the capacity for managers to ignorantly override and/or nickel-and-dime common sense industry standard best practices out of existence, for said capacity is nigh infinite.
posted by majick at 7:35 AM on October 11, 2009 [8 favorites]

FishBike: in some cases the cloud is too big to back up, and in others there is just so much trust in the system that backups seem pointless. It does seem odd that data like address books, which even for 100 million people could backup to a couple TB drives, would have no non- cloud backup. But then, if you had multiple datacenters with sharded clouds, would you bother backing up what is highly volatile data? And with SK, there's probably little distinction between small data like phone numbers and large data like all the pictures you took- and suddenly your backup set is too large to actually archive. I'd also argue that phone #'s are the easiest to lose, since they'll be fairly recreatable if inconvenient.

Sure in retrospect there are a lot of people wishing they'd taken a nightly or even weekly dump of the critical data, but in a sense a well- designed cloud seems as or more reliable than tape/ data backups. The practices of a small company archiving nightly Exchange or Perforce db to tape/disk and sending offsite don't really apply for the data footprints of a cloud- the only thing tape/disk backups buy you are snapshots of older data- and I've been in that IT seat of realizing your last usable backup of Exchange is actually from 3 weeks earlier. Worst Christmas vacation ever!

That said, even the example of google's 9-way sharding is still susceptible to a software bug or a disgruntled ex-employee. I wouldn't be surprised if one of the two is the case here.
posted by hincandenza at 7:37 AM on October 11, 2009

"Exchange is pretty damned rock solid in my experiences. I've got dozens of clients running Exchange 2003 and Exchange 2007 in SBS and standalone environments and over the last seven years I can count exactly zero corruption issues, zero downtime due to Exchange and zero support calls about Exchange."

How big are these clients? Most of Exchange's problems is it doesn't scale terribly well. And I'm surprised you've never experienced any down time. Most Exchange shops I've worked for seem to schedule a day or two a year for maintenance and updates.
posted by Mitheral at 7:38 AM on October 11, 2009

" But then, if you had multiple datacenters with sharded clouds, would you bother backing up what is highly volatile data?"

Yes. Point-in-time backups exist for a reason. The data loss described in the headlines here today is that reason. Besides, contact lists really are not that volatile. My wife's got a little hardbound booklet thing from when we dated in the 1980s. If Danger or T-Mobile Ops had a backup that was ten years out of date they'd still have better quality backups to draw on than she does. And I dunno about you, but every time I'm in charge of a core part of my company's functionality I back it up more than every decade.

" I've been in that IT seat of realizing your last usable backup of Exchange is actually from 3 weeks earlier. Worst Christmas vacation ever!"

You and me both, buddy. Anyone who has dodged that bullet should make damned well sure they keep dodging it -- frequent tested point-in-time backups of your volatile core data are how you do so. And those of us who have taken the bullet? We're supposed to be even more gun-shy now.

Sheeit, restoring from a 3 week old backup would be an outright improvement for the victims of this fuck-up. Your "screw up my vacation" backup practices of the past are still better than these clowns could manage. Ridicule is very much in order, although of course we can all sympathize and nod our heads sadly for the victims who made a perfectly reasonable assumption that their data storage provider took basic fucking precautions.
posted by majick at 7:47 AM on October 11, 2009

Add me to the list of 'can't believe this would happen to a company in the year 2009'.

I think I've met exactly one person, ever, who had owned a Sidekick - but I'm in Canada and I don't believe they ever really got popular here.

I have a Blackberry and use Gmail so even if I was to lose my Blackberry I would have the Gmail site backup. And even if Gmail lost their servers, I would have every single email in .pst files because I download them via POP3.

I don't have an iphone (blasphemer!) - does itunes do a backup of you contacts and all phone content when you sync? Blackberry desktop manager prompts for a full backup weekly but I know most BB users don't bother.
posted by jeffmik at 7:47 AM on October 11, 2009

"in some cases the cloud is too big to back up"

I've never worked something this big. Is this really the case? It would seem that, professionally speaking, "too big to back up" is a nicer way of saying "only a crazy sociopath blinded by short term revenue streams would deploy".
posted by Mitheral at 7:52 AM on October 11, 2009 [1 favorite]

Dennis Murphy: In this case, yes, having an iPhone would have prevented this problem.

But feel free to continue the little OHMYGOD IHATE APPLE SO LET ME GET MY HATERZ ON AMIRITE?! schtick you got goin' on there.

To be fair, any windows mobile device would have similar protection, or most any other smart phone or dumb phone with a sim, as would Outlook users, since in all cases it's considered normal to have your data principally stored or mirrored locally, or backed up on every sync.

It's an unusual element that SK seems to make data unarchivable, since users never even Bluetooth sync to a nearby mac/pc. It's the only phone I know that does that. Webmail is another service with similar vulnerabilities. And while I *think* Outlook now has support for hotmail/windows live access and thus potential offline storage.... I'm not sure gmail does, shirt of POP3 download which removes it from the cloud.

I'm curious if we'll learn the details of the loss. Even with sharded data, doing data recovery on the disks of machines holding collectively at least one full copy would let you recover most of what's lost. It might cost $1-2 million for a ~petabyte of recovery, but that'd be worth it in this case.
posted by hincandenza at 7:53 AM on October 11, 2009

"I think I've met exactly one person, ever, who had owned a Sidekick "

Me too, exactly one person, possibly even the same one person since it's such an obscure niche device, but I'm given to understand the devices were or are immensely popular in the hearing impaired community. And that's not a small community.

"- does itunes do a backup of you contacts and all phone content when you sync"

Yes. Plus if you're using it properly, you're doing over-the-air sync to a backing store of some kind for contact and calendar data. So iPhone users can have three copies of their data effortlessly: One on the phone, one with their service providers, and another write-only point-in-time snapshot backup that's not in the hands off sync workflow. It can certainly be done, but you have to go out of your way to lose your data with an iPhone. You'd have to blow away your desktop backups, do something bad to your IMAP or Exchange store, and drop your phone in the toilet all at the same time.
posted by majick at 7:54 AM on October 11, 2009

hincandenza - downloading via POP3 from Gmail does not delete from their server, it archives the message and moves it to 'all mail' folder. At least, thats how it works on my accounts.
posted by jeffmik at 7:54 AM on October 11, 2009

Microsoft Outlook and Exchange are like two of the four software titles they make that are actually good.

I am afraid to ask what the others are.

Age of Empires was fun.
posted by Jugwine at 7:58 AM on October 11, 2009 [1 favorite]

I don't think any system is so big that it literally can't be backed up. But I can certainly see how it may not be economically viable to back it up in a similar manner to what we rely on in smaller environments. And that people can implement system architectures, regardless of size, where it's not at all obvious how to back them up effectively, and make a conscious decision not to include backups as a mandatory functional requirement.

Traditional backups could raise the cost of the service enough that people would use other, cheaper services instead. Especially in a market where the user base doesn't care about backups (until something goes wrong) and where it doesn't seem like any of the service providers disclose their backup strategy, for those users who would care about this.

So I could see a careful discussion of business and technical risks vs. market realities leading to a decision to make the system as reliable as possible and not do traditional backups.

I strongly suspect whatever went wrong here was not a result of that kind of careful decision-making, though.
posted by FishBike at 8:10 AM on October 11, 2009

Most of Exchange's problems is it doesn't scale terribly well.

I've designed an Exchange deployment for a 90,000 seat organization, and have sanity-checked the architecture of Exchange/ActiveDirectory organizations quadruple that size. If anyone thinks that Exchange doesn't scale, they're doing it wrong.
posted by deadmessenger at 8:22 AM on October 11, 2009 [3 favorites]

"I'm not sure gmail does, shirt of POP3 download which removes it from the cloud."

My experience is the same as jeffmik. All my mail, even that I've deleted locally, is still available trhough the web interface even though I've down loaded it via Mozilla. It's the best of both worlds: I can use a real mail client when I've got my laptop and in a pinch where all I've got is a web browser I still have access to all my mail. Google pulls a yahoo and decides to kill off a service next week and I've still got all my mail.
posted by Mitheral at 8:30 AM on October 11, 2009

Mitheral: "in some cases the cloud is too big to back up"

I've never worked something this big. Is this really the case? It would seem that, professionally speaking, "too big to back up" is a nicer way of saying "only a crazy sociopath blinded by short term revenue streams would deploy".

Well, as FishBike says right above, the data is only rarely literally too big to backup- it just becomes prohibitively more difficult to do so, to the point that it makes little financial sense. It would be for these companies the equivalent of paying half your salary in insurance premiums for nuclear holocaust.

And the thing to understand about a cloud is that it is already a number of redundant copies of the data. The cloud is comprised of many copies, such that the usual failures smaller companies protect against- a server or disk dying, a local office catching fire- are eliminated in a well structured cloud: there are multiple datacenters in distant geographic locations, there are multiple copies of "the data" in each DC, etc. Many cloud storage systems even have journaled or versioned methods of storing data, so that data is largely not even deleted so much as marked archive and pushed to less (but not non-) redundant systems.

Google simply can't back themselves up, because they already do. Making point-in-time backups of GMail would be incomprehensible. Why would Google do that, when their entire company is built on a storage system that is legendarily redundant and fast and geolocated? If that storage system has a software fault and that fault occurs, the data loss would be unimaginable- and unrestorable. They are betting their company that won't happen, and so far have been lucky.

And do you think Flickr has backups, or do you think Flickr just trusts that their architecture has enough copies- and that their users have their own backups anyway- that it's not worth backing up what may be 90-95% reproduceable data by their user base? If the cloud that hosts Flickr ever went down in totum, I suspect that would be that: no more Flickr data, until and unless people trusted them enough to start re-uploading their pictures.

majick rants above that yeah, even been a week old would be a boon in this circumstance- and SK likely has orders of magnitude less data than GMail- but it's always easy to say that in retrospect. I'd be surprised if Danger really was so badly built that they didn't have at least two datacenters holding instances of the cloud, and such an architecture is pretty solid. That's why "unbelievable software bug/corruption" or "malicious act by technically savvy insider" seems more likely to me.


I currently work for [LARGE_ONLINE_TRAVEL_COMPANY], and while I don't work in the enterprise data storage area you'd better believe that data is backed up like crazy. It would literally be the end of [LARGE_ONLINE_TRAVEL_COMPANY]'s existence if that user and travel data was lost, whereas T-Mobile has other users besides SK users, and even though their SK userbase may be royally pissed now after a couple of weeks of "Hay, send me ur nmbrs pls!" the address book data is largely recreated. Hell, I wouldn't be surprised if their contracts not only protected T-Mobile from liability, but didn't even let them cancel their contracts so they could go get another device/provider.

So yeah, in some cases the data is not seen as worth backing up. I hate to reiterate it, but if I were a SK user (and I'm not, so not terribly familiar with their offerings) about the only data I'd truly miss would be photos. Past mail/text messages aren't super critical but missed, contact information can mostly be recreated (really, the only numbers you can't get back are people who you never talk to that you don't also have as a friend on MySpace, LiveJournal, Facebook, etc), etc.

But data that can't be remade and was meant to be preserved, such as pictures taken, that'd be the one great loss. And also an absolute bitch to backup the photo stores of millions of people.
posted by hincandenza at 8:37 AM on October 11, 2009 [1 favorite]

If everything is dependent on a central operator, then eventually everything is going to fall apart.

Matt, you've got backups here, right?

If you conceive your platform as an online service and your devices as dumb terminals for accessing it, then the idea of having a persistent store of anything locally is absurd.

I think the conception is the absurd part, here. At least keep a local cache, and don't wipe it out when the server returns 0 data, jesus.

Also: what's the scope for this being a rogue employee? That's what concerns me most about cloud -- the pissed-off underpaid joe tech, wandering through the filesystem.
posted by fightorflight at 8:56 AM on October 11, 2009

Let me also add that gmail data, for example, is replicated 9 times over geographically separated areas... you'd need at least three nukes to wipe out any individual gmail account.

Or just wipe out one copy and have the deletion replicate to the others. Replication != backup.
posted by smackfu at 8:56 AM on October 11, 2009 [1 favorite]

But...I thought the cloud was supposed to be backed-up to the rainbow.
posted by Thorzdad at 9:01 AM on October 11, 2009 [4 favorites]

I think the conception is the absurd part, here. At least keep a local cache, and don't wipe it out when the server returns 0 data, jesus.

I trust you have a local backup of all your Metafilter comments, contacts and history, and only open the homepage in a new tab rather than clicking refresh, in case the server has gone down in the mean time?
posted by cillit bang at 9:11 AM on October 11, 2009

And the thing to understand about a cloud is that it is already a number of redundant copies of the data. The cloud is comprised of many copies, such that the usual failures smaller companies protect against- a server or disk dying, a local office catching fire- are eliminated in a well structured cloud: there are multiple datacenters in distant geographic locations, there are multiple copies of "the data" in each DC, etc. Many cloud storage systems even have journaled or versioned methods of storing data, so that data is largely not even deleted so much as marked archive and pushed to less (but not non-) redundant systems.

One of my requirements for backups is that at least some of them should be offline. That is, on media which are physically removed from the system once the backup has been made. It just makes me nervous if there is even a theoretical possibility for software at some level to go haywire and destroy everything, or for a rogue employee to do that if all storage is accessible online.

That this type of system requires people to manually move media around is a feature, not a bug, in my opinion. For these large environments, I'd want to see this going on at multiple locations. That way, destroying the backups would require lots of manual work from many people. I don't know if any of these large cloud environments do this, and I can see how it would be too expensive if they tried. So I hope they are really, really careful about their online backups (or equivalent thereof), in that case.

I should mention that I am also nervous about walking in front of my car when the engine is running. The transmission is computer-controlled and it has a throttle-by-wire system. I get nervous that the only reason it doesn't put itself in gear and run me over is that the computer doesn't feel like doing that today. I always put the handbrake on, too, so that there's something outside of the computer's control holding it back. And so it should be with backups.
posted by FishBike at 9:19 AM on October 11, 2009 [1 favorite]

I trust you have a local backup of all your Metafilter comments, contacts and history, and only open the homepage in a new tab rather than clicking refresh, in case the server has gone down in the mean time?

You try to snark, but I lived through the jrun era, so yes, I automatically copy all of my comments before hitting post, in case the server has died while I've been writing, I load new threads in new tabs, and I *do* have an export of all my comments. I also back up my GMail via POP, and in general don't trust or assume that I'm always going to have internet access.

Why the makers of a mobile phone in a country with unreliable data access and nothing like universal coverage would assume the contrary baffles me. And to not even put in a sanity check -- "The phone has 250mb of data, the server says it has 0mb, should I ask the user about this?" -- is just damn stupid. But evolution is about to do its work on these chumps, so I guess it's OK in the long run.
posted by fightorflight at 9:27 AM on October 11, 2009

Dennis Murphy: "But feel free to continue the little OHMYGOD IHATE APPLE SO LET ME GET MY HATERZ ON AMIRITE?! schtick you got goin' on there."

I think you missed the comment he was replying to, you clod.
posted by boo_radley at 9:35 AM on October 11, 2009

That's what concerns me most about cloud -- the pissed-off underpaid joe tech, wandering through the filesystem.

Security cam photo of suspect rogue tech in cloud.
posted by rokusan at 9:44 AM on October 11, 2009 [1 favorite]

I trust you have a local backup of all your Metafilter comments, contacts and history

My preferred technique is to never make any comments worth saving.
posted by rokusan at 9:45 AM on October 11, 2009 [5 favorites]

Let me also add that gmail data, for example, is replicated 9 times over geographically separated areas... you'd need at least three nukes to wipe out any individual gmail account.

So what you're saying is that the apocalypse will be triggered not by socio-political strife, but by some guy with a few loose nukes who just changed his mind about that "fuck you, I quit" email he just sent his boss?
posted by billyfleetwood at 9:50 AM on October 11, 2009

My preferred technique is to never make any comments worth saving.

A lot of people seem to think a lot of your comments are worth saving. Maybe there is a solution here. They could set it up so that clicking the "favorite" button saves a local copy of the comment being favorited.

FAVORITES ARE RUINING REPLICATING METAFILTER!
posted by FishBike at 9:51 AM on October 11, 2009

Heh. Age of Empires II is one of very few MS products I actually use regularly.
posted by rokusan at 9:56 AM on October 11, 2009

I automatically copy all of my comments before hitting post, in case the server has died while I've been writing, I load new threads in new tabs, and I *do* have an export of all my comments.

I believe you just outed yourself as a crazy person.

Why the makers of a mobile phone in a country with unreliable data access and nothing like universal coverage would assume the contrary baffles me.

They've shipped millions of the things, so obviously the idea is sound.

And to not even put in a sanity check -- "The phone has 250mb of data, the server says it has 0mb, should I ask the user about this?" -- is just damn stupid.

In an online architecture the device's data is an incomplete, unreliable cache and the server's version is the master copy. So no, you do not ask.

And in any case it would appear the device is able to revert to its cache when the network is unavailable, but if the battery dies, the cache gets wiped.
posted by cillit bang at 9:58 AM on October 11, 2009

Oh, an actual comment, right.

A few surprising things here, aside from the general "how does this happen in the nascent information age" commentalia above:

• Apparently the real fuck-up was on a third party. From gdgt forums,"It's not a server failure. They were upgrading their SAN, and they outsourced it to a Hitachi consulting firm. There was room for a backup of the data on the SAN, but they didn't do it (some say they started it but didn't wait for it to complete). They upgraded the SAN, screwed it up and lost all the data."
• It's amazing, personally, that the sidekick apparently has no storage at all. What would drive such a design decision? I have a G1, and even though it uses the cloud, it for synchronization to local storage. That seems like the Least Surprising implementation to me, I can't see a good rationale for Danger's choice.
• Wikipedia's page on the Hiptop notes, "This article documents a current disaster." which is kinda pompous. It's terrible, but I guess I think of disasters as earthquakes and tsunamis and such.

posted by boo_radley at 10:04 AM on October 11, 2009

I believe you just outed yourself as a crazy person.
Ah yes, backups and not trusting in a server that's burned you before. The secret mark of the loon.

They've shipped millions of the things, so obviously the idea is sound.
You know, they sold a lot of thalidomide and Ford Pintos too. I do love fallacies, got any more?

In an online architecture the device's data is an incomplete, unreliable cache and the server's version is the master copy. So no, you do not ask.
And then you lose your job when the master copy turns out to be a piece of shit. Sync is hard, it's never going to be perfect, so yes unless you believe in the perfection of your code and systems, you ask. Apple are terrible at sync, and even they manage to pop up alerts warning you that a background sync is about to nuke all your contacts and check you're sure you're OK with it.

At the very least, you don't wipe the goddamn cache unless you have something to replace it with -- what if the battery dies when the user is out of reception? Gah, the shortsightedness, it burns.
posted by fightorflight at 10:06 AM on October 11, 2009 [2 favorites]

Apparently the real fuck-up was on a third party. From gdgt forums,"It's not a server failure. They were upgrading their SAN, and they outsourced it to a Hitachi consulting firm. There was room for a backup of the data on the SAN, but they didn't do it (some say they started it but didn't wait for it to complete). They upgraded the SAN, screwed it up and lost all the data."

Well, if this turns out to be right, I'd argue there are nested fuck-ups going on here, and that the larger one is on the part of whoever decided a SAN is 100% reliable even during maintenance activities, and therefore doesn't need to be backed up externally. It's the same "we have RAID so we don't need backups" mentality that has caused so many other data disasters.

In the grand scheme of things, though, this is still only a Category 1 Big Mistake (on the FishBike scale of Big Mistakes) so far.
posted by FishBike at 10:18 AM on October 11, 2009

There are thousands - millions - of companies whose business is 100% reliant on a central data store of their customers' information. Your implicit suggestion that all of these business are being stupid by not pushing this information out to their frigging handhelds for backup purposes is laughable. Just because Danger happen to be in the handhelds business does not change this.

what if the battery dies when the user is out of reception

The device is useless if it's outside reception anyway.
posted by cillit bang at 10:22 AM on October 11, 2009

There are thousands - millions - of companies whose business is 100% reliant on a central data store of their customers' information. Your implicit suggestion that all of these business are being stupid by not pushing this information out to their frigging handhelds for backup purposes is laughable

There's no such suggestion there, implicit or not. Of course you can have centralised data. But you – and Danger, clearly – are thinking about the ownership of the data incorrectly. It's not their data, it's the users'. Microsoft doesn't store everybody's Exchange server for them, does it?

Now you're right about the online architecture thing as well, to be fair: GMail's gears setup doesn't really need to ask before it wipes out its temporary local cache. But that's because there are expected to be many clients, and the server really does need to be the Truth.

The real problem here is that Danger chose a wildly unsuitable model for their phone set up. There aren't multiple clients -- people don't change their phones all that often -- and the benefits of all-eggs-in-one-basket don't come close to outweighing the dangers.

The device is useless if it's outside reception anyway.
This is the sort of idiotic short-sighted thinking that got Danger into this mess. No reception? Well they won't be able to make a phonecall then, why would they need into their address books? A landline? What's that? They're in the doctors and need to look up the notes they'd been keeping on their child's fits? Well, if there's no reception in the doctors they won't need to do that then.

It's a personal mobile information device. The information needs to be with its owner at all times, regardless of battery status, network access or server status. A thorough network backup is good, perhaps essential, but The Truth needs to be in the user's pockets. Not floating untethered in the cloud.
posted by fightorflight at 10:33 AM on October 11, 2009 [3 favorites]

They upgraded the SAN, screwed it up and lost all the data.

This is still hard to believe. This is why Iron Mountain exists as a company, so you can take your backups and put them somewhere where it will still be OK if all your servers are wiped off the earth. They'll even come and pick them up from the tape drive in your data center every day, at some stupid cost. Maybe the data will be old or out of date but it won't be lost.

Honestly, most of the bad hardware issues I've seen have been in spite of redundant hardware. A $10k disk controller with redundant everything tells you one of its controller boards is dying, and calls in a tech itself. So the tech does a hotswap replace and it dies once it is running on the "good" board. I've also seen super-duper redundant power system fail to kick in during a disaster recovery test, which is both good and bad.
posted by smackfu at 10:51 AM on October 11, 2009

"And do you think Flickr has backups, or do you think Flickr just trusts that their architecture has enough copies- and that their users have their own backups anyway- that it's not worth backing up what may be 90-95% reproduceable data by their user base? If the cloud that hosts Flickr ever went down in totum, I suspect that would be that: no more Flickr data, until and unless people trusted them enough to start re-uploading their pictures."

Thanks for the perspective hincandenza. I admit to be a little surprised that so many companies wouldn't keep at least, oh I don't know, weekly back ups a couple of generations deep instead electing to just roll their admittedly heavily weighted dice. Taking Flickr specifically it is the metadata that comprises maybe 1% of the data set that is the whole point behind using them. Permanent links, tagging and sets are what make the service more than simple image sharing. Holy avacodo imagine how much of the internet would break if Yahoo lost all that data and users were forced to re upload all their images generating new permanent links in the process. Places like metafilter where users don't have the ability to change or even up date links after a certian amount of time had pasted would be permanently degraded. RAID, however distributed, isn't a backup. I hope they are paying their employees well.

Trusting the users to have a copy is insane though I don't doubt their terms of use expect that. It's not hard to find advice even on AskMe recommending that the only copy of once in a life time pictures be stored on Flickr's or gmail's disk.
posted by Mitheral at 11:03 AM on October 11, 2009

" But then, if you had multiple datacenters with sharded clouds, would you bother backing up what is highly volatile data?"

Why not? My perspective is perhaps coloured by working for financial institutions, but yeah, we backup to multiple replicating DataDomains for the "oops I fucked up and wiped a bunch of data" backups, but we still archive to off-site media.

I've also seen super-duper redundant power system fail to kick in during a disaster recovery test, which is both good and bad.

Oh yes. Been bit by UPS/City/Gen power misbehaving all at once more than a few times.
posted by rodgerd at 11:06 AM on October 11, 2009

(Incidentally, everyone arguing that backing this cloud data up might be too hard or expensive? Well, it's cheaper and easier than ceasing to exist as a viable business. Which is pretty much what just happened to these guys.)
posted by rodgerd at 11:19 AM on October 11, 2009 [2 favorites]

cillit bang: "There are thousands - millions - of companies whose business is 100% reliant on a central data store of their customers' information. Your implicit suggestion that all of these business are being stupid by not pushing this information out to their frigging handhelds for backup purposes is laughable. Just because Danger happen to be in the handhelds business does not c hange this."

If this were centralized business data that Danger/ MS owned, you'd have a really good point, and I can understand it completely. My business' ERP data is nobody's business but my business, absolutely.

The lost data here isn't Danger's though; it's their users' data. Right now, Sally Sidekick has, from what I've read, one means of backing up her data: manually. There's no other way to get anything in or our of this device currently.

I posit that it is your implicit equivalence between this situation and all other business data in the world is laughable. They're not really similar.

cillit bang: "The device is useless if it's outside reception anyway."

Not right now it's not: it's the only existing copy of your data. Put it this way, if you had a sidekick Right Now and you were out of your service area, would you turn it off to save battery?
posted by boo_radley at 11:20 AM on October 11, 2009 [1 favorite]

GMail is about the only 'cloud' data I rely on, and I use it as most others above: pop3 it down and treat it as if it's been removed from GMail, but really I know it's been stored in "All Mail", which means I can re-retrieve from a different device if I wish, go find some old bit of info from an airport web terminal, and so on. And it's all neatly kept there forever so that the NSA can check it out whenever they like. (I hope they like LOLcats.)

Which means, yes, I have a bunch of inherent backups on each device that's been pop3ing, without ever doing a deliberate backup. Every time the mail folder on my Mac hits a couple of gigs, I burn a DVD.


(Using preview, for a change: 'pop3ing' looks like a word the Vatican would use when attempting to be hip.)
posted by rokusan at 11:33 AM on October 11, 2009 [1 favorite]

It's not hard to find advice even on AskMe recommending that the only copy of once in a life time pictures be stored on Flickr's or gmail's disk.

Really? Who on earth recommended that?
posted by rokusan at 11:35 AM on October 11, 2009

Matt, you've got backups here, right?

Heh. We do, yes. Nightly.

And it's trivial to grab an export of your comments from the preferences page if you like; and the infodump externalizes a lot of e.g. favoriting data and contact data if you want hardcopies of those; and I copy my longer comments to buffer before hitting post Just In Case, too (esp. on non-mefi sites, actually, if I don't know how well their commenting systems work).

Not that mefi is in the same business as T-Mobile, naturally. But there's space between the endpoints of Never Back Anything Up and Completist Digital Hoarder, and a lot of personal and professional best-practices fall in that space.
posted by cortex at 12:09 PM on October 11, 2009

This is really a management failure and not a technology failure, at least given what we know. When you have layoffs, key members of a team leaving and general neglect within your parent company, this is a recipe for something bad to happen. In normal companies this usually results in a liability like a lawsuit or at best, a loss of a client. Being a tech company, I'm going to surmise that low level employees were trying to find ways to tighten their budgets and the expensive, time consuming offsite backups were the first on the chopping block ... hey it is a shared nothing architecture, right? Except the guy who designed this knew that under certain situations shared nothing can be corruptible and wasn't totally sure they'd always work so in a prudent measure he decided to keep offsite backups, but he left 9 months ago and your new manager from a different division just cares about making the numbers look good so he can get on a better project within the company.
posted by geoff. at 12:18 PM on October 11, 2009 [2 favorites]

We store our off-site backups at Gringotts. We have never had a problem.
posted by double block and bleed at 12:25 PM on October 11, 2009 [2 favorites]

In this case, yes, having an iPhone would have prevented this problem.

So would buying a G1, a Pre, a Nokia, a Blackberry, or any other phone that wasn't a sidekick. On the other hand, if apple ever has another problem then buying an alternative would have prevented it. Apple's record with MobleMe is not stellar, it was a huge failure on launch which actually caused the destruction of data that people originally had on their hard drives. There was also a more recent problem where people lost data after a trial period ended.

And do you think Flickr has backups, or do you think Flickr just trusts that their architecture has enough copies

I'm pretty sure they don't have backups. There was a controversy a while ago where they nuked someone's account for posting detainee abuse photos in the whitehouse's photo stream comments, and then claimed they couldn't get the pictures back because the deletion propagated. Or maybe they just felt he wasn't worth the time to pull his data from tape.

Places like metafilter where users don't have the ability to change or even up date links after a certian amount of time had pasted would be permanently degraded. RAID, however distributed, isn't a backup. I hope they are paying their employees well.

Actually, if you go back and look at years-old archives, tons and tons of the links are broken.
posted by delmoi at 1:23 PM on October 11, 2009

Those applications would be Microsoft Outlook, coupled with Exchange, Excel, RDP, and Windows Media Center (not Windows itself Media Center). I've actually had really good experiences with Exchange and Outlook and using it remotely through BlackBerry on BES.

I do IT for a medical clinic here, and one of the doctors asked about hosting BES. So, first thing I did was get the price out of the way. That's when that conversation stopped. I'd like it better if the price were not so astronomical. I realize in many markets it's probably par for the course, but this is a small resort-type town with a lot of agriculture, and I can't think of a single business here which would be able to justify the cost, but then again, I can't think of a single business with more than a dozen or so smart phone users. Maybe the hospital, but that's a single exception.

As for the weakness of BlackBerry, RIM has had a rather outstanding uptime performance record for push mail and PIM and I've never had issues with that and remotely accessing Exchange through it.

Yes, no complaints so far, but having an extra man in the middle seems a bit excessive and prone to failure at multiple points.
posted by krinklyfig at 1:30 PM on October 11, 2009

"Actually, if you go back and look at years-old archives, tons and tons of the links are broken."

True and it is very irritating. I wouldn't be surprised if the number of broken links instantly doubled though if flickr had a catastrophic failure. Undying links is why I mostly prefer to host images on my own domain. It always feels icky though to host other peoples content though I've done that a few times if I strongly suspected the originals were going to disappear. Photobucket, RapidShare and other companies who offer free hosting but remove content if it isn't accessed for a while should be taken out behind the barn and shot.

rokusan: "

"It's not hard to find advice even on AskMe recommending that the only copy of once in a life time pictures be stored on Flickr's or gmail's disk."

Really? Who on earth recommended that?"

Check the linkage. That's just from one recent thread but, as lack of data protection is one of my tear my hair out pet peeves, I see it pretty often.
posted by Mitheral at 1:38 PM on October 11, 2009

The device is useless if it's outside reception anyway.

Your usability license has just been revoked.
posted by furtive at 1:43 PM on October 11, 2009

Ironically, T-Mobile effectively has the monopoly on (subsidized) iPhone plans in the Netherlands.
posted by goodnewsfortheinsane at 2:07 PM on October 11, 2009

Ay, geez, one little comment and the thread gets derailed. Allow me to elaborate to avoid further snark.

My wife had a Sidekick with T-Mobile and liked it, but the Apple sync services was a much better fit for her (even with the MobileMe debacle). Now she still has T-Mobile but with an unlocked iPhone that is a little cranky at times.

A future alliance between T-Mobile and Apple would be great for both firms and for us. T-Mobile gets to retaliate quite effectively against Microsoft. Apple gets a wireless partner instead of/in addition to whiny ol' AT&T. My wife and I get one cheaper bill and a carrier with better customer service.
posted by infinitewindow at 2:45 PM on October 11, 2009

Your usability license has just been revoked.

Should tell Steve Jobs that. I count 10 out of the 20 built-in iPhone apps as requiring a network/cell connection to serve any useful purpose, including all of the important ones.
posted by cillit bang at 3:15 PM on October 11, 2009

Should tell Steve Jobs that. I count 10 out of the 20 built-in iPhone apps as requiring a network/cell connection to serve any useful purpose, including all of the important ones.

Which is why your licence got revoked. It's not 10, it's 5 that require network -- YouTube, Stocks, Weather, App Store and iTunes. All 5 of which are straight interfaces to websites, basically.

The others are all useful to a greater or lesser degree without any connection. When abroad I've loaded up Maps with a map of the area while on wifi and then used the GPS with no data connection outdoors while walking. I've loaded Safari with long MeFi threads and then read them on the underground. The Phone app gives you numbers and missed calls even if you can't call them, the SMS app lets you read your archive and compose drafts, etc etc etc.

Good usability involves thinking about the edge cases, what happens when everything's not rosy. That's what you're not doing and it's what Danger didn't do.
posted by fightorflight at 3:32 PM on October 11, 2009

> I count 10 out of the 20 built-in iPhone apps as requiring a network/cell connection to serve any useful purpose, including all of the important ones.

Accessing stored data should not require a live data connection, and on the iPhone (and, to get this out of the way now, the Blackberry, Android, and WinMo phones) it doesn't. Accessing streaming data -- fresh SMS messages, video feeds, current weather reports -- by definition require life data connections, and they are equally useless without one on any device, whether it's a mobile phone, desktop computer, or Dial-A-Weather-Report.

You're making silly-ass claims here and derailing the thread. Cut that out.
posted by ardgedee at 3:49 PM on October 11, 2009 [1 favorite]

My iPod Touch does some kind of magical triangulation without having GPS capability, probably based on WiFi availability, and can determine my location without difficulty. It's a bit creepy on some level, but it's like magic, and doesn't demand that I have cell connectivity or GPS to function. Just a WiFi connection, which I can leech nearly anywhere these days.
posted by hippybear at 4:01 PM on October 11, 2009

When abroad I've loaded up Maps with a map of the area while on wifi and then used the GPS with no data connection outdoors while walking. I've loaded Safari with long MeFi threads and then read them on the underground.

Er... Half the time you try to do that, Safari will have cleared its local cache when you go back to page, on the assumption it can easily retrieve again from the network. It is not an app designed with any real accommodation for offline use. The Maps and Mail caches aren't much better.

Good usability involves thinking about the edge cases, what happens when everything's not rosy.

Edge cases are what you make engineering trade-offs against. Danger's USP is their entirely online architecture, which they and their customers clearly believed offered enough advantages to outweigh the drawbacks. And, having some experience designing synchronisation algorithms for offline apps vs building online web services, I concur.
posted by cillit bang at 4:37 PM on October 11, 2009

Er... Half the time you try to do that....
Tell you what it won't ever do: wipe out all your shit if Apple fucks up a migration of MobileMe and your battery goes flat.

Danger's USP is their entirely online architecture, which they and their customers clearly believed offered enough advantages to outweigh the drawbacks.
Uniquely stupid point, perhaps. It certainly wasn't their selling point -- that was the same as it is for iPhone, Blackberry and all the other smartphones, although more aimed at young people. It wasn't at all made clear that what you were getting was in effect the equivalent of a dumb terminal. They didn't say people "all your life, in your pocket, (well, so long as our servers are up, you have endless power, you have network connectivity and we haven't buggered up our disks)". Because that's not a selling point, it's a big danger flag.

As for your concurring, well, no shit, sync is hard. And getting around that by assuming that the network will always be there and your servers will always work is easy. Fucking stupid, but easy. But you're clearly deeply invested in the stupid route, so I'll leave you to your derail.
posted by fightorflight at 5:12 PM on October 11, 2009

Once more we can see evidence that Charlie Brooker is right.
posted by rodgerd at 6:45 PM on October 11, 2009

fightorflight: Tell you what it won't ever do: wipe out all your shit if Apple fucks up a migration of MobileMe and your battery goes flat.

Wait, wasn't it noted multiple times earlier in the thread that MobileMe did exactly that- and worse, not just to a mobile device, but actually deleting data off of Mac user's hard drive that wasn't part of the MobileMe system.

As bad as Danger has been here, what MobileMe did seems far, far, far worse: it went beyond failing itself to fail other systems and data as well.
posted by hincandenza at 6:57 PM on October 11, 2009

fightorflight: "Because that's not a selling point, it's a big danger flag."

DANGER POINT
posted by subbes at 7:11 PM on October 11, 2009

Wait, wasn't it noted multiple times earlier in the thread that MobileMe did exactly that- and worse, not just to a mobile device, but actually deleting data off of Mac user's hard drive that wasn't part of the MobileMe system.

Wait (and this goes to you too, rodgerd), I was one of the people taking a pop at Apple for shitty sync upthread ... but from what I can see most of the references to MobileMe in this thread are positive. It's delmoi's links that are the damning ones, and they're nothing like what you describe. The problems in the first two are about them losing user email -- which is unforgivably stupid, but it's nothing we haven't seen before, and is very different from the Danger case in that if you had local caches and knew what you were doing, you could recover it. If you'd used POP, you were fine, too.

I don't know where "deleting data off a Mac user's hard drive that wasn't part of the MobileMe system" comes from -- unless you mean it emptied the iDisk folder when the account expired. Well, no shit, Flickr does pretty much the same thing. If you pay up, you got access to the folder again. It's arguably bad design and a bit crap, but it wasn't a failure as such -- you get plenty of warning emails, too.

Still, the only reason I picked out the iPhone in the comment was because we'd been talking about it. There's nothing special about it, it's the same as virtually everything else in the mobile phone world -- ie, it won't irretrievably lose its mind if it can't talk to the server when it wakes up. In contrast to the hiptop, whose design I've only learned about in the past few days and am still boggling at.
posted by fightorflight at 7:15 PM on October 11, 2009 [2 favorites]

also, hincandenza, this MobileMe trial expiry thing still flashes up the huge "syncing will delete lots of your shit -- OK or no?" alert and allows you to stop it. It's worlds different from this.
posted by fightorflight at 7:24 PM on October 11, 2009

Couple details about the iPhone, fairly irrelevant to the larger issue: 1) With Maps, as long as you don't quit it, the cache should stay in place, allowing you to use however much you've loaded into it, even when offline; you can also use backgrounder with jailbreak to keep Maps running in the background -- I've used Maps for many days between connections chugging happily along with many tens of megabytes of map data loaded into it. 2) If during a backup there is more than a 5% discrepancy between the backup and the phone's Contacts or Calendar data, it explicitly asks you which one you want to go with. In fact, once something went wrong with my phone and I casually hit the wrong button and erased my backup -- but at least it alerted me and allowed me to do the screwing up. But I'm sure most other cell companies take similar precautions -- I look forward to reading what synergies of incompetence cooperated to produce this massive failure.
posted by chortly at 7:41 PM on October 11, 2009

cillit bang: "I trust you have a local backup of all your Metafilter comments, contacts and history, and only open the homepage in a new tab rather than clicking refresh, in case the server has gone down in the mean time?"

You know, I actually have a folder in my RSS reader with a bunch of feeds related to me. AskMeFi questions. My blog. Slashdot comments. DeviantArt posts. They're all set to archive indefinitely, and though the value is low, it's there. It's a bit creepy that someone else could do the same, but I guess that's the price of fame.

Google used to host a firefox plugin that would sync history and bookmarks across computers. I used it to sync between desktop, laptop and the university lab, but it broke with FF 3.0 never to return. Thank you for reminding me to check out Weave, and perhaps set it up this weekend.
posted by pwnguin at 8:31 PM on October 11, 2009 [1 favorite]

Google used to host a firefox plugin that would sync history and bookmarks across computers. I used it to sync between desktop, laptop and the university lab, but it broke with FF 3.0 never to return.

It morphed into some newer solution: http://www.xmarks.com/ - I currently use it at the end of a slightly insane sequence of things to sync my MobileMe Safari bookmarks with my netbook's Firefox, and aside from random parts of the chain stopping on the XP side (what part of "sync daily" means stop after a week? rar grr), it's been working like a champ, just like I used to use the google bookmark sync. I mean, you still have the whole "do I trust J. Random Company with all my bookmarks/passwords/physical characteristics" issue, but you had that with Google, too.
posted by Kyol at 8:50 PM on October 11, 2009

Ooh, thanks for the reminder, Danger. I've recently started using Apple Time Machine, but I haven't yet run through a recovery scenario. Those automatic backups have made me lazy. I really should verify them before I give up on manual backups completely.

(Why yes, my time machine storage is offsite.)
posted by ryanrs at 4:48 AM on October 12, 2009

I mean, you still have the whole "do I trust J. Random Company with all my bookmarks/passwords/physical characteristics" issue, but you had that with Google, too.

XMarks (which is truly great) uses an open protocol. You can set up your own central bookmark server and keep your data off theirs completely.

(That said, I don't bother. The old security vs convenience dilemma.)
posted by rokusan at 12:49 PM on October 12, 2009

Having user data in a 3rd place (user's personal computer) requires the user have one of those. Some people want some smarts on their phone but either don't have a computer or would rather only do webby things with it.
posted by morganw at 3:15 PM on October 12, 2009

Exchange and Outlook are probably Microsoft's most solid products, and on their own merits work really well if you know what the hell you're doing when you set them up properly.

Agreed.

And add SQLServer to that list.
posted by Civil_Disobedient at 3:38 PM on October 12, 2009

Ooh, thanks for the reminder, Danger. I've recently started using Apple Time Machine, but I haven't yet run through a recovery scenario. Those automatic backups have made me lazy. I really should verify them before I give up on manual backups completely.

When I swapped my hard drive out in my MacBook Pro I booted the OS X Install CD, went to "Utilities" and then "Restore System from Backup" and Time Machine went and did its thing. My whole system was back in an hour. I did have to reauthorize a couple of apps (i.e. Airport needed to redo its keychain access authentication token) but in general it's the best backup and restore I've had in ages.
posted by Talez at 4:15 PM on October 12, 2009

My experience mirror's Talez's almost exactly, except it took more than hour. But the old hard drive was FULL, so that's no surprise.
posted by flaterik at 5:38 PM on October 12, 2009

Apple insider has a "secret insider" update. It's got a pretty confused definition of "dogfooding", but it's an interesting read.

Civil_Disobedient: "And add SQLServer to that list."
Which was originally from Sybase. Sure, they rewrote it at version 7, but Sybase gave them a huge headstart for stability and performance.
posted by boo_radley at 6:09 AM on October 13, 2009

"Apple insider has a "secret insider" update. It's got a pretty confused definition of "dogfooding", but it's an interesting read."

That article has so many factually incorrect and disprovable statements that I can't take any of its conjecture seriously. And that's even leaving out the bizarre one-off redefinition of "dogfood," which should otherwise just sink it as being written by an incompetent or confused person anyhow.

As to "too big to back up?"

If you figure 200 million customers (a ridiculous over-estimate) each had 32 kilobytes of address book data (another absurd over-estimate) that's still something you could back up to 8 spindles without compressing it. Export that table to a scratch LUN; snapshot the scratch LUN; export the snapshot to a blockfile. Stick the file on the spindles in a foam-lined suitcase and take them off-site. Rotate every month. Done.

I'll do it for them for one thousandth of the cost of losing the entire company's core data. Hell, I'll throw in the hardware and insurance with the labor.
posted by majick at 9:15 AM on October 15, 2009

Looks like they found a backup.
posted by smackfu at 3:48 PM on October 15, 2009