How Not To Decom A Server Farm
September 11, 2023 11:33 AM Subscribe
In an excerpt from his biography of Elon Musk, Walter Issacson recounts how the mercurial owner of Twitter and his supporters personally went to Sacramento to decommission the company's server farm collocated there. (SLCNBC)
Dollars to donuts Alex from Uzbekistan gets fired for letting some random person who looks like Elon Musk into a client's secured server room. I mean the fact that it apparently WAS Elon Musk doesn't change the fact he socially engineered his way into a secure facility.
posted by muddgirl at 11:40 AM on September 11, 2023 [31 favorites]
posted by muddgirl at 11:40 AM on September 11, 2023 [31 favorites]
that shmuck is fuckin un-bee-lievable
posted by lalochezia at 11:43 AM on September 11, 2023 [10 favorites]
posted by lalochezia at 11:43 AM on September 11, 2023 [10 favorites]
Twitter "had 70,000 hard-coded references to Sacramento. And there's still shit that's broken because of it."
posted by zenon at 11:44 AM on September 11, 2023 [6 favorites]
posted by zenon at 11:44 AM on September 11, 2023 [6 favorites]
Well, that explains all of the instability during that period.
posted by grumpybear69 at 11:44 AM on September 11, 2023 [4 favorites]
posted by grumpybear69 at 11:44 AM on September 11, 2023 [4 favorites]
What's the GDPR rule for "your data is in a semi truck on the interstate in northern california"
posted by dis_integration at 11:46 AM on September 11, 2023 [36 favorites]
posted by dis_integration at 11:46 AM on September 11, 2023 [36 favorites]
Indistinguishable from sabotage.
posted by subdee at 11:49 AM on September 11, 2023 [26 favorites]
posted by subdee at 11:49 AM on September 11, 2023 [26 favorites]
Not mentioned in the article: How many server racks are currently usable AND operational in Portland after this move.
Also also: How many will be usable and operational in a year because the installation was sloppy and half assed.
posted by Slackermagee at 11:53 AM on September 11, 2023 [30 favorites]
Also also: How many will be usable and operational in a year because the installation was sloppy and half assed.
posted by Slackermagee at 11:53 AM on September 11, 2023 [30 favorites]
'maniacal sense of urgency'
That's Isaacson saying that Musk is bipolar without actually saying it, right?
posted by 1970s Antihero at 11:55 AM on September 11, 2023 [23 favorites]
That's Isaacson saying that Musk is bipolar without actually saying it, right?
posted by 1970s Antihero at 11:55 AM on September 11, 2023 [23 favorites]
I don't want to read this because I'm done caring about Musk and I can't respect Isaacson after he sat on the bit about how Musk used Starlink to interfere with Ukraine's operations. And I don't do hate reads.
But I do want to read this because professionally I need to be prepared to deal with people who'd do things like this.
Sigh.
posted by fedward at 11:58 AM on September 11, 2023 [15 favorites]
But I do want to read this because professionally I need to be prepared to deal with people who'd do things like this.
Sigh.
posted by fedward at 11:58 AM on September 11, 2023 [15 favorites]
That's Isaacson saying that Musk is bipolar without actually saying it, right?
From the article:
Musk had the mix of glee and anger that often accompanied one of his manic surges.
I would say that Isaacson does actually say it.
posted by subdee at 11:59 AM on September 11, 2023 [16 favorites]
From the article:
Musk had the mix of glee and anger that often accompanied one of his manic surges.
I would say that Isaacson does actually say it.
posted by subdee at 11:59 AM on September 11, 2023 [16 favorites]
Please don’t insult bipolar people by ascribing that to Elon musk when he’s obviously just an indulgent man-child
posted by armoir from antproof case at 12:00 PM on September 11, 2023 [25 favorites]
posted by armoir from antproof case at 12:00 PM on September 11, 2023 [25 favorites]
Porque no los dos? He can be both.
posted by subdee at 12:00 PM on September 11, 2023 [13 favorites]
posted by subdee at 12:00 PM on September 11, 2023 [13 favorites]
Thinking about how all of my infosec-adjacent peeps moved immediately off of Twitter after Musk took over.
posted by credulous at 12:03 PM on September 11, 2023 [22 favorites]
posted by credulous at 12:03 PM on September 11, 2023 [22 favorites]
Also also: How many will be usable and operational in a year because the installation was sloppy and half assed.
And there-in lies the problem: the failures will be inconsistent and spread out enough, and far enough out, it can’t be obviously tracked back to this move. Hard drives fail? That’s what they do! Power supply died? Well, they are redundant. Blame won’t go to Musk. Either it will be regarded as typical failures, shoddy hardware, or blame some poor soul at the new site. OF COURSE it wasn’t the move.
I ran my then-company’s operations in several data centers. I inisisted that, any time we ship gear, it get professionally packed. I was overruled on more than one occasion—just get some interns and a roll of bubble wrap. The folks turning the screwdrivers at the destination knew the difference, and they knew it was not my call when gear arrived bent (literally).
Musk gets by because a lot of not-smart folks thinks being a disrupter who bulldozes over rules and Best Practice to get stuff done is how progress is made. They don’t realize those rules exist from hard-won experience. That those not-smart folk happen to be on the right probably explains part of his shift in that direction.
posted by MrGuilt at 12:05 PM on September 11, 2023 [46 favorites]
And there-in lies the problem: the failures will be inconsistent and spread out enough, and far enough out, it can’t be obviously tracked back to this move. Hard drives fail? That’s what they do! Power supply died? Well, they are redundant. Blame won’t go to Musk. Either it will be regarded as typical failures, shoddy hardware, or blame some poor soul at the new site. OF COURSE it wasn’t the move.
I ran my then-company’s operations in several data centers. I inisisted that, any time we ship gear, it get professionally packed. I was overruled on more than one occasion—just get some interns and a roll of bubble wrap. The folks turning the screwdrivers at the destination knew the difference, and they knew it was not my call when gear arrived bent (literally).
Musk gets by because a lot of not-smart folks thinks being a disrupter who bulldozes over rules and Best Practice to get stuff done is how progress is made. They don’t realize those rules exist from hard-won experience. That those not-smart folk happen to be on the right probably explains part of his shift in that direction.
posted by MrGuilt at 12:05 PM on September 11, 2023 [46 favorites]
I don't know if poor Alex was or is a True Believer, but I am certain he has seen things in this job that left him with a thousand-yard stare.
Why would you think executives should do something like this? And their bewildered cousins who bought all the supplies at retail? Unless maybe you got your ideas of leadership from Star Trek, where the top officers go on all the adventures instead of sending specialists because they're the main characters of the show ...
posted by Countess Elena at 12:10 PM on September 11, 2023 [9 favorites]
Why would you think executives should do something like this? And their bewildered cousins who bought all the supplies at retail? Unless maybe you got your ideas of leadership from Star Trek, where the top officers go on all the adventures instead of sending specialists because they're the main characters of the show ...
posted by Countess Elena at 12:10 PM on September 11, 2023 [9 favorites]
"They were somewhere over Las Vegas when James made his suggestion that they could move them now. It was the type of impulsive, impractical, surge-into-the-breach idea that Musk loved."
posted by inflatablekiwi at 12:12 PM on September 11, 2023 [25 favorites]
"In retrospect, the whole Sacramento shutdown was a mistake."
Oh, hey. At least you've recognized that you made a--
"I was told we had redundancy across our data centers. What I wasn't told was that we had 70,000 hard-coded references to Sacramento. And there's still shit that's broken because of it."
Oh, somebody else's fault. Right. It's not like people who could have told you were shouted down by you or anything.
posted by brundlefly at 12:15 PM on September 11, 2023 [41 favorites]
Oh, hey. At least you've recognized that you made a--
"I was told we had redundancy across our data centers. What I wasn't told was that we had 70,000 hard-coded references to Sacramento. And there's still shit that's broken because of it."
Oh, somebody else's fault. Right. It's not like people who could have told you were shouted down by you or anything.
posted by brundlefly at 12:15 PM on September 11, 2023 [41 favorites]
That's Isaacson saying that Musk is bipolar without actually saying it, right?
Personally my ADHD self was reading that piece like "there but for the grace of god go I" because that is the exact feeling of wanting a result, being aggravated that there's a bajillion little tasks to get there, explodey head emoji, mull stew turn it over 'cause i can't let it go, BAM I've got a really stupid brilliant idea. Do the slapdash shortcut, accomplish part (but never all!) of the project, and laugh maniacally.
Some time later, be like "yeeeeaaaahhhh I should've just done it properly".
I primarily fuck up my own life tho. Lack of adequate funding.
posted by Baethan at 12:18 PM on September 11, 2023 [19 favorites]
Personally my ADHD self was reading that piece like "there but for the grace of god go I" because that is the exact feeling of wanting a result, being aggravated that there's a bajillion little tasks to get there, explodey head emoji, mull stew turn it over 'cause i can't let it go, BAM I've got a really stupid brilliant idea. Do the slapdash shortcut, accomplish part (but never all!) of the project, and laugh maniacally.
Some time later, be like "yeeeeaaaahhhh I should've just done it properly".
I primarily fuck up my own life tho. Lack of adequate funding.
posted by Baethan at 12:18 PM on September 11, 2023 [19 favorites]
I guess Chaos Monkey needs to get a new feature that makes random bad management decisions, to ensure that engineers implement their services to be resilient to senior executive failures.
posted by FishBike at 12:30 PM on September 11, 2023 [21 favorites]
posted by FishBike at 12:30 PM on September 11, 2023 [21 favorites]
Ah, yes, the vaunted "move fast, break things" ethos.
Except the "move fast" part is illusory.
posted by smcdow at 12:36 PM on September 11, 2023 [5 favorites]
Except the "move fast" part is illusory.
posted by smcdow at 12:36 PM on September 11, 2023 [5 favorites]
I feel like I’ve been on both sides of this kind of meeting. From a Meyers Briggs perspective this sounds like classic ENTJ thinking clashing with people with the I/S/P attributes. Meyers-Briggs is pseudoscientific horseshit is is occasionally useful as a means/framework to mediate and think about issues like this in the workplace. This should have been handled in a way that was less confrontational / drama fueled and more professional.
posted by interogative mood at 12:46 PM on September 11, 2023
posted by interogative mood at 12:46 PM on September 11, 2023
I was reminded of the crack "Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway," and wondered what the bandwidth of a semi full server racks (that should have been—but were not—wiped) trundling down the highway might be.
I was also reminded that Musk is nuts.
posted by adamrice at 1:04 PM on September 11, 2023 [7 favorites]
I was also reminded that Musk is nuts.
posted by adamrice at 1:04 PM on September 11, 2023 [7 favorites]
> I guess Chaos Monkey needs to get a new feature
Adding "Musk mode" to chaos monkey where it fires engineers at random and hires a guy on fiver to hit your servers with a hammer
posted by dis_integration at 1:07 PM on September 11, 2023 [20 favorites]
Adding "Musk mode" to chaos monkey where it fires engineers at random and hires a guy on fiver to hit your servers with a hammer
posted by dis_integration at 1:07 PM on September 11, 2023 [20 favorites]
Is there an english word for "nobody is the hero here"?
Obstructionist IT infrastructure staff are a thing, as well as Dunning-Kruger bowling ball managers.
posted by The Power Nap at 1:07 PM on September 11, 2023
Obstructionist IT infrastructure staff are a thing, as well as Dunning-Kruger bowling ball managers.
posted by The Power Nap at 1:07 PM on September 11, 2023
They are extraordinarily lucky no one was injured (if indeed no one was). Most racks aren't built for high-speed maneuvers over uneven terrain, and they probably did exactly that. I know because I've seen over-eager kids making minimum wage fucking around in data centers. I've had smashed fingers, cuts from cheapass sheet metal, bruises from crawling under floors, etc. just doing normal things, let alone a midnight chicken run with multi-ton equipment. If this had been demanded of me I'd have quit on the spot, because I can all too easily visualize getting squished into paste by a falling rack of equipment.
So yeah, literally moving fast and breaking stuff. Like bad burglars.
posted by dragstroke at 1:07 PM on September 11, 2023 [26 favorites]
So yeah, literally moving fast and breaking stuff. Like bad burglars.
posted by dragstroke at 1:07 PM on September 11, 2023 [26 favorites]
I don't think highlighting a lack of capacity at a datacenter is being obstructionist. The DC has to be able to handle the power density and cooling requirements for 5800(?!!) racks. I don't know what datacenter in the Portland Metro area has that kind of unused capacity but I've been out of the biz for a while.
posted by tmt at 1:11 PM on September 11, 2023 [3 favorites]
posted by tmt at 1:11 PM on September 11, 2023 [3 favorites]
A few days later : Twitter suffered outage after Elon Musk made ‘significant’ backend server architecture changes. The 'backend changes' being this clown show.
posted by plep at 1:12 PM on September 11, 2023 [17 favorites]
posted by plep at 1:12 PM on September 11, 2023 [17 favorites]
Is there an english word for "nobody is the hero here"?
I don't know that I can think of one exactly, but this seems like the perfect situation for the Russian idiom for trying to help and causing a disaster instead: BEAR JOB
posted by BlackLeotardFront at 1:14 PM on September 11, 2023 [15 favorites]
I don't know that I can think of one exactly, but this seems like the perfect situation for the Russian idiom for trying to help and causing a disaster instead: BEAR JOB
posted by BlackLeotardFront at 1:14 PM on September 11, 2023 [15 favorites]
Obstructionist IT infrastructure staff are a thing,
I don't think it's obstructionist to say "we need to do this right."
posted by NoxAeternum at 1:18 PM on September 11, 2023 [31 favorites]
I don't think it's obstructionist to say "we need to do this right."
posted by NoxAeternum at 1:18 PM on September 11, 2023 [31 favorites]
This should have been handled in a way that was less confrontational / drama fueled and more professional.
Sir, this is Twitter.
posted by plep at 1:20 PM on September 11, 2023 [24 favorites]
Sir, this is Twitter.
posted by plep at 1:20 PM on September 11, 2023 [24 favorites]
> Ah, yes, the vaunted "move fast, break things" ethos. Except the "move fast" part is illusory.
mr. musk was least-unfamiliar with how computing works back in the late 1900s, and doesn't realize that the stuff he's dealing with now is vastly more complex, with the consequences for breaking things much more severe.
it's like if someone who used to take a yacht out on weekends became convinced he knows how to captain an aircraft carrier and then bluffed/bought his way into actually commanding an aircraft carrier.
posted by bombastic lowercase pronouncements at 1:21 PM on September 11, 2023 [15 favorites]
mr. musk was least-unfamiliar with how computing works back in the late 1900s, and doesn't realize that the stuff he's dealing with now is vastly more complex, with the consequences for breaking things much more severe.
it's like if someone who used to take a yacht out on weekends became convinced he knows how to captain an aircraft carrier and then bluffed/bought his way into actually commanding an aircraft carrier.
posted by bombastic lowercase pronouncements at 1:21 PM on September 11, 2023 [15 favorites]
"and nothing of value was lost"
posted by rozcakj at 1:22 PM on September 11, 2023 [5 favorites]
posted by rozcakj at 1:22 PM on September 11, 2023 [5 favorites]
Silicon Valley S4E10 seems somehow relevant.
posted by credulous at 1:26 PM on September 11, 2023 [6 favorites]
posted by credulous at 1:26 PM on September 11, 2023 [6 favorites]
mr. musk was least-unfamiliar with how computing works back in the late 1900s, and doesn't realize that the stuff he's dealing with now is vastly more complex, with the consequences for breaking things much more severe.
IMO, it's less so. You used to have to buy really expensive servers for the type of work Twitter is doing, but now, each of those servers on the rack are just regular computers, made up with distributed computing and whatnot. So if you break a few CPUs, it's only a couple of thousand each dollars one vs one server that is millions of dollars.
posted by The_Vegetables at 1:30 PM on September 11, 2023 [6 favorites]
The observation that Isaacson should be making about this episode is that the git'er-done approach is less about moving the servers and shutting down the Sacramento operation, than it is about the feeling of control it gave Musk. It fed a manic high.
There's a middle ground between hacking it all up on Christmas Eve, and waiting six months for it to happen. That's ordering the infra folks to come up with a plan to get it done ASAP, and it's okay if they break things doing it. Give them permission to take big risks, let them know you'll backstop them, and cut them loose. Let them form a war-room, and show up only enough to ask "do you have everything you need?"
The conceit Musk was chasing here is that it could be done fast and cheap, that the correct way of doing it was about laziness and cost-padding, rather than organizational reactions and lessons learned. There's obstructionist IT, and then there's IT that knows that safe practices are about minimizing remote but non-trivial risks, among other things.
Of course, then we get back to Musk, behaviourally self-medicating his narcissism, and realize it doesn't matter.
posted by fatbird at 1:31 PM on September 11, 2023 [29 favorites]
There's a middle ground between hacking it all up on Christmas Eve, and waiting six months for it to happen. That's ordering the infra folks to come up with a plan to get it done ASAP, and it's okay if they break things doing it. Give them permission to take big risks, let them know you'll backstop them, and cut them loose. Let them form a war-room, and show up only enough to ask "do you have everything you need?"
The conceit Musk was chasing here is that it could be done fast and cheap, that the correct way of doing it was about laziness and cost-padding, rather than organizational reactions and lessons learned. There's obstructionist IT, and then there's IT that knows that safe practices are about minimizing remote but non-trivial risks, among other things.
Of course, then we get back to Musk, behaviourally self-medicating his narcissism, and realize it doesn't matter.
posted by fatbird at 1:31 PM on September 11, 2023 [29 favorites]
I have now read the thing, but I wish I hadn't read the thing. What a nightmare.
posted by fedward at 1:39 PM on September 11, 2023 [7 favorites]
posted by fedward at 1:39 PM on September 11, 2023 [7 favorites]
I considered sending this to my cousin, but I think as a non-techie she would just see this as "yeah, they did it, it worked, they were right."
posted by rednikki at 1:57 PM on September 11, 2023
posted by rednikki at 1:57 PM on September 11, 2023
Hardware agnostic is not hardware indifferent.
Virtualization has made some things simpler, others more complex.
Sometimes, needlessly.
And when everything is containerized, people also tend to lose track of what's happening inside the containers.
Of course, so went the argument between source and binary distribution. And I'm not trying to be a Gentoo user.
posted by snuffleupagus at 2:03 PM on September 11, 2023
Virtualization has made some things simpler, others more complex.
Sometimes, needlessly.
And when everything is containerized, people also tend to lose track of what's happening inside the containers.
Of course, so went the argument between source and binary distribution. And I'm not trying to be a Gentoo user.
posted by snuffleupagus at 2:03 PM on September 11, 2023
So yeah, literally moving fast and breaking stuff. Like bad burglars.
Oooh that's a keeper.
posted by whuppy at 2:10 PM on September 11, 2023 [7 favorites]
Oooh that's a keeper.
posted by whuppy at 2:10 PM on September 11, 2023 [7 favorites]
"If you got a godd--- U-Haul, you could probably do it by yourself."
A company I worked for had a client who did exactly this and learned the hard way that when you crash a U-Haul the supplemental insurance doesn't come close to covering the millions of dollars of servers that got wrecked in the accident.
posted by peeedro at 2:33 PM on September 11, 2023 [39 favorites]
A company I worked for had a client who did exactly this and learned the hard way that when you crash a U-Haul the supplemental insurance doesn't come close to covering the millions of dollars of servers that got wrecked in the accident.
posted by peeedro at 2:33 PM on September 11, 2023 [39 favorites]
You know who I’m fucking tired of hearing about?
posted by chasing at 2:39 PM on September 11, 2023 [21 favorites]
posted by chasing at 2:39 PM on September 11, 2023 [21 favorites]
Also, let us not forget that all of this pulling of servers and lashing into rental trucks?
Is happening on fucking Christmas Eve.
posted by NoxAeternum at 2:43 PM on September 11, 2023 [17 favorites]
Is happening on fucking Christmas Eve.
posted by NoxAeternum at 2:43 PM on September 11, 2023 [17 favorites]
I hate hate hate the word “scrappy” which makes an appearance in the article a few times. I worked for an ad agency that constantly talked about how scrappy they were which was code for making your employees work far too much. The thing is that this company was at this point the second largest in town. It was no longer “scrappy” just a sweat shop that churned through employees. Other hated term - “rock star”. You know who’s a rock star? A freaking rock star, stop saying that about someone doing their job. Rant done.
posted by misterpatrick at 2:51 PM on September 11, 2023 [21 favorites]
posted by misterpatrick at 2:51 PM on September 11, 2023 [21 favorites]
This stunt certainly ruined a lot of holidays for Twitter staff -- the Musk goons only moved a *fraction* of the servers to the new location, and this is certainly not a failure mode the system architects had in mind. "Hard-coded references to Sacramento" probably means there are clusters that expect their nodes not to be on a local network and not suddenly hundreds of miles away.
posted by credulous at 2:57 PM on September 11, 2023 [18 favorites]
posted by credulous at 2:57 PM on September 11, 2023 [18 favorites]
The thing that really boggles my mind is that the data centre owners spent (probably) millions on securing the facility against any and all possible incursions but, in the end, every single measure was subverted because someone was talked into letting a total stranger walk into the secure facility and start unplugging things. At every level, people are always the weakest link and can't be trusted.
posted by dg at 3:13 PM on September 11, 2023 [36 favorites]
posted by dg at 3:13 PM on September 11, 2023 [36 favorites]
>
What the absolute previous is Isaacson on about here? Is this a colonic irrigation thing?
posted by scruss at 3:16 PM on September 11, 2023 [5 favorites]
… launch fire-drill surges that extruded all obstacles
What the absolute previous is Isaacson on about here? Is this a colonic irrigation thing?
posted by scruss at 3:16 PM on September 11, 2023 [5 favorites]
The thing that really boggles my mind is that the data centre owners spent (probably) millions on securing the facility against any and all possible incursions but, in the end, every single measure was subverted because someone was talked into letting a total stranger walk into the secure facility and start unplugging things.
The person who let them in was a Twitter employee with access to the corporate vault, so it wasn't a breach on the end of the datacenter. And when the datacenter owners found out, they did try to stop things and protect their facility - but it turns out that a manic manchild with more wealth than some sovereign nations does not react well to "no".
posted by NoxAeternum at 3:20 PM on September 11, 2023 [9 favorites]
The person who let them in was a Twitter employee with access to the corporate vault, so it wasn't a breach on the end of the datacenter. And when the datacenter owners found out, they did try to stop things and protect their facility - but it turns out that a manic manchild with more wealth than some sovereign nations does not react well to "no".
posted by NoxAeternum at 3:20 PM on September 11, 2023 [9 favorites]
I wouldn't be surprised if Elon-vulnerable facilities are updating their security protocols against Elon-shaped individuals, including the eponymous one.
posted by credulous at 3:22 PM on September 11, 2023 [13 favorites]
posted by credulous at 3:22 PM on September 11, 2023 [13 favorites]
I'm waiting for part 2, where Elon goes to Best Buy to purchase power cords and portable AC units to prove he can fire it all up in the new data center and it will all "just work."
posted by Chuffy at 3:25 PM on September 11, 2023 [6 favorites]
posted by Chuffy at 3:25 PM on September 11, 2023 [6 favorites]
I don't know what datacenter in the Portland Metro area has that kind of unused capacity
tmt I'm guessing they're lumping Hillsboro into Portland. There are a few data centers out there, including an NTT one.
posted by shenkerism at 3:28 PM on September 11, 2023
tmt I'm guessing they're lumping Hillsboro into Portland. There are a few data centers out there, including an NTT one.
posted by shenkerism at 3:28 PM on September 11, 2023
Chuffy: I'm waiting for part 2, where Elon goes to Best Buy to purchase power cords and portable AC units to prove he can fire it all up in the new data center and it will all "just work."
Did you and I work at the same company?
Anyway, the portable AC units were a step up from one of the company owners - a self-proclaimed expert in all things AC - feeling heroic for coming up with the idea of carrying buckets of ice up the stairs to cool the servers.
He apparently didn't know that "1 ton" of air conditioning is the amount of heat you remove by melting 1 ton of ice every 24 hours, despite his expert status. Or maybe he was just aiming for the world's most Sisyphean air conditioner.
posted by clawsoon at 3:39 PM on September 11, 2023 [12 favorites]
Did you and I work at the same company?
Anyway, the portable AC units were a step up from one of the company owners - a self-proclaimed expert in all things AC - feeling heroic for coming up with the idea of carrying buckets of ice up the stairs to cool the servers.
He apparently didn't know that "1 ton" of air conditioning is the amount of heat you remove by melting 1 ton of ice every 24 hours, despite his expert status. Or maybe he was just aiming for the world's most Sisyphean air conditioner.
posted by clawsoon at 3:39 PM on September 11, 2023 [12 favorites]
Elon-shaped individuals
You know, maybe that would explain a lot about how Twitter’s been run
posted by Countess Elena at 3:55 PM on September 11, 2023 [5 favorites]
You know, maybe that would explain a lot about how Twitter’s been run
posted by Countess Elena at 3:55 PM on September 11, 2023 [5 favorites]
...though now that I think more about it, the Twitter admins probably said "good job Elon you were right we were wrong" and just left the relocated servers unprovisoned, then just reconfigured the old data center until they could do a controlled move the next month. Thus the lack of resources for peak traffic events like Deathsantis announcements.
posted by credulous at 3:56 PM on September 11, 2023 [5 favorites]
posted by credulous at 3:56 PM on September 11, 2023 [5 favorites]
This is like all those home renovation shows where they start by smashing the bathroom with sledgehammers.
I worry that CEOs will read it and be inspired by it.
posted by clawsoon at 4:35 PM on September 11, 2023 [9 favorites]
I worry that CEOs will read it and be inspired by it.
posted by clawsoon at 4:35 PM on September 11, 2023 [9 favorites]
Oh my god. That's so much more unhinged than I expected. And I expected a lot.
posted by rmd1023 at 4:56 PM on September 11, 2023 [4 favorites]
posted by rmd1023 at 4:56 PM on September 11, 2023 [4 favorites]
From the article:
posted by moonbiter at 6:03 PM on September 11, 2023 [13 favorites]
That said, X survived.Did it, though? Did it really?
posted by moonbiter at 6:03 PM on September 11, 2023 [13 favorites]
"Christ, what an asshole."
posted by AsYouKnow Bob at 7:22 PM on September 11, 2023 [12 favorites]
posted by AsYouKnow Bob at 7:22 PM on September 11, 2023 [12 favorites]
I do not care for this Elon Musk fellow.
posted by Multicellular Exothermic at 8:03 PM on September 11, 2023 [6 favorites]
posted by Multicellular Exothermic at 8:03 PM on September 11, 2023 [6 favorites]
and how much money has he lost so far?
posted by AlbertCalavicci at 9:34 PM on September 11, 2023
posted by AlbertCalavicci at 9:34 PM on September 11, 2023
not enough
posted by ryanrs at 9:39 PM on September 11, 2023 [7 favorites]
posted by ryanrs at 9:39 PM on September 11, 2023 [7 favorites]
"Never underestimate the bandwidth of a…”
Chevy Suburban (4.1 cubic meters) filled with MicroSD cards (1.5TB per 165 cubic mm, about 0.25g each). While it might be possible to fit about 24,000,000 of those cards in that all-American monstrosity, the maximum payload of 1612 pounds is about 3,000,000 cards. 4.5 exabytes at 120kph is 1.5 EB/s.
posted by Revvy at 9:44 PM on September 11, 2023 [5 favorites]
Chevy Suburban (4.1 cubic meters) filled with MicroSD cards (1.5TB per 165 cubic mm, about 0.25g each). While it might be possible to fit about 24,000,000 of those cards in that all-American monstrosity, the maximum payload of 1612 pounds is about 3,000,000 cards. 4.5 exabytes at 120kph is 1.5 EB/s.
posted by Revvy at 9:44 PM on September 11, 2023 [5 favorites]
... 3,000,000 cards ...
Are there that many? If so, it might be hard to gather them. How long would it take to produce that many of them and ship them to you?
posted by Galaxor Nebulon at 4:13 AM on September 12, 2023 [1 favorite]
Are there that many? If so, it might be hard to gather them. How long would it take to produce that many of them and ship them to you?
posted by Galaxor Nebulon at 4:13 AM on September 12, 2023 [1 favorite]
4.5 exabytes at 120kph is 1.5 EB/s.
Seek time and latency is goddamn awful though.
posted by Absolutely No You-Know-What at 4:47 AM on September 12, 2023 [2 favorites]
Seek time and latency is goddamn awful though.
posted by Absolutely No You-Know-What at 4:47 AM on September 12, 2023 [2 favorites]
Seek time and latency is goddamn awful though.
just put a redis node in the station wagon in front, it'll be fine
posted by clawsoon at 6:19 AM on September 12, 2023 [7 favorites]
just put a redis node in the station wagon in front, it'll be fine
posted by clawsoon at 6:19 AM on September 12, 2023 [7 favorites]
> I hate hate hate the word “scrappy”
I always try to respond with “scrappy-with-a-capital-C.”
posted by jimfl at 8:10 AM on September 12, 2023 [6 favorites]
I always try to respond with “scrappy-with-a-capital-C.”
posted by jimfl at 8:10 AM on September 12, 2023 [6 favorites]
wondered what the bandwidth of a semi full server racks (that should have been—but were not—wiped) trundling down the highway might be
AWS Snowmobile (literally a container full of drives) claims "up to 100 petabytes in as little as a few weeks." If "a few" means no less than 3, then the maximum throughput is 33 PB/week or approximately 1.65 gigabytes per second.
posted by fedward at 9:32 AM on September 12, 2023 [1 favorite]
AWS Snowmobile (literally a container full of drives) claims "up to 100 petabytes in as little as a few weeks." If "a few" means no less than 3, then the maximum throughput is 33 PB/week or approximately 1.65 gigabytes per second.
posted by fedward at 9:32 AM on September 12, 2023 [1 favorite]
Besides the whole "christ what an asshole" response to any description of Musk, the repeated use of "Alex the Uzbek" is really weird in this article.
posted by signal at 9:36 AM on September 12, 2023 [7 favorites]
posted by signal at 9:36 AM on September 12, 2023 [7 favorites]
the repeated use of "Alex the Uzbek" is really weird in this article.
Belittling and possibly borderline racist, I'd say.
posted by plep at 9:45 AM on September 12, 2023 [9 favorites]
Belittling and possibly borderline racist, I'd say.
posted by plep at 9:45 AM on September 12, 2023 [9 favorites]
And it appears that Mr. Musk's Wild Colo Decom may have violated Twitter's consent decree:
Multiple employees testified that Musk gave directives that were at odds with the company’s normal processes and policies. In December 2022, Musk directed that company servers be moved from one data center to another. Company policy was to wipe data before removing servers from a center, but the relocated servers were transferred without being wiped because employees did not have “enough time to put together a process that would be in compliance with [their] own policies,” according to the testimony.posted by NoxAeternum at 12:25 PM on September 12, 2023 [3 favorites]
Here's John Siracusa on his Hypercritical podcast in 2011 discussing the failings in Issacson's biography of Steve Jobs.
posted by neuron at 12:27 PM on September 12, 2023 [1 favorite]
posted by neuron at 12:27 PM on September 12, 2023 [1 favorite]
Company policy was to wipe data before removing servers from a center
Hoo boy, not wiping PII is illegal as shit.
posted by Abehammerb Lincoln at 2:51 PM on September 12, 2023 [2 favorites]
Hoo boy, not wiping PII is illegal as shit.
posted by Abehammerb Lincoln at 2:51 PM on September 12, 2023 [2 favorites]
Hoo boy, not wiping PII is illegal as shit.
Especially in California.
posted by Chuffy at 5:17 PM on September 12, 2023 [2 favorites]
Especially in California.
posted by Chuffy at 5:17 PM on September 12, 2023 [2 favorites]
sniff sniff...
I bet there's a class action lawsuit in here. I have some time, maybe I should figure out how to sue Xitter/Musk for violating the CCPA...
posted by Chuffy at 5:33 PM on September 12, 2023 [1 favorite]
I bet there's a class action lawsuit in here. I have some time, maybe I should figure out how to sue Xitter/Musk for violating the CCPA...
posted by Chuffy at 5:33 PM on September 12, 2023 [1 favorite]
So any ex twitter datacenter people on metafilter
Asking for a friend
posted by Skrubly at 9:39 PM on September 12, 2023
Asking for a friend
posted by Skrubly at 9:39 PM on September 12, 2023
Mike Masnick reacts: The Batshit Crazy Story Of The Day Elon Musk Decided To Personally Rip Servers Out Of A Sacramento Data Center
posted by fedward at 7:04 AM on September 13, 2023 [2 favorites]
Note the pattern: a willingness to ignore the details of what could go wrong, YOLO it and just test it out, and the assumption that if nothing goes wrong when you do that, it means that everything is fine and nothing else could possibly go wrong.He would have ridden in one before?
I might never even ride in a Tesla after this.
posted by fedward at 7:04 AM on September 13, 2023 [2 favorites]
after he sat on the bit about how Musk used Starlink to interfere with Ukraine's operations.
Did the author walk this accusation back after Musk complained it was not true?
posted by 922257033c4a0f3cecdbd819a46d626999d1af4a at 7:36 AM on September 13, 2023
Did the author walk this accusation back after Musk complained it was not true?
posted by 922257033c4a0f3cecdbd819a46d626999d1af4a at 7:36 AM on September 13, 2023
It's seems rich for Musk to complain that the Crimea request came in the middle of the night, and that he would need to wait for authorization from the highest levels to act.
posted by muddgirl at 9:47 AM on September 13, 2023
posted by muddgirl at 9:47 AM on September 13, 2023
Anyway, nothing is ever his fault. Great optics for a CEO but his fanbase doesn't care.
posted by muddgirl at 9:53 AM on September 13, 2023
posted by muddgirl at 9:53 AM on September 13, 2023
The fact that Elon has changed his story to explain the Starlink / Crimea issue suggests that at least he’s finally realized that he fucked up and has shifted into “escape consequences” mode typical of children.
posted by interogative mood at 4:37 PM on September 13, 2023 [7 favorites]
posted by interogative mood at 4:37 PM on September 13, 2023 [7 favorites]
Walter Isaacson’s insight-free doorstop makes at least one thing clear: the richest man in the world has a lot of growing up to do.
posted by scruss at 7:52 PM on September 13, 2023
posted by scruss at 7:52 PM on September 13, 2023
...at least he’s finally realized that he fucked up and has shifted into “escape consequences” mode typical of children.
posted by interogative mood
Something he tried to do just before facing the consequences of purchasing Twitter in the first place. Add his problems at Tesla and Hyperloop to the mix, and it's apparent that the only thing he has going for him is his billionaire status, which allows quite a bit of "escape consequences" behavior.
posted by Chuffy at 1:07 PM on September 14, 2023
posted by interogative mood
Something he tried to do just before facing the consequences of purchasing Twitter in the first place. Add his problems at Tesla and Hyperloop to the mix, and it's apparent that the only thing he has going for him is his billionaire status, which allows quite a bit of "escape consequences" behavior.
posted by Chuffy at 1:07 PM on September 14, 2023
Hyperloop had only one purpose which it did to a T - to throttle high speed rail in California.
Also, apparently the book details such brilliant Elongineering insights such as using a car wash pump for fuel pumps on a rocket engine.
posted by NoxAeternum at 10:14 AM on September 15, 2023
Also, apparently the book details such brilliant Elongineering insights such as using a car wash pump for fuel pumps on a rocket engine.
posted by NoxAeternum at 10:14 AM on September 15, 2023
Speaking of Elongineering, in his recent meeting with the Israeli prime minister (a move meant to deflect the growing cries of antisemitism with regards to both Twitter and Musk), the man has floated the idea of charging for using Twitter:
posted by NoxAeternum at 11:29 AM on September 18, 2023 [2 favorites]
Elon Musk revisiting an idea apparently floated privately in the past -- charging *everyone* to use Twitter. A lower tier than premium. "We’ve moving to a small monthly payment for use of the X system," he just told Benjamin Netanyahu, saying it's only way to stamp out bots.He truly and fundamentally does not understand this business he overpaid for.
posted by NoxAeternum at 11:29 AM on September 18, 2023 [2 favorites]
he also doesn’t understand bots. bots are there to make money for people. the only spam that will kill is the spam that doesn’t earn enough money to cover the cost of posting. already you see tons of blue check 8$/mo bots posting crypto scam shit everywhere.
posted by dis_integration at 11:34 AM on September 18, 2023
posted by dis_integration at 11:34 AM on September 18, 2023
Elon had employees make a sizzle video as part of a campaign to bring advertisers back to the platform. The video (warning links to x) shows the experience of a real person “Claire” and opens with a scroll down their actual “For You” page. They didn’t spend a lot of time vetting the tweets grabbed as they scrolled down the timeline. One post is a user complaining about how the timeline is dying with mediocre engagement and low effort reposts and dating advice, another is a retweet of Elon Musk’s threat to sue the ADL for 22 billion calling him out for blaming the sites problems on “the Jews”.
posted by interogative mood at 9:31 AM on September 24, 2023 [2 favorites]
posted by interogative mood at 9:31 AM on September 24, 2023 [2 favorites]
Watch Linda Yaccarino’s wild interview at the Code Conference
posted by jenfullmoon at 1:13 PM on September 29, 2023 [1 favorite]
posted by jenfullmoon at 1:13 PM on September 29, 2023 [1 favorite]
Elon Musk Hit With Lawsuit For Falsely Accusing Man Of Being Part Of Neo-Nazi Brawl
Brody is being represented by attorney Mark Bankston of the Texas law firm Farrar & Ball. Bankston previously represented two Sandy Hook parents who won $45 million in damages against conspiracy theorist Alex Jones
A thread by Mark explaining the lawsuit.
posted by LostInUbe at 2:17 PM on October 2, 2023
Brody is being represented by attorney Mark Bankston of the Texas law firm Farrar & Ball. Bankston previously represented two Sandy Hook parents who won $45 million in damages against conspiracy theorist Alex Jones
A thread by Mark explaining the lawsuit.
posted by LostInUbe at 2:17 PM on October 2, 2023
« Older Botch | "The young'uns call it country/The Yankees call it... Newer »
This thread has been archived and is closed to new comments
posted by NoxAeternum at 11:38 AM on September 11, 2023 [30 favorites]