Facebook is down
October 4, 2021 2:40 PM Subscribe
Facebook is down, along with Whatsapp and Instagram. Post your conspiracy theories here.
A now-deleted redditor, presumably a FB employee, is quoted here as saying
Here's someone on Twitter explaining BGP (Border Gateway Protocol).
A now-deleted redditor, presumably a FB employee, is quoted here as saying
As many of you know, DNS for FB services has been affected and this is likely a symptom of the actual issue, and that's that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC). There are people now trying to gain access to the peering routers to implement fixes, but the people with physical access is separate from the people with knowledge of how to actually authenticate to the systems and people who know what to actually do, so there is now a logistical challenge with getting all that knowledge unified. Part of this is also due to lower staffing in data centers due to pandemic measures.Apparently people can't gain physical access because their security cards aren't working!
Here's someone on Twitter explaining BGP (Border Gateway Protocol).
The soundtrack of today.
posted by Your Childhood Pet Rock at 2:44 PM on October 4, 2021 [14 favorites]
posted by Your Childhood Pet Rock at 2:44 PM on October 4, 2021 [14 favorites]
It's starting to come back up for me. I can get my feed to load, and some individual posts – but there are a lot of broken images, timeouts, things that just don't work, etc.
posted by escape from the potato planet at 2:45 PM on October 4, 2021
posted by escape from the potato planet at 2:45 PM on October 4, 2021
WaPo: "Facebook’s internal communication platform, Workplace, went down altogether, said a person familiar with the matter who spoke on condition of anonymity because they weren’t authorized to speak publicly. As employees turned to third-party tools such as Slack, many found themselves locked out of even those, because Facebook’s mechanism for signing on to them was not working."
posted by obloquy at 2:45 PM on October 4, 2021 [4 favorites]
posted by obloquy at 2:45 PM on October 4, 2021 [4 favorites]
another external explanation of the outage symptoms from cloudflare blog
posted by are-coral-made at 2:45 PM on October 4, 2021 [6 favorites]
posted by are-coral-made at 2:45 PM on October 4, 2021 [6 favorites]
I just don't want it to come back. This is going to be just like the day we found out that guy had Covid but didn't die.
posted by bleep at 2:47 PM on October 4, 2021 [88 favorites]
posted by bleep at 2:47 PM on October 4, 2021 [88 favorites]
and the beast is back. *sigh*
1540-2140 - 6 hours. Ooof.
All I can think of is the endless hours of postmortems that way too many people are going to be trapped in. I've done 3 day sessions before for outages not nearly on this scale.
posted by drewbage1847 at 2:49 PM on October 4, 2021 [10 favorites]
1540-2140 - 6 hours. Ooof.
All I can think of is the endless hours of postmortems that way too many people are going to be trapped in. I've done 3 day sessions before for outages not nearly on this scale.
posted by drewbage1847 at 2:49 PM on October 4, 2021 [10 favorites]
It’s back, dammit.
posted by leotrotsky at 2:53 PM on October 4, 2021 [5 favorites]
posted by leotrotsky at 2:53 PM on October 4, 2021 [5 favorites]
At least all the Oculus Rift users who were in VR when it went down will be able to get out before they starve.
posted by joeyh at 2:53 PM on October 4, 2021 [66 favorites]
posted by joeyh at 2:53 PM on October 4, 2021 [66 favorites]
Is BGP the kind of resource that Randall Munroe described in this strip?
posted by Countess Elena at 2:54 PM on October 4, 2021 [6 favorites]
posted by Countess Elena at 2:54 PM on October 4, 2021 [6 favorites]
I hate Facebook, but I feel kinda sick for the poor sucker who pressed ENTER on those BGP updates.
posted by aramaic at 2:55 PM on October 4, 2021 [21 favorites]
posted by aramaic at 2:55 PM on October 4, 2021 [21 favorites]
I have it on good authority that Facebook was hacked by X Æ A-12 with possible help from John Travolta, for Maersk. Did you notice the date? 10-4, that's right.
Do your own research, man. Follow the money.
posted by swift at 2:55 PM on October 4, 2021 [16 favorites]
Do your own research, man. Follow the money.
posted by swift at 2:55 PM on October 4, 2021 [16 favorites]
The revolution will not be Finsta.
You will not be able to stay home, brother
You will not be able to like, share, or subscribe
You will not be able to lose yourself in memes and
Skip out for beer during shitposts
Because the revolution will not be finsta
posted by interogative mood at 2:57 PM on October 4, 2021 [23 favorites]
You will not be able to stay home, brother
You will not be able to like, share, or subscribe
You will not be able to lose yourself in memes and
Skip out for beer during shitposts
Because the revolution will not be finsta
posted by interogative mood at 2:57 PM on October 4, 2021 [23 favorites]
BGP: Better Get PostingYourResume
posted by armoir from antproof case at 3:01 PM on October 4, 2021 [9 favorites]
posted by armoir from antproof case at 3:01 PM on October 4, 2021 [9 favorites]
Facebook is back here, but it's not working right.
Okay, who buried Facebook in the pet sematary? You gotta stop doing that!
posted by betweenthebars at 3:02 PM on October 4, 2021 [18 favorites]
Okay, who buried Facebook in the pet sematary? You gotta stop doing that!
posted by betweenthebars at 3:02 PM on October 4, 2021 [18 favorites]
Thus spake a friend with his first post after it picked back up:
"Okay! Who else got their elderly relatives vaccinated while this place was down?"
posted by EmpressCallipygos at 3:08 PM on October 4, 2021 [75 favorites]
"Okay! Who else got their elderly relatives vaccinated while this place was down?"
posted by EmpressCallipygos at 3:08 PM on October 4, 2021 [75 favorites]
OK, I’ll say it. The timing is suspicious. It’s a nice little reminder of how dependent society is on Facebook just when they’re getting congressional interest in them again.
posted by Tell Me No Lies at 3:09 PM on October 4, 2021 [10 favorites]
posted by Tell Me No Lies at 3:09 PM on October 4, 2021 [10 favorites]
I hear they took it down because Stan from Peoria was about to post his conclusions about what really happened in the 2020 elections.
posted by biogeo at 3:09 PM on October 4, 2021 [10 favorites]
posted by biogeo at 3:09 PM on October 4, 2021 [10 favorites]
Is this where I check in to let you all know I’m safe?
posted by Capt. Renault at 3:10 PM on October 4, 2021 [19 favorites]
posted by Capt. Renault at 3:10 PM on October 4, 2021 [19 favorites]
One comment about the outage that made me laugh out loud: "quickly, get your elderly family vaccinated"
posted by They sucked his brains out! at 2:43 PM on October 4
Same joke made in the other thread, was he reading that? I wonder how many people are making the same jokes. And how fast they are trying to make them to be "first!" back on the platform.
posted by tiny frying pan at 3:10 PM on October 4, 2021 [1 favorite]
posted by They sucked his brains out! at 2:43 PM on October 4
Same joke made in the other thread, was he reading that? I wonder how many people are making the same jokes. And how fast they are trying to make them to be "first!" back on the platform.
posted by tiny frying pan at 3:10 PM on October 4, 2021 [1 favorite]
I can't log into FB, as I always use a @facebook.com login because I probably gave them a bogus phone number over a decade ago. FWIW, my password is 'Facebooksucks'.
posted by Catblack at 3:14 PM on October 4, 2021 [1 favorite]
posted by Catblack at 3:14 PM on October 4, 2021 [1 favorite]
Is BGP the kind of resource that Randall Munroe described in this strip?
BGP is actually a remarkable piece of technology that doesn’t require much upkeep. However, it is an extremely powerful tool and you can cause widespread chaos with a few wrong keystrokes.
posted by Tell Me No Lies at 3:15 PM on October 4, 2021 [3 favorites]
BGP is actually a remarkable piece of technology that doesn’t require much upkeep. However, it is an extremely powerful tool and you can cause widespread chaos with a few wrong keystrokes.
posted by Tell Me No Lies at 3:15 PM on October 4, 2021 [3 favorites]
Is BGP the kind of resource that Randall Munroe described in this strip?
Not exactly, I'd describe it more like a pillar shaped like this. Imagine each one of those castellers wearing a jersey of the major net player they represent: Google, IBM, Giant Tech University, the UN, AT&T, etc.
Not really any less fragile, but with a ton of invested manual support to keep it propped up and not just one overlooked hobbyist.
posted by traveler_ at 3:15 PM on October 4, 2021 [1 favorite]
Not exactly, I'd describe it more like a pillar shaped like this. Imagine each one of those castellers wearing a jersey of the major net player they represent: Google, IBM, Giant Tech University, the UN, AT&T, etc.
Not really any less fragile, but with a ton of invested manual support to keep it propped up and not just one overlooked hobbyist.
posted by traveler_ at 3:15 PM on October 4, 2021 [1 favorite]
Aside from the usual encouragement to delete/abandon if you can: might be a good idea to reset your passwords when you can.
posted by scaryblackdeath at 3:19 PM on October 4, 2021 [3 favorites]
posted by scaryblackdeath at 3:19 PM on October 4, 2021 [3 favorites]
I have nulled
the routes
that were in
the BGP configuration
and which
you were probably
relying upon
for everything
Forgive me
they were borkable
so fragile
and so incomprehensible
posted by Joe in Australia at 3:22 PM on October 4, 2021 [105 favorites]
This whole thing has really drawn the obnoxious "I don't care because I'm not on [social media platform]" takes out of the woodwork, while people in some countries have had their entire lifeline to their friends and families disappear without warning, and small communities have faced the possibility that important connections and support networks might be gone for good. Point and laugh all you want, but there are some functions of FB and its monopoly that have (unfortunately) become vital lifelines for many.
I'm taking it as a reminder to backup important photos and take screenshots of important messages and posts (mostly from loved ones who are no longer around). And maybe spruce up my old Dreamwidth account.
posted by fight or flight at 3:26 PM on October 4, 2021 [34 favorites]
I'm taking it as a reminder to backup important photos and take screenshots of important messages and posts (mostly from loved ones who are no longer around). And maybe spruce up my old Dreamwidth account.
posted by fight or flight at 3:26 PM on October 4, 2021 [34 favorites]
might be a good idea to reset your passwords when you can.
I'm having a DEVIL of a time getting onto Instagram, actually - I have two-factor authentication set up, but I've tried repeatedly now to have them send a code to my cell phone on record and I never get it.
posted by EmpressCallipygos at 3:26 PM on October 4, 2021 [1 favorite]
I'm having a DEVIL of a time getting onto Instagram, actually - I have two-factor authentication set up, but I've tried repeatedly now to have them send a code to my cell phone on record and I never get it.
posted by EmpressCallipygos at 3:26 PM on October 4, 2021 [1 favorite]
Do your own research, man.
Now how the hell am I supposed to do that with FB down
posted by mcrandello at 3:27 PM on October 4, 2021 [31 favorites]
Now how the hell am I supposed to do that with FB down
posted by mcrandello at 3:27 PM on October 4, 2021 [31 favorites]
Another blast from the past:
The Website is Down #1 -
"Did you not get my email saying Do Not Reboot the Web server?"
posted by bartleby at 3:27 PM on October 4, 2021 [5 favorites]
The Website is Down #1 -
"Did you not get my email saying Do Not Reboot the Web server?"
posted by bartleby at 3:27 PM on October 4, 2021 [5 favorites]
If you want to download a copy of your information from Facebook, you can use the Download Your Information tool.
posted by Monochrome at 3:28 PM on October 4, 2021 [8 favorites]
posted by Monochrome at 3:28 PM on October 4, 2021 [8 favorites]
Is BGP the kind of resource that Randall Munroe described in this strip?
No, it's not a critical-yet-barely-supported piece of infrastructure. Rather, it's a critical-but-kinda-imperfect protocol that literally organizes the connectivity of the entire internet at the highest level. BGP itself didn't break. It sounds like Facebook used BGP to tell the rest of the internet that Facebook's networks weren't there any more. Oops. And because BGP is kinda imperfect, the rest of the internet was just like, "Okay, cool. Laters."
Big networks and organizations "advertise" information about their networks to the other big networks and organizations using BGP. Everyone then builds a map of where things are on the internet and how to reach them using this information. But also everyone just trusts what they hear. So if Facebook's network says something dumb like "I'm invisible! You can't see me! 👻" The internet just plays along and says "Ahh! Who said that?! It couldn't have been Facebook. Facebook does not exist."
This leads to other fun things like the time Pakistan told the internet, "Hey, Youtube is over here, folks!" The internet said, "Yup, cool, all your videos are in Pakistan, folks." Pakistan got a bunch of traffic meant for Youtube and just dropped it, and Youtube broke. In that instance, Youtube had to show up and say "Nope nope, not true, we're still over here" but in a specific way that overrode what Pakistan was saying, while other parts of the internet said "Pakistan is full of it. Don't listen to them." But the problem is that everyone just trusted what Pakistan said at first, because that's how the system works.
A whole lot of the internet is made out of protocols built on trust. We've moved into an era where that trust is a baaad assumption. We've been here a long time, actually. But lots of those protocols are still running things. Some have had bits and bobs of security and authentication hacked on (like SMTP, which is responsible for sending and receiving emails), but others are still just blithely going along without a care in the world.
posted by whatnotever at 3:30 PM on October 4, 2021 [79 favorites]
No, it's not a critical-yet-barely-supported piece of infrastructure. Rather, it's a critical-but-kinda-imperfect protocol that literally organizes the connectivity of the entire internet at the highest level. BGP itself didn't break. It sounds like Facebook used BGP to tell the rest of the internet that Facebook's networks weren't there any more. Oops. And because BGP is kinda imperfect, the rest of the internet was just like, "Okay, cool. Laters."
Big networks and organizations "advertise" information about their networks to the other big networks and organizations using BGP. Everyone then builds a map of where things are on the internet and how to reach them using this information. But also everyone just trusts what they hear. So if Facebook's network says something dumb like "I'm invisible! You can't see me! 👻" The internet just plays along and says "Ahh! Who said that?! It couldn't have been Facebook. Facebook does not exist."
This leads to other fun things like the time Pakistan told the internet, "Hey, Youtube is over here, folks!" The internet said, "Yup, cool, all your videos are in Pakistan, folks." Pakistan got a bunch of traffic meant for Youtube and just dropped it, and Youtube broke. In that instance, Youtube had to show up and say "Nope nope, not true, we're still over here" but in a specific way that overrode what Pakistan was saying, while other parts of the internet said "Pakistan is full of it. Don't listen to them." But the problem is that everyone just trusted what Pakistan said at first, because that's how the system works.
A whole lot of the internet is made out of protocols built on trust. We've moved into an era where that trust is a baaad assumption. We've been here a long time, actually. But lots of those protocols are still running things. Some have had bits and bobs of security and authentication hacked on (like SMTP, which is responsible for sending and receiving emails), but others are still just blithely going along without a care in the world.
posted by whatnotever at 3:30 PM on October 4, 2021 [79 favorites]
The revolution will not be Finsta.
“finsta”, other than a stupid word, is a way of equating anonymity with fraud, and should be resisted with great vigour.
posted by acb at 3:32 PM on October 4, 2021 [3 favorites]
“finsta”, other than a stupid word, is a way of equating anonymity with fraud, and should be resisted with great vigour.
posted by acb at 3:32 PM on October 4, 2021 [3 favorites]
catherine yronwode on Facebook: "Looks like under cover of Mercury Retrograde someone scrubbed their servers in anticipation of someone's testimony to Congress"
posted by Serene Empress Dork at 3:35 PM on October 4, 2021 [17 favorites]
posted by Serene Empress Dork at 3:35 PM on October 4, 2021 [17 favorites]
Still can't log in, and the 'why are you seeing this' link to their FAQ is 404.
posted by Catblack at 3:35 PM on October 4, 2021
posted by Catblack at 3:35 PM on October 4, 2021
fight or flight: "while people in some countries have had their entire lifeline to their friends and families disappear without warning, and small communities have faced the possibility that important connections and support networks might be gone for good. "
Well yes, but the fact that anybody relies on Facebook for anything more important than LOLing at the latest dank memes is completely bonkers. As in, this should be the wake up call for all those communities and countries to actually get some sort of usable internet infrastructure in place that doesn't depend on a company that views its users as an inferior grade of livestock.
posted by signal at 3:36 PM on October 4, 2021 [8 favorites]
Well yes, but the fact that anybody relies on Facebook for anything more important than LOLing at the latest dank memes is completely bonkers. As in, this should be the wake up call for all those communities and countries to actually get some sort of usable internet infrastructure in place that doesn't depend on a company that views its users as an inferior grade of livestock.
posted by signal at 3:36 PM on October 4, 2021 [8 favorites]
As in, this should be the wake up call for all those communities and countries to actually get some sort of usable internet infrastructure in place that doesn't depend on a company that views its users as an inferior grade of livestock.
I'm guessing that the same could be said about many of the companies that are responsible for supplying the food I eat.
posted by clawsoon at 3:42 PM on October 4, 2021 [15 favorites]
I'm guessing that the same could be said about many of the companies that are responsible for supplying the food I eat.
posted by clawsoon at 3:42 PM on October 4, 2021 [15 favorites]
As in, this should be the wake up call for all those communities and countries to actually get some sort of usable internet infrastructure
Speaking from the third world perspective I don’t believe that’s a reasonable ask.
posted by Tell Me No Lies at 3:43 PM on October 4, 2021 [36 favorites]
Speaking from the third world perspective I don’t believe that’s a reasonable ask.
posted by Tell Me No Lies at 3:43 PM on October 4, 2021 [36 favorites]
This whole thing has really drawn the obnoxious "I don't care because I'm not on [social media platform]" takes out of the woodwork, while people in some countries have had their entire lifeline to their friends and families disappear without warning, and small communities have faced the possibility that important connections and support networks might be gone for good. Point and laugh all you want, but there are some functions of FB and its monopoly that have (unfortunately) become vital lifelines for many.
The other thread is full of this kind of sanctimony, and this thread too is gaining some. Honestly ever since the "is this something I'd have to own a TV to care about" days, MeFi has been a hotbed of "well EYE am ethically pure here, and I see no reason why everybody else can't be too!"
posted by wemayfreeze at 3:43 PM on October 4, 2021 [20 favorites]
The other thread is full of this kind of sanctimony, and this thread too is gaining some. Honestly ever since the "is this something I'd have to own a TV to care about" days, MeFi has been a hotbed of "well EYE am ethically pure here, and I see no reason why everybody else can't be too!"
posted by wemayfreeze at 3:43 PM on October 4, 2021 [20 favorites]
Is Friendster safe?
posted by Don.Kinsayder at 3:44 PM on October 4, 2021 [5 favorites]
posted by Don.Kinsayder at 3:44 PM on October 4, 2021 [5 favorites]
This tweet is a nice succinct explanation of what happened.
posted by Thorzdad at 3:45 PM on October 4, 2021 [11 favorites]
posted by Thorzdad at 3:45 PM on October 4, 2021 [11 favorites]
there are some functions of FB and its monopoly that have (unfortunately) become vital lifelines for many
There are lots of ways for reasonable people (and responsive governments) to respond to a social media network as a utility, in the precise language you just used. People could just accept the situation and move on, or reflect and act upon that realization, or call for their governments to do the same. Can you think of some other options?
posted by They sucked his brains out! at 3:46 PM on October 4, 2021 [5 favorites]
There are lots of ways for reasonable people (and responsive governments) to respond to a social media network as a utility, in the precise language you just used. People could just accept the situation and move on, or reflect and act upon that realization, or call for their governments to do the same. Can you think of some other options?
posted by They sucked his brains out! at 3:46 PM on October 4, 2021 [5 favorites]
Orkut and Bebo send their best wishes for your recovery.
posted by k3ninho at 3:46 PM on October 4, 2021
posted by k3ninho at 3:46 PM on October 4, 2021
Tell Me No Lies: " Speaking from the third world perspective I don’t believe that’s a reasonable ask."
Speaking (also) from the third world, it actually is. Lots of smart people in the third world.
posted by signal at 3:48 PM on October 4, 2021 [17 favorites]
Speaking (also) from the third world, it actually is. Lots of smart people in the third world.
posted by signal at 3:48 PM on October 4, 2021 [17 favorites]
This reminds me how much I side-eye Facebook's internet.org project.
posted by Monochrome at 3:48 PM on October 4, 2021
posted by Monochrome at 3:48 PM on October 4, 2021
If I know anything, an unannounced break in communications in the Northern Hemisphere autumn is probably the first stages of a Warsaw Pact move on NATO. I would assume that Soviet T-72s are driving west through Saxony right now and that Tu-95s are establishing a picket around Iceland. The real question is will the French carry out a nuclear first-strike if the Soviets don't comply with their stated intention to halt at the western border of the Federal Republic?
[rolls dice]
posted by Fiasco da Gama at 3:48 PM on October 4, 2021 [32 favorites]
[rolls dice]
posted by Fiasco da Gama at 3:48 PM on October 4, 2021 [32 favorites]
I wonder if they tried ivermectin?
posted by NotMyselfRightNow at 3:49 PM on October 4, 2021 [4 favorites]
posted by NotMyselfRightNow at 3:49 PM on October 4, 2021 [4 favorites]
>Aside from the usual encouragement to delete/abandon if you can: might be a good idea to reset your passwords when you can.
??? for what kind of threat -- I really can't guess, maybe many attenpts to login might have sent credentials to bad actors? (Flawed because whoever would man-in-the-middle had no Facebook at the other end and nobody sends raw user/pass combos without going out of business.)
??? just "fresh password hygiene" complexity is preferred because remembering new complex things drives people to predictable and re-used passwords.
posted by k3ninho at 3:49 PM on October 4, 2021
??? for what kind of threat -- I really can't guess, maybe many attenpts to login might have sent credentials to bad actors? (Flawed because whoever would man-in-the-middle had no Facebook at the other end and nobody sends raw user/pass combos without going out of business.)
??? just "fresh password hygiene" complexity is preferred because remembering new complex things drives people to predictable and re-used passwords.
posted by k3ninho at 3:49 PM on October 4, 2021
Rife with the riff: Let's keep Facebook down until we reach herd immunity
posted by They sucked his brains out! at 3:50 PM on October 4, 2021 [5 favorites]
posted by They sucked his brains out! at 3:50 PM on October 4, 2021 [5 favorites]
Conspiracy theory? My chips go on these bingo squares:
* they befouled last might's snapshot of the panopticon security system and had to freeze everyone out while they chased down associates of the whistleblower using the everyday system
* .. or scrubbing something incriminating from the system
* the whistleblower upset a nation state who's relying on FB to amplify chaos among the "rule-based consensus" nations
posted by k3ninho at 3:51 PM on October 4, 2021 [4 favorites]
* they befouled last might's snapshot of the panopticon security system and had to freeze everyone out while they chased down associates of the whistleblower using the everyday system
* .. or scrubbing something incriminating from the system
* the whistleblower upset a nation state who's relying on FB to amplify chaos among the "rule-based consensus" nations
posted by k3ninho at 3:51 PM on October 4, 2021 [4 favorites]
>
Tell Me No Lies: " Speaking from the third world perspective I don’t believe that’s a reasonable ask."
Speaking (also) from the third world, it actually is. Lots of smart people in the third world.
"Are there smart people" may not be the obstacle in question - "are there RICH people" or "are there governments who won't block things " may be.
posted by EmpressCallipygos at 3:51 PM on October 4, 2021 [6 favorites]
Tell Me No Lies: " Speaking from the third world perspective I don’t believe that’s a reasonable ask."
Speaking (also) from the third world, it actually is. Lots of smart people in the third world.
"Are there smart people" may not be the obstacle in question - "are there RICH people" or "are there governments who won't block things " may be.
posted by EmpressCallipygos at 3:51 PM on October 4, 2021 [6 favorites]
Ah, so it turns out that the reason that the facebook.com registrar entry disappeared is that facebook is its own registrar and they ran literally everything from the DNS down to the doorlocks themselves without external third party redundancies?
Fly a little too close to the sun there Zuckerberg? Because those are Olympian levels of hubris.
posted by Horkus at 3:52 PM on October 4, 2021 [27 favorites]
Fly a little too close to the sun there Zuckerberg? Because those are Olympian levels of hubris.
posted by Horkus at 3:52 PM on October 4, 2021 [27 favorites]
As in, this should be the wake up call for all those communities and countries to actually get some sort of usable internet infrastructure in place that doesn't depend on a company that views its users as an inferior grade of livestock.
I'm sure those small communities of disabled people with rare diseases, neighbours trying to feed the elderly and infirm in their communities, and marginalised folks looking for somewhere to live after being kicked out by their families will get right on that.
This is a major problem and the issue of how this monopoly came about should (if we lived in an ideal world) be tackled by governments to secure infrastructure for their citizens. But we don't live in that world and in the meantime, on a micro level, I think some of us need to stop and reflect on certain classist attitudes we have towards the kinds of people who have no choice but to use what's available to them right now.
posted by fight or flight at 3:52 PM on October 4, 2021 [44 favorites]
I'm sure those small communities of disabled people with rare diseases, neighbours trying to feed the elderly and infirm in their communities, and marginalised folks looking for somewhere to live after being kicked out by their families will get right on that.
This is a major problem and the issue of how this monopoly came about should (if we lived in an ideal world) be tackled by governments to secure infrastructure for their citizens. But we don't live in that world and in the meantime, on a micro level, I think some of us need to stop and reflect on certain classist attitudes we have towards the kinds of people who have no choice but to use what's available to them right now.
posted by fight or flight at 3:52 PM on October 4, 2021 [44 favorites]
In unrelated news, antivax information on the net is down significantly today. Experts are puzzled.
posted by tommasz at 3:53 PM on October 4, 2021 [1 favorite]
posted by tommasz at 3:53 PM on October 4, 2021 [1 favorite]
It's a good time to reflect on how to connect with your groups, family members, community, without relying on one tool in particular.
posted by tiny frying pan at 3:58 PM on October 4, 2021 [12 favorites]
posted by tiny frying pan at 3:58 PM on October 4, 2021 [12 favorites]
Good
posted by firstdaffodils at 3:58 PM on October 4, 2021 [2 favorites]
posted by firstdaffodils at 3:58 PM on October 4, 2021 [2 favorites]
If I know anything, an unannounced break in communications in the Northern Hemisphere autumn is probably the first stages of a Warsaw Pact move on NATO. I would assume that Soviet T-72s are driving west through Saxony right now and that Tu-95s are establishing a picket around Iceland. The real question is will the French carry out a nuclear first-strike if the Soviets don't comply with their stated intention to halt at the western border of the Federal Republic?Gonna have to change your username to Fiasco da Gamma World
[rolls dice]
posted by Fiasco da Gama at 3:48 PM on October 4 [+] [!]
posted by eckeric at 3:59 PM on October 4, 2021 [19 favorites]
>>Speaking from the third world perspective I don’t believe that’s a reasonable ask.
>Speaking (also) from the third world, it actually is. Lots of smart people in the third world.
It’s not a matter of technical know-how, it’s a matter of an enormous initial outlay and then continual maintenance. Facebook has 58,000 employees and while I’m sure you could get away with 10,000 for a country-specific replacement that is still a pretty big payroll.
And there actually are staffing issues as well. Etsy is opening in Mexico and is having a hell of a time landing PHP developers. The problem is that smart people with mad skillz all take off for more lucrative countries as soon as they can.
posted by Tell Me No Lies at 4:02 PM on October 4, 2021 [7 favorites]
>Speaking (also) from the third world, it actually is. Lots of smart people in the third world.
It’s not a matter of technical know-how, it’s a matter of an enormous initial outlay and then continual maintenance. Facebook has 58,000 employees and while I’m sure you could get away with 10,000 for a country-specific replacement that is still a pretty big payroll.
And there actually are staffing issues as well. Etsy is opening in Mexico and is having a hell of a time landing PHP developers. The problem is that smart people with mad skillz all take off for more lucrative countries as soon as they can.
posted by Tell Me No Lies at 4:02 PM on October 4, 2021 [7 favorites]
Just to note that you can have governments in charge of communication infrastructure, but governments can just shut it down. There are no guarantees.
posted by Nancy Lebovitz at 4:05 PM on October 4, 2021 [4 favorites]
posted by Nancy Lebovitz at 4:05 PM on October 4, 2021 [4 favorites]
>Ah, so it turns out that the reason that the facebook.com registrar entry disappeared is that facebook is its own registrar and they ran literally everything from the DNS down to the doorlocks themselves without external third party redundancies?
The redundancy is interesting as is the extra load that got put on other services when people tried again and again to ask "are you sure it's not back yet?" which could use better time-to-live caching. (But they're not a 'rent you a domain and manage those records for customers' registrar, they run their own computers to turn names into numerical computer addresses.)
posted by k3ninho at 4:07 PM on October 4, 2021 [1 favorite]
The redundancy is interesting as is the extra load that got put on other services when people tried again and again to ask "are you sure it's not back yet?" which could use better time-to-live caching. (But they're not a 'rent you a domain and manage those records for customers' registrar, they run their own computers to turn names into numerical computer addresses.)
posted by k3ninho at 4:07 PM on October 4, 2021 [1 favorite]
they run their own computers to turn names into numerical computer addressesYeah, so do I.
But those names don't work unless the root record delegates to my registrar which then delegates to my nameservers. I don't have to clout to get the registry for .com to answer my phone calls, so I have to go though Network Solutions, or GoDaddy or some other registrar. Facebook does have that clout and is their own registrar (registrarsafe.com is facebook), so their DNS problem happened at a higher level than any DNS problem that I could ever possibly be the victim and/or perpetrator of.
posted by Horkus at 4:21 PM on October 4, 2021 [6 favorites]
I've been thinking for a while that it would be very easy to run social media over email. All it would require would be for someone to hack an open-source email client like, say, Thunderbird, so that it displayed appropriately tagged and formatted emails like social media posts in a feed.
So in the days Before Facebook, yea, before MySpace and even before LiveJournal there were such things as you describe. If you had Internet access (originally it was hard to get unless you were associated with a university or major institution) you had access to Usenet, which was like a giant decentralised Metafilter moderated by Wikipedia editors, if you can imagine such a thing. I probably haven’t used no it for fifteen years or so, but it sort of still exists although from what I understand it’s mostly used to distribute pirated files.
There was also Fidonet, a network of BBSs that used a similar store-and-forward protocol to distribute posts in forums shared between BBSs. Like Usenet, the system was run by volunteers; unlike Usenet, each BBS was a distinct site with its own credential system for its users and only served the forums they selected. Fidonet technically still exists but I don’t know if it’s actually used for anything.
Anyway, these and other protocols exist and are still supported and running to some extent, and Usenet in particular is right there. I don’t know how decentralised social media ought to work but the path is open for someone to specify a way of curating data drawn via existing protocols for private email, semi-public mailing lists, and public newsgroups, and presenting it to users identified via a broadcast public key system. It wouldn’t be so dissimilar to modern social media experience, although you’d need to figure out how to subsidise the network costs in the absence of Facebook’s marvellous money machine.
posted by Joe in Australia at 4:27 PM on October 4, 2021 [9 favorites]
So in the days Before Facebook, yea, before MySpace and even before LiveJournal there were such things as you describe. If you had Internet access (originally it was hard to get unless you were associated with a university or major institution) you had access to Usenet, which was like a giant decentralised Metafilter moderated by Wikipedia editors, if you can imagine such a thing. I probably haven’t used no it for fifteen years or so, but it sort of still exists although from what I understand it’s mostly used to distribute pirated files.
There was also Fidonet, a network of BBSs that used a similar store-and-forward protocol to distribute posts in forums shared between BBSs. Like Usenet, the system was run by volunteers; unlike Usenet, each BBS was a distinct site with its own credential system for its users and only served the forums they selected. Fidonet technically still exists but I don’t know if it’s actually used for anything.
Anyway, these and other protocols exist and are still supported and running to some extent, and Usenet in particular is right there. I don’t know how decentralised social media ought to work but the path is open for someone to specify a way of curating data drawn via existing protocols for private email, semi-public mailing lists, and public newsgroups, and presenting it to users identified via a broadcast public key system. It wouldn’t be so dissimilar to modern social media experience, although you’d need to figure out how to subsidise the network costs in the absence of Facebook’s marvellous money machine.
posted by Joe in Australia at 4:27 PM on October 4, 2021 [9 favorites]
Tally a "nation-state reminds Facebook who's in charge" for me.
posted by seanmpuckett at 4:29 PM on October 4, 2021 [4 favorites]
posted by seanmpuckett at 4:29 PM on October 4, 2021 [4 favorites]
Maybe Zuck's just throwing a tantrum over the whistleblower.
posted by Phlegmco(tm) at 4:42 PM on October 4, 2021 [2 favorites]
posted by Phlegmco(tm) at 4:42 PM on October 4, 2021 [2 favorites]
Could someone explain why “Facebook is down for a couple hours” deserves an FPP, and isn’t just the weakest of chatty newsfilter?Yeah, I hear what you're saying. "Facebook down" is not generally front page Metafilter material. But in this case Facebook's (probably) self inflicted outage was unprecedented in cause, duration, and comedy.
Not to mention that, what with all the other news currently afoot about Facebook, a cynical person might find the timing suspicious. I'm personally grateful that there's a second post in which to say haha.gif at Facebook's follies so that it doesn't drown out the thread about Facebook's crimes against humanity.
posted by Horkus at 4:45 PM on October 4, 2021 [11 favorites]
OK Boomer?
You may ask yourself, "What is that beautiful house?"
You may ask yourself, "Where does that highway go to?"
And you may ask yourself, "Am I right? Am I wrong?"
And you may say to yourself, "My God! What have I done?"
posted by thecincinnatikid at 4:48 PM on October 4, 2021 [7 favorites]
You may ask yourself, "What is that beautiful house?"
You may ask yourself, "Where does that highway go to?"
And you may ask yourself, "Am I right? Am I wrong?"
And you may say to yourself, "My God! What have I done?"
posted by thecincinnatikid at 4:48 PM on October 4, 2021 [7 favorites]
I guess I’m just in the minority in thinking that 50% of the day’s FPP’s being about FB is too many. Carry on…
posted by a box and a stick and a string and a bear at 4:50 PM on October 4, 2021
posted by a box and a stick and a string and a bear at 4:50 PM on October 4, 2021
Okay, who buried Facebook in the pet sematary? You gotta stop doing that!
Sometimes, dead is better.
posted by bendy at 4:51 PM on October 4, 2021 [8 favorites]
Sometimes, dead is better.
posted by bendy at 4:51 PM on October 4, 2021 [8 favorites]
I think folks here have been pretty good overall about couching their "I don't even OWN a Facebook!" talk in concessions toward those who do feel like other options are untenable. Personally, it was seeing a growth in comments on MF about leaving FB/IG/AMA/Twitter that made me feel I could do it as well. Amusing now, years later, that it seemed impossible. Such comments can be motivating and normalizing in a world in which "so how can I contact you??!" is still a common reaction to being off social media.
posted by youarenothere at 4:51 PM on October 4, 2021 [13 favorites]
posted by youarenothere at 4:51 PM on October 4, 2021 [13 favorites]
Given the withdrawn routes and the failure of their DNS infrastructure and the apparent failure of even their internal network, I'd say it's worth discussing.
Any one of those things, meh, whatever, shit happens. All of them together? That says bad things about the broader state of what passes for network engineering these days.
posted by wierdo at 4:52 PM on October 4, 2021 [3 favorites]
Any one of those things, meh, whatever, shit happens. All of them together? That says bad things about the broader state of what passes for network engineering these days.
posted by wierdo at 4:52 PM on October 4, 2021 [3 favorites]
I guess I’m just in the minority in thinking that 50% of the day’s FPP’s being about FB is too many.
One of them is about he absence of Facebook, so, on average, there are no FPPs about Facebook.
posted by ChurchHatesTucker at 4:59 PM on October 4, 2021 [40 favorites]
One of them is about he absence of Facebook, so, on average, there are no FPPs about Facebook.
posted by ChurchHatesTucker at 4:59 PM on October 4, 2021 [40 favorites]
I agree, there was a day a couple years back where Amazon Web Services had a failure of one of their older and simpler services called S3. Unfortunately (1) a huge fraction of web sites ended up being critically dependent on it, and (2) part of Amazon's own page for "what's down and why" ended up being dependent on S3, leading to a meta level of confusion.
The spectacular shadenfreude of Facebook (nowadays the ur-example of a walled garden) being too dependent on itself to easily recover from a failure of itself is fun. And noteworthy!
posted by traveler_ at 5:02 PM on October 4, 2021 [7 favorites]
The spectacular shadenfreude of Facebook (nowadays the ur-example of a walled garden) being too dependent on itself to easily recover from a failure of itself is fun. And noteworthy!
posted by traveler_ at 5:02 PM on October 4, 2021 [7 favorites]
My FB is back up, but seems to be ultra showcasing only posts from a mushroom ID group I belong to. Goodlord there are a lot of mushrooms.
posted by tarantula at 5:08 PM on October 4, 2021 [11 favorites]
posted by tarantula at 5:08 PM on October 4, 2021 [11 favorites]
once you start working in tech you start getting hints of how it works and your realize it's all so fucking piecemeal and duct-taped together. it's all shitty documentation, retention issues where you lose the last person who actually knows how large swathes of your codebase works and what dependencies you need to account for, internecine power-hungry pushes by product owners that are short-sighted and ruinous (Google Pay is a good recent example of this), infrastructure that is literally decades old and horrible but still in wide use because rebuilding a better version of it is a huge task that would require a level of coordination across thousands of companies that would never happen because of how horribly unregulated it all is, and etc
shit like FB going down and it's just like yeah, of course they did, and of course it's because they bought out an accredited registrar so they could manage their own DNS records
the business logic of buying out a registrar is that if fuckups like these happen, at least you can directly go in and make the fixes you need instead of having to wait for your vendor to get their people on it. bc you know your vendors are definitely not telling you what happened or how they're going to prevent it in the future because you'd see too much of their sausage making (ie how shittily cobbled together everything is). plus now you can make sure infra/devops aren't just sitting around waiting for the 504 issues to magically resolve on their own like they always do because they have no fucking clue how it all works or what's actually causing the issue because the last dev who did now works for another company
but then what happens is that the penny pinching geniuses in leadership do the exact same fucking thing they didn't want their vendors to do - they don't give you enough money or time to make important fixes, they want results asap, and shit just ends up being cobbled together because sitting down and taking a year or two to rebuild, heavily document, and triple check everything means $$ down the drain
so the reason you inhoused the thing (ie so you could manage things competently since you have way more $$ and resources than some tiny lil registrar) gets forgotten because leadership has their eyes on the next shiny innovation that'll bring in the big $$
it's almost like having a society function on a quarterly basis results in a lot of just cobbled together crap that barely works because nobody's interested in taking their time to get things right, and the $$ you're dumping into UX design is the only thing convincing society that your gilded turd is actually a big ol nugget of $$ venture capital/wall street gold
posted by paimapi at 5:11 PM on October 4, 2021 [97 favorites]
shit like FB going down and it's just like yeah, of course they did, and of course it's because they bought out an accredited registrar so they could manage their own DNS records
the business logic of buying out a registrar is that if fuckups like these happen, at least you can directly go in and make the fixes you need instead of having to wait for your vendor to get their people on it. bc you know your vendors are definitely not telling you what happened or how they're going to prevent it in the future because you'd see too much of their sausage making (ie how shittily cobbled together everything is). plus now you can make sure infra/devops aren't just sitting around waiting for the 504 issues to magically resolve on their own like they always do because they have no fucking clue how it all works or what's actually causing the issue because the last dev who did now works for another company
but then what happens is that the penny pinching geniuses in leadership do the exact same fucking thing they didn't want their vendors to do - they don't give you enough money or time to make important fixes, they want results asap, and shit just ends up being cobbled together because sitting down and taking a year or two to rebuild, heavily document, and triple check everything means $$ down the drain
so the reason you inhoused the thing (ie so you could manage things competently since you have way more $$ and resources than some tiny lil registrar) gets forgotten because leadership has their eyes on the next shiny innovation that'll bring in the big $$
it's almost like having a society function on a quarterly basis results in a lot of just cobbled together crap that barely works because nobody's interested in taking their time to get things right, and the $$ you're dumping into UX design is the only thing convincing society that your gilded turd is actually a big ol nugget of $$ venture capital/wall street gold
posted by paimapi at 5:11 PM on October 4, 2021 [97 favorites]
I’m going to just go back to updating my .plan file. Those who know me will figure it out.
posted by interogative mood at 5:30 PM on October 4, 2021 [19 favorites]
posted by interogative mood at 5:30 PM on October 4, 2021 [19 favorites]
finger interogative mood
posted by traveler_ at 5:38 PM on October 4, 2021 [25 favorites]
posted by traveler_ at 5:38 PM on October 4, 2021 [25 favorites]
Is BGP the kind of resource that Randall Munroe described in this strip?
Not quite since there are several implementations including proprietary ones, but a significant chunk of the internet (quite possibly including Facebook) relies on these four people in Prague.
posted by Slogby at 5:39 PM on October 4, 2021 [6 favorites]
Not quite since there are several implementations including proprietary ones, but a significant chunk of the internet (quite possibly including Facebook) relies on these four people in Prague.
posted by Slogby at 5:39 PM on October 4, 2021 [6 favorites]
once you start working in tech you start getting hints of how it works and your realize it's all so fucking piecemeal and duct-taped together.
tbh you can delete "in tech" from this without making it untrue. replace with "in capitalism" if you like, I suppose.
posted by We put our faith in Blast Hardcheese at 5:41 PM on October 4, 2021 [13 favorites]
tbh you can delete "in tech" from this without making it untrue. replace with "in capitalism" if you like, I suppose.
posted by We put our faith in Blast Hardcheese at 5:41 PM on October 4, 2021 [13 favorites]
They needed an angle grinder to get to the servers.
posted by Nelson at 5:48 PM on October 4, 2021 [10 favorites]
posted by Nelson at 5:48 PM on October 4, 2021 [10 favorites]
So they hacked their way in?
posted by Joe in Australia at 6:01 PM on October 4, 2021 [13 favorites]
posted by Joe in Australia at 6:01 PM on October 4, 2021 [13 favorites]
i heard from my cousin who has a friend who works at facebook that what happened is that an admin pushed this bgp configuration that made his testicles get really swollen, and
posted by phooky at 6:03 PM on October 4, 2021 [14 favorites]
posted by phooky at 6:03 PM on October 4, 2021 [14 favorites]
They needed an angle grinder to get to the servers.
why were their door locks to their servers dependent on their fucking DNS recs working
why are their door locks connected to the internet at all
posted by paimapi at 6:04 PM on October 4, 2021 [26 favorites]
why were their door locks to their servers dependent on their fucking DNS recs working
why are their door locks connected to the internet at all
posted by paimapi at 6:04 PM on October 4, 2021 [26 favorites]
Holy shit, the IoT has become Skynet. Terminators, but stupid.
posted by Don.Kinsayder at 6:07 PM on October 4, 2021 [10 favorites]
posted by Don.Kinsayder at 6:07 PM on October 4, 2021 [10 favorites]
I've worked in data centers for two companies with their own transit networks. I'm having such trouble with the idea they could design their system so badly that a botched BGP update could take out not just the front end but all their back end access as well, AND their physical access for good measure. How do you have a single point of failure for your entire company?
posted by fedward at 6:11 PM on October 4, 2021 [10 favorites]
posted by fedward at 6:11 PM on October 4, 2021 [10 favorites]
Facebook is their own registrar..? Wow, did not know that. I guess it doesn't totally surprise me—it looks like the ICANN accreditation process isn't all that complex, likely just expensive—but I guess I'm not sure why you'd bother to in-house something like that unless you were using a lot of domain names, and FB doesn't seem to use all that many. Interesting.
FWIW, I've always been a little sketchy on how exactly the domain registration process works and how data (and money) actually flows from end customer to registrar to registry operator.
Working from the "top" down, you have the DNS Root Zone. It's the theoretical starting point for all DNS queries, and is resolved using the Root Zone File, typically named "named.cache", generally preinstalled on Internet-facing devices. The file contains IP addresses for a.root-servers.net through m.root-servers.net. The authoritative version of the Root Zone File is created by ICANN and distributed by Verisign.
At the next level are Top Level Domains (TLDs) such as com, org, net, edu, mil and the other original TLDs created in 1985, plus the ccTLDs and all the miscellaneous low-rent ones created a few years ago. TLDs are defined/created by ICANN and provided to the root DNS servers (a-m) by Verisign. Each TLD has a corresponding entity to administer its registry and dole out domains under that TLD.
The registry operator for .com is (again) Verisign, under some sort of agreement with ICANN, and Verisign maintains the .com zone file, currently running about 158M entries. For reasons that are unclear to me, rather than just having Verisign handle the assignment of domain names, it's farmed out to a whole mess of registrars. It seems like just about anyone can become a registrar, if they really want to and have the cash. Registrars are companies like GoDaddy or Namecheap (or, apparently, Facebook), and are seemingly allowed to charge customers whatever they want to register domains. As the registry operator, Verisign charges each registrar a fee of around $8 per transaction, and then on top of that ICANN gets its cut in the form of a $4,000/yr "accreditation fee" and $1,125/quarter "variable fee" from each registrar, and a $0.18 "transaction fee" per domain.
My takeaway from this is that there's a shitload of money flowing upwards, mostly towards Verisign, and that Verisign's cut of domain-registration fees is gigantic compared to ICANN's. This seems totally backwards to me, on top of the sheer insanity of having a for-profit corporation sit around and charge rents for critical Internet infrastructure.
posted by Kadin2048 at 6:20 PM on October 4, 2021 [9 favorites]
FWIW, I've always been a little sketchy on how exactly the domain registration process works and how data (and money) actually flows from end customer to registrar to registry operator.
Working from the "top" down, you have the DNS Root Zone. It's the theoretical starting point for all DNS queries, and is resolved using the Root Zone File, typically named "named.cache", generally preinstalled on Internet-facing devices. The file contains IP addresses for a.root-servers.net through m.root-servers.net. The authoritative version of the Root Zone File is created by ICANN and distributed by Verisign.
At the next level are Top Level Domains (TLDs) such as com, org, net, edu, mil and the other original TLDs created in 1985, plus the ccTLDs and all the miscellaneous low-rent ones created a few years ago. TLDs are defined/created by ICANN and provided to the root DNS servers (a-m) by Verisign. Each TLD has a corresponding entity to administer its registry and dole out domains under that TLD.
The registry operator for .com is (again) Verisign, under some sort of agreement with ICANN, and Verisign maintains the .com zone file, currently running about 158M entries. For reasons that are unclear to me, rather than just having Verisign handle the assignment of domain names, it's farmed out to a whole mess of registrars. It seems like just about anyone can become a registrar, if they really want to and have the cash. Registrars are companies like GoDaddy or Namecheap (or, apparently, Facebook), and are seemingly allowed to charge customers whatever they want to register domains. As the registry operator, Verisign charges each registrar a fee of around $8 per transaction, and then on top of that ICANN gets its cut in the form of a $4,000/yr "accreditation fee" and $1,125/quarter "variable fee" from each registrar, and a $0.18 "transaction fee" per domain.
My takeaway from this is that there's a shitload of money flowing upwards, mostly towards Verisign, and that Verisign's cut of domain-registration fees is gigantic compared to ICANN's. This seems totally backwards to me, on top of the sheer insanity of having a for-profit corporation sit around and charge rents for critical Internet infrastructure.
posted by Kadin2048 at 6:20 PM on October 4, 2021 [9 favorites]
why were their door locks to their servers dependent on their fucking DNS recs working
More specifically, why were they dependent on their global DNS infrastructure to work? You'd think they'd have name servers locally that aren't dependent on the root, gtld, or even their SLD authority being functional. Preferably authoritative, but that isn't by any means required.
I get not wanting to configure IPs in the locks, but you should only need one working nameserver for the locks to be able to find a local AAA server and the AAA server should be able to survive being temporarily isolated from its own master database server.
posted by wierdo at 6:21 PM on October 4, 2021 [7 favorites]
More specifically, why were they dependent on their global DNS infrastructure to work? You'd think they'd have name servers locally that aren't dependent on the root, gtld, or even their SLD authority being functional. Preferably authoritative, but that isn't by any means required.
I get not wanting to configure IPs in the locks, but you should only need one working nameserver for the locks to be able to find a local AAA server and the AAA server should be able to survive being temporarily isolated from its own master database server.
posted by wierdo at 6:21 PM on October 4, 2021 [7 favorites]
The angle grinder story has been retracted
posted by Nelson at 6:31 PM on October 4, 2021 [6 favorites]
posted by Nelson at 6:31 PM on October 4, 2021 [6 favorites]
I refuse to be shamed for feeling schadenfreude about the big evil tech company stubbing its toe and going “ouch” for a couple of hours. And if it made the day even a tiny bit crummier for the Asshole-in-charge, then on balance it was a good day for humanity.
posted by Atom Eyes at 7:04 PM on October 4, 2021 [18 favorites]
posted by Atom Eyes at 7:04 PM on October 4, 2021 [18 favorites]
I wish I could find the article I read recently that describes that every user in some countries considers Facebook the internet. I googled and found articles about this dating back to 2015.
Facebook Subscriber Stats as of Mar. 31, 2021
posted by bendy at 7:43 PM on October 4, 2021 [3 favorites]
Facebook Subscriber Stats as of Mar. 31, 2021
posted by bendy at 7:43 PM on October 4, 2021 [3 favorites]
In my version of history engineers had to run to Home Depot to get some tools, get the wrong tools, go back get several angle grinders and then return. Then have to go back a third time because they didn’t get any disks for it.
Then as they begin try to cut through the alloy steel locks on the cages they become extremely worried that the smoke, sparks and heat from the cutting will set off the fire suppression system; a system designed to rapidly purge all the oxygen and flood the room with argon to put out an electrical fire. And this whole scene is playing out in a few dozen data centers and cages because you can’t just bring one route up as it will get instantly DDOSED as a billion Facebook users try to route through it.
Who cares if this actually happened this way. It might of happened and reality can’t be that entertaining.
posted by interogative mood at 7:57 PM on October 4, 2021 [17 favorites]
Then as they begin try to cut through the alloy steel locks on the cages they become extremely worried that the smoke, sparks and heat from the cutting will set off the fire suppression system; a system designed to rapidly purge all the oxygen and flood the room with argon to put out an electrical fire. And this whole scene is playing out in a few dozen data centers and cages because you can’t just bring one route up as it will get instantly DDOSED as a billion Facebook users try to route through it.
Who cares if this actually happened this way. It might of happened and reality can’t be that entertaining.
posted by interogative mood at 7:57 PM on October 4, 2021 [17 favorites]
Wheeeeee. Today was exciting. Well, vaguely exciting.
This didn't just break Facebook properties and resources.
My cohort and I had a bunch of cloud based SaaS and CMS stuff go down today alongside whatever else the fuck was going on.
And even better it happened just as we were launching and going fully live on two new projects right in the middle of the whole kerfuckle that was apparently bad enough to take down some of our SaaS widgets that somehow relied on parts of whatever broke. Things that were basically invisible to us as resellers and users of said SaaS components.
And it happened in ways that didn't make any sort of sense at all, as some established properties that used the same SaaS and CMS stuff, properties we didn't even touch at all stayed up and online, others with the same system failed entirely, and others had modules or components fail while staying mostly functional.
Whatever happened threw my coworking team for a total loop because of how variable the failures were between the newly launched properties and wildly variable failures in long established projects and properties.
It's been a long time since the last time I woke up to an all hands on deck and pound coffee and strings until you're really awake and alert sort of large scale outage emergency. I haven't had to use traceroute, ping, whois, dig and so on so much in one day in something like 10+ years.
Thankfully my boss is chill and reliably sane and after 30 minutes of that nonsense we all pivoted to available or offline tasks in our task lists.
I don't have caps or logs but the general sort of internet/router/lag health monitoring sites were all over the f'ing place this morning around 9:30A PST, with a focus on the NE US and Canada regions.
There was a few minutes where I was hitting refresh on some of the monitoring and reporting sites where it seems like I could see it spreading from that geographic area and links to/from the area were going down and the DNS/BGP floppiness was spreading over North America.
posted by loquacious at 8:07 PM on October 4, 2021 [14 favorites]
This didn't just break Facebook properties and resources.
My cohort and I had a bunch of cloud based SaaS and CMS stuff go down today alongside whatever else the fuck was going on.
And even better it happened just as we were launching and going fully live on two new projects right in the middle of the whole kerfuckle that was apparently bad enough to take down some of our SaaS widgets that somehow relied on parts of whatever broke. Things that were basically invisible to us as resellers and users of said SaaS components.
And it happened in ways that didn't make any sort of sense at all, as some established properties that used the same SaaS and CMS stuff, properties we didn't even touch at all stayed up and online, others with the same system failed entirely, and others had modules or components fail while staying mostly functional.
Whatever happened threw my coworking team for a total loop because of how variable the failures were between the newly launched properties and wildly variable failures in long established projects and properties.
It's been a long time since the last time I woke up to an all hands on deck and pound coffee and strings until you're really awake and alert sort of large scale outage emergency. I haven't had to use traceroute, ping, whois, dig and so on so much in one day in something like 10+ years.
Thankfully my boss is chill and reliably sane and after 30 minutes of that nonsense we all pivoted to available or offline tasks in our task lists.
I don't have caps or logs but the general sort of internet/router/lag health monitoring sites were all over the f'ing place this morning around 9:30A PST, with a focus on the NE US and Canada regions.
There was a few minutes where I was hitting refresh on some of the monitoring and reporting sites where it seems like I could see it spreading from that geographic area and links to/from the area were going down and the DNS/BGP floppiness was spreading over North America.
posted by loquacious at 8:07 PM on October 4, 2021 [14 favorites]
And this whole scene is playing out in a few dozen data centers and cages because you can’t just bring one route up as it will get instantly DDOSED as a billion Facebook users try to route through it.
Hello I would like to subscribe to "Facebook DCs, CoLos and Routers Catching on Fire Even if it Takes Down the Internet Facts".
posted by loquacious at 8:09 PM on October 4, 2021 [4 favorites]
Hello I would like to subscribe to "Facebook DCs, CoLos and Routers Catching on Fire Even if it Takes Down the Internet Facts".
posted by loquacious at 8:09 PM on October 4, 2021 [4 favorites]
But Tumblr, like a post-apocalyptic cockroach, just keeps rollin' along.
posted by The Underpants Monster at 8:14 PM on October 4, 2021 [5 favorites]
posted by The Underpants Monster at 8:14 PM on October 4, 2021 [5 favorites]
My FB is back up, but seems to be ultra showcasing only posts from a mushroom ID group I belong to. Goodlord there are a lot of mushrooms.
NaNoWriMo is coming up. People should kick it off with this paragraph!
paimapi: once you start working in tech you start getting hints of how it works and your realize it's all so fucking piecemeal and duct-taped together. it's all shitty documentation, retention issues where you lose the last person who actually knows how large swathes of your codebase works and what dependencies you need to account for
In all deadly seriousness, I was in the data centre of a major bank the day there was a very severe outage of the bank's primary online banking app.
I was in a meeting with the exec in charge of said things , a person who'd worked his way up and done a whole bunch of hands-on jobs over the years.
As I walked into this meeting, he simultaneously had the network management call for the outage in question on his speakerphone.
"If it's a bad time, I can go," I said.
"Nah, it's fine," he said. "I've been on this call since early this morning."
Our meeting continued.
Then, someone said something over the phone that made him get this faraway look in his face. He gave me the "talk to the hand" motion.
I stopped talking.
He took the speakerphone off mute.
Then, he spoke.
"Hey -- you know that Sun box on the raised floor on 2?"
Some incident management or infrastructure person on the phone said, "You mean here, at [X data centre]? Yeah."
"Recycle it."
And that was literally the fix.
If you wanted to do online banking with that bank that day, or any day after it, until someone else figured it out? The guy who fixed it was a guy knew where that old Sun Microsystems box that he'd last touched a decade (or more) ago was still running something that he'd dealt with before.
He, and only he, knew that the outage could be fixed by turning it off, and then and on again.
There was no documentation for this, obviously.
To paimapi's point, that is but another example of how companies with billions of dollars at their disposal handle this stuff.
I still have an account with said bank. Whenever there's a hiccup, I think of someone running into that room and resetting that Sun box.
posted by mandolin conspiracy at 8:18 PM on October 4, 2021 [81 favorites]
NaNoWriMo is coming up. People should kick it off with this paragraph!
paimapi: once you start working in tech you start getting hints of how it works and your realize it's all so fucking piecemeal and duct-taped together. it's all shitty documentation, retention issues where you lose the last person who actually knows how large swathes of your codebase works and what dependencies you need to account for
In all deadly seriousness, I was in the data centre of a major bank the day there was a very severe outage of the bank's primary online banking app.
I was in a meeting with the exec in charge of said things , a person who'd worked his way up and done a whole bunch of hands-on jobs over the years.
As I walked into this meeting, he simultaneously had the network management call for the outage in question on his speakerphone.
"If it's a bad time, I can go," I said.
"Nah, it's fine," he said. "I've been on this call since early this morning."
Our meeting continued.
Then, someone said something over the phone that made him get this faraway look in his face. He gave me the "talk to the hand" motion.
I stopped talking.
He took the speakerphone off mute.
Then, he spoke.
"Hey -- you know that Sun box on the raised floor on 2?"
Some incident management or infrastructure person on the phone said, "You mean here, at [X data centre]? Yeah."
"Recycle it."
And that was literally the fix.
If you wanted to do online banking with that bank that day, or any day after it, until someone else figured it out? The guy who fixed it was a guy knew where that old Sun Microsystems box that he'd last touched a decade (or more) ago was still running something that he'd dealt with before.
He, and only he, knew that the outage could be fixed by turning it off, and then and on again.
There was no documentation for this, obviously.
To paimapi's point, that is but another example of how companies with billions of dollars at their disposal handle this stuff.
I still have an account with said bank. Whenever there's a hiccup, I think of someone running into that room and resetting that Sun box.
posted by mandolin conspiracy at 8:18 PM on October 4, 2021 [81 favorites]
A whole lot of the internet is made out of protocols built on trust. We've moved into an era where that trust is a baaad assumption. We've been here a long time, actually. But lots of those protocols are still running things.
This is a pretty good metaphor for late-stage capitalism as a whole, TBH.
posted by mstokes650 at 8:30 PM on October 4, 2021 [17 favorites]
This is a pretty good metaphor for late-stage capitalism as a whole, TBH.
posted by mstokes650 at 8:30 PM on October 4, 2021 [17 favorites]
One of them is about he absence of Facebook, so, on average, there are no FPPs about Facebook.
"Whenever Facebook isn't onscreen, the other characters should all be asking where Facebook is!"
posted by Ghidorah at 8:41 PM on October 4, 2021 [15 favorites]
"Whenever Facebook isn't onscreen, the other characters should all be asking where Facebook is!"
posted by Ghidorah at 8:41 PM on October 4, 2021 [15 favorites]
Ceefax remained unaffected.
The non-public meme group I belong to, which for the last few years has been my favourite place online, is back up. Hurrah! And it contains Facebook outage memes; splendid.
posted by Wordshore at 9:26 PM on October 4, 2021 [2 favorites]
The non-public meme group I belong to, which for the last few years has been my favourite place online, is back up. Hurrah! And it contains Facebook outage memes; splendid.
posted by Wordshore at 9:26 PM on October 4, 2021 [2 favorites]
Too many to share, but the QTs and replies to this tweet is worth sharing: I think many Americans can’t appreciate just what a big deal it is that WhatsApp is down
posted by cendawanita at 9:51 PM on October 4, 2021 [11 favorites]
posted by cendawanita at 9:51 PM on October 4, 2021 [11 favorites]
MetaFilter: an extremely powerful tool and you can cause widespread chaos with a few wrong keystrokes
posted by kirkaracha at 10:20 PM on October 4, 2021 [8 favorites]
posted by kirkaracha at 10:20 PM on October 4, 2021 [8 favorites]
i feel like the weirdest part of today was how suddenly everyone on twitter --where people spend most days screaming "twitter is a steaming nazi shitpile! we are all bad people for using the bad internet" --was like, "but twitter! twitter is Good, it is not Bad, like the other internets, which are all Bad. Look at all of those fools on not-twitter, how terribly Bad and also Uncool they are."
it's almost like the problem is "anything humans touch."
posted by We put our faith in Blast Hardcheese at 10:57 PM on October 4, 2021 [3 favorites]
it's almost like the problem is "anything humans touch."
posted by We put our faith in Blast Hardcheese at 10:57 PM on October 4, 2021 [3 favorites]
Sorry for the superfluous FB FPP! I somehow managed to not realize the other post was about FB.
posted by mpark at 11:17 PM on October 4, 2021
posted by mpark at 11:17 PM on October 4, 2021
>A whole lot of the internet is made out of protocols built on trust. We've moved into an era where that trust is a baaad assumption. We've been here a long time, actually. But lots of those protocols are still running things.
We’ve been here a long time and despite people feeling like there should be problems with the trust model there have been few enough to count on one hand. The sky has been falling for a very long time.
posted by Tell Me No Lies at 12:18 AM on October 5, 2021 [2 favorites]
We’ve been here a long time and despite people feeling like there should be problems with the trust model there have been few enough to count on one hand. The sky has been falling for a very long time.
posted by Tell Me No Lies at 12:18 AM on October 5, 2021 [2 favorites]
>>Speaking from the third world perspective I don’t believe that’s a reasonable ask.
>Speaking (also) from the third world, it actually is. Lots of smart people in the third world.
It’s not a matter of technical know-how, it’s a matter of an enormous initial outlay and then continual maintenance. Facebook has 58,000 employees and while I’m sure you could get away with 10,000 for a country-specific replacement that is still a pretty big payroll.
Oh for heavens sake, you do not need to build an entire Facebook clone in order to stop using Facebook. There are plenty of other social media companies offering services for free, the point is not to rely one one company for everything - a single point of failure.
posted by Lanark at 12:53 AM on October 5, 2021 [5 favorites]
>Speaking (also) from the third world, it actually is. Lots of smart people in the third world.
It’s not a matter of technical know-how, it’s a matter of an enormous initial outlay and then continual maintenance. Facebook has 58,000 employees and while I’m sure you could get away with 10,000 for a country-specific replacement that is still a pretty big payroll.
Oh for heavens sake, you do not need to build an entire Facebook clone in order to stop using Facebook. There are plenty of other social media companies offering services for free, the point is not to rely one one company for everything - a single point of failure.
posted by Lanark at 12:53 AM on October 5, 2021 [5 favorites]
In unrelated news, antivax information on the net is down significantly today. Experts are puzzled.
It's probably best if we keep it down until society reaches herd immunity.
posted by acb at 1:09 AM on October 5, 2021 [1 favorite]
It's probably best if we keep it down until society reaches herd immunity.
posted by acb at 1:09 AM on October 5, 2021 [1 favorite]
you had access to Usenet, which was like a giant decentralised Metafilter moderated by Wikipedia editors, if you can imagine such a thing. I probably haven’t used no it for fifteen years or so, but it sort of still exists although from what I understand it’s mostly used to distribute pirated files.
IIRC, USENET had become unusable due to spam drowning out content by 1997 or so, which is when people retreated to private mailing lists. This did not help the signal to noise ratio, and accelerated its collapse into a radioactive wasteland populated only by marauding mutants.
posted by acb at 1:17 AM on October 5, 2021 [3 favorites]
IIRC, USENET had become unusable due to spam drowning out content by 1997 or so, which is when people retreated to private mailing lists. This did not help the signal to noise ratio, and accelerated its collapse into a radioactive wasteland populated only by marauding mutants.
posted by acb at 1:17 AM on October 5, 2021 [3 favorites]
Just hearing several reports of the military entering Facebook head quarters and retrieving all the servers — someone by the name @endtimepatriot
posted by acb at 1:21 AM on October 5, 2021 [2 favorites]
posted by acb at 1:21 AM on October 5, 2021 [2 favorites]
The doorlocks being entangled with the network reminded me of the story of early corporate swipe-cards. All the doors in MesoCorp were networked and only Grade-6-and-above had access to the C-suite. Then someone had the bright idea of using the cards to pay at the coffee machines. An unintended consequence was the the techs had access to biophysical data - how long it took to filter the good coffee: subtracting purchase time from time of swiping into the Executive toilet.
posted by BobTheScientist at 3:23 AM on October 5, 2021 [2 favorites]
posted by BobTheScientist at 3:23 AM on October 5, 2021 [2 favorites]
I think many Americans can’t appreciate just what a big deal it is that WhatsApp is down
It me. I had some vague idea that it was another chat system that FB owned but I've never met anyone who uses it.
posted by octothorpe at 3:43 AM on October 5, 2021 [1 favorite]
It me. I had some vague idea that it was another chat system that FB owned but I've never met anyone who uses it.
posted by octothorpe at 3:43 AM on October 5, 2021 [1 favorite]
i'm not being too hyperbolic, but i'm very shocked by this, when i went thru that tweet and the replies, which indicates this is a US-only condition. everyone everywhere else were joking about fb and ig of course, but whatsapp being down was legitimately distressing. our government's covid-19 information channels* include telegram and sms as well, but that's unusual for countries.
*what more my delivery notifications!
posted by cendawanita at 3:58 AM on October 5, 2021 [2 favorites]
*what more my delivery notifications!
posted by cendawanita at 3:58 AM on October 5, 2021 [2 favorites]
I’m curious; does anybody know what the security team at MySpace was doing yesterday morning?
posted by TedW at 4:07 AM on October 5, 2021 [1 favorite]
posted by TedW at 4:07 AM on October 5, 2021 [1 favorite]
Digitaltrends:
But according to the Pew Research Center, the number of adults using Facebook plateaued in 2018, and WhatsApp user number decreased: Only 20% of U.S. adults use WhatsApp in 2019, down from 22% in 2018. This is vastly smaller than the 73% who use YouTube, and the 69% who use Facebook. The only social media network that was less popular in the U.S. was Reddit.
Everything that I've read in the last day seems to say that cheap SMS, the prevalence of iPhones (with iMessage) and the fact that most Americans never travel abroad have all contributed to WhatsApp never catching on.
posted by octothorpe at 4:16 AM on October 5, 2021
But according to the Pew Research Center, the number of adults using Facebook plateaued in 2018, and WhatsApp user number decreased: Only 20% of U.S. adults use WhatsApp in 2019, down from 22% in 2018. This is vastly smaller than the 73% who use YouTube, and the 69% who use Facebook. The only social media network that was less popular in the U.S. was Reddit.
Everything that I've read in the last day seems to say that cheap SMS, the prevalence of iPhones (with iMessage) and the fact that most Americans never travel abroad have all contributed to WhatsApp never catching on.
posted by octothorpe at 4:16 AM on October 5, 2021
That makes sense, but that's also illuminating for me in understanding the dynamics on metafilter whenever the Facebook monopoly comes up as a subject. Most american commenters here must really assume it's a matter of consumer choice. I mean, I'm not the only country where the phone providers have social media data plans that really just include unlimited Facebook, IG, twitter, and WA (YT's a different video package), which makes WA the most effective messaging app worldwide (no international message fees or for MMS either).
posted by cendawanita at 4:43 AM on October 5, 2021 [6 favorites]
posted by cendawanita at 4:43 AM on October 5, 2021 [6 favorites]
Phone plans in the US just give you voice, SMS and data, they don't distinguish between specific apps.
posted by octothorpe at 4:49 AM on October 5, 2021 [4 favorites]
posted by octothorpe at 4:49 AM on October 5, 2021 [4 favorites]
I spent the whole day like loquacious, huddled with my team trying frantically to patch our system to cope with the disappearance of what our past engineers had clearly considered a permanent part of the landscape. We finally deployed our last fix literally minutes before the Facebook SDK came back online.
Icing on the cake was later last night when I decided to unwind with some beat saber and remembered who owns oculus 😭
posted by ook at 5:00 AM on October 5, 2021 [4 favorites]
Icing on the cake was later last night when I decided to unwind with some beat saber and remembered who owns oculus 😭
posted by ook at 5:00 AM on October 5, 2021 [4 favorites]
Phone plans in the US just give you voice, SMS and data, they don't distinguish between specific apps.
We have those too, but it's somewhat seen as the 'well-to-do' plans. Cheaper ones have app-specific data NON-throttling after the first GB (or whatever number) at 4g speed. I'm reading now that in other countries it's even more of a mess. This one tweet is passing on what their mobile provider told them that for Australia the cost of international calls were jacked up and consumers were told to go on WA instead.
No hell is too much for Zuckerberg, and the US govt shouldn't have allowed the purchase of WA to go through.
posted by cendawanita at 5:03 AM on October 5, 2021 [7 favorites]
We have those too, but it's somewhat seen as the 'well-to-do' plans. Cheaper ones have app-specific data NON-throttling after the first GB (or whatever number) at 4g speed. I'm reading now that in other countries it's even more of a mess. This one tweet is passing on what their mobile provider told them that for Australia the cost of international calls were jacked up and consumers were told to go on WA instead.
No hell is too much for Zuckerberg, and the US govt shouldn't have allowed the purchase of WA to go through.
posted by cendawanita at 5:03 AM on October 5, 2021 [7 favorites]
App specific data plans sound like a super monopolistic move that, sadly, individual consumers can’t do much about:( Here in Canada, cell plans are just super expensive compared to the rest of the world, so the lower income folks I know either go without, have non-smart phones still and pre-paid plans (generally sms is much cheaper than talking, so most include a reasonable amount of texting ability at least), or have old smartphones but no plan (or minimal prepaid plan as with the non-smart phones) and just use them where wifi is available. This means using Facebook messenger for a lot of folks who haven’t heard of other options, probably TikTok or something I haven’t even heard of for the younger crowd, or Signal for the lefties I know. But most folks haven’t heard of Signal (or Telegram, or any of the other alternate text messaging options).
posted by eviemath at 5:17 AM on October 5, 2021 [1 favorite]
posted by eviemath at 5:17 AM on October 5, 2021 [1 favorite]
I actually use Signal myself but no one I know uses it so it's totally useless as anything but a generic SMS client.
posted by octothorpe at 5:23 AM on October 5, 2021 [2 favorites]
posted by octothorpe at 5:23 AM on October 5, 2021 [2 favorites]
This tweet is just a screenshot of a Kenyan redditor saying mobile plans in their country all comes with 'free' WhatsApp. SMS and minutes aren't included automatically otoh.
Let's break it up, let's nationalize it, but let's not pretend users in many locales are swimming in choices. For once, as it's actually a 'mere' American company and not the DoD, Americans can do something about it more directly? This is your post-ww2 soft power at work.
posted by cendawanita at 5:50 AM on October 5, 2021 [4 favorites]
Let's break it up, let's nationalize it, but let's not pretend users in many locales are swimming in choices. For once, as it's actually a 'mere' American company and not the DoD, Americans can do something about it more directly? This is your post-ww2 soft power at work.
posted by cendawanita at 5:50 AM on October 5, 2021 [4 favorites]
Oh for heavens sake, you do not need to build an entire Facebook clone in order to stop using Facebook. There are plenty of other social media companies offering services for free, the point is not to rely one one company for everything - a single point of failure.
I can't communicate with friends and family using a tool they don't use. "Just use something else" effectively means "stop talking to anyone you don't see face to face".
So... no it's not a matter of building an entire Facebook clone, it's harder than that: it's a case of migrating an entire userbase of other people, most of whom don't give two shits about the problems with Facebook and their properties.
posted by Dysk at 6:40 AM on October 5, 2021 [13 favorites]
I can't communicate with friends and family using a tool they don't use. "Just use something else" effectively means "stop talking to anyone you don't see face to face".
So... no it's not a matter of building an entire Facebook clone, it's harder than that: it's a case of migrating an entire userbase of other people, most of whom don't give two shits about the problems with Facebook and their properties.
posted by Dysk at 6:40 AM on October 5, 2021 [13 favorites]
Count me as a person for whom this whole brouhaha over Facebook being down felt very U.S.-centric. The Facebook ecosystem just isn't that big in Korea compared to say, Naver or Kakao. Now if Kakao were to go down, this would bring the country to a standstill (literally - they even have a taxi-hailing app, Kakao T). I don't know of any Koreans who don't have a KakaoTalk account, then add in usage of services such as KakaoPay or their online bank KakaoBank, and it's entirely possible for a Korean person's online activities to largely remain within the Kakao ecosystem.
Edited to add: I had to dig through Naver and Nate news sites to find Korean coverage of the Facebook outage, it wasn't exactly top news.
posted by needled at 6:56 AM on October 5, 2021 [4 favorites]
Edited to add: I had to dig through Naver and Nate news sites to find Korean coverage of the Facebook outage, it wasn't exactly top news.
posted by needled at 6:56 AM on October 5, 2021 [4 favorites]
The idea that Facebook is US-centric is hilarious. It (and specifically WhatsApp) is the most global Internet product there is. As the Rohingya can attest, for one terrible example. China is the big exception to Facebook dominance. Kakao is unusual; in most countries that's Facebook or WeChat.
For folks who experienced other weird outages yesterday: there were reports of global DNS problems as various resolvers got overwhelmed by traffic for failed facebook.com addresses. I don't have a link but I remember Cloudflare having a bit about it.
posted by Nelson at 7:52 AM on October 5, 2021 [5 favorites]
For folks who experienced other weird outages yesterday: there were reports of global DNS problems as various resolvers got overwhelmed by traffic for failed facebook.com addresses. I don't have a link but I remember Cloudflare having a bit about it.
posted by Nelson at 7:52 AM on October 5, 2021 [5 favorites]
Needled: Interesting. What’s the Korean version of WhatsApp?
posted by Tell Me No Lies at 7:53 AM on October 5, 2021
posted by Tell Me No Lies at 7:53 AM on October 5, 2021
That's a great map. WeChat has less global reach than I thought! Whole lotta people though.
posted by Nelson at 7:58 AM on October 5, 2021
posted by Nelson at 7:58 AM on October 5, 2021
So... no it's not a matter of building an entire Facebook clone, it's harder than that: it's a case of migrating an entire userbase of other people, most of whom don't give two shits about the problems with Facebook and their properties.
Imagine if telephones worked, but only if the person you called used the same brand of phone. No one would accept that.
This is not a hard problem to solve per se. We're on a site, for instance, that we all read via a shared, open protocol.
Rather, if Facebook and its properties genuinely make up a communication utility that people are dependent upon, as some seem to argue, then the government needs to set up and enforce the same rules it sets up for other vital utilities.
posted by They sucked his brains out! at 7:59 AM on October 5, 2021 [2 favorites]
Imagine if telephones worked, but only if the person you called used the same brand of phone. No one would accept that.
This is not a hard problem to solve per se. We're on a site, for instance, that we all read via a shared, open protocol.
Rather, if Facebook and its properties genuinely make up a communication utility that people are dependent upon, as some seem to argue, then the government needs to set up and enforce the same rules it sets up for other vital utilities.
posted by They sucked his brains out! at 7:59 AM on October 5, 2021 [2 favorites]
Facebook being down interfered with my daily nap because it was just too much twitter fun.
posted by srboisvert at 8:01 AM on October 5, 2021 [3 favorites]
posted by srboisvert at 8:01 AM on October 5, 2021 [3 favorites]
Phone plans in the US just give you voice, SMS and data, they don't distinguish between specific apps.
This is not true. I have unlimited data these days so it doesn't matter to me but T-Mobile has a specific feature that offers unlimited streaming for specific apps called Binge-On on specific plans.
posted by srboisvert at 8:06 AM on October 5, 2021 [2 favorites]
This is not true. I have unlimited data these days so it doesn't matter to me but T-Mobile has a specific feature that offers unlimited streaming for specific apps called Binge-On on specific plans.
posted by srboisvert at 8:06 AM on October 5, 2021 [2 favorites]
I don't have a link but I remember Cloudflare having a bit about it.
The full Cloudflare analysis of the outage (from their perspective, as indicated by their own tools) says that they were seeing 30 times the usual number of DNS queries for Facebook-owned domains. They attribute some of that to human behavior and some to software:
posted by fedward at 8:14 AM on October 5, 2021 [2 favorites]
The full Cloudflare analysis of the outage (from their perspective, as indicated by their own tools) says that they were seeing 30 times the usual number of DNS queries for Facebook-owned domains. They attribute some of that to human behavior and some to software:
Now human behavior and application logic kicks in and causes another exponential effect. A tsunami of additional DNS traffic follows.I haven't looked to see if anybody else has analyzed it in depth but those human and software factors were going to be pretty consistent no matter what resolvers they use.
This happened in part because apps won't accept an error for an answer and start retrying, sometimes aggressively, and in part because end-users also won't take an error for an answer and start reloading the pages, or killing and relaunching their apps, sometimes also aggressively.
posted by fedward at 8:14 AM on October 5, 2021 [2 favorites]
Rather, if Facebook and its properties genuinely make up a communication utility that people are dependent upon, as some seem to argue, then the government needs to set up and enforce the same rules it sets up for other vital utilities.
I do not think literally one single person on this entire thread disagrees with this. Truly do not understand how people who continue to use Facebook and WA and IG, or even just point out the difficulty of extricating oneself from them, are being characterized here--the assumptions being made are getting kind of gross in addition to being off-base.
posted by We put our faith in Blast Hardcheese at 8:26 AM on October 5, 2021 [7 favorites]
I do not think literally one single person on this entire thread disagrees with this. Truly do not understand how people who continue to use Facebook and WA and IG, or even just point out the difficulty of extricating oneself from them, are being characterized here--the assumptions being made are getting kind of gross in addition to being off-base.
posted by We put our faith in Blast Hardcheese at 8:26 AM on October 5, 2021 [7 favorites]
I came to bury Facebook, not to praise it.
posted by mazola at 8:28 AM on October 5, 2021 [3 favorites]
posted by mazola at 8:28 AM on October 5, 2021 [3 favorites]
> Truly do not understand how people who continue to use Facebook and WA and IG, or even just point out the difficulty of extricating oneself from them, are being characterized her
Yeah. I use a shitty system because it's the best I've got for what I need to do, but that doesn't make me a shitty person.
posted by The corpse in the library at 8:29 AM on October 5, 2021 [3 favorites]
Yeah. I use a shitty system because it's the best I've got for what I need to do, but that doesn't make me a shitty person.
posted by The corpse in the library at 8:29 AM on October 5, 2021 [3 favorites]
Rather, if Facebook and its properties genuinely make up a communication utility that people are dependent upon, as some seem to argue, then the government needs to set up and enforce the same rules it sets up for other vital utilities.
It's never going to happen, which is why I wrote above that it's "too big to fail". Just like the big banks, Facebook will probably not be significantly regulated or broken up by the government (and which government would do it, since it's global!) because it's built a machine that's so interconnected and so complicated that the only Facebook itself can manage it, and even then it just manages to do a less bad job than the alternatives. And FB will also be pretty good at lobbying to keep itself intact too.
posted by FJT at 8:30 AM on October 5, 2021
It's never going to happen, which is why I wrote above that it's "too big to fail". Just like the big banks, Facebook will probably not be significantly regulated or broken up by the government (and which government would do it, since it's global!) because it's built a machine that's so interconnected and so complicated that the only Facebook itself can manage it, and even then it just manages to do a less bad job than the alternatives. And FB will also be pretty good at lobbying to keep itself intact too.
posted by FJT at 8:30 AM on October 5, 2021
Correction: Not above, but in the other Facebook discussion going on. Kinda hard to keep them separate now to be honest.
posted by FJT at 8:34 AM on October 5, 2021
posted by FJT at 8:34 AM on October 5, 2021
> What’s the Korean version of WhatsApp?
KakaoTalk. It's interesting the dominance of KakaoTalk in Korea led Naver to take their messaging application to Japan, renamed as LINE and became dominant in Japan.
Between China, Japan, and Korea (as well as Taiwan, another country where LINE is popular), that's a large number of users who are using something other than WhatsApp.
posted by needled at 8:49 AM on October 5, 2021 [4 favorites]
KakaoTalk. It's interesting the dominance of KakaoTalk in Korea led Naver to take their messaging application to Japan, renamed as LINE and became dominant in Japan.
Between China, Japan, and Korea (as well as Taiwan, another country where LINE is popular), that's a large number of users who are using something other than WhatsApp.
posted by needled at 8:49 AM on October 5, 2021 [4 favorites]
I can't communicate with friends and family using a tool they don't use. "Just use something else" effectively means "stop talking to anyone you don't see face to face".
As an American who spends a lot of time overseas but still has family in the US that I'm close with, this is my biggest issue as well. Most of my family are not at all tech-savvy, just getting them to learn WhatsApp and FB Messenger was a major challenge (I don't use Apple so iMessage isn't an option). I'm not nuts about FB but if that's the most viable option for staying in touch, it's what I'm going to use.
posted by photo guy at 8:49 AM on October 5, 2021 [4 favorites]
As an American who spends a lot of time overseas but still has family in the US that I'm close with, this is my biggest issue as well. Most of my family are not at all tech-savvy, just getting them to learn WhatsApp and FB Messenger was a major challenge (I don't use Apple so iMessage isn't an option). I'm not nuts about FB but if that's the most viable option for staying in touch, it's what I'm going to use.
posted by photo guy at 8:49 AM on October 5, 2021 [4 favorites]
And FB will also be pretty good at lobbying to keep itself intact too.
They have already laid out lobbyist tendrils to be involved in the regulatory process, if that ever plays out to something that Congress one day acts upon. This self-regulation approach is the along the same lines of the "stakeholder" language that allowed Boeing to make and sell death planes, as it happens.
posted by They sucked his brains out! at 8:52 AM on October 5, 2021
They have already laid out lobbyist tendrils to be involved in the regulatory process, if that ever plays out to something that Congress one day acts upon. This self-regulation approach is the along the same lines of the "stakeholder" language that allowed Boeing to make and sell death planes, as it happens.
posted by They sucked his brains out! at 8:52 AM on October 5, 2021
I had some vague idea that it was another chat system that FB owned but I've never met anyone who uses it.
As of the most recent statistics I found, two billion (with a B) people worldwide use WhatsApp. Per Statcounter, 72% of smartphone users worldwide are on Android, and WhatsApp is the dominant Android messaging platform. The iPhone has only a 26.75% share of the global market. By comparison the iPhone has 57% market share in the US (Android is at 42.59%). It seems Facebook Messenger is the dominant cross-platform messaging service in the US, but those statistics are a little old. (Those statistics also don't cover use cases like my teen niece whose "group text" niche is filled by messages on Instagram, but that's also owned by Facebook).
In short: for people who don't use iMessage (exclusive to Apple devices) the two most popular messaging platforms (global and US-centric) are both owned by Facebook. My relatives in Europe all own iPhones but primarily use WhatsApp because it's easier to use the app everybody else already uses than live in the Apple walled garden. I still send them all happy birthday texts on iMessage but I know I'm the outlier.
posted by fedward at 9:01 AM on October 5, 2021 [10 favorites]
As of the most recent statistics I found, two billion (with a B) people worldwide use WhatsApp. Per Statcounter, 72% of smartphone users worldwide are on Android, and WhatsApp is the dominant Android messaging platform. The iPhone has only a 26.75% share of the global market. By comparison the iPhone has 57% market share in the US (Android is at 42.59%). It seems Facebook Messenger is the dominant cross-platform messaging service in the US, but those statistics are a little old. (Those statistics also don't cover use cases like my teen niece whose "group text" niche is filled by messages on Instagram, but that's also owned by Facebook).
In short: for people who don't use iMessage (exclusive to Apple devices) the two most popular messaging platforms (global and US-centric) are both owned by Facebook. My relatives in Europe all own iPhones but primarily use WhatsApp because it's easier to use the app everybody else already uses than live in the Apple walled garden. I still send them all happy birthday texts on iMessage but I know I'm the outlier.
posted by fedward at 9:01 AM on October 5, 2021 [10 favorites]
While iMessage only runs on Apple devices, I'm able to send texts and photos to my Android-using parents (and receive them). WhatsApp doesn't need to be a closed system, it is just engineered to be one.
posted by They sucked his brains out! at 9:26 AM on October 5, 2021 [3 favorites]
posted by They sucked his brains out! at 9:26 AM on October 5, 2021 [3 favorites]
That's fine as long as SMS doesn't incur additional costs, but for many people worldwide SMS isn't free, or it comes with limits or extra charges for sending or receiving messages "internationally" (whether that means someone is physically roaming outside their phone's country, or just texting a friend whose phone is registered in a different country). Services like WhatsApp and iMessage may be included for "free" in a plan (as discussed above) and will work over wifi. Those factors might not be important to somebody in the US who has unlimited texting and/or never travels internationally, but they're important to two billion people worldwide counting only WhatsApp users and not people on other platforms.
posted by fedward at 9:39 AM on October 5, 2021 [4 favorites]
posted by fedward at 9:39 AM on October 5, 2021 [4 favorites]
This is not true. I have unlimited data these days so it doesn't matter to me but T-Mobile has a specific feature that offers unlimited streaming for specific apps called Binge-On on specific plans.
This is quite different from data plans I've seen in other countries that literally only allow access to certain social media properties. Sometimes it's literally just Facebook properties (these are often super cheap because Facebook subsidizes them), sometimes they include Twitter or even various Google properties other than YouTube. They're popular because they are much cheaper than even the smallest "open" data plans available and tend to be unlimited.
And as noted by others, WhatsApp is huge because international texts and calls are unaffordably expensive in much of the world. Hell, for many carriers and plans that's true here in the US as well, but most of us do little to no international calling/texting. Plus, if you've got one of those Facebook-subsidized unlimited data plans you're a hell of a lot better off using WhatsApp for everything rather than paying per minute or per text. (Even without that, it's probably cheaper to use WhatsApp, it's just not as stark a difference) Unlimited text and calls aren't so common elsewhere like they are here.
That said, none of that has any bearing on the choices most people in the US have regarding Facebook itself. People in the US have entirely different reasons for using it, often but not exclusively due to sunk costs.
posted by wierdo at 10:16 AM on October 5, 2021
This is quite different from data plans I've seen in other countries that literally only allow access to certain social media properties. Sometimes it's literally just Facebook properties (these are often super cheap because Facebook subsidizes them), sometimes they include Twitter or even various Google properties other than YouTube. They're popular because they are much cheaper than even the smallest "open" data plans available and tend to be unlimited.
And as noted by others, WhatsApp is huge because international texts and calls are unaffordably expensive in much of the world. Hell, for many carriers and plans that's true here in the US as well, but most of us do little to no international calling/texting. Plus, if you've got one of those Facebook-subsidized unlimited data plans you're a hell of a lot better off using WhatsApp for everything rather than paying per minute or per text. (Even without that, it's probably cheaper to use WhatsApp, it's just not as stark a difference) Unlimited text and calls aren't so common elsewhere like they are here.
That said, none of that has any bearing on the choices most people in the US have regarding Facebook itself. People in the US have entirely different reasons for using it, often but not exclusively due to sunk costs.
posted by wierdo at 10:16 AM on October 5, 2021
The theory I'm seeing online is that, 12 hours after being exposed by a whistleblower on 60 Minutes, Facebook HQ deliberately took their server offline for six hours to give themselves time to scrub their servers of anything that would be incriminating if they were subpoenaed. They also locked employees out of their offices during this time to keep them from saving any files.
It's a theory that makes sense to me and fits the facts so far as I know/understand them.
posted by orange swan at 10:48 AM on October 5, 2021 [6 favorites]
It's a theory that makes sense to me and fits the facts so far as I know/understand them.
posted by orange swan at 10:48 AM on October 5, 2021 [6 favorites]
The question now is how do we get them to do it again, and permanently this time.
posted by Mr.Encyclopedia at 10:59 AM on October 5, 2021 [3 favorites]
posted by Mr.Encyclopedia at 10:59 AM on October 5, 2021 [3 favorites]
it's almost like having a society function on a quarterly basis results in a lot of just cobbled together crap that barely works because nobody's interested in taking their time to get things right, and the $$ you're dumping into UX design is the only thing convincing society that your gilded turd is actually a big ol nugget of $$ venture capital/wall street goldInscribe this on a building somewhere so folks in future generations can learn from the error of our ways.
posted by blindkoala at 11:56 AM on October 5, 2021 [1 favorite]
The theory I'm seeing online is that, 12 hours after being exposed by a whistleblower on 60 Minutes, Facebook HQ deliberately took their server offline for six hours to give themselves time to scrub their servers of anything that would be incriminating if they were subpoenaed. They also locked employees out of their offices during this time to keep them from saving any files.
It's a theory that makes sense to me and fits the facts so far as I know/understand them.
I'm not prone to conspiracy theories but Facebook has proven again and again that it's not acting in good faith -- which happens to be the very subject of the whistleblower interview. That it controls all of the levers that make such a goof possible certainly doesn't make me feel any more inclined to give it the benefit of the doubt.
In my mind, this should warrant a legal investigation, but I'm not holding out much hope on that. We can't even seem to meaningfully pursue a rogue president who openly attempted a coup and incited a violent insurrection -- and who, apparently, has had a big role in convincing Facebook to be a primary vector of information warfare against the US.
posted by treepour at 12:02 PM on October 5, 2021
It's a theory that makes sense to me and fits the facts so far as I know/understand them.
I'm not prone to conspiracy theories but Facebook has proven again and again that it's not acting in good faith -- which happens to be the very subject of the whistleblower interview. That it controls all of the levers that make such a goof possible certainly doesn't make me feel any more inclined to give it the benefit of the doubt.
In my mind, this should warrant a legal investigation, but I'm not holding out much hope on that. We can't even seem to meaningfully pursue a rogue president who openly attempted a coup and incited a violent insurrection -- and who, apparently, has had a big role in convincing Facebook to be a primary vector of information warfare against the US.
posted by treepour at 12:02 PM on October 5, 2021
While iMessage only runs on Apple devices, I'm able to send texts and photos to my Android-using parents (and receive them). WhatsApp doesn't need to be a closed system, it is just engineered to be one.
Spoken by a person whose life doesn't span borders. Texting and calling internationally is expensive as fuck, and not included in standard plans. Data is, which makes whatsapp and Facebook messenger effectively free. The quality of audio is often significantly better as well, and without the often extreme latency of international phone calls.
And SMS and voice telephony won't do video chat. Or group chat. Nevermind group video chat.
posted by Dysk at 12:13 PM on October 5, 2021 [2 favorites]
Spoken by a person whose life doesn't span borders. Texting and calling internationally is expensive as fuck, and not included in standard plans. Data is, which makes whatsapp and Facebook messenger effectively free. The quality of audio is often significantly better as well, and without the often extreme latency of international phone calls.
And SMS and voice telephony won't do video chat. Or group chat. Nevermind group video chat.
posted by Dysk at 12:13 PM on October 5, 2021 [2 favorites]
Spoken by a person whose life doesn't span borders
Okay. WhatsApp is not free, it is subsidized, just as WeChat is subsidized in China. Here is an overview of the WhatsApp business model that explains how it works. "Free" or subsidized, either way, the WhatsApp platform is engineered to be closed to maintain monopoly control over that business model.
posted by They sucked his brains out! at 12:23 PM on October 5, 2021 [2 favorites]
Okay. WhatsApp is not free, it is subsidized, just as WeChat is subsidized in China. Here is an overview of the WhatsApp business model that explains how it works. "Free" or subsidized, either way, the WhatsApp platform is engineered to be closed to maintain monopoly control over that business model.
posted by They sucked his brains out! at 12:23 PM on October 5, 2021 [2 favorites]
MetaFilter: you can cause widespread chaos with a few wrong keystrokes
posted by Mr. Bad Example at 12:37 PM on October 5, 2021 [3 favorites]
posted by Mr. Bad Example at 12:37 PM on October 5, 2021 [3 favorites]
Imagine if telephones worked, but only if the person you called used the same brand of phone. No one would accept that.
Telephones did work like that at one point, and it sucked, and it's one of the things that led to the centralization of telephone service under the monopoly-regulated Bell Telephone in the US for decades. It's literally a textbook case of goverment regulation of a "natural monopoly".
Perversely though, it's some of those same regulated telephone monopolies, outside of the US, that are leading to Facebook/WhatsApp becoming a de facto public utility. For various reasons, in some cases apparently being completely detached from modern telecommunications reality, those legacy national telco operators charged stupendous rates for international calls and even texts, leading to their abandonment in favor of private Internet-based services.
I'm not sure how you know when a monopoly-regulated utility is the way to go, and when open competition is the way to go; clearly in the US, deregulation of the telecommunications industry in the 80s was pretty beneficial (and is why US users aren't having this discussion while paying per-kilobyte rates using our leased Western Electric ISDN modems from AT&T). But that said, the current cableco+Verizon duopoly seems to contain suspicious aspects of the worst of both regulated and free-market systems.
I think the answer is something like: regulated monopolies work well for common-carrier situations where interoperability is more important than other characteristics of the system, but only if they're regulated in a way that continually takes into account advances in technology and passes those advances on to consumers. Frustration at that last bit not happening is one of the things that led to the breakup of AT&T.
Since most countries don't have much leverage over Facebook, and are unlikely to be consulted by the US as to any regulatory actions over it, the solution has to come from somewhere else.
My feeling is that in places where individuals have any political voice, the best thing to push for would be reform of local telcos and mobile operators, to eliminate the artificially-high charges on plain-old voice and SMS services. Make those cheap enough, and you will eliminate the dependency on a foreign for-profit corporation in favor of entirely locally-owned infrastructure that already exists.
posted by Kadin2048 at 12:37 PM on October 5, 2021 [9 favorites]
Telephones did work like that at one point, and it sucked, and it's one of the things that led to the centralization of telephone service under the monopoly-regulated Bell Telephone in the US for decades. It's literally a textbook case of goverment regulation of a "natural monopoly".
Perversely though, it's some of those same regulated telephone monopolies, outside of the US, that are leading to Facebook/WhatsApp becoming a de facto public utility. For various reasons, in some cases apparently being completely detached from modern telecommunications reality, those legacy national telco operators charged stupendous rates for international calls and even texts, leading to their abandonment in favor of private Internet-based services.
I'm not sure how you know when a monopoly-regulated utility is the way to go, and when open competition is the way to go; clearly in the US, deregulation of the telecommunications industry in the 80s was pretty beneficial (and is why US users aren't having this discussion while paying per-kilobyte rates using our leased Western Electric ISDN modems from AT&T). But that said, the current cableco+Verizon duopoly seems to contain suspicious aspects of the worst of both regulated and free-market systems.
I think the answer is something like: regulated monopolies work well for common-carrier situations where interoperability is more important than other characteristics of the system, but only if they're regulated in a way that continually takes into account advances in technology and passes those advances on to consumers. Frustration at that last bit not happening is one of the things that led to the breakup of AT&T.
Since most countries don't have much leverage over Facebook, and are unlikely to be consulted by the US as to any regulatory actions over it, the solution has to come from somewhere else.
My feeling is that in places where individuals have any political voice, the best thing to push for would be reform of local telcos and mobile operators, to eliminate the artificially-high charges on plain-old voice and SMS services. Make those cheap enough, and you will eliminate the dependency on a foreign for-profit corporation in favor of entirely locally-owned infrastructure that already exists.
posted by Kadin2048 at 12:37 PM on October 5, 2021 [9 favorites]
The theory I'm seeing online is that, 12 hours after being exposed by a whistleblower on 60 Minutes, Facebook HQ deliberately took their server offline for six hours to give themselves time to scrub their servers of anything that would be incriminating if they were subpoenaed. They also locked employees out of their offices during this time to keep them from saving any files.
I'm afraid this doesn't make a lot of sense. Files are going to be a) backed up, b) in local copies spread all over Facebook including a lot of laptops that are offline, and c) almost certainly already copied by the whistleblower.
In addition attempts to do a widespread deletion like that would leave traces all over the place. If someone really did start poking around with a subpoena it would be obvious enough that a large event had taken place.
posted by Tell Me No Lies at 1:40 PM on October 5, 2021 [7 favorites]
I'm afraid this doesn't make a lot of sense. Files are going to be a) backed up, b) in local copies spread all over Facebook including a lot of laptops that are offline, and c) almost certainly already copied by the whistleblower.
In addition attempts to do a widespread deletion like that would leave traces all over the place. If someone really did start poking around with a subpoena it would be obvious enough that a large event had taken place.
posted by Tell Me No Lies at 1:40 PM on October 5, 2021 [7 favorites]
Okay. WhatsApp is not free, it is subsidized, just as WeChat is subsidized in China.
Literally on the first few lines of that article, it says "The app allows users to upload their contact book and message anyone who has the app installed, at no cost."
"Effectively free" is almost an understatement there. It doesn't cost anything to send a message to my parents with WhatsApp. It costs me nearly 30p per text. It costs me nothing to call them on WhatsApp, or video call them on WhatsApp, even both of them at once. I pay a pound for every few minutes if I make a phonecall.
Whatever whatsapp are charging businesses, I don't care. I can talk to my parents, the rest of my family, and my friends abroad, without having to watch the clock, and without incurring myself any expense to myself beyond the data plan I pay for anyway.
I'm sure Facebook are getting paid somehow, by selling data, by advertising, by charging premiums to businesses who want an official presence. But it is free for the end user. The Metro, as an example, is a free newspaper. They get paid too - somebody has an expense to them. Nobody objects to them being called free though. This is no different. There simply is no charge, in pounds and pence, euros and cents, for using WhatsApp. There really fucking is for any traditional telephony across borders.
posted by Dysk at 1:50 PM on October 5, 2021 [7 favorites]
Literally on the first few lines of that article, it says "The app allows users to upload their contact book and message anyone who has the app installed, at no cost."
"Effectively free" is almost an understatement there. It doesn't cost anything to send a message to my parents with WhatsApp. It costs me nearly 30p per text. It costs me nothing to call them on WhatsApp, or video call them on WhatsApp, even both of them at once. I pay a pound for every few minutes if I make a phonecall.
Whatever whatsapp are charging businesses, I don't care. I can talk to my parents, the rest of my family, and my friends abroad, without having to watch the clock, and without incurring myself any expense to myself beyond the data plan I pay for anyway.
I'm sure Facebook are getting paid somehow, by selling data, by advertising, by charging premiums to businesses who want an official presence. But it is free for the end user. The Metro, as an example, is a free newspaper. They get paid too - somebody has an expense to them. Nobody objects to them being called free though. This is no different. There simply is no charge, in pounds and pence, euros and cents, for using WhatsApp. There really fucking is for any traditional telephony across borders.
posted by Dysk at 1:50 PM on October 5, 2021 [7 favorites]
More details about the October 4 outage: the official statement.
It's a good corporate writeup. just enough detail to be interesting, but all projecting calm and competence. "This happened and we know why and hey, we were prepared for it". I guess I have to buy a Facebook engineer a few drinks to get the fun version of the story.
posted by Nelson at 1:58 PM on October 5, 2021 [1 favorite]
It's a good corporate writeup. just enough detail to be interesting, but all projecting calm and competence. "This happened and we know why and hey, we were prepared for it". I guess I have to buy a Facebook engineer a few drinks to get the fun version of the story.
posted by Nelson at 1:58 PM on October 5, 2021 [1 favorite]
I wrote...
I'm afraid this doesn't make a lot of sense. Files are going to be a) backed up, b) in local copies spread all over Facebook including a lot of laptops that are offline, and c) almost certainly already copied by the whistleblower.
and d) even if you managed to perfectly delete everything, people know that the files exist(ed) and all it takes is one to testify that they all suddenly disappeared.
posted by Tell Me No Lies at 2:10 PM on October 5, 2021 [1 favorite]
I'm afraid this doesn't make a lot of sense. Files are going to be a) backed up, b) in local copies spread all over Facebook including a lot of laptops that are offline, and c) almost certainly already copied by the whistleblower.
and d) even if you managed to perfectly delete everything, people know that the files exist(ed) and all it takes is one to testify that they all suddenly disappeared.
posted by Tell Me No Lies at 2:10 PM on October 5, 2021 [1 favorite]
Just to back up Dysk's point, I'm in the UK and have family in France.
It costs me 20p (27c) to send a text to them, 65p (89c) a minute to call, and 65p (89c) to send an MMS. Three (my UK network) will only deliver the last of those if it's to a number on Orange, Bouygues or Free Mobile.
WhatsApp might be the 'nobody really uses that' property in North America, but for a lot of the world this was essentially a major telecoms carrier going down.
Completely unscientific, but the tweets from all three accounts:
Facebook: 47.4K Retweets 23.3K Quote Tweets 174.6K Likes
Instagram: 21.8K Retweets 8,772 Quote Tweets 107.9K Likes
WhatsApp: 86.1K Retweets 40.9K Quote Tweets 332.2K Likes
Incidentally, WhatsApp is the only one of the three to put out comms in Portuguese and Spanish as well.
posted by MattWPBS at 3:10 PM on October 5, 2021 [3 favorites]
It costs me 20p (27c) to send a text to them, 65p (89c) a minute to call, and 65p (89c) to send an MMS. Three (my UK network) will only deliver the last of those if it's to a number on Orange, Bouygues or Free Mobile.
WhatsApp might be the 'nobody really uses that' property in North America, but for a lot of the world this was essentially a major telecoms carrier going down.
Completely unscientific, but the tweets from all three accounts:
Facebook: 47.4K Retweets 23.3K Quote Tweets 174.6K Likes
Instagram: 21.8K Retweets 8,772 Quote Tweets 107.9K Likes
WhatsApp: 86.1K Retweets 40.9K Quote Tweets 332.2K Likes
Incidentally, WhatsApp is the only one of the three to put out comms in Portuguese and Spanish as well.
posted by MattWPBS at 3:10 PM on October 5, 2021 [3 favorites]
I was in Spain for a project a couple years back. Two of us were from the US, everyone else were from various European countries, WhatsApp was the texting solution of choice.
Also, I was in the UK last month, and was told not to use iMessage as it didn't work properly with some carriers for group messages.
posted by CheeseDigestsAll at 3:31 PM on October 5, 2021 [1 favorite]
Also, I was in the UK last month, and was told not to use iMessage as it didn't work properly with some carriers for group messages.
posted by CheeseDigestsAll at 3:31 PM on October 5, 2021 [1 favorite]
I'm on Team Intentional. You don't just switch off BGP, not the very least because as mentioned, it's low maintenance. I don't know FB, it's certainly possible they put themselves in a fatfinger position on a Monday morning, but in the real world the things that BGP was designed to handle doesn't change very often. Plus the config is high security shit and there are usually relatively few people with the privileges to change it. It wasn't that floor's Local Mouse Replacer who tripped and elbowed a keyboard. A lot of speculation, sure, but I believe that they are indeed that bad.
The theory I'm seeing online is that, 12 hours after being exposed by a whistleblower on 60 Minutes, Facebook HQ deliberately took their server offline for six hours to give themselves time to scrub their servers of anything that would be incriminating if they were subpoenaed.
I thought the 60 Minutes Haugen interview was the sole motivator and Zuck was just pouting to drown out the news cycle, but yesterday also saw something potentially larger: the release of her 8 SEC complaints against FB became available.
Bonus Reading: Facebook acts like a hostile foreign power.
posted by rhizome at 3:33 PM on October 5, 2021 [4 favorites]
The theory I'm seeing online is that, 12 hours after being exposed by a whistleblower on 60 Minutes, Facebook HQ deliberately took their server offline for six hours to give themselves time to scrub their servers of anything that would be incriminating if they were subpoenaed.
I thought the 60 Minutes Haugen interview was the sole motivator and Zuck was just pouting to drown out the news cycle, but yesterday also saw something potentially larger: the release of her 8 SEC complaints against FB became available.
Bonus Reading: Facebook acts like a hostile foreign power.
posted by rhizome at 3:33 PM on October 5, 2021 [4 favorites]
Is there a reason email isn't really considered an alternative to proprietary messaging apps?
posted by Television Name at 4:35 PM on October 5, 2021 [1 favorite]
posted by Television Name at 4:35 PM on October 5, 2021 [1 favorite]
Is there a reason email isn't really considered an alternative to proprietary messaging apps?
They each fill their own niche. You could press email into service as a message app but the underlying protocol wasn't really designed for it.
posted by Tell Me No Lies at 4:54 PM on October 5, 2021 [2 favorites]
They each fill their own niche. You could press email into service as a message app but the underlying protocol wasn't really designed for it.
posted by Tell Me No Lies at 4:54 PM on October 5, 2021 [2 favorites]
Is there a reason email isn't really considered an alternative to proprietary messaging apps?
Poor threading (especially for group chats, and doubly so if anyone forgets to use reply all instead of reply even once), less immediate delivery (depending on provider), and no facility for voice chat or video chat, just off the top of my head.
posted by Dysk at 5:02 PM on October 5, 2021 [5 favorites]
Poor threading (especially for group chats, and doubly so if anyone forgets to use reply all instead of reply even once), less immediate delivery (depending on provider), and no facility for voice chat or video chat, just off the top of my head.
posted by Dysk at 5:02 PM on October 5, 2021 [5 favorites]
but in the real world the things that BGP was designed to handle doesn't change very often.
Ok, maybe, but what if you could move fast (and break things) with BGP? Wouldn't that be nice? 😬
posted by Nonsteroidal Anti-Inflammatory Drug at 5:09 PM on October 5, 2021 [2 favorites]
Ok, maybe, but what if you could move fast (and break things) with BGP? Wouldn't that be nice? 😬
In this paper, we present Facebook's BGP-based data center routing design and how it marries data center's stringent requirements with BGP's functionality. We present the design's significant artifacts, including the BGP Autonomous System Number (ASN) allocation, route summarization, and our sophisticated BGP policy set. We demonstrate how this design provides us with flexible control over routing and keeps the network reliable. We also describe our in-house BGP software implementation, and its testing and deployment pipelines. These allow us to treat BGP like any other software component, enabling fast incremental updates.-Running BGP in Data Centers at Scale
posted by Nonsteroidal Anti-Inflammatory Drug at 5:09 PM on October 5, 2021 [2 favorites]
(also, having previously run a BGP AS, I would disagree that BGP isn't meant to change often. It's how you indicate the physical connections of the internet. If routes didn't change, we wouldn't have all these autonomous systems figuring out how to talk to each other. The entire reason I was assigned an ASN was so that I could indicate how to connect to our handful of servers when the main fiber line went down.)
posted by Nonsteroidal Anti-Inflammatory Drug at 5:14 PM on October 5, 2021 [1 favorite]
posted by Nonsteroidal Anti-Inflammatory Drug at 5:14 PM on October 5, 2021 [1 favorite]
Ok, maybe, but what if you could move fast (and break things) with BGP? Wouldn't that be nice?
I think I speak for everyone when I say "No”. :-)
posted by Tell Me No Lies at 6:04 PM on October 5, 2021 [1 favorite]
I think I speak for everyone when I say "No”. :-)
posted by Tell Me No Lies at 6:04 PM on October 5, 2021 [1 favorite]
Is there a reason email isn't really considered an alternative to proprietary messaging apps?
This is at the forefront of my mind every time I get yet another work document in my WA. email is mobile too, but readability is poor (in terms of quickly digesting the info and attachments), is what I'm being told. I've got issues with all the accruing work culture but it's undoubtedly a fact the public and private sector ppl i encounter would rather send screenshots of documents to be reviewed rather than pdfs or doc-type files. It messes with my internal tracking but it IS faster in getting info out.
posted by cendawanita at 8:19 PM on October 5, 2021
This is at the forefront of my mind every time I get yet another work document in my WA. email is mobile too, but readability is poor (in terms of quickly digesting the info and attachments), is what I'm being told. I've got issues with all the accruing work culture but it's undoubtedly a fact the public and private sector ppl i encounter would rather send screenshots of documents to be reviewed rather than pdfs or doc-type files. It messes with my internal tracking but it IS faster in getting info out.
posted by cendawanita at 8:19 PM on October 5, 2021
Oh and that's related to mobile penetration exceeding 100% of the population in many places. Basically for a lot of ppl, laptops are still higher-tier purchases. A basic smartphone (especially in markets where iphone is also an expensive purchase, esp with entry-level androids being dirt cheap with the aforementioned app-specific plans) or an Android tablet is usually the only tech device on hand. Emails aren't really optimised for mobile devices the way messaging and socmed apps are. Not to mention for a substantial number of ppl, having a phone number is inevitable, setting up an email address isn't.
Especially in this pandemic you can really see this gap where small businesses and schools needed to go remote and virtual. And i don't know how much Americans are aware how a lot of other jurisdictions have pushed for digital financial services to the extent it's still unimaginable in the USA. WA tends to be a key support tool in the notifications etc.
posted by cendawanita at 8:32 PM on October 5, 2021 [1 favorite]
Especially in this pandemic you can really see this gap where small businesses and schools needed to go remote and virtual. And i don't know how much Americans are aware how a lot of other jurisdictions have pushed for digital financial services to the extent it's still unimaginable in the USA. WA tends to be a key support tool in the notifications etc.
posted by cendawanita at 8:32 PM on October 5, 2021 [1 favorite]
Another tangential point: talking about the pandemic also reminded me how our systems really optimised for mobile phone use over other digital devices. MANY MANY other countries associate our vaccination status (and just general COVID-19 related services) with our phone numbers via apps. My country is not the only one where paper vaccination cards are considered as mere backup (and if this is properly communicated in the public arena. Plenty of stories of ppl being turned away because they can't show their vaccination status on 'the app'). This penalises those who aren't on smartphones and there's a massive push to get them with phones.
Now, imagine, what kind of plans can they afford for these phones?
posted by cendawanita at 8:40 PM on October 5, 2021 [1 favorite]
Now, imagine, what kind of plans can they afford for these phones?
posted by cendawanita at 8:40 PM on October 5, 2021 [1 favorite]
If texting is playing hot potato, emailing is gift wrapping that potato, handing it to someone and watching them unwrap it.
posted by skyscraper at 8:41 PM on October 5, 2021 [3 favorites]
posted by skyscraper at 8:41 PM on October 5, 2021 [3 favorites]
clearly in the US, deregulation of the telecommunications industry in the 80s was pretty beneficial (and is why US users aren't having this discussion while paying per-kilobyte rates using our leased Western Electric ISDN modems from AT&T). But that said, the current cableco+Verizon duopoly seems to contain suspicious aspects of the worst of both regulated and free-market systems.
It's so much better leasing your DSL or fiber router from at&t. Having finally been deployed a decade after Carterfone, there was never going to be forced leasing of the telco's ISDN equipment. It's just too bad that the FCC under Bush decided that the physical part of an Internet connection could be considered an information service and therefore not subject to the usual regulations and nobody has considered revisiting that decision since, despite it flying in the face of the plain language of the 1996 Telecommunications Act.
I could go on about the double edged sword of deregulation and how it was good for businesses and less so for consumers, but this isn't really the place.
BTW, email works just fine for one to one chats, but not so much for groups. It takes about the same amount of time for my shitty mail server to deliver a message as it does for WhatsApp to deliver a message. But again, Facebook subsidized data plans make that a nonstarter for a lot of people and WhatsApp isn't really problematic in and of itself anyway.
posted by wierdo at 9:13 PM on October 5, 2021
It's so much better leasing your DSL or fiber router from at&t. Having finally been deployed a decade after Carterfone, there was never going to be forced leasing of the telco's ISDN equipment. It's just too bad that the FCC under Bush decided that the physical part of an Internet connection could be considered an information service and therefore not subject to the usual regulations and nobody has considered revisiting that decision since, despite it flying in the face of the plain language of the 1996 Telecommunications Act.
I could go on about the double edged sword of deregulation and how it was good for businesses and less so for consumers, but this isn't really the place.
BTW, email works just fine for one to one chats, but not so much for groups. It takes about the same amount of time for my shitty mail server to deliver a message as it does for WhatsApp to deliver a message. But again, Facebook subsidized data plans make that a nonstarter for a lot of people and WhatsApp isn't really problematic in and of itself anyway.
posted by wierdo at 9:13 PM on October 5, 2021
Jim Wright explains in some detail how valuable Facebook is for people who have mobility problems, hearing problems, vision problems (this is a *lot* of older people in particular), various other difficulties in making contact....
https://www.stonekettle.com/2021/10/recap-october-4-2021.html?m=1
He gets into more detail about how setting up Zoom is difficult for some older people, and that a lot of people don't have the physical bandwidth for it.
I will also note that Zoom has tiny text for chat.
The August before the pandemic, I contacted them about the value of being able to increase the font size in chat, and they said they might be getting to it in a year. I've assumed they were busy, but I guess I should prod them again.
posted by Nancy Lebovitz at 4:25 AM on October 6, 2021 [9 favorites]
https://www.stonekettle.com/2021/10/recap-october-4-2021.html?m=1
He gets into more detail about how setting up Zoom is difficult for some older people, and that a lot of people don't have the physical bandwidth for it.
I will also note that Zoom has tiny text for chat.
The August before the pandemic, I contacted them about the value of being able to increase the font size in chat, and they said they might be getting to it in a year. I've assumed they were busy, but I guess I should prod them again.
posted by Nancy Lebovitz at 4:25 AM on October 6, 2021 [9 favorites]
Perversely though, it's some of those same regulated telephone monopolies, outside of the US, that are leading to Facebook/WhatsApp becoming a de facto public utility. For various reasons, in some cases apparently being completely detached from modern telecommunications reality, those legacy national telco operators charged stupendous rates for international calls and even texts, leading to their abandonment in favor of private Internet-based services.
Kadin, you might find it interesting to look into Canada's situation. The phone/internet/mobile/cableTV rates are very high relative to the US because Canada never had even the limited US style Bell deregulation and has all the telecoms dominated by just a couple of companies. I've been out of country for too long to know the details anymore but I think it'd be a really interesting comparison of market & regulation effects across pretty similar economies.
posted by srboisvert at 7:12 AM on October 6, 2021 [1 favorite]
Kadin, you might find it interesting to look into Canada's situation. The phone/internet/mobile/cableTV rates are very high relative to the US because Canada never had even the limited US style Bell deregulation and has all the telecoms dominated by just a couple of companies. I've been out of country for too long to know the details anymore but I think it'd be a really interesting comparison of market & regulation effects across pretty similar economies.
posted by srboisvert at 7:12 AM on October 6, 2021 [1 favorite]
Nancy Leibovitz, I came here to share the same link.
I’m not elderly but I’ve been very cautious with in person social activity. FB has helped me stay sane. It sucks in so many ways that desperately need to be addressed, but it’s also how I find out that my friends need help, or have a new kitten, organize projects, engage in activism, etc.
Maybe this has been discussed upthread, but it would be interesting to see a conversation on what could address the risks/issues FB or a similar platform while keeping - or improving - the things that work well
posted by bunderful at 11:31 AM on October 6, 2021 [3 favorites]
I’m not elderly but I’ve been very cautious with in person social activity. FB has helped me stay sane. It sucks in so many ways that desperately need to be addressed, but it’s also how I find out that my friends need help, or have a new kitten, organize projects, engage in activism, etc.
Maybe this has been discussed upthread, but it would be interesting to see a conversation on what could address the risks/issues FB or a similar platform while keeping - or improving - the things that work well
posted by bunderful at 11:31 AM on October 6, 2021 [3 favorites]
Is there a reason email isn't really considered an alternative to proprietary messaging apps?
also security and delivery issues, which are ultimately tied to one another. it's really easy to alter your email headers and say 'hey, my name is admin@gmail.com' so what most email service providers (ESPs) do now involves delaying emails at initial mail transfer agent (MTA), screening that email which involves pinging safelisting sites like SpamHaus to see if the originating IP is a suspect one, scanning each and every link (Apple has a setting now that does this as a defauilt for every single email), pinging the registrar of the originating domain for DKIM or SPF authentication (often both), and a lot of other things, all of which involves this back and forth communication via SMTP about delivery statuses, etc, and this is all prior to pushing the email to the internal relay servers which are ultimately responsible for depositing the email into the right inbox
so you end up with 1) a lot more failures in delivery, 2) significant delays in delivery of up to multiple hours, and 3) needing more data exchange for transmission as a default on top of everything else folks have listed above
Whatsapp/other platforms are easier to manage because it's all on one platform, managed by one provider who have things like identification/etc locked down and tied to phone numbers and communication restricted only to explicitly listed contacts at the individual level
posted by paimapi at 11:34 AM on October 6, 2021 [2 favorites]
also security and delivery issues, which are ultimately tied to one another. it's really easy to alter your email headers and say 'hey, my name is admin@gmail.com' so what most email service providers (ESPs) do now involves delaying emails at initial mail transfer agent (MTA), screening that email which involves pinging safelisting sites like SpamHaus to see if the originating IP is a suspect one, scanning each and every link (Apple has a setting now that does this as a defauilt for every single email), pinging the registrar of the originating domain for DKIM or SPF authentication (often both), and a lot of other things, all of which involves this back and forth communication via SMTP about delivery statuses, etc, and this is all prior to pushing the email to the internal relay servers which are ultimately responsible for depositing the email into the right inbox
so you end up with 1) a lot more failures in delivery, 2) significant delays in delivery of up to multiple hours, and 3) needing more data exchange for transmission as a default on top of everything else folks have listed above
Whatsapp/other platforms are easier to manage because it's all on one platform, managed by one provider who have things like identification/etc locked down and tied to phone numbers and communication restricted only to explicitly listed contacts at the individual level
posted by paimapi at 11:34 AM on October 6, 2021 [2 favorites]
What happened on the Internet during the Facebook outage. An excellent analysis from Cloudflare's perspective. Some highlights:
- Cloudflare's DNS traffic went up 3x
- A map of which countries made the most Facebook requests
- A measurement of increased traffic on news sites, other social media, etc
Nice link, Nelson. Just for the sake of posterity in case that article disappears, here are the top 10 Facebook-using countries, based on the ratio of SERVFAIL to normal DNS responses during the outage period:
If I'm doing my napkin math correctly, that indicates that something like 88% of Turkey's DNS lookups were Facebook-related.
Maybe the US has more leverage over Ergodan than it knows, if it were to nationalize Facebook...
posted by Kadin2048 at 10:39 AM on October 8, 2021 [2 favorites]
Country Ratio Turkey 7.34 Grenada 4.84 Congo 4.44 Lesotho 3.94 Nicaragua 3.57 South Sudan 3.47 Syria 3.41 Serbia 3.25 Turkmenistan 3.23 UAE 3.17 Togo 3.14 French Guiana 3.00I don't know about anyone else, but I would not have guessed that zero countries from North America or Europe would be in the top ten. (Well, I guess Turkey being in "Europe" is debatable, but Wikipedia says it's mostly located in "Western Asia" so I'm going with it.)
If I'm doing my napkin math correctly, that indicates that something like 88% of Turkey's DNS lookups were Facebook-related.
Maybe the US has more leverage over Ergodan than it knows, if it were to nationalize Facebook...
posted by Kadin2048 at 10:39 AM on October 8, 2021 [2 favorites]
I would disagree that BGP isn't meant to change often. It's how you indicate the physical connections of the internet. If routes didn't change, we wouldn't have all these autonomous systems figuring out how to talk to each other.
My understanding is that the BGP configuration is what doesn't change very often. Of course the routes are going to change and be changed by BGP, that's its mission in life!
We can speculate about stray semicolons or whatever could get fatfingered into the config, or, and I think Occam would agree, we can have a pretty good, better idea that the people who have the permissions to change this stuff are less likely to make a dumb mistake, and as it turns out in one of FB's explanations:
During one of these routine maintenance jobs, a command was issued with the intention to assess the availability of global backbone capacity, which unintentionally took down all the connections in our backbone network, effectively disconnecting Facebook data centers globally. Our systems are designed to audit commands like these to prevent mistakes like this, but a bug in that audit tool prevented it from properly stopping the command.
Then later, in the recovery phase...
this took time, because these facilities are designed with high levels of physical and system security in mind. They’re hard to get into, and once you’re inside, the hardware and routers are designed to be difficult to modify even when you have physical access to them.
"Even when you have physical access" ...in addition to being difficult when modifying remotely. Also notice the mention of a command-audit tool that apparently didn't include the problem command, implying the issuer had a level of knowledge above that of the tool-maker(s). Note also that this was to get a reading on capacity? They didn't have a pre-existing tool or monitoring process for that? Unlikely, and weird to me regardless.
This is all Skip Bayless-quality analysis, but for these reasons I'm still calling it a fog machine.
posted by rhizome at 4:22 PM on October 8, 2021
My understanding is that the BGP configuration is what doesn't change very often. Of course the routes are going to change and be changed by BGP, that's its mission in life!
We can speculate about stray semicolons or whatever could get fatfingered into the config, or, and I think Occam would agree, we can have a pretty good, better idea that the people who have the permissions to change this stuff are less likely to make a dumb mistake, and as it turns out in one of FB's explanations:
During one of these routine maintenance jobs, a command was issued with the intention to assess the availability of global backbone capacity, which unintentionally took down all the connections in our backbone network, effectively disconnecting Facebook data centers globally. Our systems are designed to audit commands like these to prevent mistakes like this, but a bug in that audit tool prevented it from properly stopping the command.
Then later, in the recovery phase...
this took time, because these facilities are designed with high levels of physical and system security in mind. They’re hard to get into, and once you’re inside, the hardware and routers are designed to be difficult to modify even when you have physical access to them.
"Even when you have physical access" ...in addition to being difficult when modifying remotely. Also notice the mention of a command-audit tool that apparently didn't include the problem command, implying the issuer had a level of knowledge above that of the tool-maker(s). Note also that this was to get a reading on capacity? They didn't have a pre-existing tool or monitoring process for that? Unlikely, and weird to me regardless.
This is all Skip Bayless-quality analysis, but for these reasons I'm still calling it a fog machine.
posted by rhizome at 4:22 PM on October 8, 2021
« Older Rocket ... Man | "Until a drape of calmness furled around the earth... Newer »
This thread has been archived and is closed to new comments
And then I realized that nope work just transitioned to a new security software last week and probably twitter is just blocked now. Oh well. We can't have all the nice things.
posted by phunniemee at 2:43 PM on October 4, 2021 [4 favorites]