Hello, operator... hello?
November 5, 2014 3:01 AM Subscribe
This is what happens when 911 fails.
"For six hours, emergency services went dark for more than 11 million people across seven states. The entire state of Washington found itself disconnected from 911. The outage may have gone unnoticed by some, but for the more than 6,000 people trying to reach help, April 9 may well have been the scariest time of their lives."
Via the ever-interesting Risks Digest vol 28 issue 33
"For six hours, emergency services went dark for more than 11 million people across seven states. The entire state of Washington found itself disconnected from 911. The outage may have gone unnoticed by some, but for the more than 6,000 people trying to reach help, April 9 may well have been the scariest time of their lives."
Via the ever-interesting Risks Digest vol 28 issue 33
actually thought this was going to be about the several stories ive read about 911 operators being total assholes and hanging up on people for being "too upset".
the universe is infinite in the breadth and width of ways it can suck though, i guess.
posted by emptythought at 4:05 AM on November 5, 2014
the universe is infinite in the breadth and width of ways it can suck though, i guess.
posted by emptythought at 4:05 AM on November 5, 2014
Try Twitter?
posted by sammyo at 4:05 AM on November 5, 2014 [2 favorites]
posted by sammyo at 4:05 AM on November 5, 2014 [2 favorites]
Another story of regulators not properly regulating. If they actually tracked complaints and fined (or whatever) for problems and obliged them to let customers who had 911 problems leave their contract with not only no penalties but a significant "sorry we risked your life" cheque, the companies would suddenly fix all these problems. But since there are absolutely no consequences for them, they don't spend the money.
posted by jeather at 4:23 AM on November 5, 2014 [15 favorites]
posted by jeather at 4:23 AM on November 5, 2014 [15 favorites]
Move fast and break things.
posted by Zarkonnen at 4:31 AM on November 5, 2014 [5 favorites]
posted by Zarkonnen at 4:31 AM on November 5, 2014 [5 favorites]
On a more serious note, I absolutely agree with jeather. Anything other than hefty fines will just be absorbed as cost-of-business or a minor PR incident.
posted by Zarkonnen at 4:33 AM on November 5, 2014
posted by Zarkonnen at 4:33 AM on November 5, 2014
Why exactly should we have to put up with the offloading of vital public services onto organizations that are, definitionally, conspiracies against the public?
posted by You Can't Tip a Buick at 4:54 AM on November 5, 2014 [8 favorites]
posted by You Can't Tip a Buick at 4:54 AM on November 5, 2014 [8 favorites]
I do some work with small 911 call centers operating at a municipal and county level, and the technology available to them these days is amazing, tied into mapping and visualization software and redundant to the Nth degree, but using off-the-shelf gear so even small towns can afford big city 911 services. Lot of good it does them if the wireless companies (and also the SIP providers) can't or won't route the calls their way.
If Obama really wants to make an impact headed out the door, he should lean on the FCC to classify wireless and internet as common carriers.
posted by Slap*Happy at 5:01 AM on November 5, 2014 [20 favorites]
If Obama really wants to make an impact headed out the door, he should lean on the FCC to classify wireless and internet as common carriers.
posted by Slap*Happy at 5:01 AM on November 5, 2014 [20 favorites]
The "definition" which establishes corporations as conspiracies against the public also applies to the government; politics and bureaucracy are trades.
posted by I-Write-Essays at 5:23 AM on November 5, 2014 [1 favorite]
posted by I-Write-Essays at 5:23 AM on November 5, 2014 [1 favorite]
Wireless carriers are common carriers. The states can't regulate them, but the federal government does.
Speaking as someone who also has some experience in this area, the FCC certainly does fine companies for outages and other issues related to 911 routing. The problem with this situation is that the outages came in as a trickle, rather than as a wave. It was difficult for anyone to realize the scope of the outage in time to stop it. That's not to say that it shouldn't have been stopped, and the FCC is certainly taking a look at who could have done what, but this is not really a failure by carriers regarding routing to the call center, whether wireless or wireline. (Note that's the wireline telcos generally carry more of the burden with 911 routing, at least as far as the last mile routing to the actual call center -- the last big 911 routing failure involved Verizon switches going down in Virginia, causing outages to call centers all over the Northern Virginia suburbs). This was a failure of, basically, Internet software that everyone from the PSAP to the carrier relies on.
There's absolutely no doubt that these sorts of issues will proliferate as we move to more IP-based network infrastructures. But 911 in general is pretty complex thing, and there's no single entity that can really be blamed for these sorts of events. Everyone working on 911 is generally doing the best they can, it's just really hard, particularly when it comes to wireless.
posted by devinemissk at 5:56 AM on November 5, 2014 [6 favorites]
Speaking as someone who also has some experience in this area, the FCC certainly does fine companies for outages and other issues related to 911 routing. The problem with this situation is that the outages came in as a trickle, rather than as a wave. It was difficult for anyone to realize the scope of the outage in time to stop it. That's not to say that it shouldn't have been stopped, and the FCC is certainly taking a look at who could have done what, but this is not really a failure by carriers regarding routing to the call center, whether wireless or wireline. (Note that's the wireline telcos generally carry more of the burden with 911 routing, at least as far as the last mile routing to the actual call center -- the last big 911 routing failure involved Verizon switches going down in Virginia, causing outages to call centers all over the Northern Virginia suburbs). This was a failure of, basically, Internet software that everyone from the PSAP to the carrier relies on.
There's absolutely no doubt that these sorts of issues will proliferate as we move to more IP-based network infrastructures. But 911 in general is pretty complex thing, and there's no single entity that can really be blamed for these sorts of events. Everyone working on 911 is generally doing the best they can, it's just really hard, particularly when it comes to wireless.
posted by devinemissk at 5:56 AM on November 5, 2014 [6 favorites]
Yikes. This is why I have the regular phone numbers for police, fire, and ambulance dispatch for my county in my phone's address book. A few years ago I tried calling 911 from my cell phone and couldn't get through: the first time I got nothing but silence, the second (and third... and fourth... and fifth...) times a fast busy signal. I simply can't trust it. If I ever have such an emergency again, I'll try 911 first but I'll have the other numbers as a fallback just in case.
posted by zsazsa at 6:18 AM on November 5, 2014 [2 favorites]
posted by zsazsa at 6:18 AM on November 5, 2014 [2 favorites]
Every day they dont never come correct
Just ask my man right here with the broken neck
posted by Potomac Avenue at 6:21 AM on November 5, 2014 [6 favorites]
Just ask my man right here with the broken neck
posted by Potomac Avenue at 6:21 AM on November 5, 2014 [6 favorites]
TIL that the Risks digest is still going strong!
posted by scolbath at 6:32 AM on November 5, 2014 [6 favorites]
posted by scolbath at 6:32 AM on November 5, 2014 [6 favorites]
Puleeeze. Fining these companies is a joke. In most places they just tack on a "recovery fee" to the bill. You do know there are about 19 hidden fees allowed by the FCC including fees that allow carriers to recover regulatory costs.
posted by Gungho at 6:58 AM on November 5, 2014
posted by Gungho at 6:58 AM on November 5, 2014
This is one reason community services and some minor training are so key and important. If 911 fails, do you know how to get medical treatment? Can you stabilize a wound? What about help for the person banging down your door?
posted by corb at 6:59 AM on November 5, 2014 [1 favorite]
posted by corb at 6:59 AM on November 5, 2014 [1 favorite]
This kind of error happens all over the place. It happens in both the public sector and the private sector. If there's a major difference between the two, I'd like to see some evidence for it.
These things will continue to happen no matter how much you threaten and punish people, unless there is a cultural change and a general change in the way the engineering is done. And even then, you'll only reduce the number, because a certain amount of error is truly unavoidable when you're dealing with humans.
There are methods that can prevent a lot of stuff like this, but nobody uses them.
First of all, setting up those methods requires significant long-term investments in people, tools, and processes. The day to day pressure on managers, again both public and private, is to keep costs down and deliver features quickly. In both the public and private sectors, a lot of the pressure comes from the "money people", but politicians are just as good at it as financiers. And if you as a manager do somehow get something working, chances are those same bosses will reorganize you for political reasons and mess it up.
Furthermore, your personal incentives are wrong. You're only there for a relatively short time, and most likely nothing too horrible will happen on your watch anyhow, especially not if you're an up-and-comer who will get moved quickly. If you invest to do it right, all you're doing is sacrificing the quick "success" that controls your next promotion, for something that you may hope will make your successors look better. And, again, if you're moving around a lot, you have a good chance of being able to blame any failures that do happen on your predecessors (who are probably already invulnerable to any realistic form of accountability).
Meanwhile, at the bottom of the pyramid, there's an entrenched culture among developers of resisting high assurance methods, because they are (1) not fun for most people, (2) not something most people can just pick up on their own, (3) not something some people can ever pick up at all (4) just plain unfamiliar and scary, and (5) seen as an unrealistic pipe dream.
The hypothetical possibility of a huge fine, or being fired, or being put out of business, or being put in jail, or getting the firing squad, or whatever you want to threaten, is so remote from everybody's daily thoughts that it has basically no effect on their actions.
Anyway, since nobody does it right, everybody can claim to be following "best practices".
There may be other problems, too. I don't necessarily understand everything.
Structures like professional organizations can sometimes provide a bit of pushback on all that, but they're far from perfect... and there are no effective professional standards in software development, and it wouldn't be that easy to formulate them.
posted by Hizonner at 7:24 AM on November 5, 2014 [16 favorites]
These things will continue to happen no matter how much you threaten and punish people, unless there is a cultural change and a general change in the way the engineering is done. And even then, you'll only reduce the number, because a certain amount of error is truly unavoidable when you're dealing with humans.
There are methods that can prevent a lot of stuff like this, but nobody uses them.
First of all, setting up those methods requires significant long-term investments in people, tools, and processes. The day to day pressure on managers, again both public and private, is to keep costs down and deliver features quickly. In both the public and private sectors, a lot of the pressure comes from the "money people", but politicians are just as good at it as financiers. And if you as a manager do somehow get something working, chances are those same bosses will reorganize you for political reasons and mess it up.
Furthermore, your personal incentives are wrong. You're only there for a relatively short time, and most likely nothing too horrible will happen on your watch anyhow, especially not if you're an up-and-comer who will get moved quickly. If you invest to do it right, all you're doing is sacrificing the quick "success" that controls your next promotion, for something that you may hope will make your successors look better. And, again, if you're moving around a lot, you have a good chance of being able to blame any failures that do happen on your predecessors (who are probably already invulnerable to any realistic form of accountability).
Meanwhile, at the bottom of the pyramid, there's an entrenched culture among developers of resisting high assurance methods, because they are (1) not fun for most people, (2) not something most people can just pick up on their own, (3) not something some people can ever pick up at all (4) just plain unfamiliar and scary, and (5) seen as an unrealistic pipe dream.
The hypothetical possibility of a huge fine, or being fired, or being put out of business, or being put in jail, or getting the firing squad, or whatever you want to threaten, is so remote from everybody's daily thoughts that it has basically no effect on their actions.
Anyway, since nobody does it right, everybody can claim to be following "best practices".
There may be other problems, too. I don't necessarily understand everything.
Structures like professional organizations can sometimes provide a bit of pushback on all that, but they're far from perfect... and there are no effective professional standards in software development, and it wouldn't be that easy to formulate them.
posted by Hizonner at 7:24 AM on November 5, 2014 [16 favorites]
Puleeeze. Fining these companies is a joke. In most places they just tack on a "recovery fee" to the bill. You do know there are about 19 hidden fees allowed by the FCC including fees that allow carriers to recover regulatory costs.
Yep, carriers can recover regulatory costs in certain categories. But not all regulatory costs -- the FCC has to specifically approve the costs, and it never approves regulatory cost recovery for a forfeiture. Regulatory cost recovery is for things like the Universal Service Fund, not for enforcement actions.
posted by devinemissk at 7:46 AM on November 5, 2014 [4 favorites]
Yep, carriers can recover regulatory costs in certain categories. But not all regulatory costs -- the FCC has to specifically approve the costs, and it never approves regulatory cost recovery for a forfeiture. Regulatory cost recovery is for things like the Universal Service Fund, not for enforcement actions.
posted by devinemissk at 7:46 AM on November 5, 2014 [4 favorites]
Puleeeze. Fining these companies is a joke.
It doesn't have to be. Remember when Yahoo was threatened with exponentially-escalating fines for not opening up their users' data to the government? They were staring down the barrel at literally being on the hook for a SEXTILLION dollars.
There's no reason honestly persuasive government fines couldn't be levied against those who conspire against the public.
posted by chimaera at 9:00 AM on November 5, 2014 [5 favorites]
It doesn't have to be. Remember when Yahoo was threatened with exponentially-escalating fines for not opening up their users' data to the government? They were staring down the barrel at literally being on the hook for a SEXTILLION dollars.
There's no reason honestly persuasive government fines couldn't be levied against those who conspire against the public.
posted by chimaera at 9:00 AM on November 5, 2014 [5 favorites]
Puleeeze. Fining these companies is a joke. In most places they just tack on a "recovery fee" to the bill. You do know there are about 19 hidden fees allowed by the FCC including fees that allow carriers to recover regulatory costs.
Then what's your solution?
posted by dirigibleman at 9:25 AM on November 5, 2014
Then what's your solution?
posted by dirigibleman at 9:25 AM on November 5, 2014
The solution is more accountability of corporations to the public, rather than just to shareholders.
posted by walrus at 9:58 AM on November 5, 2014 [1 favorite]
posted by walrus at 9:58 AM on November 5, 2014 [1 favorite]
The solution is more accountability of corporations to the public, rather than just to shareholders.
So more poorly-written regulations with lax enforcement.
posted by nerdler at 10:01 AM on November 5, 2014
So more poorly-written regulations with lax enforcement.
posted by nerdler at 10:01 AM on November 5, 2014
The solution is more accountability of corporations to the public, rather than just to shareholders.
That's not a solution. That's a talking point on a political candidate's web site. What does "more accountability of corporations to the public" entail?
posted by dirigibleman at 11:07 AM on November 5, 2014
That's not a solution. That's a talking point on a political candidate's web site. What does "more accountability of corporations to the public" entail?
posted by dirigibleman at 11:07 AM on November 5, 2014
That's not a solution. That's a talking point on a political candidate's web site. What does "more accountability of corporations to the public" entail?
If the public agrees to pay you for a vital service, your fulfillment of the service is completely transparent, heavily regulated, and subject to criminal penalties and crippling civil fines if you fail in providing that service?
If you want to provide 911 tracking, the public expects it to work, or what's the point? And the way you do it has to be able to be audited by the public, or their experts. If your vital service fails due to incompetence, negligence or cost-cutting so your executive team can have another $10M in bonuses, that should be the basis for harsh criminal and civil penalties.
If you can't make money under those guidelines, boo fucking hoo. Go sell accent lighting or cars instead.
posted by maxwelton at 12:05 PM on November 5, 2014 [4 favorites]
If the public agrees to pay you for a vital service, your fulfillment of the service is completely transparent, heavily regulated, and subject to criminal penalties and crippling civil fines if you fail in providing that service?
If you want to provide 911 tracking, the public expects it to work, or what's the point? And the way you do it has to be able to be audited by the public, or their experts. If your vital service fails due to incompetence, negligence or cost-cutting so your executive team can have another $10M in bonuses, that should be the basis for harsh criminal and civil penalties.
If you can't make money under those guidelines, boo fucking hoo. Go sell accent lighting or cars instead.
posted by maxwelton at 12:05 PM on November 5, 2014 [4 favorites]
We should devise a regulatory regime that levies massive, massive, massive fines for everyday violations like this. Moreover, the standards imposed should get progressively tighter and tighter. In the short term, companies will be forced to serve the public rather than their shareholders. In the long term, if the regulations are made tight enough, every service provider will commit violations. When this happens, the state will be able to take over ownership of the newly bankrupted companies.
I call this scheme "No Corporation Left Behind."
posted by You Can't Tip a Buick at 12:44 PM on November 5, 2014 [5 favorites]
I call this scheme "No Corporation Left Behind."
posted by You Can't Tip a Buick at 12:44 PM on November 5, 2014 [5 favorites]
Anyway, since nobody does it right, everybody can claim to be following "best practices".
holy crap, Hizonner evaluates the current state-of-the-tech-industry in 25-words-or-less. Wired and Gizmodo got nuthin'.
[high assurance methods] ...not something some people can ever pick up at all
preach it. the script kiddies don't stand a chance when something life-or-limb/*very* important comes up.
posted by j_curiouser at 1:37 PM on November 5, 2014
holy crap, Hizonner evaluates the current state-of-the-tech-industry in 25-words-or-less. Wired and Gizmodo got nuthin'.
[high assurance methods] ...not something some people can ever pick up at all
preach it. the script kiddies don't stand a chance when something life-or-limb/*very* important comes up.
posted by j_curiouser at 1:37 PM on November 5, 2014
Puleeeze. Fining these companies is a joke. In most places they just tack on a "recovery fee" to the bill. You do know there are about 19 hidden fees allowed by the FCC including fees that allow carriers to recover regulatory costs.
Then what's your solution?
posted by dirigibleman 2 days ago [+]
Passing legislation that prevents companies from recovering fines from their customers... But that would mean buying off a lot of Congress, and I can't afford that.
posted by Gungho at 12:24 PM on November 7, 2014
Then what's your solution?
posted by dirigibleman 2 days ago [+]
Passing legislation that prevents companies from recovering fines from their customers... But that would mean buying off a lot of Congress, and I can't afford that.
posted by Gungho at 12:24 PM on November 7, 2014
« Older Panik to Crawford to Belt | oy Newer »
This thread has been archived and is closed to new comments
posted by Thorzdad at 3:38 AM on November 5, 2014 [24 favorites]