A Glitch Is Breaking All Firefox Extensions
May 4, 2019 7:48 AM   Subscribe

Mozilla is very sorry for the inconvenience. Slashdot and Techcrunch also discuss.
posted by Little Dawn (226 comments total) 19 users marked this as a favorite
 
I got hit by this today. Just *bang* and all my security add-ons were disabled.
I did the temporary fix, and all seems well for now.

It's really weird to see the web without the add-ons running. It really is a hot mess.
posted by Thorzdad at 7:52 AM on May 4, 2019 [36 favorites]


i've been dying. the internet is unacceptable without adblock. i've confined myself to mefi and ao3.
posted by Mizu at 8:01 AM on May 4, 2019 [26 favorites]


Here's the active bug report if you want to follow along.

Does automatic certificate expiration solve any real problems? I know it solves a theoretical problem. I know it causes a whole lot of real problems, all the time.
posted by Nelson at 8:01 AM on May 4, 2019 [8 favorites]


I was hit with this last night, but I restarted Firefox this morning and they're all back.
posted by Preserver at 8:02 AM on May 4, 2019


I was hit by this last night. Just in the middle of browsing - BAM, all extensions, including ones released by Mozilla, suddenly stopped working. It's fixed now though and I didn't do anything.

It's been pretty depressing to see the vitriol over this. I don't just mean people saying it's a mess, but the people who are just ... so over-the-top angry about their add-ons being temporarily disabled. It's inconvenient, but nobody's day was ruined worse than the devs', yanno?
posted by Kutsuwamushi at 8:03 AM on May 4, 2019 [18 favorites]


Mizu...Just do the temp fix outlined in the Mozilla link, and you will be able to resume normal browsing.
posted by Thorzdad at 8:04 AM on May 4, 2019


winterhill, I've been hit by it here in the UK
posted by amcewen at 8:04 AM on May 4, 2019


Could someone explain in a low-jargon way what a signing certificate is, why this one expired, and why that disables add-ons?
posted by medusa at 8:13 AM on May 4, 2019 [2 favorites]


Mine stopped working while reading this post, of course. I hope it is fast and easy for them to solve, but I don't see why anyone would get angry about it. Things break and need to be fixed, it's not like there is an evil cabal at Mozilla who are deliberately breaking extensions.
posted by Dip Flash at 8:13 AM on May 4, 2019 [2 favorites]


Yeah, I just used Chrome for a while.

I think that some people who used password managers that got disabled were more inconvenienced, but ... I would never put my passwords into a database I could only access with an add-on. Precisely because add-ons sometimes get yanked or disabled or bugged.
posted by Kutsuwamushi at 8:15 AM on May 4, 2019 [3 favorites]


what a signing certificate is, why this one expired, and why that disables add-ons?

A "signing certificate" is a small file on a computer that's sort of like a Notary Seal. A little rubber stamp you can apply to a document to say "this document is now notarized and therefore authentic". Only in this case instead of a paper document, it's Firefox extension code that's being stamped. We call this signing. "This extension was signed by this certificate".

Signing certificates have expiration times on them that say "this certificate cannot be used after May 4 2019". Once that expiration happens, not only can the certificate no longer be used to sign new extensions, but all the old signatures go invalid too. If this happened to Notaries it would be as if when a Notary's commission expires, then not only can they not notarize new documents, but all old documents they'd ever stamped were now also unnotarized retroactively. (We don't do this in the real world, but we do with digital certificates.)

The final piece here is that Firefox will only run signed extensions. Ie, will only accept valid notarized documents. And so if the signing certificate expires, suddenly no extensions are signed any more and nothing runs. Oops.

(One complicating factor; certificate chains. Certificates are signed by other certificates, all the way up to one of several Root Certificates with ultimate authority. It'd be like if your local Notary's stamp were itself stamped by the Governor, and the Governor's stamp was in turn stamped by the President. If any certificate in this chain expires then the whole chain falls apart. I believe this bug was an intermediate certificate in the chain expiring.)

Organizations that use a lot of certificates are supposed to have systems in place to monitor for when their certificate expires and renew it well ahead of the expiry date. That seems to have not worked in this case.
posted by Nelson at 8:23 AM on May 4, 2019 [46 favorites]


This was the last straw and I've completely switched over to Chrome. The decision to force a restructuring of plugins about a year ago was bad enough and I purposefully disabled Firefox updates to preserve the plugins I had working. It's totally unacceptable that the Firefox developers still maintained control over my browser even after I disabled updates.
posted by ElKevbo at 8:23 AM on May 4, 2019 [2 favorites]


i've confined myself to mefi and ao3.

My ao3 extension is borked! Now what??

(jk. I'm just keeping my completed tabs open until the add-on is working again. The a03 rating add-on is the only way I can keep track of what I've read.)
posted by greermahoney at 8:24 AM on May 4, 2019 [1 favorite]


Organizations that use a lot of certificates are supposed to have systems in place to monitor for when their
certificate expires and renew it well ahead of the expiry date. That seems to have not worked in this case.


That's really the root cause here. The follow-on admin process failed.

This is a management issue, not a tech issue.
posted by mikelieman at 8:29 AM on May 4, 2019 [9 favorites]


The fix instructions don't seem to apply to Firefox for Android. Anyone know if there's a solution there? I'm worried I'll have to uninstall and reinstall my ao3rdr add-on, and I'll lose all my data. I'm still recovering from losing all my data two years ago.
posted by greermahoney at 8:35 AM on May 4, 2019


> It's totally unacceptable that the Firefox developers still maintained control over my browser even after I disabled updates.

Nelson and mikelieman both have provided good explanations for what happened. The extension disablement was not due to a browser update. Although by disabling updates you've prevented Firefox from being able to rectify the problem.
posted by ardgedee at 8:35 AM on May 4, 2019 [16 favorites]


Yes, ardgedee, they've established that they make terrible technical decisions AND have significant management issues. Fool me once...

It's really sad and frustrating that a tool that was once built around ideals of giving users choices and customization now forces users to cede control of software on their computer to others who take away those choices. I understand the security tradeoffs involved but this organization has proven itself untrustworthy with these and many other decisions that have taken the focus off of providing a solid, customizable web browser and onto many unrelated, unnecessary things.

I appreciate the work of all of the developers but it's clearly time for me to uninstall this particular piece of software. Best of luck!
posted by ElKevbo at 8:43 AM on May 4, 2019 [4 favorites]


My most important extensions are to (try to) keep my privacy a bit- adblockers fall in that category of course, and it troubles me a bit that the fix is "give up some of your privacy to Firefox" by allowing studies. Forcing me to turn that back on again rubs me in exactly the wrong way.
posted by DreamerFi at 8:45 AM on May 4, 2019 [4 favorites]


chiming in to say thanks, Little Dawn, for this post. Noticed this problem late last night just before bedtime, woke up having forgotten about it and my first-thing visit to MeFi reminded me ... and with a quick-easy temp-fix.

Also: f*** advertising




or more to the point, that deficiency in the culture of the so-called west that seems to make it the only way to make cool things like the interwebs pay for themselves.
posted by philip-random at 8:46 AM on May 4, 2019 [4 favorites]


I only use Chrome when I make the calculation that my sense of privacy is less valuable than my need to use the website, e.g. Bye, Chrome: Why I’m Switching to Firefox and You Should Too (Fast Company), Millions of Chrome Users Have Installed Malware Posing as Ad Blockers (Slashdot), Browser Autofill Used to Steal Personal Details in New Phishing Attack (Guardian), Google Eavesdropping Tool Installed on Computers without Permission (Guardian).

I imagine that the chilling effects (MIT Technology Review) induced by a feeling that we've been stripped of a sense of privacy and security by the temporary loss of extensions that claim to protect us could be related to the shaking of fists and yelling (Youtube, KHAAAAAAN!).
posted by Little Dawn at 8:55 AM on May 4, 2019 [18 favorites]


I've been leaving Firefox open because of a presumably unrelated issue* - wow, literally just now as I was typing this all the extension icons reappeared in the upper right corner. The hell is going on with you, Mozilla people?

*Anytime I close Firefox and then reopen it I get a message that it's already running and would I like to close and restart? So I close it (Task Manager usually shows 5 instances running) and then I get a crash message and report ticket. I report it, and then the crash report box freezes. I submitted ten reports one day, haven't heard anything back. Sorta pisses me off.
posted by Alvy Ampersand at 8:58 AM on May 4, 2019


So when I jumped online this morning, I checked right away for an update for this issue. The note was that the fix should be rolling out in the background, and sure enough - by the time I got to this thread and read to the bottom, all of my extensions and add-ons have been restored.

Inconvenient, to be sure, but I'm glad they got it corrected.
posted by nubs at 9:00 AM on May 4, 2019


Calling Mozilla untrustworthy is an absolutely gigantic leap that I'm unwilling to make here. I belive that Mozilla's interests are far more likely to be aligned with my own than Google's are*, and the fact that they shipped one bug that was fixed in a few hours doesn't shake that feeling at all. Apart from the plugin migration (which I'll talk about in a second), I don't recall any Mozilla having any other major crisis in recent history.

Firefox is pretty clear about how it delivers updates, and allows you to disable both update mechanisms if you want (you shouldn't -- the benefits of automatic browser updates vastly outweigh the risks and drawbacks).

I switched to using Firefox full-time at home a year ago, and have had absolutely zero regrets. It's more stable, offers a bunch of neat privacy features (such as the officially-sanctioned extension Facebook sandboxing extension), and is significantly faster in many contexts. Go load up one of the megathreads in Firefox and Chrome, and compare the load times if you want to see what I'm talking about. On a low-powered system, these pages can take up to 10 seconds to render in Chrome, but render almost instantaneously in Firefox.

I know the plugin/extension migration was painful for many, but there was plenty of advance warning, and it unlocked many of these new performance/privacy features. The old Firefox extension system was based on a decade-old architecture that failed to achieve most of its original goals, and generally hadn't aged well -- it was well past its expiration date.

*I might be willing to concede that Apple's interests are relatively benign too, but Safari isn't cross-platform, and is generally a kind of shitty browser. Its spec-compliance is easily the worst of any modern browser (including Edge), and major rendering bugs have gone unfixed for years on end.
posted by schmod at 9:05 AM on May 4, 2019 [37 favorites]


Alternative to Chrome that is basically Chrome without the tracking: Brave Browser
posted by gwint at 9:05 AM on May 4, 2019 [1 favorite]


Brave is basically "Our founder used to work at Mozilla, but resigned in disgrace for being a homophobe. We block ads, but then put our own ads on the pages you visit, because we are obviously more trustworthy, and want to keep the revenue for ourselves."

Hard pass.
posted by schmod at 9:09 AM on May 4, 2019 [49 favorites]


For one reason or another, any misstep by Firefox/Mozilla seems to inspire a largely irrational anger in otherwise rational people.*

The continued existence of this browser is a testament to the cultural value of FOSS and a push back against the handful of megacorps who are taking over the rest of the web, and the literal two massive conglomerates who produce its only real competitors. It trickles along on crumbs from Google and Cisco, and yet manages to have an immense impact on the internet as we know it.

Oh jeez, this comparatively tiny non-profit accidentally allowed a certificate to expire on a piece of incredibly complex software you got for free, causing no actual damage AFAICT, let everyone know by every channel they had access to, and fixed it within a couple hours?

Quelle fuckin' horreur.

*Yes there have been legit complaints about some of Mozilla's top-level decisions, and fuck Brendan Eich. But have you looked at Google and Apple lately?
posted by aspersioncast at 9:20 AM on May 4, 2019 [73 favorites]


I might be willing to concede that Apple's interests are relatively benign
Counterpoint: the very existence of those horrid Airpods.
posted by aspersioncast at 9:23 AM on May 4, 2019 [6 favorites]


Firefox is pretty clear about how it delivers updates, and allows you to disable both update mechanisms if you want (you shouldn't -- the benefits of automatic browser updates vastly outweigh the risks and drawbacks).

I used to be quite stubborn about not letting anything update itself on my computer, but then I found out about Firefox ESR, which does exactly what I wanted -- it keeps me up to date on security but doesn't change in noticeable ways during its year+ life-cycle. That's just one of many reasons I have a huge reservoir of good will toward Mozilla that isn't going to be depleted by one screwup.
posted by aws17576 at 9:24 AM on May 4, 2019 [6 favorites]


Hi, everyone. I work at Mozilla. We've got a lot of people working on this at the moment; desktop will fix itself if it hasn't already and mobile is in-flight.

I'm not operating on a ton of sleep here but I'm happy to take questions.
posted by mhoye at 9:27 AM on May 4, 2019 [81 favorites]


Just a big thank you for putting in the extra hours - and on a weekend no less - from my side!

I'll just restrict myself to Metafilter for the time being and maybe go outside and look at some flowers for a change? :)
posted by bigendian at 9:35 AM on May 4, 2019 [9 favorites]


It's inconvenient, but nobody's day was ruined worse than the devs', yanno?

Or maybe dissidents and political activists in hostile countries whose use of certain extensions was a part of their online anonymity. But we probably won't ever hear about a handful of disappearances scattered around the world because Mozilla is run by incompetents.
posted by tclark at 9:35 AM on May 4, 2019 [2 favorites]


I'm not operating on a ton of sleep here but I'm happy to take questions.

What's going to be put in place so that something like this can never happen again?
posted by slater at 9:36 AM on May 4, 2019 [1 favorite]


Added on Preview: Thank you, mhoye! I appreciate your comments and explanations when Firefox issues come up. No real questions, and it isn't important, but if you have any insight into what might be happening with the installs on my computers as described below, I'd love to hear it. They're all on regular release.

I've been following along on Reddit, since the Firefox installation on my HTPC lost its add-ons last night in the middle of a Youtube video. I have Chrome installed on that computer as well, and since they decided not to kill adblocking extensions, I made do. My mother's laptop FF also lost extensions around the same time.

I put and use Firefox on all of the computers in the house. But the oddest part is that not all of my computers have been affected by this. My own laptop is still fine, though due to paranoia I haven't closed Firefox since last night. However, Firefox on the new computer I that I set up yesterday, that I synced to the Firefox install on the HTPC, is also still full of add-ons. I've closed that browser and restarted it several times since, and it is still fine.

From what I can tell, I use fewer add-ons than most, but they make all the difference in my browsing experience. The real, unfiltered Web is really unpleasant.
posted by monopas at 9:42 AM on May 4, 2019 [2 favorites]


What's going to be put in place so that something like this can never happen again?

That's going to wait for the postmortem - we're not going to let a problem like this happen again, but right now we're making sure it's fixed right for everyone. We know what the technical problem is here, and we're fixing that right now, but making sure that we understand the context of that problem in depth so that we can prevent a recurrence from a code, process and culture perspective will take some time.

Get yourself a glass of whatever your favourite is and a nice, long rest when all this is done.

Thanks. Soon enough, but you never celebrate until you're over the line.
posted by mhoye at 9:46 AM on May 4, 2019 [21 favorites]


By the time I finished reading this thread and scrolled to the bottom, the fix hit my browser. Thanks for the feedback mhoye and for getting it resolved.

I've been pleased with this browser for the last year and I've given up on Chrome.
posted by Fizz at 9:48 AM on May 4, 2019 [3 favorites]


So I realize it's a cert thing and my initial reaction last night was basically THIS IS IT - ADDONPOCALYPSE IS NIGH UPON US!

And ranted with a lot of "fuck"s in the reddit thread, I didn't catch the original point of it, just the "solution" being to enable unsigned in nightly, and like, fuck if that's valid.

I just want an org I can trust, and Moz was the only thing I had trust in, especially since even MS is going Blink.

Pale Moon I thought of trying, otherwise it's Vivaldi or Opera? Are they using Blink too?
Anyways, I'm still using them but keeping a more wary eye. Not cool.
posted by symbioid at 9:49 AM on May 4, 2019


If for various reasons you can't use the official remedy, a workaround (that you probably shouldn't use for long) is to set xpinstall.signatures.required to false in about:config.

ETA - I just tried this on Android and it seems to work there too.
posted by trig at 9:51 AM on May 4, 2019 [2 favorites]


Brave is basically "Our founder used to work at Mozilla, but resigned in disgrace for being a homophobe. We block ads, but then put our own ads on the pages you visit, because we are obviously more trustworthy, and want to keep the revenue for ourselves."

I can't speak to the first assertion, but the second is false. By default the Brave browser blocks ads and does not add any.
posted by gwint at 9:52 AM on May 4, 2019


The extension disablement was not due to a browser update

Except for the fact that it WAS due to an update, IIRC a year or so back, that originally implemented the change that required all extensions to use their intermediate cert to run. Which many extension devs weren't happy about. And oops, that cert expired last night! So it was absolutely broken by a browser update, quite a while ago, that didn't manifest itself until the other shoe dropped.
posted by tclark at 9:54 AM on May 4, 2019 [6 favorites]


I mean in that sense it was due to the big bang.
posted by ericost at 10:00 AM on May 4, 2019 [8 favorites]


Also, on preview - there are some bad-tempered people out there!

People care about Firefox, sometimes intensely - my inbox is a Firehose Of Caring right now - but that's the price of doing business for us. It's work, but it's a lot better than having people not care.
posted by mhoye at 10:01 AM on May 4, 2019 [21 favorites]


maybe dissidents and political activists in hostile countries whose use of certain extensions was a part of their online anonymity

This is a valid concern, but is there any evidence whatsoever that it's even likely? Did this e.g. affect Tor?
posted by aspersioncast at 10:01 AM on May 4, 2019


METAFILTER: my inbox is a Firehose Of Caring right now
posted by philip-random at 10:04 AM on May 4, 2019 [6 favorites]


It did affect Tor Browser too.
posted by Bangaioh at 10:04 AM on May 4, 2019 [1 favorite]


It affected Tor and some other partners, we're actively working with them now.
posted by mhoye at 10:05 AM on May 4, 2019 [3 favorites]


Here's one report from someone on HN, 13 hours ago:

Chirael 13 hours ago [-]

Just discovered the same message in the Tor browser, and it seems that NoScript got disabled. So people running Tor are a lot more vulnerable right now.
posted by tclark at 10:05 AM on May 4, 2019 [1 favorite]


To be clear: Tor Browser, the Firefox fork, is what's been affected.

Tor the anonymity network itself is unaffected.
posted by Bangaioh at 10:10 AM on May 4, 2019 [2 favorites]


My ao3 extension is borked! Now what??

(jk. I'm just keeping my completed tabs open until the add-on is working again. The a03 rating add-on is the only way I can keep track of what I've read.)


Wait, there's an ao3 extension? ...why did I never think to look for one before.

Features include:
- Three star rating system
- Hide works
- Blacklist by tags
- Bookmark by chapter read
- Scan bookmarks for updates
- Keep track of last visit
- Backup and restore user data
- Open source code!


Wow, this is awesome. I'm glad for this glitch; I've learned an awesome new thing because of it. Thank you, greermahoney!
posted by anthy at 10:11 AM on May 4, 2019 [3 favorites]


people who are just ... so over-the-top angry

It's a fundamental feature of western civilization. Getting murderously angry over trivialities is just the way things work -- if you don't get murderously angry at nearly everything that happens in your day to day life, then there's something wrong with you and we probably need to get murderously angry at you now.
posted by aramaic at 10:19 AM on May 4, 2019 [13 favorites]


I did the temporary fix to get an ad-blocker back - mostly because I just use chrome for a limited number of things. All of my bookmarks are on firefox.
posted by Ms. Moonlight at 10:20 AM on May 4, 2019


I mean in that sense it was due to the big bang.

Well sure, it was due to a combination of many unfortunate events:
  • The universe was created.
  • In the course of time, the web browser became a pretty big deal.
  • Someone decided that Mozilla signing all the add-ons was a good idea.
  • Someone chose to implement those signatures with a cert that expires.
  • Instead of checking signatures on installation and for cert revocation, it simply re-checks the signatures every day.
  • Against the protest of some users, the option to turn off said signature checking was removed.
  • The cert unexpectedly expired.
posted by sfenders at 10:30 AM on May 4, 2019 [8 favorites]


unacceptable that the Firefox developers still maintained control over my browser even after I disabled updates.
Oh boy you're gonna love what Google does then!
posted by 922257033c4a0f3cecdbd819a46d626999d1af4a at 10:48 AM on May 4, 2019 [30 favorites]


This has been a brief nuisance, not a reason to abandon an excellent browser and the only competitive alternative to Chromium. I switched back to FF a few months ago after years on Chrome (after a previous thread where mhoye expounded the Quantum update). It's brilliant and running ublock on my Android phone without needing to root the device has been fantastic.

mhoye, if you're listening: SWIPE DOWN TO REFRESH!!!
posted by Evstar at 10:51 AM on May 4, 2019 [3 favorites]


This really sucks, and the impact has been huge, and I hope Mozilla has both a smooth rollout for the fix and comes up with some good mitigations for the future.

That said, I have a lot of empathy for the Mozilla folks here, and I get angry when I see people calling them incompetent or screaming at them.

“The certificate expired” is one of those phrases that will make most IT ops folks shudder from remembered pain. Many, many organizations have had incidents with certs expiring, including recently the US government. Mostly, when this happens, it just means your website is down; this particular Mozilla cert had a comparatively huge blast radius.

Some reasons that keeping certs up to date include:
  • Most certificates expire on long time scales (1 or 2 years are common). Long delays between cert renewals make them easy to forget. In a lot of orgs, the state of the art in remembering this is a calendar reminder.
  • You can set up automated monitoring and warnings of cert reminders, but again — because it doesn’t happen frequently, it’s hard to test if your monitoring is still working.
  • Third-party monitoring helps (because they monitor many certs, and therefore exercise the code more). But because your cert still renews infrequently, there are still risks: I cannot count the number of cert renewal emails I have seen in spam folders.
  • If you do manage to remember to actually do it, many orgs still have cert renewal as a manual process involving arcane text commands. “I got a new cert but I fucked it up” also happens.
  • Why wasn’t it automated? Because it happens rarely, or because we tried that once and it broke when our vendor changed their code, etc.
One solution, championed by Lets Encrypt, is to make certs expire more frequently (weeks or months) and automate the process. This is really awesome for websites, but assumes the content signed by the cert can update that frequently. This is easy on a web site, but for a code signing cert, can be a nightmare. Require every extension you distribute to be updated once a month, or it stops working? I’ve done that inside a company, where we could enforce IT policies, and it was incredibly painful even then.

Another option is to remove the signing cert as a requirement. Let Firefox run extensions which are not signed by Mozilla, in the default config. (IIRC this used to be how it worked?) But at that point you reduce Mozilla’s ability to police their extension ecosystem, which is not really a good thing in 2019. People are not generally good about distinguishing trustworthy from untrustworthy extensions; and previously-trustworthy extensions can and have been purchased and loaded with surveillance code.

None of this makes this particular incident suck less. Mozilla should put a lot of effort behind making this better.

But bear in mind that this class of problem has been a pain in the ass for decades, and no one has come up with a good solution for all cases yet. It seems like a stupid problem, but it gets more and more complicated when you go down the rabbit hole. So it’s fine and good to be frustrated, but cut the actual people behind this some slack, ok?
posted by a device for making your enemy change his mind at 10:57 AM on May 4, 2019 [25 favorites]


Nelson, please let me climb on a favorite hobbyhorse here for a second:

Does automatic certificate expiration solve any real problems? I know it solves a theoretical problem. I know it causes a whole lot of real problems, all the time.

Cert rotation goes wrong all the time because validity windows are too long, so rotation (whether manual or automated) is hardly ever exercised, things get forgotten, and rot goes unnoticed a long time. This is the same reason why I've stood in line for expedited passport renewal when I've had unplanned business travel but the other guy had his trip planned for six months and just forgot his passport was expiring: the solution is not long validity times, but short validity times and frequent rotation. (Bonus: switching to frequently-expiring certs when your process is manual will finally mean you automate the pain away.) Let's Encrypt does this right, but if you're paying $100 for every cert, you naturally want to minimize your costs (self-signed in pre-prod! once a year rotation in prod! This Is Fine!), which is a perverse incentive that causes expired certs and dismayed users.

On the other hand, if you had to rotate a 90-day cert every 60 days (because you're a good admin who renews and starts trying to rotate with 30 days left of your validity window, for insurance!) you'd notice when pre-prod broke, and have time to fix it before prod was at risk. Save yourselves heartache: rotate early and often!

This message has been brought to you by the letters oh crap, and the number AGAIN???
posted by sldownard at 10:59 AM on May 4, 2019 [8 favorites]


I got a bit cranky about this, too, because I use firefox everywhere...on my phone, my surface, my work machine and my newest machine, a little box I use on my TV. I have that exclusively to watch videos from youtube, really...and youtube is unwatchable without using ublock origin via firefox.

So thanks for the hard work, guys. (My work machine never lost its add-ons, but the little machine on the TV did, they're back there now.)

The web is unusable without the ability to block ads and selectively block javascript. I kind of think that's why the walled gardens have popped up...yes, you get ads but you don't have to deal with Random Web Site trying to load JS from 321 different domains.
posted by maxwelton at 11:00 AM on May 4, 2019 [2 favorites]


As an aside, in the theme of “common IT problems that make us crazy, see this tweet thread (from before the Mozilla incident):

“Let’s play a game: using five words or less, utter a phrase that will elicit a great story from a software engineer with years of operational experience.”

Certificate expiration appears many times in the replies. ;)
posted by a device for making your enemy change his mind at 11:05 AM on May 4, 2019 [6 favorites]


My only experience with certificates has been running let’s encrypt on my home server.
What is a reasonable time frame to expect a new cert/chain to be created? I’m just surprised that the only fix available in 24hrs is various degrees of disabling cert checks, and not, like, just fixing the certificate.
posted by stobor at 11:06 AM on May 4, 2019


"I only use Chrome when I make the calculation that my sense of privacy is less valuable than my need to use the website"

Seems a bit narrow, realistically, you're making a privacy sacrifice anytime you use any net-enabled device, or exist near any internet-connected device that has has a camera or microphone, or carry your phone or use it ever. You can't have privacy if you're using the internet.
posted by GoblinHoney at 11:15 AM on May 4, 2019 [1 favorite]


So what are the good browser alternatives when Firefox is borked and Chrome is undesireable? I used Chrome to look up Waterfox, which I'd never heard of until I went to Reddit/Twitter to find out what was going on with Firefox, and Waterfox auto-played an ad, so that was out for me...
posted by TwoStride at 11:15 AM on May 4, 2019


it troubles me a bit that the fix is "give up some of your privacy to Firefox" by allowing studies.

Studies doesn't sound like it was their first choice to use as the vector to roll out a fix, but it was one that they could use fairly quickly to get people working again. I'd rather give up some of my privacy on a temporary basis to Mozilla than to almost anyone else in this industry. Now the patch is in I've turned off Studies.

Before the hotfix was in I accidentally opened up Youtube and … how do people even without an ad blocker?
posted by scruss at 11:16 AM on May 4, 2019 [6 favorites]


That said, I have a lot of empathy for the Mozilla folks here, and I get angry when I see people calling them incompetent or screaming at them.

I have no beef with the employees of Mozilla, least of all mhoye. But the C-Suite is riddled with above average even for Silicon Valley venality and incompetence. I mean, seriously. The simple fact that Jascha Kaykas-Wolff still even has a job there means that everyone he reports to is incompetent and shouldn't be let anywhere near a nonprofit organization which claims to serve a mission rather than shareholders.
posted by tclark at 11:19 AM on May 4, 2019 [2 favorites]


So what are the good browser alternatives when Firefox is borked and Chrome is undesireable

Brave
posted by 922257033c4a0f3cecdbd819a46d626999d1af4a at 11:21 AM on May 4, 2019 [1 favorite]


I am vary grateful that Firefox exists to keep the web at least slightly open and honest, but I use Safari myself. This isn't a gloat, it's just... I got so much going on in my life, tech is something I have no time for. Says the person who for the past couple of months has been hand-editing a custom Kittens Game auto-play script. Shut up.
posted by seanmpuckett at 11:29 AM on May 4, 2019


It's inconvenient, but nobody's day was ruined worse than the devs', yanno?

In another forum, it was suggested this could actually do some real damage for people using various add-ons for privacy / protection in countries with even nastier and more privacy hating governments than my own.
posted by jzb at 11:43 AM on May 4, 2019 [3 favorites]


It's inconvenient, but nobody's day was ruined worse than the devs', yanno?

We don't have the luxury of being glib about stuff like that. People depend on Firefox - and I don't mean use, I mean "depend" - in a lot more difficult situations than almost anyone realizes, and those are only the ones we know about.
posted by mhoye at 11:48 AM on May 4, 2019 [9 favorites]


mhoye, not to cross the streams, but (#potus45 Megathread), as a functionally tech-illiterate researcher, I'm finding it difficult to believe that it's just a coincidental expiration of security certificates happening all at once. I'd very much like to be wrong about this, fwiw.

My first comment on this thread linked to an MIT Technology Review article by Evgeny Morozov pointing out the potential implications for democracy, because in many ways, this is more than an inconvenience, and today also seems like a weird day (#potus45 Megathread) on the internet, to say the least.

And I used the phrase "sense of privacy" for several reasons, GoblinHoney, including because I think internet security and privacy is a First Amendment right, and this glitch is an example of how that works. "Chilling effects" (University of Richmond Law Review) is also a First Amendment reference (William & Mary Law School Scholarship Repository), so the stakes seem to be fundamental human rights, and Mozilla is carrying a hugely-appreciated burden in trying to protect us.
posted by Little Dawn at 11:53 AM on May 4, 2019 [1 favorite]


If for various reasons you can't use the official remedy, a workaround (that you probably shouldn't use for long) is to set xpinstall.signatures.required to false in about:config.

Ah! Thanks, trig, that's worked nicely for me, and my user-installed extensions (as opposed to Debian-packaged extensions, which are unaffected by the bug) are all working again.

Now to avoid installing any extensions until the Debian packaging for Firefox itself catches up (Debian builds of Firefox have the Studies thing disabled at build time and rely on Debian's standard update mechanism rather than letting Firefox run its own).
posted by flabdablet at 12:00 PM on May 4, 2019


Little Dawn, I think your conspiracy theory meter is set a little high. Security certificates expire at a known time, anything from a year (or more) downwards. To make this more than a coincidence (is there something special going on in US politics right now? I mean, more special than the baseline level of special which the rest of the world has come to ignore) someone would have to arrange the current knot in the ongoing US political debacle precisely at the same time in advance of the Mozilla security cert expiry.

That would require a level of mass competence and secrecy beyond any government.
posted by scruss at 12:04 PM on May 4, 2019 [6 favorites]


I'm finding it difficult to believe that it's just a coincidental expiration of security certificates happening all at once.

It was the expiration of just one certificate, one which was used to sign most (or all?) of the Firefox add-ons. I gather it was an intermediate cert, which could have been relied upon by more than one signing cert, but that makes no difference if you're not concerned with the technical details.

So the official story is not difficult to believe in any way. To me the only bizarre thing is the way it was designed to shut down extensions mid browser session, rather than for example checking them on startup and showing a warning.
posted by sfenders at 12:05 PM on May 4, 2019 [7 favorites]


Firefox is worth supporting as others upthread have said. One security scare I had recently was in finding the Accessibility screen reader option enabled by default but with the indicator turned off by the developers and not yet re-enabled. I worried that the screen reader had been enabled by malware to sniff passwords or such.
posted by anthill at 12:06 PM on May 4, 2019


I'm finding it difficult to believe that it's just a coincidental expiration of security certificates happening all at once. I'd very much like to be wrong about this, fwiw.

The core technical issue here, if not the root cause, is the expiry of an intermediate certificate that was signed two years ago, which... well, don't get me wrong: I also resent the fact that if you're navigating hyperconnected modernity after a while "think like a conspiracy theorist" starts to sound a lot like "wear your seatbelt". That's not what this is.

In this case, though, for whatever it's worth I work directly with a lot of the people involved and have known some of them for my entire adult life (lot of Canadians at Moz!) and I'm not seeing the connections you're inferring.
posted by mhoye at 12:07 PM on May 4, 2019 [9 favorites]


I’m just surprised that the only fix available in 24hrs is various degrees of disabling cert checks, and not, like, just fixing the certificate.

Fixing a certificate is not a thing that can happen. The entire point of certificates is to be demonstrably immutable.

The correct fix is for all extensions in whose authentication chain the expired certificate appeared to be re-issued on addons.mozilla.org after being signed with new certificates that don't rely on the expired one, then installed in browsers via the usual extension update mechanism.

Subsequently, Mozilla needs to keep track of every extension it has ever signed, and invent protocols for delivering timely updates to those whose certification is due to expire soon even if the original extension developers are long gone and their extensions are essentially abandonware. If they still work at all then many will still be relying on them.

Mozilla also needs to set in stone a promise to maintain the existence and effect of about:config settings sufficient to allow users to apply temporary workarounds for the next little surprise independent of any fix pushed by Mozilla.
posted by flabdablet at 12:24 PM on May 4, 2019 [4 favorites]


We don't have the luxury of being glib about stuff like that.

This is a valid point and I apologize for coming across as though privacy concerns aren't real - they certainly are. Addressing this as swiftly as possible was the right thing to do.

What depressed me was going onto forums to find out what was going on with Firefox, and finding a bunch of men yelling about how Firefox is crap and and generally acting as if they had been personally betrayed by Mozilla since Mozilla made a development decision they disagreed with. Those are the people that I was thinking of.
posted by Kutsuwamushi at 12:24 PM on May 4, 2019 [9 favorites]


a bunch of men yelling about how Firefox is crap and and generally acting as if they had been personally betrayed by Mozilla

There's nothing to be done about people like that. They're the same ones screaming at the Government to keep its hands off their Medicare.
posted by flabdablet at 12:31 PM on May 4, 2019 [4 favorites]


My extension icons just showed up but they aren't operational, and web dev tools aren't working, so I can't pull up a console to see what the problem is. OTOH we had yet another expiring cert problem on our CI test system at work last week and we're talking about developing a way to do that. First step was to put a test at the front of the pipeline to fail when any certs are, oh, a month away from expiring. Because updating certs is much easier before they've expired than after.
posted by morspin at 12:32 PM on May 4, 2019 [1 favorite]


morspin, I had to restart Firefox after applying the xpinstall.signatures.required workaround in order to make my extensions come alive again. If what Moz is pushing out involves some similar temporary relaxation of cert checking, perhaps it needs the same thing done after landing?
posted by flabdablet at 12:38 PM on May 4, 2019


Re: ao3rdr:

Three star rating system

Its, I think, a 5-star system, and if you choose the lowest star, it hides the fic from your search results. I love that feature.
posted by greermahoney at 12:53 PM on May 4, 2019


what does it say about me that I complain about bad institutional policies, abusive market practices irl but when software has a glitch... I quietly, patiently, passively reinstall, add the add ons and start again (eg. filtering the entire web with noScript AGAIN one script at a time).
never once thought about complaining to someone.
posted by ipsative at 12:58 PM on May 4, 2019 [3 favorites]


> I also resent the fact that if you're navigating hyperconnected modernity after a while "think like a conspiracy theorist" starts to sound a lot like "wear your seatbelt". That's not what this is.

I appreciate the clarification, because I surely don't understand what's happening, and when it looks like it's politically helpful to break adblockers on a wide scale, my own ignorance does cause me concern. I appreciate the work you're doing, and perhaps instead of resenting me, maybe see me as an example of how people who don't know how this works can interpret this kind of disruption? You are doing vitally important work, and I surely appreciate it, so thank you.
posted by Little Dawn at 1:02 PM on May 4, 2019 [1 favorite]


I appreciate the work you're doing, and perhaps instead of resenting me, maybe see me as an example of how people who don't know how this works can interpret this kind of disruption?

I'm sorry, I didn't mean to imply that I resent you - I realized what I was saying there too late, and missed my edit window. I meant "I, like you, dislike the fact that we seem to have to think like this".

And to your point - yeah, one of the challenges of communicating this stuff is that it really is esoteric and doesn't map to conventional metaphor well.
posted by mhoye at 1:16 PM on May 4, 2019 [6 favorites]


I bet all those Microsoft folks are glad this happened and drew attention away from how their DNS fuckup broke a good chunk of their own products and the parts of the internet that were reliant on them.
posted by mrzarquon at 1:31 PM on May 4, 2019 [2 favorites]


I am entirely confident that this was a mistake, not some nefarious plot, and my heart goes out to everyone who is having a miserable weekend because of it.

Stuff happens, and I'm glad a fix went out over the fastest channel available, but I can't say it's a great look that the current fix requires turning on the "Allow Firefox to send technical and interaction data to Mozilla" pref (especially since last year, opt-out was interpreted to mean "opt-in to reporting data about the fact that you opted out"), since studies can only be enabled after telemetry is turned on. I accept that's an unavoidable consequence of using the studies feature to quickly push a fix; I just wish the blog post at least acknowledged that "turn on telemetry to fix your browser" is not what anyone wants and entirely at odds with Mozilla's stated values.
posted by zachlipton at 1:32 PM on May 4, 2019 [4 favorites]


OK, that's kind of hilarious. My desktop firefox, which was not affected last night, just this second told me all my extensions are untrustworthy.

I wonder what kept it alive as long as it was? I guess I'll have to turn studies on here, too.

The conspiracy theorist in the back of my mind is not imagining government agents, but an MBA with nice hair and teeth who was damned if their telemetry graphs weren't going to show better numbers this month...

Joke. I think.
posted by maxwelton at 2:09 PM on May 4, 2019 [3 favorites]


Huh, ff Dev edition here and not a single problem. Recommended
posted by dozo at 2:18 PM on May 4, 2019


Saw this post, turned on studies (which i had off on purpose). A long while later my extensions finally get disabled, but the fix had not downloaded yet.

Went to about:config and set "app.normandy.run_interval_seconds" to five seconds, restarted the browser. The fixes were downloaded and my extensions came back. Reset the seconds setting to default.

Noticed that having studies on downloaded some study not related to the fix. Also noticed, as some people commenting on the Mozilla blog post also did, THAT MY CONTAINERS HAVE BEEN WIPED AND RESET TO DEFAULT. Not a big deal for me, personally, but WTF. I hope that they were at least reset in a way that is not leaving obsolete/corrupt settings files in my user folder for all eternity.
posted by D.C. at 2:23 PM on May 4, 2019 [1 favorite]


I wonder what kept it alive as long as it was?

It checks the signatures periodically; based on other values I remember from about:config I would guess once every 24 hours. So for each user it failed at some random time since last midnight UTC.
posted by sfenders at 2:23 PM on May 4, 2019 [1 favorite]


This thread is so helpful! Not only about the glitch but also people discussing the add-ons they miss the most, I am taking notes.
posted by Coaticass at 2:46 PM on May 4, 2019


Always a shock to be reminded how many people think that the internet is unusable without an adblocker, but the idea that they're going to quit using their web browser over not being able to use their ad blocker for a little while is a new level
posted by Kwine at 2:46 PM on May 4, 2019 [1 favorite]


Jeez, I'm glad the tone of the thread seemed to have righted itself because I was seriously thinking I had woken up in bizarro world. Shit happens, y'all. Nobody's trust was violated, a simple oversight happened to some people who usually get it right and are almost radically transparent about both process and rationale.

The worst aspects of social media, still unchecked, are destroying us, but we keep doubling down on instant outrage.
posted by wierdo at 2:46 PM on May 4, 2019 [9 favorites]


This was the last straw and I've completely switched over to Chrome.

Who the hell voluntarily chooses that Google abomination ?
posted by Pendragon at 2:53 PM on May 4, 2019 [6 favorites]


Like Devs, certs & cert renewals are the bane of an Ops/Infra teams life (effectively superseding the long held hatred for printer & fax-machines). The chaos that can ensue when a renewal doesn't occur in a timely fashion can be epic. Whats worse is the sense of impending doom when you know you need to renew a cert but all the doco on how to do it and where its required is either non-existent, full of gaps or just plain wrong. And sure, you can automate all of this but its typically held together with tape & twine - when the person that cobbled that together is gone and 3yrs later theres a cert to renew then you're in the poo again.

Nice work Mozilla - I was almost at the point of just not browsing until it was resolved - I'd forgotten how truly dire the unfiltered web can be.
posted by phigmov at 2:56 PM on May 4, 2019 [3 favorites]


It checks the signatures periodically

or else it gets the hose again
posted by dephlogisticated at 3:03 PM on May 4, 2019 [19 favorites]


Previously on MetaFilter. Traditionally, certificates use advanced mathematical calculations to expire at the least convenient time.
posted by Huffy Puffy at 3:07 PM on May 4, 2019 [4 favorites]


Does automatic certificate expiration solve any real problems?

The entire point of certificates is to be demonstrably immutable.


When there is a good reason to revoke a certificate e.g. when Google stopped trusting Symantec-issued SSL/TLS certs, it can take months and require a browser update.
I don't understand why we can't have a system where each certificate is trusted for an indefinite period into the future but can also be revoked with immediate effect?
I know that's not how the current certificate chain works but it doesn't seem like an impossible thing to engineer.
posted by Lanark at 3:20 PM on May 4, 2019


Always a shock to be reminded how many people think that the internet is unusable without an adblocker

FTFY
posted by Greg_Ace at 3:33 PM on May 4, 2019 [9 favorites]


I don't understand why we can't have a system where each certificate is trusted for an indefinite period into the future but can also be revoked with immediate effect?

One advantage of building an expiration date into the certificate, rather than relying on revocation, is that it doesn’t require any additional infrastructure.

If you want to rely on revocation, you need to have servers that host the revocation list, those servers need to be reliable and secure and performant, the system needs to be robust to attack by bad actors, etc. You also need to be damn sure you trust the people running those systems. It turns into a huge mess, and gets expensive fast. For expiration, you just include a little bit of data that is downloaded once.

(A variety of revocation mechanisms do exist, but they are in fact a huge mess to deal with.)
posted by a device for making your enemy change his mind at 3:35 PM on May 4, 2019 [8 favorites]


Lanark, that's what revocation lists and OCSP checks are for. For various reasons, one of the big ones being the unreliable nature of the Internet and the fact that there is simply no "right thing to do" when the necessary servers are unavailable for whatever reason (all options suck), they have never been implemented with a policy of strict checking in consumer software.
posted by wierdo at 3:38 PM on May 4, 2019


I don't understand why we can't have a system where each certificate is trusted for an indefinite period into the future but can also be revoked with immediate effect?

I suppose we could, but that would perhaps involve designing something new, and designing new things that do cryptography is scary. Which is probably one reason why the system used is as similar to web server certs as it is, though in practice it's not quite so identical as some people in the comments sections of the internet have assumed, e.g. those who think that simply issuing a new one to fix the problem should have been a twenty minute job.

To me it would make more sense to leave things otherwise as they are, except use the signing date when considering whether the signature is valid, rather than the current date.
posted by sfenders at 3:38 PM on May 4, 2019


Also, the belittlement I'm seeing of people's real feelings of anger and frustration at the sudden profound disruption of their entire online experience wouldn't be allowed to fly in any other Metafilter thread.
posted by Greg_Ace at 3:39 PM on May 4, 2019 [9 favorites]


Count me among the annoyed, but not enraged. I can only imagine what it's like for people who are the Technical Support Person for their aging parents/grandparents and have to try to explain this issue to them. Good luck.

I thought the problem was on my end, and I had no indication of why it happened. I actually uninstalled ublock origin, and went to reinstall it, and got an error stating that something was wrong with my connection. I wish there had been a more informative error message to point me in the right direction. I finally got it reinstalled at least.

Also the expiration of certificates - I have really not a lot of sympathy there. I don't want to express myself in a pissy manner here, but maybe offload the certificate expiration reminders to someone who isn't going to automate it, such as an administrative assistant? Or use a paper calendar or something? That won't end up in your spam folder. I mean, companies/organizations have to renew contracts and things every year, have the person keeping track of that walk over to Cert Person and say "hey, time to renew the Super Important Certificate" like a month before. Seriously, I don't get why this is so hard. Paper calendars, use them!
posted by cats are weird at 3:39 PM on May 4, 2019 [3 favorites]


There is a big difference between being understandably annoyed about having to be assaulted with ads and assuming bad faith and/or promoting baseless conspiracy theories.

I try to be understanding because Trump is literally making people crazy, bringing psyops tactics that had previously been reserved for recruiting the far right to the public at large. In my defense, it's distressing to me that it has become nearly impossible to escape the catastrophizing and the outrage machine. There once was a time when simply avoiding a relative few websites was enough to prevent oneself from being steeped in the toxic stew. Getting splashed on a regular basis was bad enough, but now we're all in the pool and there is no escape that doesn't involve not knowing what is going on in the world around us.
posted by wierdo at 3:52 PM on May 4, 2019 [13 favorites]


Parenthetically, I would like to thank my pi hole from keeping me from seeing the horrors of an adblock+/privacy badger less version of the internet!
posted by mikelieman at 4:11 PM on May 4, 2019 [4 favorites]


Also, the belittlement I'm seeing of people's real feelings of anger and frustration at the sudden profound disruption of their entire online experience wouldn't be allowed to fly in any other Metafilter thread.

Greg_Ace and I don’t agree on much, I think, but this is correct.
posted by mhoye at 4:14 PM on May 4, 2019 [5 favorites]


A firewall might be beneficial as a temporary bandage to hold the virus out while systems are upgraded?
posted by hugbucket at 4:22 PM on May 4, 2019


The last of many, many, many, many straws.

Goodbye Firefox.
posted by Pinback at 4:23 PM on May 4, 2019


Pretty strange that the web runs on .... advertising that we can't abide at all because Christ, it makes browsing terrible? This seems more of an issue to me than a Firefox glitch. If a majority start blocking ads, then - apocalypse? Nirvana? What, exactly?
posted by zenzenobia at 4:59 PM on May 4, 2019


Lots of mentions of Chrome in here. Personally I gave up on it after I discovered that the thing eating my laptop CPU time on the Windows machine I had it installed (but rarely used) on was Google's "Software Reporter Tool", a Windows service that comes with Chrome and runs in the background to scan your filesystem and report to Google what it finds. Seriously, that is a real thing that happened.

Ungoogled-chromium might be an option, I guess.
posted by sfenders at 5:00 PM on May 4, 2019 [2 favorites]


If a majority start blocking ads, then - apocalypse?

If so, welcome to the apocalypse. It's been going on for a year or two I believe.
posted by sfenders at 5:01 PM on May 4, 2019 [1 favorite]


Late to the party because I've been away from my desktop all day - but apparently this still breaks spontaneously if you have "Allow Firefox to install and run studies" disabled for privacy reasons. That's what happened when I fired it up a few minutes ago.

Enabling studies fixed the problem, and they've been disabled again, but apparently disabling studies leaves current studies still running? That feels like it might be both necessary for the hotfix, but bad for the random other study that was installed at the same time. A little extra guidance would be helpful about that.

Apple and Google reported pretty meh quarters, Microsoft had a big Azure flub the week before their annual dev conference and Mozilla has this. Been a bad week for tech giants all around.
posted by bcd at 5:36 PM on May 4, 2019


How do I get my dev tools back? https://developer.mozilla.org/en-US/docs/Tools/Tools_Toolbox talks about a "Toggle Tools" menu item under the "Tools | Web Developer >", but all I have there is "View Page Source".
posted by morspin at 6:01 PM on May 4, 2019


If a majority start blocking ads, then - apocalypse?

If so, welcome to the apocalypse. It's been going on for a year or two I believe.


So vote on it and do exactly as the majority wants. Don't try to hack growth in users and try to mould the whole world into the worst of the valley's behavioural excesses and neuroses nurtured in the hothouse of genius programmes.
posted by hugbucket at 6:09 PM on May 4, 2019


So do you have to enable studies to get this fix at all? Mozilla's page isn't clear about that. I'm too afraid of most websites at the moment to go trying to Google for an answer.

I would like to thank Metafilter for being a nice website even with no adblocker.
posted by wondermouse at 6:24 PM on May 4, 2019 [1 favorite]


How do I get my dev tools back?

Shift+Ctrl+C, where they always were, no?
posted by scruss at 6:26 PM on May 4, 2019


And sure, you can automate

And then, you also automate all the errors (and you still need to maintain the automation).
What's worse than one intermediate cert expiring? Certs going sideways on a scale and as fast as a computer can process them.

And "Why don't you just...." and it's variation are just...flames on the side of my face.
posted by MikeKD at 6:34 PM on May 4, 2019 [2 favorites]


One personal irony here is that recently I had to deal with some SSL certificates that were being revoked (basically there's been an ownership change and the previous company was like "we don't want to deal with these certs anymore"). As the calendar ticked down to the date of revocation and we still didn't have replacement certs ready for various, totally bullshit reasons, a couple of people tried to assure me that actually revocation was handled really poorly by web browsers and the worst-case scenario was that we'd have technically invalid certs but no one would notice.

The day of revocation came and we'd managed to replace all of the important certificates, but some on our development servers remained unchanged. This was acceptable to us in the interim, but it did mean we got to test the revocation theory in practice. It turns out Chrome does not, in fact, care about revocation. You can browse any site that has a revoked certificate just fine, and you won't get any warnings about it.

You know what browser correctly throws a fit and throws up a giant warning about the certificate being revoked? Firefox.
posted by chrominance at 7:02 PM on May 4, 2019 [6 favorites]


As this is a broad thread for Firefox addon/privacy stuff: last night I randomly came upon this privacytools.io page, which describes (even more) Firefox settings for privacy. If you're going to adjust these about:config settings it's good to have a general idea of what they actually do, and some of them are not relevant if you're using add-ons that block things.

(...It's also a good idea not to make numerous about:config adjustments during a period in which Firefox starts acting up for completely unrelated reasons so you don't have to spend a few hours wondering what you did wrong!)

I will also make my semi-annual plea to get a cookie manager. Without my cookie manager last night, I had accumulated 60-80 junk cookies from visiting like a handful of pages. The only sites that need to retain cookies are those that you actually want to remember you between sessions (MetaFilter if you're logged in; any site visited regularly that has a log-in; etc.) Your job is to add those sites to the whitelist. The cookie manager will then harmlessly delete all the other garbage cookies, when you close a tab, that would otherwise stay on your computer forever tracking you for profit (these cookies are also mitigated, of course, by having a strong ad/tracking blocker, so you never touch the bad-sites in the first place). I am not a web expert but I believe this is how browsers should have been designed in the first place. It's not a difficult concept to press a button to say "yes, this web site and I have a relationship! remember it!". No more difficult than saying "give this app access to photos" on your phone, where incidentally these cookies are festering and there's little you can do other than delete the works, including the 10 you'd like to keep (even Apple still won't get this right in Settings).

For me blocking ads is not really the point. I am blocking the bullshit proliferation of unscrupulous late-capitalist companies that don't do anything of value for society. Like many here, I would happily keep an ad served by the domain I visited that didn't involve getting permanently tracked by 20 other businesses.
posted by sylvanshine at 7:44 PM on May 4, 2019 [17 favorites]


I temporarily switched over to the preview version of the new Edge on my laptop. It has my essential addins and works fine. Right now I’m on my iPad, where I use (and am happy with) Safari. I’ll check out Firefox on the laptop later, see if everything’s back.

I really hate that advertising has been so important for everything, not just on the web. I know it’s been like that for a long, long time, well before the web. I don’t have an answer for this, but I think the need for ad blockers shows that web advertising is out of control and effectively ruins the web experience. I don’t know what the ultimate result of my use of ad blockers will be on the sites I block, but I’m not very sympathetic. Is it wrong of me to think those sites are there for me, I am not there for them?
posted by lhauser at 7:57 PM on May 4, 2019 [1 favorite]


Kutsuwamushi

It's been pretty depressing to see the vitriol over this. I don't just mean people saying it's a mess, but the people who are just ... so over-the-top angry about their add-ons being temporarily disabled. It's inconvenient, but nobody's day was ruined worse than the devs', yanno?


I doubt that Tor users are so relaxed about it. Any security breach can have literally life-threatening consequences.

It was a major fuck-up, in no small part because it was an easily avoidable one. Shame on Mozilla.

And that is from a long term FF user and fan.
posted by Pouteria at 8:19 PM on May 4, 2019 [2 favorites]


I switched to Opera and Chrome when Quantum came out; I keep an old version of Firefox that I use for DownThemAll and FireFTP. Quantum killed DownThemAll, and that was the only thing holding me back from using Opera as my main browser and Chrome as my "ugh, I guess I need to see this page with ads" browser.
posted by ErisLordFreedom at 8:31 PM on May 4, 2019


Seems like people would rather be angry than understand the underlying issues at hand.

I would have thought better of MetaFilter, to be honest.
posted by schmod at 10:06 PM on May 4, 2019 [7 favorites]


Also, Tor forked a tremendously complicated piece of software without considering the assumptions behind its security model, nor did they do anything to change those assumptions (namely, that Mozilla do not consider the functioning of extensions to be an critical to maintaining browser security, which, TBH, is perfectly reasonable).

A lot of very eager folks (presumably including many involved Tor) routinely audit web browsers for security. None of those people even noticed that "Hey, Mozilla's cert is about to expire, and we actually depend on those certs being valid for a whole bunch of important things."

Pointing fingers at Mozilla feels pretty weak, when a security-focused fork of the browser also wasn't able to identify and mitigate this particular weakness.
posted by schmod at 10:21 PM on May 4, 2019 [13 favorites]


"Seems like people would rather be angry than understand the underlying issues at hand.

I would have thought better of MetaFilter, to be honest.
"
Spare me the overwrought righteousness…

Maybe people are not so much angry over this, as completely fed up with all of the Firefox developer's poorly-conceived shillery shenanigans, wilfully dismissive arrogance in response to actual user feedback, and sheer don't-give-a-fuck-ery about anything that didn't spring forth from inside the nest?

In the past, they injected non-user-requested apps/plugins because they thought it was fun, and removed features/APIs that users and developers want (sometimes with ostensibly legitimate reasons, but quite often without listening to to user/developer complaints & attempting to reinstate equivalent functionality). And with large chunks of the functionality that people want being offloaded onto plugins, they've just shown through their incompetence how fragile their ecosystem really is.

I'm not angry about what Mozilla's become. I've just had a gutful of their behaviour, particularly over the last decade or so. If I'm angry at anything or anyone, it's at myself for putting up with it for so long.
posted by Pinback at 10:31 PM on May 4, 2019 [4 favorites]


Apparently there's a knock-on bug—and I'm one of the affected—where even after the hotfix is installed, either manually or through Normandy/studies, add-ons are not yet restored.

Relatedly, I get the anger and the frustration, but I look at it like this: where am I going to go? Google? I've worked diligently for almost two years to get that company out of my electronic life. To me, it's the same as "well, better dust off that browser made by Facebook." Sure, I could install Chromium and hope that the developers who build it a) keep it up to date, b) actually remove all of the Google stuff they say they do (I've no reason to doubt their sincerity but what if they miss something?), and c) continue to do so. All the while, I'm furthering the march towards a monoculture where Blink is the only browser rendering engine.

I can't do that. I totally understand that other people have different priorities and I will not hold those priorities against others. I may disagree, a lot, and it won't prevent me from making replies like this trying to persuade people to see my point of view, that Mozilla have made missteps but, in my eyes, they've a long way to go before I abandon the one piece of software that I unquestionably believe is trying to get it right for the user.
posted by fireoyster at 10:36 PM on May 4, 2019 [10 favorites]


You know what browser correctly throws a fit and throws up a giant warning about the certificate being revoked? Firefox.

In what universe is this behaviour correct? The warning is utterly irrelevant to 99% of websites; i.e., the static blogs and private sites whose owners are most likely to mess up their certificate renewals. The failure to distinguish means users have been trained to ignore the warning and just press ahead. It's like a safety light in a car I had that would light up if a door was ajar, if a seatbelt was undone, if I accelerated sharply, if I braked sharply, or - oh yes - if the radiator was busted and the engine overheating. Guess how much use that was.
posted by Joe in Australia at 2:07 AM on May 5, 2019


I am apparently one those that do not get offered studies even if I go and re-enable them, so I'm stuck without u-block and ghostery for now, among others.

I'm very surprised at how much faster pages are loading, I have some digging to do when they start working again.
posted by deadwax at 2:37 AM on May 5, 2019


"Then you were running an insecure browser." and so do you.

Similar to "observable universe", you do not ever have a secure browser, just more secure than prior to recent update (and sometimes not).
posted by filtergik at 4:02 AM on May 5, 2019 [1 favorite]


mhoye, are you able to confirm that the certificate from this reddit post, whose SHA256 fingerprint shows as B0:F7:0C:5C:36:3B:59:9B:B8:40:78:A4:35:F8:7E:F4:B8:FB:30:E5:84:18:2E:11:AD:FC:7B:07:AF:02:AE:82 in the Firefox certificate viewer, is indeed the one being automatically installed via Studies and that the manual installation procedure outlined in the reddit post and fully documented at www.velvetbug.com is sound?
posted by flabdablet at 4:20 AM on May 5, 2019


I switched to Opera and Chrome when Quantum came out

I tried Pale Moon and some others for a while, then came crawling back to Firefox a few weeks later. All it cost me was a few hours experimenting with the alternatives, a few more hours learning to use userChrome.css and some other tricks to replace everything essential to me from Classic Theme Restorer, and my immortal soul. The last of the functionality I missed was restored a year or so later.

So it's a complicated relationship we have, but I am not ready to DTMF Firefox just yet. If and when Mozilla fix the underlying problem such that this (and, one hopes, everything else) will fail safe next time even if all Mozilla servers suddenly disappear from the world and aren't around to renew certificates, I'll happily go back to not even thinking about it.
posted by sfenders at 5:23 AM on May 5, 2019


> I am apparently one those that do not get offered studies even if I go and re-enable them, so I'm stuck without u-block and ghostery for now, among others.

Try setting xpinstall.signatures.required to false in about:config.

I already had it set to false so maybe that's why my extensions were unaffected by the cert expiration and according to trig it seems to work.
posted by Bangaioh at 5:32 AM on May 5, 2019


You know what browser correctly throws a fit and throws up a giant warning about the certificate being revoked? Firefox.

In what universe is this behaviour correct? The warning is utterly irrelevant to 99% of websites; i.e., the static blogs and private sites whose owners are most likely to mess up their certificate renewals.
Two different things. Both/all browsers throw a fit if you forget to renew the certificate and let it expire. The anecdote you're quoting from was about checking for revoked certificates. Revocation is a more obscure corner.

Chrome did not enable the standard revocation check, for performance reasons. Firefox used a different compromise. I *think* both those statements are still true. There is a new standard of revocation, which (mostly?) avoids the performance problem. The new standard is only enforced when the certificate enables it (OCSP Must-staple). Technical reference: Revocation_Checking_in_Firefox.
posted by sourcejedi at 6:29 AM on May 5, 2019 [1 favorite]


Try setting xpinstall.signatures.required to false in about:config

Be aware that many builds of Firefox don't include or respect that setting.

The Debian build does, for which whew. I believe the nightly dev builds for all platforms do as well.

I have restored what appears to be full functionality on my Debian installation of Firefox 62.0.2 even with xpinstall.signatures.required reset to True, by importing this certificate and then reinstalling the affected extensions. Only two of mine were affected, the rest having been installed via Debian package management rather than via addons.mozilla.org.

I'm about to uninstall that certificate again and try manually installing the hotfix xpi to see if that works too.
posted by flabdablet at 6:46 AM on May 5, 2019 [3 favorites]


You know what browser correctly throws a fit and throws up a giant warning about the certificate being revoked? Firefox.

It's like a safety light in a car I had that would light up if a door was ajar, if a seatbelt was undone, if I accelerated sharply, if I braked sharply


No, this is about reacting to a cert that was explicitly revoked. In your analogy it's not a seatbelt being undone, it's your car failing a roadworthy and the mechanic explicitly slapping a defect notice on the windscreen.
posted by russm at 7:08 AM on May 5, 2019 [2 favorites]


Confirmed.

I removed the manually installed certificate, reset xpinstall.signatures.required and app.update.lastUpdateTime.xpi-signature-verification to their default values in about:config, and restarted Firefox. Within a few seconds it disabled the same extensions as it had done originally, complete with an identical warning.

I then clicked on the hotfix xpi linked from my previous comment, told Firefox to allow Metafilter to install it, and it did, after which both the disabled extensions immediately became available and functional.

Checked the Certificate Manager and found that the same signingca1.addons.mozilla.org certificate that I had previously installed by hand and subsequently removed had been re-installed by the hotfix, which also shows up in the Add-ons Manager as "hotfix-update-xpi-intermediate 1.0.2".

Removing the hotfix-update-xpi-intermediate add-on left the certificate installed, so I left the add-on removed for tidiness.

Given that there are substantial numbers of Firefox Quantum installations configured in such a way as not to allow Mozilla to push arbitrary code and preferences at whim via Studies, I am very disappointed that a simple link to what appears to be a fully functional hotfix is not prominently pinned to the top of the easiest-to-find Mozilla information page on this issue. There is less clickery involved in fixing it this way than by faffing about turning Studies on in Preferences.
posted by flabdablet at 7:10 AM on May 5, 2019 [4 favorites]


Not having to wait six hours is nice too.
posted by flabdablet at 7:21 AM on May 5, 2019


I am very disappointed that a simple link to what appears to be a fully functional hotfix is not prominently pinned to the top of the easiest-to-find Mozilla information page on this issue.

“Appears to work on my machine” is unfortunately not a great metric across a user base as broad as ours, and it’s far too low a bar for us to set for ourselves.

We’re testing real, proper fixes now; I know waiting sucks but shipping unproven software in haste is unlikely to improve the situation.
posted by mhoye at 7:51 AM on May 5, 2019 [7 favorites]


Does that hotfix not perform the same work as the one pushed via Studies?
posted by flabdablet at 7:53 AM on May 5, 2019


mhoye, is there likely to be a public "this is what happened, this was our temporary fix, this is why it won't in all likelihood happen again" post about this? I'm always fascinated to get a peek inside problems like this.

Not looking to pin blame, just a peek at "Jane went on vacation and Jim was at an offsite and Machine 17 had hung at blah and Bug 43874328 was postponed while Fix 63128 regressed Process Z and ... "
posted by maxwelton at 8:03 AM on May 5, 2019




So, is every pre-Quantum installation (versions 56 and older) just royally fucked, or what? Because the official messaging is to just wait if the fix hasn't been applied yet, but it would be disingenuous to tell that contingency of users to wait for something that's never going to happen.

As much as mhoye's responses are appreciated, it's hard not to believe someone else at the company would see this as an opportunity to get a lot of stubborn or reluctant users to update to the newest version. If that really is the case, then users of older versions probably deserve to know they will need to rely on an unofficial solution if they want to continue using an older version.
posted by Arson Lupine at 9:12 AM on May 5, 2019


is every pre-Quantum installation (versions 56 and older) just royally fucked, or what?

Not necessarily.
posted by flabdablet at 9:26 AM on May 5, 2019


users of older versions probably deserve to know they will need to rely on an unofficial solution if they want to continue using an older version.

AFAIU, running older versions already is an unofficial solution. I'd be surprised if Mozilla fixed pre-Quantum releases given ESR is currently at 60.

Users of those versions may just as well use one of the known workarounds, it's not like it'll be that much worse than running an unsupported browser.
posted by Bangaioh at 9:36 AM on May 5, 2019


It interesting the amount of anger at a non-profit organization that provides a service you are using absolutely for free (and ad-free). And the threats of gimme what I want right now or I'll stop using your free service I don't pay for. I wonder how many of the angriest people have ever clicked that donate button for the Mozilla Foundation. (And 10 bucks isn't really a lifetime donation).
posted by JackFlash at 10:01 AM on May 5, 2019 [18 favorites]


gimme what I want right now or I'll stop using your free service I don't pay for

and then you'll be sorry!
posted by flabdablet at 10:09 AM on May 5, 2019 [2 favorites]


I wonder how many of the angriest people have ever clicked that donate button for the Mozilla Foundation.

I was more confused than angry by the glitch, but this was a good reminder to donate. Thanks JackFlash!

And thanks mhoye for the updates along the way.
posted by sockshaveholes at 10:45 AM on May 5, 2019 [1 favorite]


Users of those versions may just as well use one of the known workarounds, it's not like it'll be that much worse than running an unsupported browser.

They didn't seem to be working for me - not the hotfix-update-xpi-intermediate workaround, or setting xpinstall.signatures.required to false, or enabling Studies (got a blank list). I'd been clinging to version 56.x on my home computer along with continuing to implement Every Security Measure Known to Humankind - or at least Greg_Ace-kind - to try to compensate for the inherent risk of not upgrading, because, sorry mhoye, Quantum's enormous disruption of my happily customized browsing experience was just too much to deal with.

But this was the final camel that broke my back and prompted me to go through the upgrade process. I spent a couple hours recording all the URLs in the various sessions I'd crafted and all the extensions I've been using, exported my bookmarks, and completely removed all trace of Firefox from my system (to expunge any lingering accumulated cruft/non-longer-needed about:config settings, etc.) before installing version 66 and re-establishing my accustomed environment as best I was able. Fortunately the extensions installed successfully, including the hotfix-update-xpi-intermediate workaround, and it looks like that less-than-fun slog resolved the issue, though I'm still looking to replace some lost functionality (RIP FLST and Tab Mix Plus, you were loved and appreciated).

I have to say, though, that after going through all that I'm not impressed with the performance - loading web pages is noticeably slower than pre-Quantum versions, in fact it's about as annoyingly slow as Chrome and definitely slower than the older version was. I'm currently researching how to improve FF performance; since I was able to do so on older versions I'm hoping the same is possible for Quantum. And I'll check out that privacytools.io page that sylvanshine linked to as well, since I still want to try to keep corporate exploitation to a minimum...call it a hobby.
posted by Greg_Ace at 10:52 AM on May 5, 2019


They didn't seem to be working for me - not the hotfix-update-xpi-intermediate workaround, or setting xpinstall.signatures.required to false, or enabling Studies (got a blank list).

If I understand correctly, hotfix-update-xpi-intermediate uses APIs that are only present in Quantum versions in order to force re-evaluation of all the disabled extensions' signatures after it's installed the new intermediate cert. My guess is that the Studies mechanism is just pushing that same hotfix and would probably be configured not do so to browsers it won't work on.

I would have expected setting xpinstall.signatures.required to false to allow you to re-download your disabled extensions one by one and have them reinstall successfully, but I don't have a pre-Quantum Firefox lying around to test this expectation against.

For anybody else who has a pre-Quantum version of Firefox that they don't want to update, you might want to try the manual certificate installation process described at www.velvetbug.com before giving up on it entirely.
posted by flabdablet at 11:01 AM on May 5, 2019 [1 favorite]


That makes sense. But in my own case, I decided to use this "opportunity" to impel myself to go ahead and upgrade.

My day job is software support; I often get annoyed - in private of course, I am a professional - at customers who stubbornly cling to older versions (some over 10 years out of development!!), especially given that their expensive yearly support contract gives them free upgrades with an absolute shit-ton of bug fixes and performance improvements. So I was feeling a tad hypocritical after something like a year and a half of refusing to make the Quantum leap.*

Interestingly, today the page-loading performance is vastly improved. Which is nice.

*sorry, I couldn't resist
posted by Greg_Ace at 11:21 AM on May 5, 2019 [3 favorites]


I think sometimes our hottest feelings are reserved for people / institutions who we either have a relationship with or claim an ideal we identify with ... and then fail on some point or another.

If I get mocked or criticized by a rando, not fun, but whatever, why do I really have to care about their opinion anyway? If I get mocked or criticized by a close friend or SO, man that's a punch in the gut.

If I get ghosted or breakup-texted by someone who I went out with a few times or even a few months, it's not nice, but then again it's hard to expect a big investment in carefully taking a relationship apart when investment in putting it together is still minimal, so I move on easy enough. If I get ghosted or breakup-texted by someone who I went with for years, someone who invoked "love" and even talked about responsibilities that should go with it, then yeah, I'm going to feel some burning resentment over that.

If Microsoft does something dumb with the UX of their products, I roll my eyes and move forward. If Apple does something equally dumb, as a longtime user, I tend to get extra upset because I'm more closely involved and I have higher expectations set in no small part by their own claims of high standards here (or at least I used to, my expectations have been reset and I'm getting used to the new reality where I'm no longer a key part of their target demographic).

Mozilla is a standard bearer for empowering / protecting users, so when users who flock to that standard have an experience which feels like a betrayal, they *feel* it. That's as much a sign that Mozilla has by and large done their job well enough for long enough to create expectations and make users care as it is a sign that they're also an organization that can occasionally make cert mistakes or let someone hijack the browser for a weird promo campaign. If people need to go see other browsers, respect, I've done it myself (Chrome when FF performance wasn't good enough for me, Safari when energy use has been paramount), but I hope they don't forget what made them care about Mozilla in the first place.
posted by wildblueyonder at 11:21 AM on May 5, 2019 [7 favorites]


Does that hotfix not perform the same work as the one pushed via Studies?

Please just don't, we're close.

mhoye, is there likely to be a public "this is what happened, this was our temporary fix, this is why it won't in all likelihood happen again" post about this? I'm always fascinated to get a peek inside problems like this.

Extremely likely.

is every pre-Quantum installation (versions 56 and older) just royally fucked, or what?

No. We haven't forgotten you.
posted by mhoye at 11:43 AM on May 5, 2019 [8 favorites]


Here's a Twitter thread by @SwiftOnSecurity giving some context to the problem & Mozilla's solution. It's a high level look at why this all happened the way it did.
I'd like to say to everyone complaining there's no way to turn unsigned extensions "back on" in normal Firefox.
The depth/scale of malware that forced this design was immense. I managed+supported Firefox for a decade, and did IT security. It has to be this way. It's unfortunate
Years ago, Firefox got to a point where it was getting completely, unfixably subverted by malware inserting custom code and extension hooks. Antivirus couldn't handle it. It's basically its own operating system. I've been on the frontlines. It's hard to describe the scale.
At the same time, yes, Mozilla made a mistake in instituting a strict operational process that wasn't fully reliable, as we see today. But do not make the mistake of thinking it was completely on a whim.
Firefox was getting destroyed and millions of users were suffering.
Chrome went through the same journey. They got to the point that to subvert it at all, malware started overwriting Chrome with custom-compiled old versions of the Canary developer builds, set to never update, and allow their injected malicious extensions. It's savage out there.
You've got to remember you are the 1%.
The problems you experience, that I experience, which we see as trivial from our contexts, can ruin the entire computer experience for others. There are just no good answers to this that doesn't involve sacrifices from somebody.
My website, written from my experience fighting browser malware at scale, literally instructs users to ---manually wipe the entire user data directory and program directory of browsers---. That's because of historical issues from malware that just couldn't be fixed any other way.
posted by scalefree at 12:22 PM on May 5, 2019 [11 favorites]


I think there's a particular thing where people are very sensitive about auto-updating software in general, because we've all been burnt by it. Especially now that so many things are web applications, unannounced updates that break everything, remove features, or just lead to confusion are the norm. And with Firefox, a bunch of people who were unhappy about the rapid release cycle became further unhappy about the loss of old extensions, and then this happens, which to users feels like Yet Another Example of Damn Software Changing On Me Unexpectedly.

And I sympathize with that, but the nature of web browsers is that they inherently have to be updated frequently to get security updates. I cringe so much when people announce they're running old unsupported versions of browsers, because they're inevitably riddled with unpatched security vulnerabilities.

But there's a trust issue here, and as this thread has shown, people have long memories and little patience for auto-updating software.
posted by zachlipton at 12:38 PM on May 5, 2019 [7 favorites]


Interestingly, today the page-loading performance is vastly improved. Which is nice.

I guess you could say that the current performance is giving me a quantum of solace...
posted by Greg_Ace at 1:00 PM on May 5, 2019 [1 favorite]


We are rolling out updates incrementally now.
posted by mhoye at 2:30 PM on May 5, 2019 [7 favorites]


Here's a Wired article highlighting why it's not unreasonable to suspect the random perturbations of an uncaring universe aren't the only possible cause when you find that all of your security and privacy measures have been shut off simultaneously via your browser vendor—“A Mysterious Hacker Group Is On a Supply Chain Hijacking Spree”.

(No connection to this Firefox event, just an explanation of what supply chain attacks are and details about a particular group carrying them out on other software vendors, including examples where systems were affected by the hundreds of thousands, but only a handful appear to have been targets for further exploitation.)
posted by XMLicious at 3:30 PM on May 5, 2019 [2 favorites]


Thanks, mhoye. I'm sure this has been very stressful. Do the updates apply to the Android version, as well?
posted by Chrysostom at 7:05 PM on May 5, 2019 [3 favorites]


Sigh. Supposedly Android v 66.0.4 is available, but when I go to the play store they’re still on 66.0.2. So... no fix yet.
posted by greermahoney at 7:35 PM on May 5, 2019 [2 favorites]


> "It interesting the amount of anger at a non-profit organization that provides a service you are using absolutely for free (and ad-free)."
Mozilla showed just how much they valued their "ad-free" reputation on December 16, 2017 - when they shoved a viral ad for Mr. Robot into their browser through the exact same back-door they're now requiring be turned on to fix their latest fuckup.
posted by Pinback at 8:36 PM on May 5, 2019 [1 favorite]


Woot! Second Firefox Fix Repairs Broken Browser Extensions For More People (Slashdot)
"Mozilla on Sunday began distributing new Firefox updates to fix a problem that broke extensions for many browser users on Friday," reports CNET
posted by Little Dawn at 8:54 PM on May 5, 2019


exiting firefox on my mac didn't fix things, but I downloaded the latest version (which I was supposedly on, but whatever) and installed that, and now things work again, with my settings and extensions built in.

Still annoyed though.
posted by gryftir at 9:23 PM on May 5, 2019 [1 favorite]


Mozilla showed just how much they valued their "ad-free" reputation on December 16, 2017 - when they shoved a viral ad for Mr. Robot into their browser through the exact same back-door they're now requiring be turned on to fix their latest fuckup.

That incident was indeed a massive breach of trust, hugely damaging to Mozilla's reputation, and the subsequent mealy-mouthed apology from Jascha Kaykas-Wolff made it perfectly clear that he had no clue about just how bad a fuckup it was. Which, you know, marketing C-suite; what can ya do.

The subsequent process changes would appear to make a similar incident less likely to happen. But even so, Studies is turned off in all my browsers and will damn well remain so even if it means I need to deploy only my own builds. If I wanted to let somebody other than me control the scheduling of changes to the software installed on my own computers I'd be running Windows, not Debian.
posted by flabdablet at 9:31 PM on May 5, 2019 [4 favorites]


We are rolling out updates incrementally now.

By what mechanism are those getting applied? Are specific updates available for manual installation?
posted by flabdablet at 9:33 PM on May 5, 2019 [1 favorite]


> "The subsequent process changes would appear to make a similar incident less likely to happen."
The "process changes" outlined there amount to "we promise to actually follow the guidelines we already had".

Consider, too, that the "fix" for the current problem also violates at least the first of those principles, and doesn't fit in with any of the other stated uses of either Shield or Studies…
posted by Pinback at 11:18 PM on May 5, 2019 [1 favorite]


I wonder what happened when everyone who was so outraged asked for their money back?
posted by bongo_x at 2:13 AM on May 6, 2019 [2 favorites]


Do Mozilla's fighty vigilante defenders think they are helping? It can be really hard to get your apologies heard over noise like that.
posted by Ashenmote at 3:02 AM on May 6, 2019 [3 favorites]


Supposedly Android v 66.0.4 is available, but when I go to the play store they’re still on 66.0.2.

We're gated on Play Store rollout rates on Android, but updates are rolling there as well.
posted by mhoye at 4:18 AM on May 6, 2019 [1 favorite]


Got the update this am and all is well. Thanks!
posted by greermahoney at 4:32 AM on May 6, 2019


Still 66.0.2 on Android in Hong Kong. uBlock Origin 1.18.6 remains crippled.
posted by Mister Bijou at 4:34 AM on May 6, 2019


I wonder what happened when everyone who was so outraged asked for their money back?

That’s not how this works. The fact that Firefox is free does not absolve us of the commitments we’ve made to the people who use it.
posted by mhoye at 4:44 AM on May 6, 2019 [7 favorites]


66.0.4. Brilliant!
Many, many thanks.
posted by Mister Bijou at 5:04 AM on May 6, 2019 [1 favorite]


Interestingly, this bug broke add-ons on my Waterfox for Android, but not on my desktop Waterfox. Then again, a friend tells me that it did affect his desktop Waterfox. So who even knows.
posted by Too-Ticky at 5:59 AM on May 6, 2019


Quantum killed DownThemAll

Rubbish. The developers of that extension killed DTA. They had 18 months to get their arses into gear and produce something new but instead they decided to dig their heels in and WOE IS ME about the whole thing. If you want to blame anyone for the loss of DTA blame them.
posted by urbanwhaleshark at 6:27 AM on May 6, 2019 [4 favorites]


I think that some people who used password managers that got disabled were more inconvenienced, but ... I would never put my passwords into a database I could only access with an add-on. Precisely because add-ons sometimes get yanked or disabled or bugged.

On this: when Quantum was nearing its cut off date LastPass's updated extension was an absolute hot mess. Mostly it just flat-out refused to work. I exported all my passwords and looked for an alternative. Eventually I switched to Keepass (with Keepass2Android on my phone) and haven't looked back.
posted by urbanwhaleshark at 7:54 AM on May 6, 2019 [1 favorite]


Got the update at work today and now my taskbar icon is a default empty page icon. But on the bright side the Studies fix seemed to fix things for me and I didn't even notice the issue here before the update installed.
posted by Nonsteroidal Anti-Inflammatory Drug at 8:08 AM on May 6, 2019


The "process changes" outlined there amount to "we promise to actually follow the guidelines we already had"

I read them as "we promise not to let Marketing pressurize the engineering team again" but that's probably just my prejudices showing; as a retired software engineer I can attest that an engineer's worldview is no less patchy than a marketroid's, our blind spots every bit as profound albeit in different places, and our propensity to act as if we had none every bit as potentially infuriating.

the "fix" for the current problem also violates at least the first of those principles, and doesn't fit in with any of the other stated uses of either Shield or Studies

And that is why I believe that leaving arbitrary code execution back doors active in stuff I rely on is a worse idea than turning them off: policy around them is no protection against abuse because it will inevitably be ignored eventually. All it takes is somebody - and in this particular case it was surely engineers, but there will always be somebody - deciding that some issue or other is so pressing that it justifies Can = Must and guidelines be buggered.

Power corrupts; arbitrary code execution power corrupts arbitrarily. I would much rather that decisions about how much to trust any given software provider only needed making at those specific times when I'm deciding whether or not to install what's on offer, and that surprising new behaviours in long-installed software were generally recognized as bugs.

It doesn't take much odd behaviour to convince the typical naive user that their machine is horribly infected and destroy their basic trust in it. I know this because since retiring from the software development game I've been keeping myself busy doing PC support and I meet a lot of typical naive users, and a great deal of what I've been doing lately has involved reassuring people that no, the bizarre behaviour you just started seeing does not mean it's no longer safe to operate your online banking, it just means that Firefox updated itself and broke NoScript again.

The Quantum cutover in particular was huge for my customers. I had long been installing NoScript on customer computers, giving them a clear explanation of what it did and why that's something they should value, and training them up to use it competently; but the revamped NoScript UI that arrived with the WebExtensions version left most of them floundering because frankly it's fucking horrible, and the new version did not inherit whitelists from the old XPCOM version so all those pages they'd all forgotten they'd whitelisted three years ago were all busted again. And sure, most of that is Giorgio Maone's fault rather than Mozilla's, but that's a distinction lost on the typical customer.

But there was an explainable reason and rationale for that inconvenience. Mozilla was forced to lock down its extensions ecosystem because of how badly it was being abused by malware developers, and there are certain things that new-style extensions can't do that old-style extensions could, and the new NoScript UI is at least in part a reflection of that. Folks understand when you tell them something like that and it leaves them trusting and appreciating their browser more, not less.

The way the current fuckup is being handled absolutely doesn't. It's far more reminiscent of the last time Studies got abused, and I don't think that's coincidence.

I'd trained my customers to cast an eye over their installed add-ons every now and then and make sure they still remembered what each one did and why it was there. So after the idiotic Mr Robot bullshit got force-installed, I had a flood of phone calls from people frightened into believing it wasn't safe to do their banking any more and could I please come for a visit.

From a browser engineer's point of view, malware is anything that somebody has found to be doing stuff that the engineer disapproves of (like harvesting passwords); but that's typically not what malware looks like from the point of view of most users. Malware developers obviously try to keep their stuff pretty well camouflaged, but Sturgeon's Law applies to malware every bit as much as it does to science fiction, so users will quite often see bits of it poking out and it's these odd little bits of new behaviour that characterize malware from the user's point of view.

Before Quantum locked things down, Firefox malware would often show up as extensions with odd names that users didn't remember installing, or as odd behaviour they didn't remember encountering before, and it's those little things that users had been trained to interpret as clear evidence of infection. Most remain blissfully unaware of the gross violations of privacy that malware is actually built to inflict, as these remain largely invisible.

Now, there's a reason that banks don't send emails with clickable links in them: it makes it possible to train people that emails that purport to be from your bank and do have clickable links in them are just phishing attempts. Appearances matter and consistently doing the right thing matters too.

So it was close to unforgivable for Mozilla to make their own browser behave in a manner indistinguishable from having suffered a malware infection from a user's point of view. Just an unspeakably stupid decision. It's hard enough to inculcate some basic level of security awareness in technologically naive people without having that effort actively undermined by the browser vendor.

Had it not been for the fact that there is no better browser available in the market I would certainly have switched my customers to something else. But as far as I'm aware there is nothing else available that does as good a job at keeping shit out of people's machines while failing to spy on them as Firefox + NoScript + uBlock Origin (leaving aside the various Firefox derivatives like PaleMoon and WaterFox, which I've been unwilling to recommend to other people on the basis of doubts about their ongoing sustainability). I'm not about to vote for Trump because of being let down by Obama on Gitmo, which is essentially how I feel about all those folks reflexively jumping ship to Chrome in the last few days.

The current certificate fuckup is nowhere near as inherently bad because the initial inconvenience was not deliberate. Shit happens. It's a temporary inconvenience only, and it comes with a specific, unambiguous warning containing a nice link to an information resource: well done Moz for that, and for making a fix available so fast. But I am still not seeing any documentation from Mozilla on what the fix actually does, having had to rely on other people to reverse engineer it to figure that out; and I am still not seeing any information about why hand-installing a new intermediate cert and then forcing affected extensions to be re-checked is in some way unsound.

Mozilla appears to be operating like the Church, doing its level best to sweep its fuckup under the rug as fast as possible to minimize the number of people who actually notice that anything's gone wrong in the first place. I think that's a mistake. I think it should have prioritized making a one-click manual fix easily available via the page that the warning linked to, along with a clear explanation of what the fix does and a set of steps for replicating it by hand in case it didn't work, instead of being all oh, you don't want to look in there. Because that's what software vendors who fuck up do and that's something users can forgive. Hell, even Microsoft gets this right.

This whole business with oh shit oh shit push out a silent fix as fast as possible and don't tell anybody what the fix actually did? Not cool. Vendors who want to be seen to be trustworthy own their fuckups, they don't try to cover them up because "convenience". Software that breaks mysteriously and then fixes itself equally mysteriously doesn't look trustworthy, it looks flaky as fuck, and Firefox is too good a browser to risk looking flaky as fuck.

And again, I'll continue to use it and recommend it because there's nothing better out there, a fact of which the dev team can be justifiably proud. But it would be good to take the wagon-circling down a notch.
posted by flabdablet at 9:30 AM on May 6, 2019 [6 favorites]


I would never put my passwords into a database I could only access with an add-on. Precisely because add-ons sometimes get yanked or disabled or bugged.

I would never put mine into a database I could only access in the cloud, for completely parallel reasons. KeePass FTW from day 1.
posted by flabdablet at 9:32 AM on May 6, 2019 [2 favorites]


Mozilla appears to be operating like the Church, doing its level best to sweep its fuckup under the rug as fast as possible

Our #1 priority is solving the problem for our users correctly and as quickly as possible. As soon as we deployed the hotfix we also posted on the addons blog about what we were doing and how it worked, and we've been updating that post to provide additional information since. When we are fully certain the incident has been resolved, we will begin a postmortem process and share those results when it's complete.

I know we're all running hot right now, but that's not an acceptable analogy at all.
posted by mhoye at 10:37 AM on May 6, 2019 [9 favorites]


Ran an update on my Debian 9 system this morning and Firefox ESR 60.6.2esr-1~deb9u1 showed up to replace 60.6.1esr-1~deb9u1, and all of my extensions were active again on re-start. So yay Mozilla and Debian and everyone else involved!
posted by XMLicious at 10:50 AM on May 6, 2019


Vendors who want to be seen to be trustworthy own their fuckups

Mozilla is not a vendor.
posted by JackFlash at 11:31 AM on May 6, 2019 [2 favorites]


It seems like we're needlessly delving into semantics now, but I'll bite: what would you call them, then? And why?
posted by Greg_Ace at 11:36 AM on May 6, 2019


It seems like we're needlessly delving into semantics

In the English language, a vendor is a person or company who sells a product or service. It's a pretty clear and simple word.

It comes from Latin vendere, to sell. In French, vendre, also means to sell.
posted by JackFlash at 11:46 AM on May 6, 2019 [1 favorite]


It comes from Latin vendere, to sell. In French, vendre, also means to sell.

The reality is slightly more subtle and involved than you might at think from that clever quip. Wiktionary says the meaning of "vend" is "to hawk, or peddle merchandise". Whether modern jargon-y usage in the software industry has drifted sufficiently far from the dictionary definition is a another question. Peddling is defined as going door to door to sell things, but the less literal meaning of peddling an idea or point of view is also current. Vend does mean to sell, but of course sell too can be interpreted with a similar meanings such as "to promote a product" or a viewpoint. The definition of merchandise is more broad, involving the trade of articles of commerce and such; surely a web browser could be included. Mozilla does have a marketing department. Can you "market" something without "vending" it?

Much like the patch notes for this release, one could go into more detail than you might at first think.
posted by sfenders at 12:08 PM on May 6, 2019 [2 favorites]


I understand there could have been serious consequences for some people. I don't know why that means other people need to be upset on behalf of these theoretical people and their theoretical problem. Could they not be upset on their own?

I was annoyed, but mostly a little distrustful of the way this all went down, everything about it, so I waited and downloaded the full update. Still not 100% on trust.

But I didn't pay for this. It was free.

That’s not how this works. The fact that Firefox is free does not absolve us of the commitments we’ve made to the people who use it.

That's admirable, seriously, but doesn't really change the view from this end. I don't know that I've ever donated to Mozilla, or thought about it. Since I've never been asked for a donation I assumed they had enough money. If I had donated a lot I might feel differently.


I feel like there's a strong "I'd like to speak to your manager" vibe to this whole reaction, but since it's not women, but men with very important tech stuff going on it's different.
posted by bongo_x at 12:19 PM on May 6, 2019 [1 favorite]


Mozilla is not a vendor.

Then s/vendor/provider/g; the point still stands.

Okay, so. Here is a simple and direct fix that I believe to be a fix, not a workaround, and which I have just walked through again and verified as working for me (Debian Firefox 62.0.2 64-bit). As far as I can tell it does the same thing as the official fixes that Mozilla is offering via various software update mechanisms. However, everything you need in order to apply it is contained in this comment, and in particular it involves neither running nor installing any software and causes no change to your Firefox version. All that's required is importing one certificate, resetting one setting in about:config, and restarting Firefox.

As far as I am aware all versions of Firefox for which extension signing is an issue are capable of doing these things, so I can think of no reason why this procedure would fail on other Firefox versions, but I'm a mere lay parishioner so what would I know. If anybody wishes to excommunicate me for offering up this heresy, so be it. But I would appreciate a MeMail from anybody else for whom the fix works; also from anybody for whom it doesn't, either because my instructions are unclear or inapplicable or because you've worked through them and your extensions are still busted.

Copy all of the following lines from the BEGIN CERTIFICATE line to the END CERTIFICATE line inclusive, paste them into a new empty document in your favourite plain text editor, then save it as icfix.pem. If your favourite plain text editor happens to be Microsoft Notepad you will need to change the Save As Type setting from Text Documents to All Files, otherwise Notepad will make icfix.pem.txt which won't work.
-----BEGIN CERTIFICATE-----
MIIHLTCCBRWgAwIBAgIDEAAIMA0GCSqGSIb3DQEBDAUAMH0xCzAJBgNVBAYTAlVT
MRwwGgYDVQQKExNNb3ppbGxhIENvcnBvcmF0aW9uMS8wLQYDVQQLEyZNb3ppbGxh
IEFNTyBQcm9kdWN0aW9uIFNpZ25pbmcgU2VydmljZTEfMB0GA1UEAxMWcm9vdC1j
YS1wcm9kdWN0aW9uLWFtbzAeFw0xNTA0MDQwMDAwMDBaFw0yNTA0MDQwMDAwMDBa
MIGnMQswCQYDVQQGEwJVUzEcMBoGA1UEChMTTW96aWxsYSBDb3Jwb3JhdGlvbjEv
MC0GA1UECxMmTW96aWxsYSBBTU8gUHJvZHVjdGlvbiBTaWduaW5nIFNlcnZpY2Ux
JjAkBgNVBAMTHXNpZ25pbmdjYTEuYWRkb25zLm1vemlsbGEub3JnMSEwHwYJKoZI
hvcNAQkBFhJmb3hzZWNAbW96aWxsYS5jb20wggIiMA0GCSqGSIb3DQEBAQUAA4IC
DwAwggIKAoICAQC/qluiiI+wO6qGA4vH7cHvWvXpdju9JnvbwnrbYmxhtUpfS68L
bdjGGtv7RP6F1XhHT4MU3v4GuMulH0E4Wfalm8evsb3tBJRMJPICJX5UCLi6VJ6J
2vipXSWBf8xbcOB+PY5Kk6L+EZiWaepiM23CdaZjNOJCAB6wFHlGe+zUk87whpLa
7GrtrHjTb8u9TSS+mwjhvgfP8ILZrWhzb5H/ybgmD7jYaJGIDY/WDmq1gVe03fSh
xD09Ml1P7H38o5kbFLnbbqpqC6n8SfUI31MiJAXAN2e6rAOM8EmocAY0EC5KUooX
KRsYvHzhwwHkwIbbe6QpTUlIqvw1MPlQPs7Zu/MBnVmyGTSqJxtYoklr0MaEXnJN
Y3g3FDf1R0Opp2/BEY9Vh3Fc9Pq6qWIhGoMyWdueoSYa+GURqDbsuYnk7ZkysxK+
yRoFJu4x3TUBmMKM14jQKLgxvuIzWVn6qg6cw7ye/DYNufc+DSPSTSakSsWJ9IPx
iAU7xJ+GCMzaZ10Y3VGOybGLuPxDlSd6KALAoMcl9ghB2mvfB0N3wv6uWnbKuxih
q/qDps+FjliNvr7C66mIVH+9rkyHIy6GgIUlwr7E88Qqw+SQeNeph6NIY85PL4p0
Y8KivKP4J928tpp18wLuHNbIG+YaUk5WUDZ6/2621pi19UZQ8iiHxN/XKQIDAQAB
o4IBiTCCAYUwDAYDVR0TBAUwAwEB/zAOBgNVHQ8BAf8EBAMCAQYwFgYDVR0lAQH/
BAwwCgYIKwYBBQUHAwMwHQYDVR0OBBYEFBY++xz/DCuT+JsV1y2jwuZ4YdztMIGo
BgNVHSMEgaAwgZ2AFLO86lh0q+FueCqyq5wjHqhjLJe3oYGBpH8wfTELMAkGA1UE
BhMCVVMxHDAaBgNVBAoTE01vemlsbGEgQ29ycG9yYXRpb24xLzAtBgNVBAsTJk1v
emlsbGEgQU1PIFByb2R1Y3Rpb24gU2lnbmluZyBTZXJ2aWNlMR8wHQYDVQQDExZy
b290LWNhLXByb2R1Y3Rpb24tYW1vggEBMDMGCWCGSAGG+EIBBAQmFiRodHRwOi8v
YWRkb25zLm1vemlsbGEub3JnL2NhL2NybC5wZW0wTgYDVR0eBEcwRaFDMCCCHi5j
b250ZW50LXNpZ25hdHVyZS5tb3ppbGxhLm9yZzAfgh1jb250ZW50LXNpZ25hdHVy
ZS5tb3ppbGxhLm9yZzANBgkqhkiG9w0BAQwFAAOCAgEAX1PNli/zErw3tK3S9Bv8
03RV4tHkrMa5xztxzlWja0VAUJKEQx7f1yM8vmcQJ9g5RE8WFc43IePwzbAoum5F
4BTM7tqM//+e476F1YUgB7SnkDTVpBOnV5vRLz1Si4iJ/U0HUvMUvNJEweXvKg/D
NbXuCreSvTEAawmRIxqNYoaigQD8x4hCzGcVtIi5Xk2aMCJW2K/6JqkN50pnLBNk
Px6FeiYMJCP8z0FIz3fv53FHgu3oeDhi2u3VdONjK3aaFWTlKNiGeDU0/lr0suWf
QLsNyphTMbYKyTqQYHxXYJno9PuNi7e1903PvM47fKB5bFmSLyzB1hB1YIVLj0/Y
qD4nz3lADDB91gMBB7vR2h5bRjFqLOxuOutNNcNRnv7UPqtVCtLF2jVb4/AmdJU7
8jpfDs+BgY/t2bnGBVFBuwqS2Kult/2kth4YMrL5DrURIM8oXWVQRBKxzr843yDm
Ho8+2rqxLnZcmWoe8yQ41srZ4IB+V3w2TIAd4gxZAB0Xa6KfnR4D8RgE5sgmgQoK
7Y/hdvd9Ahu0WEZI8Eg+mDeCeojWcyjF+dt6c2oERiTmFTIFUoojEjJwLyIqHKt+
eApEYpF7imaWcumFN1jR+iUjE4ZSUoVxGtZ/Jdnkf8VVQMhiBA+i7r5PsfrHq+lq
TTGOg+GzYx7OmoeJAT0zo4c=
-----END CERTIFICATE-----
Open Firefox's Options or Preferences, open the Privacy & Security section, scroll down to find the Certificates paragraph, then click View Certificates... to open the Certificate Manager.

In the Certificate Manager window, click the Authorities tab, then click Import. This will open a file picker window in which you should browse to the folder where you saved icfix.pem, then click on icfix.pem, then click Open.

You should see a Downloading Certificate window asking whether you want to trust "signingca1.addons.mozilla.org". Click its View button and make sure the certificate looks legit; in particular, check that its SHA1 fingerprint is 9A:70:74:99:8F:46:9F:7C:57:DB:E3:8A:E9:1E:00:0A:4B:50:F8:48. All being well, close the Certificate Viewer window, then click OK in the Downloading Certificate window to install the certificate.

Back in the Certificate Manager window, scroll through the list of certificate authorities and make sure you now have one from Mozilla Corporation named signingca1.addons.mozilla.org. All being well, close the Certificate Manager. Incidentally, if you later want to restore your Firefox to the exact condition it was in before you followed these instructions, using Certificate Manager to delete that new certificate is all you need to do.

Type about:config into the browser's address bar and press Enter. If you see a warning about "voiding your warranty" (which is an attempt at humour, the licence explicitly disclaiming that any such thing exists) then click through its acknowledgement button.

In the about:config Search box, type xpi (no need to press Enter, search filtering happens as you type).

Look for the setting named app.update.lastUpdateTime.xpi-signature-verification (it will be in bold type, indicating that it's been modified); right-click on it, then click Reset in the resulting pop-up menu.

Quit Firefox, then restart it, then wait approximately one minute. Your disabled extensions should restore themselves.

Thanks to benb at www.velvetbug.com for the reverse engineering work.

Our #1 priority is solving the problem for our users correctly and as quickly as possible.

You say that, and I'm quite sure you believe it, but if it were actually true you would have documented what needed to be done before launching into the intensive debugging effort required to get it done correctly and automatically in software. Making the materials required for a manual fix available quickly would have saved time for those of us capable of following instructions like those I've written here, and would have saved Mozilla team members time they've now had to waste on finding and undoing side effects of all those half-assed workarounds that have allegedly been causing so much grief.
posted by flabdablet at 1:11 PM on May 6, 2019 [2 favorites]


Making the materials required for a manual fix available quickly would have saved time for those of us capable of following instructions like those I've written here

Firefox has over 250 million users. I doubt that spending time on a complicated manual fix for a handful of special snowflakes was a high priority.
posted by JackFlash at 1:31 PM on May 6, 2019 [2 favorites]


Also the sort of folks who run NoScript and know how to perform surgery on their certificate stores are not exactly the median consumer that Firefox is trying to serve.
posted by Nelson at 1:49 PM on May 6, 2019 [2 favorites]


In particular, there's a lot of value in not having a bunch of people who have implemented some kind of manual fix that might:

- Conflict with the official fix
- Pose a security vulnerability
- Be a problem when the new certificate expires in a couple of years
- Leave some unspecified number of people with a non-standard configuration that has to be tested and supported forever

Since the official fix now boils down to "update your browser to the latest version," I'm not sure why following these steps is an improvement over that?

Debian Firefox 62.0.2 64-bit

Just out of curiosity, why are you using 62.0.2 in particular?
posted by zachlipton at 1:56 PM on May 6, 2019 [2 favorites]


You say that, and I'm quite sure you believe it, but if it were actually true you would have documented what needed to be done before launching into the intensive debugging effort required to get it done correctly and automatically in software.

What you’re describing is called the waterfall development model and it’s well-understood at this point to not be an effective way to ship reliable software, or to ship software reliably.
posted by mhoye at 4:13 PM on May 6, 2019 [5 favorites]


You say that, and I'm quite sure you believe it, but if it were actually true...

wow
posted by urbanwhaleshark at 5:12 PM on May 6, 2019 [1 favorite]


Firefox has over 250 million users

some of whom still think of it as free software and have chosen to use it for exactly that reason:
“Free software” means software that respects users' freedom and community. Roughly, it means that the users have the freedom to run, copy, distribute, study, change and improve the software. Thus, “free software” is a matter of liberty, not price. ... With these freedoms, the users (both individually and collectively) control the program and what it does for them. When users don't control the program, we call it a “nonfree” or “proprietary” program. The nonfree program controls the users, and the developer controls the program; this makes the program an instrument of unjust power.
- Conflict with the official fix
- Pose a security vulnerability


...by making the same changes to browser configuration that the official fix does? Not seeing it.

- Be a problem when the new certificate expires in a couple of years

The new certificate expires in April 2025. That's not a couple of years, that's six years. The number of users who will have applied not even a single official update in six years is going to be very very small indeed.

- Leave some unspecified number of people with a non-standard configuration that has to be tested and supported forever

Millions of people generate non-standard Firefox configurations every single day, simply by installing their own preferred combination of extensions. Compared to that degree of sheer configuration chaos, any mess made by folks who prefer specific minimal fixes for specific issues is going to amount to a melted special snowflake in the ocean.

Since the official fix now boils down to "update your browser to the latest version," I'm not sure why following these steps is an improvement over that?

Because it makes the absolute minimum alteration to system configuration required to address the problem at hand, and that's a strong preference for those of us who have been burned before by vendor-knows-best mystery-meat updates applied on vendors' schedules rather than our own.

why are you using 62.0.2 in particular?

Because it's what was in Debian Unstable when I last did a round of software upgrades, and I've had no pressing reason to do another round since.

What you’re describing is called the waterfall development model

That's a stretch and you know it. "We need to install this new intermediate cert and then force add-on signing to be re-checked" is not in any way comparable to the design documentation for a large system.

and it’s well-understood at this point to not be an effective way to ship reliable software, or to ship software reliably

and since fixing this specific issue does not actually require any software to be shipped, that's a moot point.

This institutionalized inability to conceive of its mission as anything other than "shipping software" is clericalism for engineers, it's toxic to the free software movement, and when Mozilla does get around to this issue's post-mortem it ought to be given serious consideration.
posted by flabdablet at 8:34 PM on May 6, 2019


-----BEGIN CERTIFICATE-----
MIIHLTCCBRWgAwIBAgIDEAAIMA0GCSqGSIb3DQEBDAUAMH0xCzAJBgNVBAYTAlVT
MRwwGgYDVQQKExNNb3ppbGxhIENvcnBvcmF0aW9uMS8wLQYDVQQLEyZNb3ppbGxh
...
eApEYpF7imaWcumFN1jR+iUjE4ZSUoVxGtZ/Jdnkf8VVQMhiBA+i7r5PsfrHq+lq
TTGOg+GzYx7OmoeJAT0zo4c=
-----END CERTIFICATE-----
Cool!

/me loads a cert somebody posted in a comment on the internet into his root trust store...
posted by russm at 8:46 PM on May 6, 2019 [11 favorites]


some of whom still think of it as "free software"

Well, some of you can think all you want but using the definitions of that very same web site you linked to, gnu.org, they explicitly denounce Firefox as not complying with their "free software" requirements. So it's really a mystery why anyone would think so. In fact, Gnu makes their own fork called IceCat which you are free to use as "free software", meeting your cited definition, if that is so important to you, instead of whining about Firefox.
posted by JackFlash at 9:45 PM on May 6, 2019 [1 favorite]


If you want to use an old version of Firefox with known security vulnerabilities and follow a moderately complicated series of steps to install random certificates copy/pasted into metafilter comments into your root trust store, knock yourself out, but I think there are rather solid reasons why that's a course of action Mozilla isn't going to recommend to its users in place of "let the automatic updater fix it."

Anyway, since it is, the FSF's objections aside, free software, you're welcome to use your freedoms to do with it as you please, and it seems like that's working out for you.
posted by zachlipton at 2:31 AM on May 7, 2019 [3 favorites]


/me loads a cert somebody posted in a comment on the internet into his root trust store...

That's exactly what many people have done, which is why at least the basic info including that certificate should have come from Mozilla at some point. Communicating to the public what the bug fix is going to do even before issuing it would not constitute an adoption of the "waterfall" model. In fact, I hear some other development methodologies even allow some limited use of actual design documents!

The arguments against going further and publishing the more detailed instructions for how to install the new cert serve equally well as arguments against having a certificate store UI enabled at all. Perhaps it should be shut down, except in the developer build, or only on Windows. Then we'd have had even more fun ad-hoc workarounds.
posted by sfenders at 5:07 AM on May 7, 2019 [1 favorite]


So, they fixed the glitch?
posted by Huffy Puffy at 5:08 AM on May 7, 2019 [1 favorite]


This institutionalized inability to conceive of its mission as anything other than "shipping software" is clericalism for engineers,

That’s the second time you’ve gone to the well for that catholic-church analogy, so I guess I didn’t fully convey how offensive it was the first time. Regardless, I don’t think this is going to be a productive conversation and the blue deserves better.

If anyone else has questions about where we are or what’s next, I’m happy to talk about them.
posted by mhoye at 5:35 AM on May 7, 2019 [7 favorites]


So, they fixed the glitch?

We’re 100% rolled out for release and current ESR; because of an odd bit of numerology over at the Play Store we’re at 99.99% rollout on Android. Apparently you don’t have a rollback option if you go to 100% deployment there, but anything less than that leaves the option open? So we’re keeping both feet well inside that line for a few days to keep that option open.

We still have patches to sort out for older versions of Firefox, going back I’m not sure how far; under normal circumstances we’d be asking people to update to the current release, but these aren’t normal circumstances and we’re going to sort that out shortly.
posted by mhoye at 5:47 AM on May 7, 2019 [7 favorites]


Congratulations mhoye and thanks for your communication with us here on MeFi.
posted by Nelson at 7:32 AM on May 7, 2019 [3 favorites]


Just stopping in to say I loves me some FF. That it borked, and the Interwebs was awful for a few minutes, was just proof. Smooches to the 'zills!
posted by petebest at 7:35 AM on May 7, 2019 [1 favorite]


Do Mozilla's fighty vigilante defenders think they are helping?

I'm seeing a lot more fightiness from the detractors, but helping what? This conversation?

Hopefully slightly more than "Fuck you, I'm downloading Ka-Rome!"
posted by aspersioncast at 7:47 AM on May 7, 2019 [3 favorites]


I’m about to joke about somebody making and/or subsequently breaking an add-on to Lynx to show images, but then I realized it almost definitely exists already.
posted by Huffy Puffy at 8:14 AM on May 7, 2019


I'm seeing a lot more fightiness from the detractors, but helping what? This conversation?

Hopefully slightly more than "Fuck you, I'm downloading Ka-Rome!"
posted by aspersioncast at 12:47 AM on May 8 [2 favorites +] [!]
Eponysterical…
posted by Pinback at 3:34 PM on May 7, 2019


/me loads a cert somebody posted in a comment on the internet into his root trust store

In general I agree with you that doing that is howlingly unsound. Which is why I also strongly agree with sfenders that this same certificate should be made readily available, in some form compatible with Firefox's inbuilt Certificate Manager, from a demonstrably trustworthy Mozilla source.

I also agree that technically speaking, this particular cert has no business being in a root trust store; I am quite sure that the internal Mozilla security processes employed to keep its private key secret do not meet the rigorous standard one would generally require for a root cert because if they did, there would be no point in issuing these intermediate certs in the first place.

However, given that extensions updated as late as May 7th are still being signed using the old, expired intermediate certificate, then having Firefox unconditionally accept either this new cert or any other bearing the same ID and public key as the expired one probably really is the only way to make add-on authenticity checks succeed again for the time being.

If you use openssl to take apart the certificate I posted and check it against the intermediate cert contained in any add-on already installed and/or available via addons.mozilla.org, you will find that its ID and public key are indeed identical to those from the old, expired intermediate certificate. I am happy to write up the process by which I did that before posting it, if anybody's interested.
posted by flabdablet at 4:36 AM on May 8, 2019


Looks like today's 66.0.5 release has taken a new approach to forcing absolute trust in this new intermediate cert, by hard-coding it into Firefox as has long been done for its root (which is why root-ca-production-amo isn't visible in the Certificate Manager).

I wonder what the plan is for the day when signingca1.addons.mozilla.org's private key does eventually leak.

If Mozilla is going to continue taking on centralized responsibility for locking malware out of the add-ons ecosystem, it would be polite to offer Firefox users some UI controls for dealing with broken add-on signatures that are at least as granular as those we already have for dealing with broken SSL certificate chains on https websites. Big scary warnings: good. Absolute failure to chooch: not so much.
posted by flabdablet at 5:13 AM on May 8, 2019 [1 favorite]


It should also be the case that the entire certificate chain required for validating the end-entity certs that ship with signed add-ons should always be available, in a format compatible with the Certificate Manager, via some well-known page at addons.mozilla.org; and that all the certificates made available that way get updated well before expiry.
posted by flabdablet at 5:40 AM on May 8, 2019


Found it. It was on display in the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying "Beware of the Leopard", but addons-public-intermediate.crt is the new add-on signing intermediate cert good until April 2025.

Because this cert is hardcoded into Firefox itself as of 66.0.5, you might not see it in Certificate Manager going forward. Users of unsupported versions should be able to download and import it as-is using Certificate Manager to restore add-on functionality, or convert it from binary to PEM format to verify that it does indeed match the one I posted earlier.
posted by flabdablet at 9:03 AM on May 8, 2019 [1 favorite]


Yeah but I want to know what happened to the leopard.
posted by Greg_Ace at 10:09 AM on May 8, 2019


What happened?
posted by hugbucket at 8:09 AM on May 9, 2019




Yeah but I want to know what happened to the leopard.

It couldn't read the fox's signature, so it exploded.

Anyway, congratulations and thanks to the Firefox team. While I think one or two things could've been done differently, now that the initial shock and horror is over that's overshadowed by respect and admiration for the determination and speed with which this got patched up. Fixing it this quickly without making things worse wasn't an easy job.
posted by sfenders at 2:29 PM on May 9, 2019 [2 favorites]


And here’s an additional post from our VP of engineering.

One thing to note is that we’re deleting all the telemetry data we received over the course of this incident. Some people - probably only a handful- were clearly opting into studies because they felt they had to do so to get their Firefox working again; that looks a lot like coerced consent, and that’s not ok. But we can’t tell that presumably tiny amount of telemetry data from any other, and as a result we have to throw out all of it.
posted by mhoye at 2:30 PM on May 9, 2019 [9 favorites]


From that additional post:
Finally, we’ll be looking more generally at our add-on security architecture to make sure that it’s enforcing the right security properties at the least risk of breakage.
Seems to me that the principles ought to be as follows:

1. Add-ons can't be installed into Firefox unless (a) they've passed a Mozilla review and been digitally signed or (b) they've been okayed by a user who deliberately overrides a big scary warning, as is currently offered for assorted kinds of SSL breakage on https websites or (c) a user has turned off installation-time add-on checking entirely by setting a property in about:config, which must always be available and respected.

Revocation of parts of the installation certificate chain should only ever be done explicitly, not via expiration dates, and only when Mozilla has good reason to believe that a private key has leaked and installation certificates are being forged in the field. In no case should revocation occur before affected add-ons have been made available on addons.mozilla.org with new signatures.

2. Once installed and enabled by the user, an add-on can get disabled only when (a) the user disables or uninstalls it or (b) Mozilla specifically blacklists it by ID or (c) the content included in the installation-time hash checks turns out to hash to a different value on a before-use check. There should be no re-validation of the original signature chain, just a tripwire-style content check against a locally stored hash database.

Downloaded blacklist entries should include a clear description of exactly why Mozilla has blacklisted that add-on. Checking for blacklisted add-ons should be done instead of the current signature re-validation process, and could be done at about the same frequency.

3. Any add-on that has been automatically disabled must be able to be re-enabled manually, the process to include specifically overriding a big scary warning.

4. Baked-in add-ons like Pocket or Hello or the execrable PDF.js or that search-harvesting thing whose name escapes me ought to appear in the Add-Ons list, just like stuff I added myself does, so that I can disable or (preferably) uninstall them. The principles here are that just because I choose to trust Mozilla, that shouldn't mean I'm forced to trust Mozilla's marketing affiliates as well; and just because something can be done in JavaScript doesn't mean it should be. As a school IT technician, PDF.js wasted countless hours of my time and that of the teachers I was supporting, mainly due to its spontaneous replacement of a fully functional PDF reader with something that totally failed to understand the school printers.

5. Add-ons should never be pushed by Mozilla by any mechanism without specific, informed, push-time opt-in. The only thing that should ever resemble infection by malware should be infection by malware.
posted by flabdablet at 4:31 PM on May 10, 2019


As promised, Here's a fix for pre-57 versions of Firefox, hot off the QA presses. I hope it sorts out those of you stuck on earlier versions. There's an excellent chance it will work on pre-52 Firefox, if you're stuck that far back, but we can't guarantee it.

Please consider upgrading to current Firefox if you have that option! It's a lot better, for real.
posted by mhoye at 1:44 PM on May 13, 2019 [3 favorites]


The pre-57 fix performs the following steps during installation:

1. If the user has set a master password, log in with it.
2. Inject the new add-on signing intermediate certificate into the certificate database.
3. If certificate injection failed and the current date is before 20-May-2019, set about:config preference app.update.lastUpdateTime.xpi-signature-verification to the current time to defer signature re-verification for another 24 hours.
4. If certificate injection succeeded, force add-on signatures to be re-verified immediately.

It doesn't do anything after being installed, and it doesn't remove the injected certificate when uninstalled.

Successful installation and subsequent uninstallation of this fix makes exactly the same changes to your Firefox configuration as manually installing the new intermediate certificate.
posted by flabdablet at 9:28 PM on May 13, 2019 [1 favorite]


As it happens I had an issue with an expired certificate while using Mozilla today, and I'd like to share it together with my thoughts.

I went to this page of Tema Smith's blog, which appears to include an element from cowbird.com. Cowbird.com was a sort of semi-curated publishing platform that closed a couple of years ago. It's in archive mode now.

Here's the message I received within the element linking to cowbird.com:
Warning: Potential Security Risk Ahead

Firefox detected an issue and did not continue to cowbird.com. The website is either misconfigured or your computer clock is set to the wrong time.

It’s likely the website’s certificate is expired, which prevents Firefox from connecting securely. If you visit this site, attackers could try to steal information like your passwords, emails, or credit card details.

What can you do about it?

Your computer clock is set to 5/14/2019. Make sure your computer is set to the correct date, time, and time zone in your system settings, and then refresh cowbird.com.

If your clock is already set to the right time, the website is likely misconfigured, and there is nothing you can do to resolve the issue. You can notify the website’s administrator about the problem.

Learn more…

☐ Report errors like this to help Mozilla identify and block malicious sites
The message about my computer time is practically useless. I suppose that an incorrect would hardly ever cause a problem like this, and in any case it would be easy for Mozilla to check that the date is correct. I have two buttons below the error message: "Go Back (Recommended)", and "Advanced..." (sic). Neither is of any use. "Go Back" takes me to my home page - I have no idea why. "Advanced" gives me this message:
Websites prove their identity via certificates, which are valid for a set time period. The certificate for cowbird.com appears to be expired.

Error code: SEC_ERROR_EXPIRED_CERTIFICATE
View Certificate
This time I have three choices. Clicking on "SEC_ERROR_EXPIRED_CERTIFICATE" gives me a page of base-64 gobbledegook that is apparently the actual certificate. "View Certificate" helpfully tells me that I can't view the certificate, because it is expired. No, duh. And "Go Back (Recommended)" takes me to my home page again.

NONE OF THESE ARE USEFUL OR HELPFUL. This is just a stupid blog page. I don't think it even has any fields that I could use to madly enter passwords, emails, and credit card details. I have no way to override the decision the browser made for me, and the buttons are less than helpful. This is an experience only an engineer could love. I have no doubt that some users here could figure out a way around the roadblock; I can't. I wouldn't bother, anyway, because it's just some blog page. The page itself is a couple of years old, cowbird.com is closed; there's not much chance anyone is going to fix the certification problem.

I've been using Mozilla since I switched from Netscape Navigator. I am deeply grateful to them, and to the engineers that have worked on its predecessors all the way back to Mosaic. I don't think that precludes me having an opinion on this, though: the certification process is intrinsically bad, but the way Mozilla handles expired certificates makes it worse. It is stupid, obstructive, patronising, and confusing. It breaks the internet. The user interface is meaningless and repetitive and false. The "Learn more..." option is actively wrong. Expired certificates are a common problem and Mozilla should be ashamed that their program handles them so badly.
posted by Joe in Australia at 5:05 AM on May 14, 2019 [1 favorite]


The message about my computer time is practically useless. I suppose that an incorrect would hardly ever cause a problem like this

In my experience, date-related certificate failures have indeed been caused by an incorrectly set local time more often than not. I've seen this happen about four times as often as I've seen actual expired certificates. This might be because I encounter other people's old computers with busted clock batteries more often than most people do, but it's true all the same.

and in any case it would be easy for Mozilla to check that the date is correct.

It's not Mozilla (the organization) checking the current date against the validity range recorded in the certificate, but Firefox (the local browser); if your computer is giving Firefox bad information about the current date and time, there's not much Firefox can do about that.

In fact, one of the standard workarounds for HTTPS sites with expired SSL certificates involves deliberately though temporarily setting your system clock to some time within the certificate's validity range, and reloading the page. It shouldn't need to be done, because anybody serving HTTPS content really does have a responsibility to keep their certs up to date, but it does work.

This time I have three choices. Clicking on "SEC_ERROR_EXPIRED_CERTIFICATE" gives me a page of base-64 gobbledegook that is apparently the actual certificate. "View Certificate" helpfully tells me that I can't view the certificate, because it is expired. No, duh. And "Go Back (Recommended)" takes me to my home page again.

NONE OF THESE ARE USEFUL OR HELPFUL.


Clicking the Advanced button usually presents a fourth option as well: a button labelled "Add exception..." in a grey bar at the bottom of the same text box containing the three clickables you've already identified.

I suspect that what's happened here is that temasmith.com has embedded a page from cowbird.com and has used CSS to style the embed in such a way that overflow is simply cropped off, and as a result you can only just see the top of the grey bar and can't see the "Add exception..." button at all. If you visit the cowbird.com page concerned (whose address I got from above the base-64 gobbledegook in the certificate dump) in its own window, you get to see the whole thing. Exercising the now-visible "Add exception..." option then gets you access to the cowbird.com page, and reloading the temasmith.com page then works as it should have done from the get-go.

Perhaps there might be ways that Firefox could render its substitute content for failed embeds in some fashion that makes it more resilient in this kind of situation, but I can't see how Mozilla could even begin to guarantee that whatever they did would always work. A sufficiently creative site designer would always be able to find some way to fuck their efforts up.

This is just a stupid blog page. I don't think it even has any fields that I could use to madly enter passwords, emails, and credit card details. I have no way to override the decision the browser made for me, and the buttons are less than helpful. This is an experience only an engineer could love.

Quite so, and this is one of the reasons that I have in the past argued quite vociferously that the lemming-like drive toward total replacement of HTTP with HTTPS is a mistake. There is simply no reason that I can see for using a more complicated protocol with extra failure modes to do jobs that the simpler, insecure protocol handles perfectly adequately.

That said: in this particular instance, HTTP would not be safe because cowbird.com relies heavily on scripts, and allowing scripts to be intercepted and possibly modified on their way from server to browser is unacceptably poor security practice in 2019. No, the site doesn't have fields for the collection of PII but do you really want to give the creepy dude three seats away from you at Starbucks the ability to inject his own scripts into your browser? I wouldn't.

Expired certificates are a common problem and Mozilla should be ashamed that their program handles them so badly.

I agree with you that the layout of the Big Scary Warnings that Firefox uses to discourage people from connecting to HTTPS sites of dubious provenance is perhaps less bulletproof than it could be for people interested in pushing on past the warning, but to the extent that it does indeed discourage people from pushing on past the warning, it's actually doing the job it was designed to do.

I also agree that it shouldn't prohibit bypassing the warning, and the fact that in this instance it effectively does so is unfortunate, but again I see the fault here lying way way way more with temasmith.com and cowbird.com than with Mozilla. I don't think it's fair to yarg on Mozilla for implementing a browser that in most cases actually does a really good job of achieving exactly what certificate expiry was designed to achieve i.e. time-based validation expiry.

The page itself is a couple of years old, cowbird.com is closed; there's not much chance anyone is going to fix the certification problem.

It might be closed to the uploading of new content, but it's still out there on the Web serving the old content it already has, its domain registration was last updated less than a year ago, and whoever is currently operating it should absolutely not be doing so with an expired SSL certificate.

Expired certificates are a common problem and Mozilla should be ashamed that their program handles them so badly.

I think you might as well demand that Mozilla do something about link rot in general. It's the Web. Shit breaks. If worse were not better we'd all be using Xanadu instead, but it isn't so we're not. Sigh.
posted by flabdablet at 6:46 AM on May 14, 2019


Joe in Australia, the page is breaking because it loads a completely separate page in an iframe and that page has a broken SSL certificate. Firefox is showing you its standard SSL warning page, but for some reason the "Advanced" option does not include "Accept the Risk and Continue". Edge and Chrome both just show a big "broken content" grey box with no options to diagnose the problem at all.

All three browsers are treating this SSL error in the context of an iframe differently than if you load it as the main page. I'm guessing none of the browsers present the option to ignore the error because in the context of an iframe there's extra risk involved, specifically to the containing page.

I agree entirely expired SSL certificates are user-hostile and 99% of the time aren't real problems. But, well, here we are.
posted by Nelson at 7:08 AM on May 14, 2019 [1 favorite]


On further investigation, it seems that the issue with the missing "Add Exception..." button in this instance is not caused by anything CSS-related, but by the fact that the definition of that button includes a hidden="true" attribute. Exposing the definition via Inspect Element and editing away that attribute causes the button to appear, and it's fully functional.

I can't think of a good reason why Firefox should hide that button for a certificate error in an iframe when it doesn't hide it normally, so when I have time I'll be off to the Googles to see if I can find any discussion of this decision in a Firefox bug report. Perhaps mhoye would care to elucidate?
posted by flabdablet at 9:38 AM on May 14, 2019


flabdablet I'm assuming it's because of a risk of the (potentially compromised page) accessing something in the parent document, the one with the iframe element? I don't understand this part of the browser security model very well. But as I noted, both Chrome and Edge also don't give the user any way to override the SSL expiration in an iframe.
posted by Nelson at 9:40 AM on May 14, 2019


I can't think of a good reason why Firefox should hide that button

Somebody else could, though, eight years ago in CVE-2012-1964.

To test your browser's behaviour for certificate errors affecting an entire page: https://access.techsmith.com has a self-signed certificate that your browser should reject.

To test behaviour for certificate errors inside an iframe, paste the following into your address bar (MeFi doesn't allow me to create a clickable data: URL):
data:text/html,<iframe width="700" height="700" src="https://access.techsmith.com"></iframe>
Workaround for the iframe case, which I've verified as working for the blog site originally reported by Joe in Australia: if you see a certificate warning, and you click Advanced but don't then see an "Add Exception..." button, try right-clicking anywhere on the certificate warning and choosing This Frame > Open Frame in New Tab. That should get you a page with nothing but the certificate warning on it, whose Advanced button will then expose the "Add Exception..." option.

And while I agree that this is utterly non-intuitive, and that no reasonable person could be expected to find this workaround without research and/or guidance, I still don't think it's a design flaw in Firefox so much as an unfortunate compromise that's been pretty much forced by the state of the Web today. I also agree with Nelson's point that Firefox at least gives you some information about why it won't load a broken iframe, which puts it ahead of its major competitors.
posted by flabdablet at 10:30 AM on May 14, 2019 [2 favorites]


I can't think of a good reason why Firefox should hide that button

Somebody else could, though, eight years ago in CVE-2012-1964.


That's a very cunning attack and a real bodge of a solution.
posted by Joe in Australia at 9:03 PM on May 14, 2019


> "Clicking the Advanced button usually presents a fourth option as well: a button labelled "Add exception..." in a grey bar at the bottom of the same text box containing the three clickables you've already identified."
The truly stupid thing about that?

The default, at least in earlier versions of FF, is to accept that potentially-dodgy certificate permanently
posted by Pinback at 10:02 PM on May 14, 2019


You might not agree with their rationale, but at least they have one.

Dialog fatigue is a thing. The canonical examples both come from Microsoft Windows, whose standard software installation process just trains people to click Next Next Next Next until they can click Done, never bothering to examine any of the boxes where the Next buttons are; this has been thoroughly exploited by foistware vendors.

And when Vista first came out, its new User Account Control feature was rapidly dubbed User Annoyed Constantly and widely derided as useless on the basis that all it achieved, in the end, was training people to click Allow every time without thinking about what they were doing.

Whether or not the overall security of a large user base is going to be worsened more by dialog fatigue than by having untrustworthy certificates pinned by accident strikes me as a question with no immediately obvious answer, especially given that the people most likely to pay attention to details like this in the first place are also going to be those most inclined to turn off Mozilla's telemetry.
posted by flabdablet at 11:16 PM on May 14, 2019 [1 favorite]


Also worth noting is that ssh is generally well-regarded as security protocols go, and it relies completely on trust-on-first-use rather than any kind of centralized trust infrastructure. Storing faulty certificates permanently by default sure smells like bad security practice, but it seems to me that the scenarios where it might cause harm would be encountered far less often than site operators failing to keep their certs up to date or getting their domain names wrong.
posted by flabdablet at 11:21 PM on May 14, 2019 [1 favorite]


> "You might not agree with their rationale, but at least they have one."
Yeah, saw that (and, IIRC, other similar statements) back in the day. I didn't agree with the reasoning then, and I agree with it even less now, 10 years later.

Fundamentally, I believe that the browser - or any software, for that matter - shouldn't be trying to second-guess the users' intentions or reasons. If it must - and, if pressed, I might agree that the matter in question is an edge-case - then it should at least default to the option that provides them with the most security. Ideally, it shouldn't even give the user that option.

(For those who already know all the following, bear with me - I'm saying it by way of explaining my reasoning…)

Really, in practice there's only 4 reasons for a certificate to fail (in decreasing order of importance):
  1. The certificate has been revoked by the issuing authority (i.e. the chain of trust has been lost for some reason - the cert issuer / holder has been compromised or found untrustworthy; the cert holder superseded/revoked old/unused certs e.g. on domain or owner change; etc).
  2. The certificate doesn't match the one previously accepted for that site.
  3. The certificate has expired (or the user app thinks it's expired e.g. the time/date issues brought up earlier).
  4. The certificate is trivially invalid for that domain (e.g. misconfigured cert; server using cert issued for www.example.com on forum.example.com; etc).
The whole thing is a chain of trust - you trust Mozilla to vet the cert authority and accept their certs; and their trust extends blah blah blah down the chain to the cert on the webserver. But, realistically, the only thing the chain is certifying is that your browser A trusts authority B to vouch for owner C of website D from time/date E to time/date F.

If any one of those is invalid, then the trust is invalid…

In cases 1, 3, and 4, the most-safe - and, Id argue, most-correct - way of dealing with the cert failure is to prevent the user from trusting it i.e. refuse to connect to the website, and give no option of bypassing that. Anything else is insecure, and leaves the user open to pwnage.

Case 2 is a bit different - absent of any way for the user to independently confirm validity, you kinda have to accept that the certificate being presented is validated & trusted by a browser-accepted cert issuer. I assume FF already does that - it looks like it does - but I don't have an easy way to check it handy.

Now, personally, I'd prefer that there was some leeway given for points 3 and 4 (i.e. trust that I can tell whether my time is wrong or the website's misconfigured their cert, and allow the option of creating a time- or session-limited exception). But that definitely isn't the safest or most secure option, so I would never advocate it as a blanket solution.

(And, since I'm currently faffing around with FF 66.0.5 & reminding myself why I stopped pre-Australis, I'll note that Firefox no longer even gives you the option of not accepting expired certs (i.e. point 3) permanently. If you click on "Advanced", "Accept the Risk and Continue", it now goes straight to accepting the expired certificate permanently <facepalm.gif> …)
> "Also worth noting is that ssh… relies completely on trust-on-first-use rather than any kind of centralized trust infrastructure."
And since it (generally) doesn't rely on that trust infrastructure, in every user-facing implementation I've encountered it encourages and implores and expects you to verify trust through a back-channel e.g. real-life key exchange, a previously-established and still secure connection, etc) before permanently storing the key. That's difficult/impossible to do online without some sort of chain of trust (i.e. exactly what a Cert Authority & user/website certs are for).

And even so, user-implementations also warn the hell out of you about trusting keys that no longer match or are similarly invalid, and implore you to independently check them before accepting them…
posted by Pinback at 12:08 AM on May 16, 2019


Case 2 is a bit different - absent of any way for the user to independently confirm validity, you kinda have to accept that the certificate being presented is validated & trusted by a browser-accepted cert issuer. I assume FF already does that - it looks like it does - but I don't have an easy way to check it handy.

We kind of do, in the form of the fix for the issue this very post was prompted by.

The new intermediate cert for add-on signing does not match the old one because its expiration date is different. But Firefox trusts it (silently) anyway because its signature validates against the baked-in root cert for add-on signing.

As an experiment I just deleted the new intermediate cert, and re-imported a version I'd modified by changing the last 4 in the PEM file to a 5. That change is in the intermediate cert's signature region and it was enough to make Firefox disable add-ons again, demonstrating beyond doubt that Firefox is not trusting the new intermediate cert purely by virtue of its having been imported into the certificate database as both russm and myself had assumed was the case.

Rather, when Firefox encounters a failed certificate it will search the pool of certs it already knows about in order to find a valid chain back to a trusted root, as outlined by Eric Rescorla in Technical Details on the Recent Firefox Add-on Outage. Corrupting the new intermediate cert's signature was enough to make that search fail - as it should, because the modified cert no longer validates against its issuer.

[ssh] user-implementations also warn the hell out of you about trusting keys that no longer match or are similarly invalid, and implore you to independently check them before accepting them

When you tell Firefox to browse a site whose SSL cert it can't validate, the first thing it presents is a page titled Insecure Connection, with
Your connection is not secure
in great big text and the short sharp explanation
The owner of access.techsmith.com has configured their website improperly. To protect your information from being stolen, Firefox has not connected to this website.
in smaller text underneath. There's a Go Back button in a nice friendly blue, and an Advanced button you have to look twice at before convincing yourself it's not actually greyed out.

It's pretty clear to me that a lot of thought has been put into this design, and that it has a very high likelihood of guiding the majority of Firefox users toward choosing security over immediate gratification.

I'll let you and Joe in Australia have the fistfight about whether Firefox needs to be doing more or less to make bypassing a dodgy cert possible. I'm pretty happy with it as it is; I think it strikes a good balance between We Know What You Really Need and Here Are Chainsaws For You To Juggle.
posted by flabdablet at 5:16 AM on May 16, 2019 [1 favorite]


« Older Cite Your Sauce   |   The rise of conspiracy entrepreneurs and their... Newer »


This thread has been archived and is closed to new comments