Cloudflare not happy times
February 23, 2017 5:57 PM   Subscribe

Cloudflare CDN has been dumping uninitialized memory from its reverse proxies, including all sorts of things that are supposed to be under HTTPS. Like passwords and private messages from dating sites and cookies and online password manager data.

Affected websites include OKCupid, Yelp, Pingdom, Montecito Bank and Trust, Uber, Lyft, Coinbase, Bitpay, Product Hunt, Udemy, Crunchyroll, Fitbit, Hacker News, Stack Overflow, Zendesk and Discord.

On HN.
posted by hleehowon (135 comments total) 42 users marked this as a favorite
 
A fair few hosting services as well. An incomplete list is on stackshare.

Basically, change all your damned passwords.
posted by hleehowon at 5:59 PM on February 23, 2017


As far as fuckups go, this one is really up there. Been changing passwords for the last hour or so and I'm sure I've only scraped the surface. The only upside to it is that it can't be actively exploited, but that still leaves cached pages laying around everywhere. Google has been doing their best to sanitize their archives, but there are other search engines and private spiders and such. (Not to mention the Internet Archive probably has a gargantuan task on their hands as well.)
posted by Rhomboid at 6:07 PM on February 23, 2017 [1 favorite]


Why exactly did we think it was a good idea to have a single company MITM a giant chunk of the internet?
posted by zachlipton at 6:08 PM on February 23, 2017 [29 favorites]


And BTW for those of you wondering, Metafilter uses Cloudfront, not Cloudflare. That's not to say that it's absolutely certain that no Mefi data has leaked, but change your passwords just to be sure.
posted by Rhomboid at 6:12 PM on February 23, 2017 [3 favorites]


1password has apparently been almost-compromised, where the SSL stuff got pwned but they encrypt the message further within the SSL stuff just for this sort of contingency. It's in the HN thread
posted by hleehowon at 6:15 PM on February 23, 2017 [1 favorite]


Notable quote from the bug tracker:

Cloudflare pointed out their bug bounty program, but I noticed it has a top-tier reward of a t-shirt.
https://hackerone.com/cloudflare
Needless to say, this did not convey to me that they take the program seriously.

posted by destrius at 6:21 PM on February 23, 2017 [35 favorites]


Why exactly did we think it was a good idea to have a single company MITM a giant chunk of the internet?

You dare question the Divine and Invisible Hand of the Free Market?
posted by Evilspork at 6:21 PM on February 23, 2017 [49 favorites]


From the main person in the first link:

"Cloudflare pointed out their bug bounty program, but I noticed it has a top-tier reward of a t-shirt."
posted by Evilspork at 6:24 PM on February 23, 2017 [9 favorites]


Why exactly did we think it was a good idea to have a single company MITM a giant chunk of the internet?

Because botnets and DDOS. Your choices now are go with a big CDN or have your service be at the mercy of anyone with fifty bucks worth of bitcoin.
posted by phooky at 6:27 PM on February 23, 2017 [32 favorites]


I know, but we've gotten ourselves in a ridiculous situation. It used to be: "The Net interprets censorship as damage and routes around it." Now it's: "censorship [in the form of DDOS attacks] caused us to route bloody everything through one company that knows how to mitigate it."
posted by zachlipton at 6:40 PM on February 23, 2017 [13 favorites]


Jesus mother of fuck...
posted by acb at 6:42 PM on February 23, 2017 [4 favorites]


My working theory was that this was related to their "ScrapeShield" feature which parses and obfuscates html - but because reverse proxies are shared between customers, it would affect *all* Cloudflare customers.

We fetched a few live samples, and we observed encryption keys, cookies, passwords, chunks of POST data and even HTTPS requests for other major cloudflare-hosted sites from other users. Once we understood what we were seeing and the implications, we immediately stopped and contacted cloudflare security.
I have been a very heavy user of ScienceDaily; I'm sure I've clicked on every Space and Time offering for at least the last 4 years, most of the biology pieces, and many others; and a lot of this has come in 2 or 3 hour binges at a time.

Maybe 8 months ago, from one day to the next SD started loading much more slowly for me, and a week or so after that, I started getting CAPTCHAs from Cloudflare before a page would load at all a few times in every session. I figured SD was worried I was copying their stuff, and one day I said to myself 'aha! I'm experiencing a man in the middle attack -- but a benign one.'

Not so benign, as it turns out.
posted by jamjam at 6:43 PM on February 23, 2017


Some days it's like the more I know about how the internet works, the more I want drop everything and make small figurines out of wood and clay.
posted by fifteen schnitzengruben is my limit at 6:44 PM on February 23, 2017 [60 favorites]


Welp! Good time to finally start using lastpass
posted by Going To Maine at 6:51 PM on February 23, 2017 [5 favorites]


Oh, incidentally, this explains the note I saw on 1Password’s blog about how the Cloudflare bug hadn’t affected them. So maybe also use that.
posted by Going To Maine at 6:58 PM on February 23, 2017


Can someone ELI5 what a CDN is?
posted by quaking fajita at 6:58 PM on February 23, 2017


Tavis Ormandy strikes again.

Tavis is part of Google's Project Zero, and previously has been punching holes in most of the major anti-virus providers.

Welp! Good time to finally start using lastpass

Eh, he, he....well.... Tavis was looking over password managers, but got blow back from the community. What's worse, the bugs you know about or the bugs you don't?
posted by zabuni at 6:59 PM on February 23, 2017 [8 favorites]


A cool kid is scraping stuff there for a fuller listing

https://github.com/pirate/sites-using-cloudflare
posted by hleehowon at 6:59 PM on February 23, 2017 [1 favorite]


I really, really don't want to use a password manager, because I'm worried about situations where I don't have my phone available, and it's just more convenient not to... but between this, and the recent news about JScript applications getting around write protection... I don't know any more.

Also, I got a notification from Google today that "Something has been changed" on my g-everything account. I checked recent device usage, and it didn't report anything other than what I'd done. My Google password isn't used by anything else. And I did see that an app that's native to my phone recently requested more permissions to my account, so I think it's that...

Is it still worth changing my password?
posted by codacorolla at 7:01 PM on February 23, 2017 [2 favorites]


quaking fajita: A CDN is a way for websites to get faster. If you live in say Singapore or something and you want to Google something, Google has servers in Singapore specifically for you so the speed of light and the speed of networking doesn't slow down your search to take 10 seconds.

Cloudflare is basically a CDN service for other websites, so that you can pull off that same trick without having a few billion dollars handy.
posted by hleehowon at 7:01 PM on February 23, 2017 [3 favorites]


so, from a practical standpoint, if we do use something like lastpass already, then what are the next actions? Immediately change master password and re-generate passwords for all sites?

or would this need to be patched first before it's "safe" to change passwords?
posted by subversiveasset at 7:01 PM on February 23, 2017


Can someone ELI5 what a CDN is?

A Content Delivery Network hosts multiple proxy copies of your site so a) if it’s under high demand it doesn’t go down and b) you get data from a proxy that is closer to your location. Cloudflare also prided itself on being particularly resistant to Distributed Denial of Service Attacks, as well as providing some other security features that I know nothing about.
posted by Going To Maine at 7:02 PM on February 23, 2017 [1 favorite]


AgileBits, the 1Password people, posted that data in 1password wasn't compromised.
posted by ZeusHumms at 7:03 PM on February 23, 2017


so, from a practical standpoint, if we do use something like lastpass already, then what are the next actions? Immediately change master password and re-generate passwords for all sites?

or would this need to be patched first before it's “safe” to change passwords?*

It depends on if your master password was getting leaked. You would presumably want to regenerate the child passwords, though, since those would be leaked. (An assumption on my part, not a fact. See 1Password’s blog post.)
posted by Going To Maine at 7:03 PM on February 23, 2017


Tavis Ormandy‏ @taviso

Could someone from cloudflare security urgently contact me.
7:11 PM · Feb 17, 2017


Oof. That's not the sort of thing you want to see on a Friday afternoon.
posted by ethand at 7:09 PM on February 23, 2017 [8 favorites]


Ironically, Travis on password managers.
posted by jaduncan at 7:10 PM on February 23, 2017


Thanks hleehowon and Going To Maine.
posted by quaking fajita at 7:12 PM on February 23, 2017


Note that as far as I can tell, Cloudflare has already fixed the bug, so going forward there will no longer be bits of your private data being sent out with other sites. The problem is that the bug has been around for a while, so any sites that cache historical Internet traffic might contain some private data belonging to you. If you change your password now, you'll be safe as your new password won't be leaked, at least via this particular bug.
posted by destrius at 7:12 PM on February 23, 2017 [1 favorite]


or would this need to be patched first before it's "safe" to change passwords?

It's been patched. Project Zero practices responsible disclosure, which means they wait until the software developer patches, or a set date. So it's safe to change passwords.

Is it still worth changing my password?

Google doesn't use Cloudflare, they are large enough to have servers everywhere. So your g-everything account should be ok, as long as you didn't reuse that password. Might I suggest two-factor authentication though? While this requires your phone occasionally, and the first time you sign on any device, it is pretty painless after that.
posted by zabuni at 7:13 PM on February 23, 2017 [3 favorites]


Yeah, Amazon, Facebook, Google and Microsoft do have a few billion dollars handy so none of them are affected at all
posted by hleehowon at 7:15 PM on February 23, 2017 [3 favorites]


(KeePass is his recommendation, if you're keeping track.)
posted by jaduncan at 7:15 PM on February 23, 2017 [12 favorites]


Google doesn't use Cloudflare, they are large enough to have servers everywhere. So your g-everything account should be ok, as long as you didn't reuse that password. Might I suggest two-factor authentication though? While this requires your phone occasionally, and the first time you sign on any device, it is pretty painless after that.

Yep, I do use 2-factor, and it's occasionally a pain (trying to log onto a hard wired terminal when I'm in a building with no cell service), but I'm glad I have it. Looking through my account it seems like it was the app update that pinged the security alert.

Anyway, it turns out cyberpunk got here, and instead of being sexy, cool, dangerous, or emancipatory it's just a bunch of dumb banal bullshit and digital paperwork.
posted by codacorolla at 7:17 PM on February 23, 2017 [20 favorites]


Welp! Good time to finally start using lastpass KeePass
posted by Going To Maine at 7:17 PM on February 23, 2017 [3 favorites]


My Google account also required me to reauthenticate today, with no explanation.
posted by Yowser at 7:17 PM on February 23, 2017 [9 favorites]


Since the horrified tones suggests this matters for people who aren't tech geeks, could someone please explain it in English, for people who aren't tech geeks?

Like if you haven't logged into a site in forever, then presumably your password wasn't put through whatever this is and you're good?

I don't see how "change all your passwords" is follow-able advice. I would imagine my lastpass has at least a couple of hundred passwords.
posted by If only I had a penguin... at 7:18 PM on February 23, 2017 [3 favorites]


My Google account also required me to reauthenticate today, with no explanation.

Odd, and concerning... Do you have an LG? I do, and it looks like my memo software that's built into the phone had an update where it escalated the privileges it has with my Google account. I think that's the cause. But, I'm definitely open to changing my password.
posted by codacorolla at 7:20 PM on February 23, 2017


I think the problem with "haven't logged into a site in forever" is that we don't know how long this bug has been active (do we? I haven't read through everything yet).
posted by quaking fajita at 7:21 PM on February 23, 2017


Like if you haven't logged into a site in forever, then presumably your password wasn't put through whatever this is and you're good?

I don't see how "change all your passwords" is follow-able advice. I would imagine my lastpass has at least a couple of hundred passwords.
posted by If only I had a penguin... at 2:18 on February 24 [+] [!]


As per cloudflare:
The three features implicated were rolled out as follows.
The earliest date memory could have leaked is 2016-09-22.

2016-09-22 Automatic HTTP Rewrites enabled
2017-01-30 Server-Side Excludes migrated to new parser
2017-02-13 Email Obfuscation partially migrated to new parser
2017-02-18 Google reports problem to Cloudflare and leak is stopped

You may wish to treat passwords as compromised if they relate to services served by CF that you've logged into between those dates. Also, insert standard comment about reuse of passwords and usernames.
posted by jaduncan at 7:25 PM on February 23, 2017 [1 favorite]


Cloudflare's blog post claims:
The three features implicated were rolled out as follows. The earliest date memory could have leaked is 2016-09-22.

2016-09-22 Automatic HTTP Rewrites enabled
2017-01-30 Server-Side Excludes migrated to new parser
2017-02-13 Email Obfuscation partially migrated to new parser
2017-02-18 Google reports problem to Cloudflare and leak is stopped
posted by XMLicious at 7:25 PM on February 23, 2017


My Google account also required me to reauthenticate today, with no explanation

Yeah, mine did too, on my (Samsung) phone. But only one of my two accounts. I have two-factor on them already, though.
posted by soren_lorensen at 7:26 PM on February 23, 2017


FYI, the Google account log in again thing is a bug that happened that is unrelated to this.
posted by reluctant early bird at 7:33 PM on February 23, 2017 [13 favorites]


AgileBits, the 1Password people, posted that data in 1password wasn't compromised.

The problem with the above and "haven't logged in in months" is that according to Tavis, they were able to recover API information from 1Password. Which gives them keys to the kingdom. Now, 1password will probably dispute that, but Tavis, as mentioned above, has a rather frosty relationship with password manager creators.
posted by zabuni at 7:36 PM on February 23, 2017


The other question here for logins (and I don’t know the answer to this): If I’ve used the account via the android app, has the password been compromised? Or is there a separate token that’s being used?
posted by Going To Maine at 7:42 PM on February 23, 2017


That's hard to know without the specifics and examining the .apk; any given app could be implemented on a token or a user/pass basis. I'd like to think that session tokens were issued (and thus you could log out and in) but this might be incorrect.

If you're asking about Uber, I would guess it's token based rather than sending the details every time. More details are likely to come out about Uber in particular.
posted by jaduncan at 7:47 PM on February 23, 2017


Uh. In fact, I know it's token based for Uber because I've just idly harvested a whole bunch of Uber authentication tokens* whilst looking around to see the implications of the bug.

* for the record, I'm about to purge them rather than exploit things.
posted by jaduncan at 7:49 PM on February 23, 2017 [1 favorite]


What would be in the Uber data? Unsalted CC numbers?
posted by codacorolla at 7:56 PM on February 23, 2017


Hmm, further reading says it was just the 1Password API sessions, which were hardened. Sorry. Twitter makes all of this move fast.
posted by zabuni at 7:56 PM on February 23, 2017


Uber was, as far as I can see, *OAuth IDs and secrets*, as well as location of the pickup with time/date, phone model and various other less important bits. So, to put it mildly, log out and in.
posted by jaduncan at 8:00 PM on February 23, 2017


Ok, can we condense this down to actionable advice for people following along who don't know enough to figure out how bad this is? Because we've gone from "change all your passwords right now" to "change all your passwords, except for Amazon, Google, Microsoft, Facebook, [anyone else big enough not to use Cloudflare]"

Is the issue that we can't possibly know what sites are affected? Would this include banks? Companies like GoDaddy?
posted by schadenfrau at 8:01 PM on February 23, 2017 [5 favorites]


Is the issue that we can't possibly know what sites are affected? Would this include banks? Companies like GoDaddy?

This list -linked above, and evolving- is kind of badly sorted but covers the Alexa Top 1000 and some of the other biggies that could have been exposed.
posted by Going To Maine at 8:03 PM on February 23, 2017 [1 favorite]


So, to put it mildly, log out and in.

I don't know how Uber does it, but this won't necessarily clear an oauth token. The tokens are good usually for a set period of time, and the site is allowed to keep them and keep re-presenting them until they run out. Logging out of the site does not mean they have to throw it away.
posted by RustyBrooks at 8:03 PM on February 23, 2017


Although I suppose you're talking about the app. That probably does throw away the oauth token. Sorry, I'm more used to thinking about apis in terms of integrating them with other websites.
posted by RustyBrooks at 8:05 PM on February 23, 2017


I would very, very much hope that a site like Uber that deals with billing information and location history purges tokens when they aren't using them and regenerates on the next login.
posted by jaduncan at 8:05 PM on February 23, 2017


Seeing #cloudbleed as a name/hashtag for this.
posted by Celsius1414 at 8:11 PM on February 23, 2017


From an HN comment:

> The greatest period of impact was from February 13 and February 18 with around 1 in every 3,300,000 HTTP requests through Cloudflare potentially resulting in memory leakage (that’s about 0.00003% of requests).

1) From the metrics I recalled when I interviewed there, and assuming the given probability is correct, that means a potential of 100k-200k paged with private data leaked every day.

2) What's the probably that a page is served to a cache engine? Not a clue. Let's assume 1/1000.

3) That puts a bound around a hundred leaked pages saved per day into caches.

4) Do the cache only provide the latest version of a page? I think most do but not all. Let's ignore that aspect.

5) What's the probably that a page contains private user information like auth tokens? Maybe 1/10?

6) So, that's 10 pages saved per day into the internet search caches.

7) That's on par with their announcement: "With the help of Google, Yahoo, Bing and others, we found 770 unique URIs that had been cached and which contained leaked memory. Those 770 unique URIs covered 161 unique domains." Well, not that we know for how long this was running.

8) Now, I don't want to downplay the issue, but leaking an dozen tokens per day is not that much of a disaster. Sure it's bad, but it's not remotely close to the leak of the millennia and it's certainly not internet scale leak.

9) For the record, CloudFlare serves over one BILLION human beings. Given the tone and the drama I expected way more data from this leak. This is a huge disappointment.

Happy Ending: You were probably not affected.


Seems to suggest we're not quite in Krusty-spitting-in-burgers territory and I can leave my passwords be.

Anyone care to speak for the math one way or the other?
posted by bunbury at 8:13 PM on February 23, 2017 [2 favorites]


Ok, can we condense this down to actionable advice for people following along who don't know enough to figure out how bad this is? Because we've gone from "change all your passwords right now" to "change all your passwords, except for Amazon, Google, Microsoft, Facebook, [anyone else big enough not to use Cloudflare]"

This article might help; I assume we'll see more articles with clear instructions of what to do pop up soon. Remember this bug dropped only about 5 hours ago, so there's not been a lot of time to react.

Despite it being a big deal, I don't think it's particularly time sensitive for an average user. Your passwords or other auth data might sitting in some server out there in the clear, but until today nobody knew that they could find it. It will take attackers time to crawl through various archives to harvest passwords, and many archive sites will also be purging their records while this happens. No need to panic, but do take action soon.

So best advice for now is to just change all the passwords of sites that you care about, and change the passwords for other sites as and when you remember them. It's always good to rotate your passwords regularly anyway, so take the chance to do so.
posted by destrius at 8:15 PM on February 23, 2017 [2 favorites]


That maths fine as far as it goes, but doesn't account for the possibility of intentional attacks. If you know that one page is malformed/triggers the bug, you can just repeatedly request that without it entering search engines. If you absolutely have to, you can craft the page and put it on CF. That won't necessarily trigger search engines, and you can prevent it being listed via robot.txt in any case. Your server can then sit there and demand that page all day, just waiting to see what falls out.

Has there been active exploitation? Hard to know, and it's not the end of the world, but changing your PW or cycling tokens where CF-served sites have your billing details is a good idea.
posted by jaduncan at 8:21 PM on February 23, 2017 [1 favorite]


Note that Cloudflare's post about this downplays it quite a lot. Probably been edited by some sneaky PR.
Like if you haven't logged into a site in forever, then presumably your password wasn't put through whatever this is and you're good?
Passwords are the easy part. The scope of potential exposures is impossibly broad.

Suppose you were using some forum, that used Cloudflare, and in a private message to someone there who you trust, you mentioned you are in an affair. Then Duck Duck Go cached some other page on an entirely other website, which thanks to the bug, got that message leaked into it. Then someone searched DDG and found it (which people have been doing today). Now someone knows you're having an affair and proceeds with the blackmail.
posted by joeyh at 8:41 PM on February 23, 2017 [2 favorites]


I need more uncertainty with my tech! More!
posted by Strange_Robinson at 9:00 PM on February 23, 2017 [1 favorite]


Good lord, the underlying bug is really lame. One of those things where what you instinctively type first is WRONG and you have to develop the habit of always going back and changing it.
posted by lastobelus at 9:16 PM on February 23, 2017 [2 favorites]


Various websites occasionally expire user sessions for perfectly valid reasons. If you find yourself suddenly logged out of a site, it's not automatically a cause for concern.
posted by schmod at 9:20 PM on February 23, 2017 [1 favorite]


There's nothing to be concerned about if you get a CAPTCHA when visiting a site that is served by Cloudflare or any other CDN (or Google, for that matter.) That's part of the service that they provide, mitigating high levels of traffic that might be due to attacks. If you are asked to prove you're human it doesn't mean there's anything wrong with your system, it just means that that site might have been experiencing high levels of traffic (potentially from your netblock, which could mean that someone else on your ISP was compromised and is being used as a zombie, but again, that doesn't mean that you are.)
posted by Rhomboid at 9:22 PM on February 23, 2017 [1 favorite]


So my bank just texted me and said my password was just changed, and if it wasn't me who changed it to call them. It wasn't me.

Edited to add it wasn't actually changed. Interesting.
posted by maxwelton at 9:42 PM on February 23, 2017


maxwelton: Yiyiyi! Does your bank's website use cloudflare?
posted by joeyh at 9:45 PM on February 23, 2017


Good lord, the underlying bug is really lame. One of those things where what you instinctively type first is WRONG and you have to develop the habit of always going back and changing it.

The C code was auto-generated though, from the Ragel parser generator. If you look at the Ragel source, it seems a bit harder to spot (at least for somebody with no prior knowledge of Ragel like me). There's a moral in this somewhere.
posted by destrius at 10:06 PM on February 23, 2017


OK, false alert. Geez, nice timing. However it did have me download the text file from github, and as far as I can tell, none of the "bank level" players in my password manager (keepass) are on the list. Not that it means anything.

I mean, if I was an IT guy at a bank or health provider you'd really, really, really have to convince me using a third-party for anything secure was a good idea. Assuming I was convinced our IT department was competent. And if I was on the team...hm.
posted by maxwelton at 10:10 PM on February 23, 2017


maxwelton: competent IT departments in regulated industries are as rare as hen's teeth. Mainframes with COBOL are the order of the day still in many banks, Epic uses MUMPS, etc etc.
posted by hleehowon at 10:15 PM on February 23, 2017 [3 favorites]


bank.com might partner with another company to host part of their website. Say a live web chat service. If that is set up on chat.bank.com, and the login cookie for bank.com is valid for subdomains too, login cookies leak to chat.bank.com. If the partner uses cloudflare, you can still be exposed even though bank.com doesn't.
posted by joeyh at 11:28 PM on February 23, 2017


Mainframes with COBOL are the order of the day still in many banks, Epic uses MUMPS, etc etc.

FWIW, the same mainframes that are running VM's of legacy COBOL code are running VMs for linux hosting containers with tomcat/java stacks, so I don't find the fact that proven code in production is left running, even if it's in COBOL.

MUMPS has been "M" for a decade or so now, IIRC. It's been a while since I deployed VistA, but the VA's EHR rocks. And #1, we could give it away , for free, to every healthcare provider and increase efficiency 50 - 100%

With that said, I'm pretty confident the NSA's knew about this for a long time.
posted by mikelieman at 11:45 PM on February 23, 2017 [1 favorite]


Can someone tell me if my understanding is correct? I know something about computers but communication protocols bore me and I've never needed to learn.

So I thought the whole point of https is that it was secure against man in the middle attacks? An eavesdropper could only get encrypted info (presumably because one side is using a public/private key combo or something?) So leaking my page http transactions normally shouldn't matter?

But the issue here is that a CDN like Cloudfare isn't just routing information back to my bank, it's running the web site so it's actually encrypting and decrypting pages? So if they are leaking things back someone is in fact getting my plaintext info like bank account numbers, recent transactions and mothers' maiden name?

Wouldn't my password still be hashed in that case with a competent design? I mean I didn't think a bank employee could see my password just by looking at the request. (I get that there are incompetent designs, and bad people getting hashed passwords to decrypt at their leisure is also an issue.)
posted by mark k at 11:53 PM on February 23, 2017 [1 favorite]


So I thought the whole point of https is that it was secure against man in the middle attacks?

In theory it is, although where 'the middle' happens to be is kinda flexible. I've seen at least one configuration like this. ( NB: there's nothing inherently *wrong* with this, but it illustrates an aspect of the issue. )


Your browser gets https://www.example.com

DNS gives you an IP address of www.example.com

That IP address is a server running a reverse proxy, and will forward the request to one worker in a backend cluster, depending on which is up and maybe using a round-robin to distribute work.

The HTTPS connection is **terminated** on the reverse proxy. From the reverse proxy -> cluster is plain old http.

That's where, I'd say, the data is leaking, from the 'cleartext' side.

And while this stack is wonderful on say, a VPS in a datacenter talking among themselves on a private network, as you can see, can have issues when just deployed on a CDN, depending on the CDN's coding.

Who checked in the bad code? Temp employee from the NSA maybe?
posted by mikelieman at 12:18 AM on February 24, 2017 [4 favorites]


In order to do its job, the CDN has to have the unencrypted contents of the pages it serves. So that means that when you request example.com, your browser makes a secure connection to the CDN, and then the CDN makes a separate secure connection to the origin server. But those are two different encrypted connections; internally the CDN decrypts the data from the origin server, stores a copy of it, and then sends it back to you, re-encryped over the other connection. The CDN is essentially a man-in-the-middle that is explicitly endorsed by the origin server, e.g. they would have arranged to give the CDN access to their certificate's private key, so that the CDN can pretend to be example.com. This isn't a violation of any of the design principles of TLS, because they are two separate connections; there is no man in the middle of any one of those two connections.
posted by Rhomboid at 1:09 AM on February 24, 2017 [9 favorites]



So I thought the whole point of https is that it was secure against man in the middle attacks?


Your connection to Cloudfare is secure. Cloudfare have a (valid) certficate that says they're authorized to serve content for yourbank.com (which they are). They are an authorized MitM, which https doesn't really protect against.

The connection between Cloudfare and yourbank.com's actual origin servers is also secure. But, everything is in cleartext at Cloudfare, at least briefly.

For some period of time, the communication between you and your bank is in memory on the servers/proxies that are brokering these connections between you and your bank and caching reusable content (like the gif that is your bank's logo).

This bug caused cloudfare to serve up content from unexpected sections of memory, which sometimes contained data that wasn't supposed to be reusable, and certainly wasn't supposed to be served to anyone but the user that requested it initially.

So, if google's crawler happened to crawl a site that also used cloudfare, and happened to trigger this bug, it may have received data that was supposed to go only to you. The search terms that make it easy to find this sort of thing are looking for http headers, so the most likely leaked payload are Cookies/sessions and things like Oauth tokens (that are contained near the headers that are being searched for)... but that's hardly the full extent of what might have leaked.
posted by toxic at 1:14 AM on February 24, 2017 [8 favorites]


I might be misunderstanding something but I thought it was a standard security practise to zero out buffers after you finish with them to stop/mitigate data leaks from these kinds of bugs, especially buffers containing stuff like passwords. There shouldn't be sensitive stuff just sitting in memory after the process is finished with it.
posted by L.P. Hatecraft at 2:05 AM on February 24, 2017


I thought it was a standard security practise to zero out buffers after you finish with them to stop/mitigate data leaks from these kinds of bugs

The rogue data wasn't from your own session, so it could still be live at the point of duplication. The report talks confidently about 'uninitialised memory', and no doubt the reporter knows a lot more than they've put into that document, but noting else there talks about how stale the leaked data was.

Not that whatever process is handling your transaction should be able to even see another process's data areas. I don't know how you build a CDN, but considering the amount of data they have to move quickly around the place in a massively parallel fashion, one wonders if good architecture is secondary to raw performance and memory reuse efficiency.

But yeah, this 'bug' is far too useful as a surveillance tool to escape suspicion that it was actually a bug in a surveillance tool.
posted by Devonian at 2:50 AM on February 24, 2017 [4 favorites]


Who checked in the bad code? Temp employee from the NSA maybe?
How does a bunch of random, untargeted, and uncorrelated data serve intelligence objectives? I'm honestly asking; I was under the impression that massive data management and the parsing of said data for actionable intelligence is something of a problem for the NSA these days.
posted by xyzzy at 3:54 AM on February 24, 2017


Everything is compromised. The green service code on the NBA and clear text server creates a random seven digits when the request is made. The module then randomly pipes a clear text to the ACU. A clear RMI is served to the XPT. The XPT green code is then forwarded to the server (which is DMI) and serves a further notification to GreenPark resulting in an easily identifiable clear text (Pando code) residing on the AGC. Nothing you can do about it now. As I said, everything is compromised.
posted by unliteral at 4:36 AM on February 24, 2017


I use PasswordMaker Pro to generate passwords. It's convenient and nothing is stored in the cloud.
posted by Pendragon at 4:59 AM on February 24, 2017


(Insert grumpy comment about the wisdom of giving all your passwords to an online password manager service.)
posted by iffthen at 5:18 AM on February 24, 2017 [3 favorites]


How does a bunch of random, untargeted, and uncorrelated data serve intelligence objectives? I'm honestly asking; I was under the impression that massive data management and the parsing of said data for actionable intelligence is something of a problem for the NSA these days.

The reported discovery may not have been the primary effect of whatever was doing this. Surveillance systems have bugs too - probably more than most, due to the curtailed opportunities for testing, code review and bug reporting - and there's no way that the discoverer would know of the normal mode of operation of what was going on, assuming it was a deliberate attack.

For example, we know it could be triggered by 'unbalanced tags'; we don't know if those tags had their own syntax and parameters in normal use to control more precisely what was grabbed and reported. All we know is that there's an unlikely behaviour evinced in a system that happens to leak sensitive data at scale and with huge scope.

That this fits in exactly with the motives, MO and published history of the NSA and friends could just be a coincidence.
posted by Devonian at 5:21 AM on February 24, 2017 [2 favorites]


I see. The implication being that NSA guy could ask CDN to serve everything on UserX if given the correct parameters. It does fit with the "Google is the frenemy and we must mine them for all they're worth" narrative served up by leaked NSA data. Thanks for clarifying.
posted by xyzzy at 5:31 AM on February 24, 2017


It' could be a real bug, it could be the NSA operating with the co-operation of the CDN management, it could be the NSA operating without such co-operation, it could be another agency, it could be another agency from another state. If it was a deliberate attack, it could have been introduced remotely via another vulnerability, locally by a contractor, or locally by a full-time employee. History has examples of all of the above, and I don't doubt history is a very incomplete record.

tl;dr - assume this phone is tapped.

(It occurs to me that there's a 'simple and cheap' detection system for this sort of problem, where you checksum the HTTP data as it comes into the CDP and compare that to a checksum calculated just before egress. But I doubt it would be either really simple or really cheap, an attacker capable of inserting code capable of traducing the CDN's operations so intimately would presumably also be able to frig such a test, and if it was inserted with the connivance of the CDP's management there'd be no point, so at best it would catch this sort of bug if it was truly innocent. I do wonder what the company will report.)
posted by Devonian at 5:55 AM on February 24, 2017 [2 favorites]


The features that caused all this were related to dynamically rewriting the content, so it's expected to change and a checksum would not be effective.
posted by Rhomboid at 6:07 AM on February 24, 2017




I thought it was a standard security practise to zero out buffers after you finish with them to stop/mitigate data leaks from these kinds of bugs

Fun thing that even a lot of relatively experienced C programmers don't realize: if you memset() a buffer to zero, if the compiler can determine that you aren't actually going to use that buffer again, the compiler will optimize that call out. You have to different call, memset_s(), to make sure that the compiler does exactly what you want.
posted by HighLife at 6:46 AM on February 24, 2017 [8 favorites]


My first programming teacher told me that learning C would be like getting a really great knife. You're cutting up all the things - only problem is, why is there blood all over your hands? Eventually, you'd discover "Oh wow, this thing has a handle on it! I could grab it by that end".
posted by thelonius at 6:52 AM on February 24, 2017 [16 favorites]


Lies, thelonius.

C does not have a handle.
posted by ryanrs at 7:07 AM on February 24, 2017 [16 favorites]


My Google account also required me to reauthenticate today, with no explanation.

Me too, on my phone and on chrome on all my other devices. I have two-factor on with a hard key, and there was no suspicious access in the logs. That's three people in this thread. From what I've read, Google purged their authentication cache (in whole or in part, and that's almost certainly not a technically accurate description) to deal with this, so maybe that's what we're seeing.
posted by The Bellman at 7:07 AM on February 24, 2017 [1 favorite]


> Not that whatever process is handling your transaction should be able to even see another process's data areas. I don't know how you build a CDN, but considering the amount of data they have to move quickly around the place in a massively parallel fashion, one wonders if good architecture is secondary to raw performance and memory reuse efficiency.

At that scale you wouldn't be able to do any per-process isolation. You're talking about one process on the reverse proxy for each client site just to make sure when you leak data it goes to a user of the same site.
posted by Horselover Fat at 7:15 AM on February 24, 2017


Add me, and it's four people from the thread. Google made me reauth a couple of accounts on my phone.
posted by mikelieman at 7:15 AM on February 24, 2017 [1 favorite]


bank.com might partner with another company to host part of their website.

My credit union's entire "online banking" website is outsourced to Intuit. I doubt they're the only ones.
posted by indubitable at 7:22 AM on February 24, 2017


The features that caused all this were related to dynamically rewriting the content, so it's expected to change and a checksum would not be effective.

Do we know this? The report says that this was the initial suspicion (because something most certainly was dynamically rewriting the content!) but does not say those suspicions were right or wrong.

Scrapeshield seems to have all the functionality to perpetrate this problem, though. While it most certainly not work with a simple checksum test, though, and while you can think up ways to get around that, if Scrapeshield has strcpy problems then your basic defensive programming skills are not going to be amenable to that sort of sticking plaster. And
posted by Devonian at 7:26 AM on February 24, 2017


MUMPS has been "M" for a decade or so now, IIRC. It's been a while since I deployed VistA, but the VA's EHR rocks.

The EHR is great, true. Doesn't make the language, or what I saw of the VA's preferred coding style when I worked there, any better. :P
posted by Mr. Bad Example at 7:27 AM on February 24, 2017 [2 favorites]


codacorolla: "What would be in the Uber data? Unsalted CC numbers?"

So, here's an interesting question: Have credit card processors finally adopted the equivalent of an authentication token for services that "Save" card numbers for their customers?

If not, why?
posted by schmod at 7:28 AM on February 24, 2017 [1 favorite]


I might be misunderstanding something but I thought it was a standard security practise to zero out buffers after you finish with them to stop/mitigate data leaks from these kinds of bugs, especially buffers containing stuff like passwords. There shouldn't be sensitive stuff just sitting in memory after the process is finished with it.

It turns out that this is very difficult to do in practice. For one such discussion from a respected cryptographer: How to zero a buffer
posted by indubitable at 7:29 AM on February 24, 2017 [3 favorites]


Thanks to those taking the time to explain what this means, especially Rhomboid and Toxic for very helpful responses to my specific question. Sure, my bank account will probably be drained tomorrow but at least I've learned a bit more about the modern net.
posted by mark k at 7:31 AM on February 24, 2017


I thought it was a standard security practise to zero out buffers after you finish with them to stop/mitigate data leaks from these kinds of bugs...

"What do we want? Cache invalidation!
When do we want it? Cache invalidation!"
-Steven Frank, just now.
posted by The Bellman at 7:39 AM on February 24, 2017 [7 favorites]


Google made me reauthorize a corporate account but none of my vanilla accounts, fwiw.
posted by RustyBrooks at 7:42 AM on February 24, 2017 [1 favorite]


C pointer arithmetic is like crystal meth, one hit and you’re hooked.

Eventually, your teeth fall out.
posted by tommasz at 7:42 AM on February 24, 2017 [2 favorites]


My android phone forced me to sign in on my primary (2fa-enabled) gmail account, but not on my work or other ones. It is happening to a fair number of people, it seems.
posted by Pryde at 7:49 AM on February 24, 2017


Google says the deauth bug is unrelated to the Cloudflare issue. Makes sense, given that they are not CF users, so it doesn't directly affect them, aside from having sensitive data in their cache. OTOH, some affected sites definitely used Google Oauth2 tokens, which unlike OG OAuth could be used to impersonate you on those sites if you were incredibly unlucky.
posted by wierdo at 8:11 AM on February 24, 2017 [1 favorite]


Cloudflare posted an incident report with full technical details this morning.

There was a comment on the Ars Technica story with what is apparently the text of an email from Cloudflare to a customer who (Cloudflare says) wasn't affected, and that the company has reached out directly to those who were:
In our review of these third party caches, we discovered data that had been exposed from approximately 150 of Cloudflare's customers across our Free, Pro, Business, and Enterprise plans. We have reached out to these customers directly to provide them with a copy of the data that was exposed, help them understand its impact, and help them mitigate that impact.
posted by Celsius1414 at 8:17 AM on February 24, 2017


Stack Overflow/Stack Exchange sites were not directly affected, having migrated away from CloudFlare in June 2016.
posted by fantabulous timewaster at 8:45 AM on February 24, 2017


Cloudflare posted an incident report with full technical details this morning.
It turned out that the underlying bug that caused the memory leak had been present in our Ragel-based parser for many years but no memory was leaked because of the way the internal NGINX buffers were used. Introducing cf-html subtly changed the buffering which enabled the leakage even though there were no problems in cf-html itself.

Once we knew that the bug was being caused by the activation of cf-html (but before we knew why) we disabled the three features that caused it to be used. Every feature Cloudflare ships has a corresponding feature flag, which we call a ‘global kill’. We activated the Email Obfuscation global kill 47 minutes after receiving details of the problem and the Automatic HTTPS Rewrites global kill 3h05m later. The Email Obfuscation feature had been changed on February 13 and was the primary cause of the leaked memory, thus disabling it quickly stopped almost all memory leaks.
I think things like Harry Potter appeal to a lot of people because we intuitively understand that bubbling underneath the surface strata of our daily lives there is a whole ton of stuff that might as fucking well be wizards.
posted by Shepherd at 8:45 AM on February 24, 2017 [4 favorites]


So, here's an interesting question: Have credit card processors finally adopted the equivalent of an authentication token for services that "Save" card numbers for their customers?

Yes, this is a thing.
posted by brennen at 8:48 AM on February 24, 2017


I think things like Harry Potter appeal to a lot of people because we intuitively understand that bubbling underneath the surface strata of our daily lives there is a whole ton of stuff that might as fucking well be wizards.

Any sufficiently advanced technology, etc. etc.
posted by The Bellman at 9:21 AM on February 24, 2017


Every feature Cloudflare ships has a corresponding feature flag, which we call a ‘global kill’.

Now I sort of want to work at Cloudflare just so I can run into a room one day and shout: "Activate the global kill!"
posted by The Bellman at 9:23 AM on February 24, 2017


How does a bunch of random, untargeted, and uncorrelated data serve intelligence objectives?

One of the more interesting pieces of data that was leaking is Uber location data. If this was somehow introduced intentionally (and I doubt that it was, but it's certainly possible), it would open an opportunity to track users' locations and ride histories.

Plenty of intelligence objectives there.
posted by toxic at 9:45 AM on February 24, 2017


So we are left to wonder, "Am I part of the 0.00003%??"
posted by hank at 9:58 AM on February 24, 2017


Interesting, on our project we have a runtime switch in non public builds to enable overflow detection and freed memory accesses. Bascically we replace all allocs/free to calls to VirtualAlloc/VirtualFree, this means that when we free we actually decommit the memory and keep the virtual page but with read/write/ex access, so any read/write to freed memory is a crash. We also allocate an additional page at the end of the buffer on the initial alloc (and align the alloc so it ends just at the page boundary) which is never committed and has no access, this traps all page size and smaller overflows.

This setup is incredibly useful at tracking bad accesses (it's all C++ so we firing shotguns in close proximity of our feets all day). Obviously the perf hit is significant (but it clearly steers you toward keeping allocations in check while you're developing, so not a bad thing) and in it's rawest form but a cap on your process lifetime (virtual space isn't infinite). We also have a variation that does it in limited scopes, on random allocations and another variations that ends up freeing the pages of old allocations at some point so that we can run the process for a longer time.

This is obviously not a panacea since you actually have to encounter the bug to catch it, but it's proven very useful at catching issues early on developers machine. I'm unsure how feasible it would for those devs to use systems like this (volume alone would kill the server quickly I think unless you recycle through the virtual memory), but using it all the time in a limited scope in dev builds has proven to be very useful and made our code a lot more robust.
posted by coust at 10:10 AM on February 24, 2017 [3 favorites]


I've learned a lot today about what goes on in big CDNs...
posted by Devonian at 11:25 AM on February 24, 2017


FWIW, Google states that the reauth on various accounts (some coworkers and friends of mine got this as well) is unrelated.

I think the "NSA did it!" spin on this is probably multiplying causes unnecessarily. There's a pretty reasonable narrative about this as an accidental bug, and while paranoia about state actors is generally justified, I think in this instance it probably obscures a lot of the ways that an architecture like Cloudfront makes us less secure.

The place to look for agency here probably isn't "did a badguy do this specific thing". It's "does the industry keep making decisions that consolidate power and vulnerability to failure modes", and "is the architecture of the modern web broken".
posted by brennen at 12:02 PM on February 24, 2017 [6 favorites]


The place to look for agency here probably isn't "did a badguy do this specific thing". It's "does the industry keep making decisions that consolidate power and vulnerability to failure modes", and "is the architecture of the modern web broken".

Yes to all of the above. I think that the company report is reasonably compelling (to me, who doesn't do these things but understands the concepts and some of the specifics, mod ignorance) and on the one hand I'm glad that it was a bug and not a GCHQ intercept program called GIANT JELLY or whatever. On the other... I'm sure that there's now a GIANT JELLY in the works, because this incident has highlighted a lot of potential for mischief. This was a lovely concatenation of sleeping bugs, new features and reconfigurations that accidentally mimicked a very plausible attack. (And could still have been, although given that most - all? - of the software involved is open source, that is now available for forensics.)

And yes, the architecture of the modern web is broken, and it always has been - inasmuch as it does its basic task so darn well it's constantly overburdened with inappropriate and unsafe usages, which create a lot of problems and rather fewer good solutions. In one way,the whole saga has demonstrated some impressive resilience (which as Schneier has been preaching for a while, is a good thing to aim for, rather than prevention of detedction alone.)

My favourite analogy here is the immune system - and those early analogies to biological viruses were apt. Are our biologies broken? Oh boy, yes. But evolution is very bad at fixing past mistakes at source; instead, it's come up with a really complicated (and capable fo self-harm) system of interlocking prevention, detection and remediation subsystems. Result: we stay alive long enough as individuals to survive as a species. If I had to put money on the long-term nature of all this digital stuff, I'd go down that route.
posted by Devonian at 12:36 PM on February 24, 2017 [2 favorites]


iffthen: "(Insert grumpy comment about the wisdom of giving all your passwords to an online password manager service.)"

Says the person with only one device.
posted by Samizdata at 1:22 PM on February 24, 2017 [1 favorite]


On the other... I'm sure that there's now a GIANT JELLY in the works, because this incident has highlighted a lot of potential for mischief.

Between this and the whole 'Passwords please!" airport security protocol that seems to be becoming a thing it does seem that consciously monitoring and shaping the boundaries of one's internet footprint (and having multiple of them for different porpoises - possibly via multiple VPN services) is becoming a necessity.

Currently debating whether to go with the Yubikey 4, or just the Fido for now and update later.

Also thinking I should finally get round to finding out what the whole "dark web" thing is. Don't suppose it's anything like warezing via IRC bots and invisible FTP dirs? I used to rock at that back in the day... once spent ~3 months getting Windows NT from this South Korean server on a 14.4 via my (at the time) Shetland based ISP. Turned out NT wasn't significantly less shit than WFW 3.11, but hey, it's the taking part that counts.


2) What's the probably that a page is served to a cache engine? Not a clue. Let's assume 1/1000.

Umm, analysis... ur doin it wrong.
posted by Buntix at 1:30 PM on February 24, 2017


in this instance it probably obscures a lot of the ways that an architecture like Cloudfront makes us less secure.

Understatement. If the NSA wanted to pull this trick off, they would waltz in, pay the Cloudfront guys money and handshakes and smiles all around.
posted by hleehowon at 1:36 PM on February 24, 2017 [2 favorites]


I can't find the password setting on half of the flagged sites, and the lastpass extension is giving me duplicate entries when I do change, and the vault seems to lag behind other tabs making it difficult to figure out which password I need to clean up.
posted by CBrachyrhynchos at 1:43 PM on February 24, 2017


Understatement. If the NSA wanted to pull this trick off, they would waltz in, pay the Cloudfront guys money and handshakes and smiles all around.

Possibly not. It also seems like the alt-right GamerGate crowd hate Cloudflare, which is not a bad sign (albeit not good by default).
posted by Buntix at 1:56 PM on February 24, 2017


I'm getting the impression that the main reason this isn't looking like an NSA job is this: there was no way to control what data got leaked, and there's just far, far too much data going through Cloudflare to ever hope you'd happen to find something actionable.

I mean, those Uber leaks sound super tasty at first - oauth tokens! locations and times! - but the data is memory cruft from random sessions between random users and random services. Even if it came with metadata sufficient to identify the user involved, it would be vanishingly unlikely for any of them to turn out to be someone the feds cared about.
posted by reprise the theme song and roll the credits at 1:57 PM on February 24, 2017 [2 favorites]


Your choices now are go with a big CDN or have your service be at the mercy of anyone with fifty bucks worth of bitcoin.

$60,000 seems a bit high.
posted by MiltonRandKalman at 2:57 PM on February 24, 2017


"fifty bucks worth" not "fifty bitcoin"
posted by RustyBrooks at 3:24 PM on February 24, 2017 [2 favorites]


Possibly useful (although it's likely better to assume that any important passwords you have should be changed): http://www.doesitusecloudflare.com
posted by bitmage at 3:54 PM on February 24, 2017 [1 favorite]


Just tested, and the http://www.doesitusecloudflare.com site is hitting servers with a query of:
"GET / HTTP/1.1" 200 549 "-" "curb"
so it's doing a live test and not working from a list. If a provider stops using Cloudflare then this site may report them clean even though they were vulnerable previously.
posted by bitmage at 4:00 PM on February 24, 2017 [3 favorites]


KeePass is his recommendation, if you're keeping track.

I work in an industry that requires pretty extreme infosec testing/verification both internally and externally, and this is the ONLY one they approved. We had a temporary recommendation to switch while they patched a bug that lasted less than a day.

It's the real deal.
posted by emptythought at 4:47 PM on February 24, 2017 [1 favorite]


Just tested, and the http://www.doesitusecloudflare.com site is hitting servers with a query of:
"GET / HTTP/1.1" 200 549 "-" "curb"
so it's doing a live test and not working from a list. If a provider stops using Cloudflare then this site may report them clean even though they were vulnerable previously.


In fairness, the site isn’t diditusecloudflare.com or wasitusingcloudflare.com
posted by Going To Maine at 5:36 PM on February 24, 2017 [2 favorites]


Keepass is available for Windows, Android and Linux. It's what I've used for years. I put it on my Dropbox account so I can access anywhere.
posted by 922257033c4a0f3cecdbd819a46d626999d1af4a at 5:51 PM on February 24, 2017


Keepass. Hosted on Sourceforge? Really? I mean, I know they're under new management, but that's still not a chance I'd be willing to take today.
posted by Celsius1414 at 6:24 PM on February 24, 2017


KeepassX is the currently maintained code you're looking for.
posted by wierdo at 6:51 PM on February 24, 2017 [1 favorite]


Err, stupidity strikes. It's what I use on non-Windows platforms. KeePass is actually maintained (and 2.x is cross platform, but inertia is a bitch)
posted by wierdo at 6:55 PM on February 24, 2017 [1 favorite]


Yeah, I used to do IT support, and ever since I've used KeePass.
posted by krinklyfig at 7:06 PM on February 24, 2017


Users of 1Password, can click on "show" next to Security Audit and then enable Watchtower it will show you which accounts you should consider changing. I found it helpful.
posted by terrapin at 11:39 AM on February 27, 2017 [1 favorite]


Anybody have recommendations for a good KeePass android client?
posted by Going To Maine at 1:49 PM on February 28, 2017


Anybody have recommendations for a good KeePass android client?

I've been using Keepass2Android. Seems pretty good so far.
posted by reprise the theme song and roll the credits at 12:32 AM on March 1, 2017 [2 favorites]


K2A is great. It has sync options for many different storage providers (including self hosting with SFTP, OwnCloud, WebDAV, or plain FTP) and can make the process of entering passwords reasonably easy. Not quite as automatic as something on desktop due to the limitations of the platform, but if you're rooted it can even do that.

It has an accessibility provider that (optionally) allows it to detect password fields in web pages and even many apps and an input provider to go along with it so the password can be provided without the very real risk of clipboard snooping. Fantastic app, all around, and open source too.
posted by wierdo at 2:11 AM on March 1, 2017 [1 favorite]


A list of 23.5k companies possibly affected - Cloudflare Customers
posted by ggiaco at 3:40 PM on March 21, 2017 [1 favorite]


« Older Archaeogenomic evidence reveals prehistoric...   |   Triumph of the Will and the Cinematic Language of... Newer »


This thread has been archived and is closed to new comments