Github Survived The Biggest DDoS Attack Ever Recorded

March 1, 2018 2:41 PM

On Wednesday, at about 12:15pm ET, 1.35 terabits per second of traffic hit the developer platform GitHub all at once. It was the most powerful distributed denial of service attack recorded to date—and it used an increasingly popular DDoS method, no botnet required.

ZDNet: Memcached is a tool meant to cache data and reduce strain on heavier data stores, like disk or databases. It is only intended to be used on systems that are not exposed to the internet, as there is no authentication required. However, there are currently more than 50,000 known vulnerable systems, according to Akamai.

When a system receives a memcached get request, it forms a response by collecting the requested values from memory and then sending them back in an uninterrupted stream.

However, DDoS attackers have now realised they can use this protocol to launch attacks. First, the attacker implants a large payload on an exposed memcached server. Then, the attacker spoofs the "get" request message with a target's source IP, and thus, a short request to the exposed server can result in a pile of traffic arriving at the victim.

Bleeping Computer: The good news is that mitigating such attacks should be extremely easy. All a victim has to do is block incoming connections from port 11211, the DDoS attack's reflection source. [...]

As Akamai warns, the size of these attacks is going up, and it might soon reach even bigger numbers.

In fact, this is what appears to be happening. DDoS attacks leveraging the Memcached vulnerability started over the weekend. Cloudflare reported mitigating the first such attacks, which reached 260 Gbps, on Monday, February 26.

Qrator Labs then reported mitigating another Memcached-based DDoS attack that reached over 500 Gbps, followed by the one that Akamai detected of 1.3 Tbps.

Cloudflare: Memcrashed - Major amplification attacks from UDP port 11211
GitHub Engineering: February 28th DDoS Incident Report
Akamai: Memcached-Fueled 1.3 Tbps Attacks

Previous GitHub DDoS: China vs GitHub

posted by not_the_water (8 comments total)

relatedly, there's a podcast called Darknet Diaries that covers hacks and infosec stuff that's really fascinating

the details are really complicated if you don't have the education but it seems like you can pretty much chalk every single mistake to carelessness and the subsequent response to laziness. like with ASUSgate and their complete unwillingness to give a single fart about their extremely hackable topshelf routers even after someone went out of their way, repeatedly, to let them know about the vulnerability. or here when people use an unsecured tool in an inappropriate way
posted by runt at 2:49 PM on March 1, 2018 [4 favorites]

but it seems like you can pretty much chalk every single mistake to carelessness and the subsequent response to laziness. like with ASUSgate and their complete unwillingness to give a single fart about their extremely hackable topshelf routers even after someone went out of their way, repeatedly, to let them know about the vulnerability. or here when people use an unsecured tool in an inappropriate way

Unfortunately it's not quite that simple. In the wake of these attacks more attention will be devoted to security memcached servers but this type of reflection is a pretty easy type of attack to make via any unauthenticated UDP service.

DNS is one of the classic ones, as it's more of a problem to block DNS answers entering your network from elsewhere than it is to block memcached responses and if you query for DNSSEC-signed answers you can achieve a pretty respectable amplification factor, though not nearly as high as this memcached attack.

Until effective anti-spoofing methods for UDP are widely deployed there's not a lot that will stop this type of attack, though this particular instance was a bit notable for its high amplification factor.
posted by Nerd of the North at 2:56 PM on March 1, 2018 [1 favorite]

This rule is normally reserved for things like web-forms, but should really be considered by anything open/web facing: You must consider how your feature/service will be abused by others..

I don't think memcache can just throw up their hands and blame users with poorly configured/insecure systems. If it's a "feature" out-of-the-box, onus is on them (memcache) to make it secure out-of-the-box. (cf how Microsoft has tried to get better with each server OS release -- last I tried, you really had to work to shoot yourself in the foot by setting something up insecurely. Not that you couldn't, but .. )

(edit to add: yeah, the "until we can fix IP spoofing .." is memcache's saving grace)
posted by k5.user at 3:52 PM on March 1, 2018 [1 favorite]

Also the ISPs need to take responsibility in keeping known malicious traffic the fuck off their network. I shouldn't have to shovel money in CloudFlare's direction because a Tier 1 or 2 ISP is too stupid, lazy or cheap to keep blatantly obvious DDoS traffic from traversing their pipes.
posted by Slap*Happy at 4:29 PM on March 1, 2018 [1 favorite]

Until effective anti-spoofing methods for UDP are widely deployed there's not a lot that will stop this type of attack, though this particular instance was a bit notable for its high amplification factor.

A rudimentary handshake with a random 128-bit number before blasting your target with the requested data would be enough.
posted by Talez at 5:03 PM on March 1, 2018 [2 favorites]

Aww memcached. Now I remember the Livejournal.
posted by alex_skazat at 8:02 PM on March 1, 2018 [3 favorites]

Someday, people will learn to stop doing boneheaded things like opening overly-trusting services to the internet, but apparently that day is not today.

Kudos to Akamai for this, though. It must feel amazing when you've specced your system to handle ridiculous load 5X and it comes in at slightly over that and you just sort of deal with it. (Although I've often wondered if working in DDoS mitigation kind of damages your faith in humanity over time. Like, your job is literally just to deal with assholes trying to break things for lulz. There's intellectual challenge in the tech to defeat the assholes, but then there's just the fact that there's always, always a new bunch of assholes.)
posted by sldownard at 11:36 PM on March 1, 2018 [6 favorites]

This rule is normally reserved for things like web-forms, but should really be considered by anything open/web facing: You must consider how your feature/service will be abused by others..

My current boss is better at this than anyone I've ever met - he was a teenage phone hacker and script kiddie who graduated to actually running a technical division at our company and he has a sixth sense for sniffing out avenues of abuse in the systems we build. I thought I was ok at it, but it's fun to watch a master at work. His brain works in strange and terrifying ways.
posted by protocoach at 1:20 PM on March 2, 2018 [2 favorites]