Join 3,561 readers in helping fund MetaFilter (Hide)


BGP Hijacking for fun and profit!
August 13, 2014 12:36 PM   Subscribe

Dell SecureWorks has discovered that a blackhat used spoofed BGP announcements in order to steal cryptocurrency worth about $83,000 over the course of a few months earlier this year.

Border Gateway Protocol (BGP) is the protocol used by ISPs to tell each other which IPs they own, so as to make internetwork routing work. If there are overlapping announcements, whichever one covers the smaller total space wins.
Without BGP, there is no Internet as we know it. But that doesn’t mean it can’t cause problems—our reliance on the accuracy of the information provided by BGP routers means that anyone who can gain access to one can redirect some portion of online traffic by advertising a sufficiently small set of addresses whose traffic it wants to target. In other words, if you want access to some piece of online traffic directed to someone else, you can use BGP to announce that you will deliver it to its intended recipients—in the same way that Comcast announces it can deliver traffic to me—and the rest of the Internet will believe you. So this is probably what happened in the bitcoin theft incidents investigated by SecureWorks—the thief used the credentials of someone who worked at a Canadian ISP to send out false routing announcements. Using those announcements, the thief redirected the traffic of groups dedicated to bitcoin mining and was able to retain the bitcoins harvested by those groups’ machines rather than paying them out to the owners of the mining computers.
posted by Chocolate Pickle (40 comments total) 13 users marked this as a favorite

 
Did this get fixed in IPV6?
posted by Chocolate Pickle at 12:46 PM on August 13 [2 favorites]


Oh man, I love Bitcoin threads!
posted by Sangermaine at 12:51 PM on August 13 [3 favorites]


No. BGP remains the core of everything.

Besides which there's not much to " fix" when the problem is stolen credentials.
posted by Tell Me No Lies at 12:51 PM on August 13 [2 favorites]


The first time I had to care about BGP was when the voip company I worked for got taken off the internet for a day when our request to a customer that they add our IPs to their firewall allow list somehow got morphed into "assign these ips to your router", which somehow got escalated to someone at AT&T who was stupid enough to update their routing tables with those ips, causing 100% of our voice traffic to flow down this poor customer's 1.5meg T-1 link.
posted by empath at 12:54 PM on August 13 [34 favorites]


Besides which there's not much to " fix" when the problem is stolen credentials.

The problem is access to BGP routers period, though, right? Not necessarily "stolen credentials"? Or am I misunderstanding?
posted by saulgoodman at 12:56 PM on August 13


A BGP announcement for a single IP address is an unstoppable force. I wonder if NSA has used this, on occasion?
posted by Chocolate Pickle at 12:56 PM on August 13


BGP is basically based on the assumption that there are no malicious actors. Anybody who wants to steal an ip can do it, until someone upstream notices and decides to disconnect them/ignore them.
posted by empath at 12:58 PM on August 13 [3 favorites]


Please the Anton Kapela & Alex Pilosov's talk at Defcon 16 on the matter of BGP hijacking.

Yes, the NSA does this, they most likely have been doing it for the last 20 years.
No, It's not operationally possible to BGP announce a single IP address.
This is a fixable problem with BGP route signing I believe.
It does not necessarily involve hacking a router or stealing credentials.
This attack can be perform by anyone with sufficent bandwidth, skill, peering and will.
posted by Agent_X_ at 1:03 PM on August 13 [12 favorites]


This attack can be perform by anyone with sufficent bandwidth, skill, peering and will.

Basically anybody with access to a core router at a reasonably sized ISP.
posted by empath at 1:05 PM on August 13 [1 favorite]


That's funny, parts of the Internet went down yesterday due to BGP problems.

Think Y2K-style problems, but in router firmware.
posted by blue_beetle at 1:11 PM on August 13 [3 favorites]


Think Y2K-style problems, but in router firmware.

Here's the configuration after applying the Cisco-recommended patch:

IPv4 - 1000k
MPLS - 8k (default)
IPv6 + IP Multicast - 8k (default)


Aww, only 8k for IPv6. No Internet of Things for you!
posted by RobotVoodooPower at 1:37 PM on August 13 [1 favorite]


A BGP announcement for a single IP address is an unstoppable force.

No, BGP announcements involve an autonomous system number and an network address/prefix . The maximum prefix that people will accept from the world is a /24, which corresponds to the old BGP3 Class C address, and has a net mask of 255.255.255.0

Any higher prefix (read, smaller network) announcement will be dropped. If you want to announce a /30, you have to aggregate it with other network, announce at least a /24, then, on the inside, break it apart.

The problem we're running into now is the full table runs about 500K entries, and a lot of older Juniper and Cisco routers can't handle over 512K entries. This isn't a protocol issue, this is an implementation issue. Nobody but nobody believed we'd see that many prefixes in the table when they wrote BGP4. Outages have been occurring recently because of this -- someone screws up, stops aggregating routes and announces them individually, the table climbs up over 512K, and older routers choke.

Aggregation. Say I'm an ISP, and I have 172.16.0.0/24 assigned to one customer, and 172.16.1.0/24 assigned to another. Both are routable**, and my routers know where the customer is on my network.

So, instead of announcing both of them, I can announce 172.16.0.0/23, and the world will then know to send everything to me on that address -- and I'm only announcing one prefix, not two.

If I then sell networks running 172.16.3.0/24 and 172.16.4.0/24, I can then announce 172.16.0.0/22 and get all four into the global table with just one entry.

Part of the reason Routing Is Hard is that we *have* to aggregate, and it's not a trivial problem. So, my customer who has 172.16.3.0/24 is multi homed. I get to announce the 172.16.3.0/24 as part of my larger aggregation, but that other network he's homed on doesn't have 172.16.2.0/24, so they have to announce 172.16.3.0/24.

Worse, if my line to that customer dies, I need to stop announcing 172.16.3.0/24, which means I have to stop announcing 172.16.0.0/22 and start announcing 172.16.0.0/23 and 172.16.4.0/24.

Of course, this propagates upstream. My upstream may well have been announcing 172.16.0.0/20, but now they have to break that apart.

This is why Route Flapping is bad -- everytime you drop or announce a new route, everyone in the path has to reaggregate and renounce. If you go down, stay down for a while. After a certain number of Up-Down-Up-Down in a given period, you're upstream will just say "You're down" and not accept announcements from you for a while. After that, they'll listen for a bit, and if you're still flapping, they'll leave you down. If you're stable, they'll accept, reaggregate, renounce, and you get your Internet back.

So, it's a non-trivial problem, and BGP4 has done an amazing job at quite literally building the Internet. "Internetwork" is a network of networks, and BGP is how those networks find each other.

(I helped setup and run a few ASN in the not too distant past. Last one was 32423)



** No, these aren't routable, they're RFC1918, but the example rule is DO NOT USE REAL NETWORKS EVER. To BGP, 172.16.0.0/16 is fake.
posted by eriko at 1:39 PM on August 13 [42 favorites]


Filter Gauntlets and companies that provide near real time BGP route change monitoring are just about our only line of defense. There's BGP over GRE which can add some measure of protection, but for the most part BGP is the internet of old where you just trusted everything and figured everyone else running the internet was a punk anarchist hippie like yourself that wouldn't dream of being assy. Those days are gone by at least 20 years...
posted by Annika Cicada at 1:55 PM on August 13 [3 favorites]


Also worth mentioning that it isn't the Bitcoin protocol which was hacked, but a commonly-used mining protocol which is unencrypted and susceptible to these kinds of attacks.

(The Bitcoin protocol isn't encrypted either, but since nodes don't trust each other very much attacking it with BGP isn't as simple as "redirect traffic steal money")
posted by RobotVoodooPower at 1:57 PM on August 13


RobotVoodooPower: redirect traffic steal money Bitcoin

FTFY
posted by tonycpsu at 1:58 PM on August 13 [1 favorite]


eriko: no BGP peer is under any obligation to change their route aggregation based on your incoming route updates, I can ignore the request to update my routing table from /23 to /22 and you'll be stuck, unless we have agreed upon community strings, but even then I can still choose to disrespect your route update and aggregate however I want.
posted by Annika Cicada at 2:00 PM on August 13 [1 favorite]


BGP also relies on the people configuring routing information to be smart enough to assign reasonable route preferences. If memory serves it allows operators to specify arbitrary preferences in terms of the routes advertised by neighboring autonomous systems, which if configured wrong, can lead to routing loops, which is very bad.
posted by axiom at 2:27 PM on August 13


The problem is access to BGP routers period, though, right? Not necessarily "stolen credentials"? Or am I misunderstanding?

Getting access to a BGP router that is part of the trusted circle is a problem, yes.

But consider this: This situation has existed for twenty years now, and you can count the number of critical incidents on one hand.

Part of the reason for that is that getting kicked out of the BGP trusted circle is the end of your ISP business. If you regularly advertise busted routes nobody is going to talk to you, and there is no internet without connectivity. Whoever let those Canadian keys get compromised is looking for a new job right now.

From a technical standpoint you could narrow the potential damage by having credentials per route rather than per BGP router, but a) it would only narrow the problem and b) the overhead is massive and would likely end up being a greater source of disruption than the problem it purports to solve.
posted by Tell Me No Lies at 2:36 PM on August 13 [1 favorite]


these types of attacks are going to last for a few hours tops before getting fixed. They require access to large ISP tier 1 networks, so we are talking government sponsored/corruption required in order to pull off an attack like this.

Yes it's uncommon, it's highly visible within the large scale routing world, and the risk is mitigated by end users using encrypted protocols for sensitive data being transported.
posted by Annika Cicada at 2:46 PM on August 13 [1 favorite]


Um, what the fuck is everyone here talking about?
posted by latkes at 3:00 PM on August 13 [3 favorites]


BGP
posted by Annika Cicada at 3:03 PM on August 13 [3 favorites]


Oh man, I love Eve Online threads!
posted by TheWhiteSkull at 3:06 PM on August 13 [10 favorites]


Um, what the fuck is everyone here talking about?

The secret underpinnings of the fragile web that sustains your very life.
posted by Tell Me No Lies at 3:41 PM on August 13 [10 favorites]


(online life)
posted by Tell Me No Lies at 3:41 PM on August 13


Bitcoin: banking reinvented by people who didn't think they needed verify connections.
posted by adamsc at 3:59 PM on August 13 [1 favorite]


I wondered why SSL didn't protect against the BGP hijacking attack until I read on to "the easiest option for pool servers is to require miners to use the Secure Socket Layer (SSL) protocol". /facepalm. I guess no one running these money printing machines considered someone else might want to hijack their money printers. Although naïve SSL isn't sufficient; you need to verify the certificate really belongs to the entity you are expecting.

BGP is one of the scarier parts of the Internet fabric. It's complex, and brittle, and has a very precarious security model. It's also served us quite well for over 20 years and so much depends on understanding how it works, it's hard to imagine it being replaced.

Here's an article on how BGP spoofing works and some live Internet examples of it being exploited. And since someone mentioned Eve Online, here's some clever things they've done with BGP to improve game performance.
posted by Nelson at 4:04 PM on August 13


Bitcoin: banking reinvented by people who didn't think they needed verify connections.

OK so, that's not what happened here, but fun fact: real electronic bank transfers actually do pretty much work without verification.
posted by odinsdream at 6:22 PM on August 13 [3 favorites]


eriko: no BGP peer is under any obligation to change their route aggregation based on your incoming route updates, I can ignore the request to update my routing table from /23 to /22 and you'll be stuck, unless we have agreed upon community strings, but even then I can still choose to disrespect your route update and aggregate however I want.

True enough. You have to filter rationally, of course. BGP as written assumes you won't, but in the real world, sometimes, you have to. Remember when Iran announced 8.0.0.0/4 and wondered why their tiny internet lines saturated?

In general, you aggregate conservatively and try to get the packets through. Unless your Verizon, who seems to keep deaggregating and pushing the table over 512K. Yeah, I get that we need to fix all the routers, but seriously, stop that.

I also get the sense that you've run much larger networks that I have. Most peers I've dealt with is five. So, people, if I say X and Annika Cicada says !X, I'd bet on Annika Cicada.

One of the important lessons in life is there is always someone better than you at something, listen to them and learn. The most important skill I have is the ability to say "I don't know. How does that work?"
posted by eriko at 7:40 PM on August 13 [3 favorites]


Um, what the fuck is everyone here talking about?

You got you this big-ass computer that was designed by big-brained dweebs to make money out of, I shit you not, thin-fucking-air.

Now, this ain't folding money, this is the kind of money bankers and shit put down in ledgers, only there ain't no more ledgers, that shit's all computers on the internet now. So instead of hiring Sean and Vinnie to take a paper bag of the folding stuff to the bosses, it's got to go over the internet, one computer to another.

Now, computers generally don't talk to each other direct - they hand off like runners and bag-men. So, the big-ass computer pulls money almost literally out of it's ass, and then hands it off to a bag-man, who stuffs it in a bag and puts the Boss' name and organization on it. He hands it off to a runner, who runs up to the corner, and goes, "Hey, any of you guys know this dude?"

Bad-ass at the corner goes, "Nah, man, but I heard of the dude and his crew. Hard core motherfuckers. Head on over five blocks east, and ask there."

This works, up until the runner comes across someone who got duped. He heard from someone important that the Boss works out of "The Cafe" out on the docks, but someone who seemed legit, but was a fucking weasel, just now told him that The Boss at the Organization was now running out of some garage just outside town.

"Who you running for, kid?"
"Big-Ass Computer's bag-man, by way of the dude at that corner! This is for The Boss, at The Organization!"
"Hey, hey, you're in luck! I know where that's going! Just heard about it! Hand it over, guy, and you're done for the night! My runner will take it from here! Good job!"

So, the next day, the Boss rolls on up to an abandoned garage, all the money the computer pulled out of its ass is gone. The weasel got snuffed, but even he didn't know where the money was headed.

These modern times, I tell ya.
posted by Slap*Happy at 8:18 PM on August 13 [20 favorites]


Eriko, I've worked on BGP in a lot of different scenarios, but I wouldn't categorize myself as some BGP expert :-)
posted by Annika Cicada at 8:53 PM on August 13 [1 favorite]


Eriko, I've worked on BGP in a lot of different scenarios, but I wouldn't categorize myself as some BGP expert :-)

Let's just tell them it's all Unicorn Horns and Fairy Dust.

Because I think those are better documented than most BGP implementations.
posted by eriko at 9:47 PM on August 13 [4 favorites]


One of the important lessons in life is there is always someone better than you at something, listen to them and learn. The most important skill I have is the ability to say "I don't know. How does that work?"

One of my favorite bits of 1990s cisco lore is the time when a candidate claiming a complete theoretical and functional understanding of BGP-4 ended up interviewing with Tony Li. The guy got really snippy when Tony kept suggesting that his descriptions of the protocol were incorrect.

Things came to a head when the candidate decided to directly challenge Tony as to why he thought his understanding of BGP-4 was so much better. Tony pointed out that he was an author of the protocol and things sort of went downhill from there. It's the only time I've heard of an engineer directly walking a candidate to the door, and it set a standing record for interviews by having the candidate last only 30 minutes into a 5 hour schedule.
posted by Tell Me No Lies at 11:41 PM on August 13 [8 favorites]


It's the only time I've heard of an engineer directly walking a candidate to the door

We hired him. Made it to CIO eighteen months later.
posted by Devonian at 1:22 AM on August 14 [6 favorites]


> I wouldn't categorize myself as some BGP expert

I know a guy whose name is on a number of BGP-related IETF RFCs. Of all the conversations over dinners and drinks we've had through the years, nearly none of them covered BGP. In my experience, the threshold for sufficient comprehension of BGP to be able to discuss it socially over drinks is much higher than the knowledge needed to, say, discuss microprocessor design theory with people who do that for a living.
posted by ardgedee at 4:22 AM on August 14


OK so, that's not what happened here, but fun fact: real electronic bank transfers actually do pretty much work without verification.
olinsdream: you should read the post again - that's exactly what happened here. A fundamental concept in security is that systems are attacked at the weakest point, not the strongest. It doesn't matter if the core bitcoin protocol is secure if the system itself has flaws.

More broadly, this continues the pattern of Bitcoin services being built by amateurs who then tell the world they're the future of banking. Just like the exchanges which didn't keep records, it's unlikely that someone who didn't realize that they needed to secure connections on the internet made just that one mistake. Bitcoin promoters would be much better off finding experts to audit their systems than engaging in PR after the fact.
posted by adamsc at 5:50 AM on August 14


Bitcoin promoters would be much better off finding experts to audit their systems than engaging in PR after the fact.

hmm that doesnt sound very Disruptive of u
posted by p3on at 9:45 AM on August 14 [1 favorite]


On pondering...This whole event lends more credence to my notion that techno-libertertarianism is a terribly misguided ideology that totally discounts why we have the society and laws we have in place today, replaced instead with a petulant childish attitude largely based on the rhetorical question "WHY DO I HAVE TO?!" that the person asking assumes "I DON'T" is the only (il)logical answer.
posted by Annika Cicada at 11:03 AM on August 14 [3 favorites]


"WHY DO I HAVE TO?!"

On the other hand that is not a bad question to ask, vigorously and often.

It is the libertarian's inability to tell the difference between useful and non-useful systems, and definitely their designs to replace both types with ideas that would make Pollyana blush, that earn them the rolled eyes they almost universally inspire.
posted by Tell Me No Lies at 1:07 PM on August 14 [1 favorite]


One of my favorite bits of 1990s cisco lore is the time when a candidate claiming a complete theoretical and functional understanding of BGP-4 ended up interviewing with Tony Li

I was at a conference where Microsoft was announcing Services For Unix on the Windows platform, and as part of the pitch, they announced they had a completely ksh compatible shell.

A voice pipes up. "No, it isn't. I know of at least three cases where it is not true to ksh."

The presenter, thrown off course, but not lost, asks "Oh really? Can you document this?"

The voice in the audience says "Of course I can. I'm David Korn."

To Microsoft's credit, the presenter said "Oh really? We'd really like to know what those are and get those fixed." Djk, later, opined that they'd actually done a very good job with it, but there are damn few 100% correct implementations*.

He also explained the new "insert STDOUT of command here" syntax he'd developed. He'd fought arthritis since his early 20's, and getting the pinky up to the back tick was an amazingly hard thing for him to do. I will note that ksh fully supports the Bourne shell syntax.

I will also note that djk got the O'Reilly reference book on ksh signed by the members of KoRn.

* The bitch of ksh is that a subset of it is 100% Bourne shell compatible. If you write to that subset, your script will work on any Unix system since Version 7, unlike csh, which was completely incompatible with sh. GNU proved later the correct answer is to steal from the best, and bash, like ksh, is basically a superset of sh. The biggest non ksh88 thing in bash is brace expansion, which is from csh, but ksh93 has that as well. And, of course, there are two thing you should do with bash.

1) set -o vi
2) cd /usr/ports/shell/ksh93; make install
posted by eriko at 9:08 PM on August 14 [1 favorite]


BGP's been in the news a lot lately. This week the Internet has had rolling brownouts because of a 512k BGP route limit. eriko talked about this upthread, but now it's in the popular press and seems to be making a continuing mess.
posted by Nelson at 7:29 PM on August 17


« Older GenCon, the best four days of gaming starts tomorr...  |  Despite the comment collecting... Newer »


This thread has been archived and is closed to new comments