Juniper systems hack
December 18, 2015 3:53 PM   Subscribe

Firewalls from Juniper Network have been revealed as having a back door which permitted system-level access and the ability to read VPN traffic. Juniper has released emergency patches to fix the vulnerability. The affected equipment is heavily used by the US government.
posted by Chocolate Pickle (47 comments total) 9 users marked this as a favorite
 
And there's the paradox - The same hardware that the government would rely on to secure their own data is the same hardware that they wish to be deliberately vulnerable as well. This is the same hardware that we hear many in positions of power say must have deliberate back doors on them for law enforcement purposes. This is the same hardware that the NSA and other covert agencies are tasked with backdooring themselves, and they have been known to "encourage" vulnerabilities from the inside for these reasons - Ars covers the background quite nicely.

The backdoor itself may not have been engineered by the US, but it's entirely possible that the vulnerability was encourage or discovered and held tightly by one of the covert teams - our policies certainly support that.
posted by MysticMCJ at 4:04 PM on December 18, 2015 [13 favorites]


I would think that this as likely to be the work of the US government as it is a foreign government. What does the NSA care if it's made a hole for the rest of the government? They only care about increasing the NSA's reach, and as the Ars Technica article says:
Der Spiegel reported that an NSA operation known as FEEDTROUGH worked against Juniper firewalls and gave the agency persistent backdoor access.
Other things to be avoided: IPSec.
posted by Llama-Lime at 4:07 PM on December 18, 2015


Cue mass queries on Shodan and Censys.io to check how many identifiable Juniper Firewall/VPN ScreenOS devices with SSH or telnet exposed are out there.....and who's potentially pwnable.
posted by inflatablekiwi at 4:10 PM on December 18, 2015 [1 favorite]


I would think that this as likely to be the work of the US government as it is a foreign government. What does the NSA care if it's made a hole for the rest of the government? They only care about increasing the NSA's reach

Yeah, after reading the Ars Technica article my immediate thought was, "huh, so the NSA doesn't just want to spy on citizens, they want to spy on the rest of the government." Hell, they probably either saw how effectively the CIA spied on the Senate Intelligence Committee, or helped them do it.
posted by Existential Dread at 4:10 PM on December 18, 2015


If this turns out to be yet more dirty work by the usual suspect (ie NSA, whose charter includes protection of U.S. government communications and information systems against penetration), could such an intentional sabotage of their own purpose (not to mention the public good) be leveraged in any realistic way to rein in the rogue agency?
posted by anonymisc at 4:12 PM on December 18, 2015


Juniper should be able to trace which employee checked in that code if they have proper source control. The question is for which agency did that employee put that code there for?
posted by LoveHam at 4:17 PM on December 18, 2015 [2 favorites]


Juniper should be able to trace which employee checked in that code if they have proper source control.

Unless their source repository was compromised and tampered with. Or a developer's credentials were compromised and used without their knowledge. Is it common for committers to cryptographically sign every commit they make to source repos in large organizations?
posted by indubitable at 4:26 PM on December 18, 2015 [10 favorites]


could such an intentional sabotage of their own purpose (not to mention the public good) be leveraged in any realistic way to rein in the rogue agency?

You can't sue the US government.
posted by a lungful of dragon at 4:27 PM on December 18, 2015 [1 favorite]


Huh, my first reaction was if the backdoor was spotted, it probably wasn't NSA. Most of their attacks we know about seem to be a lot more subtle and hard to detect. At first blush it seems equally likely that it's some random hacker who got lucky inside Juniper.

But it turns on an analysis of the malware and how subtle and unique it is. I imagine that's forthcoming from someone soon.
posted by Nelson at 4:27 PM on December 18, 2015


Juniper should be able to trace which employee checked in that code if they have proper source control.

Possibly not. Source control within a corporation doesn't really expect and isn't set up to prevent malicious actors inserting something covertly. The expectation of the system is that employees use their own credentials, while the expectation of employees when protecting their credentials is that they don't have to treat their fellow co-workers like hostile spooks. The nets are set up to catch mistakes and accidents rather than sabotage and criminals.

OTOH, intelligence people are notoriously incompetent, so who knows.
posted by anonymisc at 4:27 PM on December 18, 2015 [2 favorites]


OTOH, intelligence people are notoriously incompetent, so who knows.

The ones you hear about are. Sample bias perhaps?
posted by Dysk at 4:29 PM on December 18, 2015 [2 favorites]


The ones you hear about are. Sample bias perhaps?

I actually put more weight on testimony from those with sufficient security clearance to observe how more often than not "national security" is abused to shield dirty laundry from accountability. But you're correct that there is certainly never any shortage of culture-of-incompetence/politics/law-breaking/corruption scandals either.
posted by anonymisc at 4:38 PM on December 18, 2015


Juniper should be able to trace which employee checked in that code if they have proper source control.

I'm not familiar with everything out there, but none of the source control I have ever used has trustworthy proof of which user made a given change. The security layer has been the fact that I can connect to the shared repo, after that it trusts whatever claims I make about who I am without question. Maybe if you are lucky you have ssh access logs for the day the commit was pushed, so you can see whose key was used to push the commit?
posted by idiopath at 4:43 PM on December 18, 2015


Early reports are that the change was made more than three years ago. It's unlikely they still have things like SSH logs from that far back.
posted by Chocolate Pickle at 4:45 PM on December 18, 2015


Heh, I hope their IT people are taking notes, because sleuthing this out might end up being book material, "The Cuckoo's Egg"-style. :)
posted by anonymisc at 4:50 PM on December 18, 2015 [2 favorites]


You know, in a sane world the NSA would be helping American governments and companies to secure their networks.
posted by ob1quixote at 4:56 PM on December 18, 2015 [3 favorites]


Everybody seems to be sure it's the NSA. Surely they would have deeper and more subtle holes (like, say, silicon in CPUs which executes in privileged mode any code in a memory buffer that has a specific cryptographic checksum, allowing them to subtly pwn the router/server/phone on demand). I'd suspect it'd be more likely to be someone else; the prime suspects would be the Chinese PLA hacking battalion or the Russian FSB, with various frenemies as other possible suspects (how much, for example, do the Israelis trust the US? Or the French, for that matter?)
posted by acb at 5:13 PM on December 18, 2015


Friends in the sec biz made a few jokes about Juniper re-thinking off-shoring their dev work. A whole second network setup, buildings, etc to secure can be tough.
posted by k5.user at 5:21 PM on December 18, 2015 [1 favorite]


those with sufficient security clearance to observe how more often than not "national security" is abused to shield dirty laundry from accountability.

Exactly, what's top secret that every government and expert in that category does not already know? TS to keep 'them' from knowing that we know that they know that..... silly. The secrecy is to keep stuff hidden from 535 specific people who would laugh at an appropriation bill if they knew what it was actually for.

Now would a poor foreign government want credit card numbers and pins? (actually a on a slide of a security researcher I just saw at a talk last night, but the discussion didn't go down that path)
posted by sammyo at 5:45 PM on December 18, 2015


You can't sue the US government.

The Federal Tort Claims Act has something to say against that. The real issue is whether this was a deliberate policy decision (immune) or the result of somebody in a three-letter agency screwing up. (Not immune, generally.)
posted by fifthrider at 6:54 PM on December 18, 2015


Everybody seems to be sure it's the NSA. Surely they would have deeper and more subtle holes (like, say, silicon in CPUs which executes in privileged mode any code in a memory buffer that has a specific cryptographic checksum, allowing them to subtly pwn the router/server/phone on demand).

Maybe they have those too, but if they're caught they are burned forever, so they have others that are simpler and easier to deny if they are discovered?

Also backdoors in sofware can end up running on different silicon, so instead of pwning every piece of silicon, just pwn the software that runs on all the silicon.
posted by BungaDunga at 7:57 PM on December 18, 2015


What does the NSA care if it's made a hole for the rest of the government? They only care about increasing the NSA's reach...

and...

You know, in a sane world the NSA would be helping American governments and companies to secure their networks.

The NSA is problematic in a ton of ways (look at some of my commenting history to see me clearly articulate this), but this critique seems unwarranted. Part of their mandate is providing technical assistance to the US Govt on helping secure them. They care about the security of the US governments infrastructure.

They even make guides to help people harden/configure software/hardware/operating systems.

Now I feel dirty, please don't make me say nice things about them again.
posted by el io at 9:19 PM on December 18, 2015 [2 favorites]


The NSA is problematic in a ton of ways (look at some of my commenting history to see me clearly articulate this), but this critique seems unwarranted. Part of their mandate is providing technical assistance to the US Govt on helping secure them. They care about the security of the US governments infrastructure.

The problem is that the NSA has two mandates: breaking everyone else's security and promoting the security of US Government communications. These two goals are, to some extent, fundamentally incompatible.

We know that the NSA has discovered security vulnerabilities in commercial products, including products used by the government, and sat on them so they could exploit them themselves. We know the NSA has promoted weak and/or broken encryption. And we know the NSA spends hundreds of millions of dollars to get backdoors into encryption technology.

Clearly, NSA's collection and exploration arm is winning over its security mandate.
posted by zachlipton at 9:29 PM on December 18, 2015 [2 favorites]


The NSA is apparently turning into the IT security boogeyman. They're the new China.

So not all Juniper devices are affected by this. A while ago Juniper acquired NetScreen. Specifically, ScreenOS is made by NetScreen, and only ScreenOS is discovered to have a backdoor. ScreenOS is apparently used heavily for VPNs in the financial sector specifically (SWIFT money transfers). I'm not sure where else they're used.

There are two backdoors. One is apparently hard coded credentials in the sshd service (remote administration) or something similarly easy. This has been reversed out of the patches in private, and I expect will be public before the weekend is over.

The second backdoor has the potential to be much more interesting. The wording is somewhat vague, and it basically describes the ability to passively decrypt VPN traffic. There aren't a lot of ways to do this, that I can think of that would have avoided detection for close to four years. This potentially means it is Something Cool. This could be replacing some hard coded crypto constants with specific values, which would indicate pretty high sophistication, and potentially nation state level attacker. Keep in mind, this is all speculation though...

The same person is not necessarily responsible for both of these. They were discovered at the same time, and just due to proximity people are likely to associate them. However, if it does turn out to be a sophisticated attack against the crypto and a basic backdoor, there will be a pretty huge gap in attack complexity which would be hard to explain. Basically, if you understand crypto well enough to backdoor the VPN implementation, why not put a bug in the VPN implementation that gives you the backdoor as well? It would be much, much harder to discover at a fixed install cost.

What I would expect happened is someone discovered the hard coded creds, then they did an audit and discovered the VPN tampering. The most hilarious timeline is the NSA or similar got their fancy crypto backdoor discovered because some random shmuck independently hacked Juniper and put in a blatant backdoor.

Some notes on hardware backdoors because they came up earlier -- I'm not an expert on hardware at this level at all so take it with a grain of salt, but the problem with hardware backdoors is if they symmetrically authenticated those shared secrets can be reversed out of the silicon and used against you. This means your backdoor likely has to be asymmetric, and asymmetric operations are very hard to hide in silicon just due to the amount of gates they require and the relative smallness of the die.
posted by yeahwhatever at 10:26 PM on December 18, 2015 [11 favorites]


"huh, so the NSA doesn't just want to spy on citizens, they want to spy on the rest of the government."

"No wonder they don't want to share with the other children."
posted by snuffleupagus at 10:41 PM on December 18, 2015


The problem is that the NSA has two mandates: breaking everyone else's security and promoting the security of US Government communications. These two goals are, to some extent, fundamentally incompatible.

I certainly agree with this assessment. My comment was to remind folks that indeed they do care about defensive security (and particularly the US govt systems vulnerabilities).*

*On the other hand, I do remember the end of Sneakers.
posted by el io at 10:42 PM on December 18, 2015


The problem with the NSA's dual mandate is, anger at security breaches is generally diffuse and it's hard to prove the NSA wasn't doing its best. Also, closing security breaches is generally not the sort of thing you want to publicise. In contrast failure to provide backdoors to law enforcement is particular: it's easy for people to complain that they would have caught Osama McBadguy if it weren't for the white knights at the NSA, and the complainants have no problem going public. The consequence of this is what we see: vast resources thrown at breaking security; little or no assistance with improving it.
posted by Joe in Australia at 4:20 AM on December 19, 2015


i have been involved in chip design for years and i think it would be very difficult to put backdoors secret in hardware as described above. the codebase for a given chip is relatively small and only lives for a year or so as you are working on it. also making small changes to the code can disrupt things in such a way that the whole thing simply no longer works anymore without corresponding changes to other parts of the design. finally getting the design to close timing is such a bitch that almost all the (hard) logic is gone over with a fine-tooth comb and fixed and refactored to meet the required cycle time.

tl;dr i think it would be hard to sneak something in given the (relatively small) size of the codebase and the level at which it is scrutinized during the design process.
posted by joeblough at 10:35 AM on December 19, 2015 [6 favorites]


There is speculation that this backdoor may involve the definitely-intentionally-backdoored-by-NSA/NIST Dual EC random generator.
posted by RobotVoodooPower at 10:56 AM on December 19, 2015


That Dual EC backdoor the NSA created is a gift that's going to keep on giving. I just hope the secret skeleton key NSA holds never leaks to the bad guys. Assuming it hasn't already. (On edit, noting that my language betrays I've apparently accepted the NSA aren't "the bad guys".)
posted by Nelson at 11:35 AM on December 19, 2015




From Juniper's KB re: the second backdoor:

The following product families do utilize Dual_EC_DRBG, but do not use the pre-defined points cited by NIST:

ScreenOS*

* ScreenOS does make use of the Dual_EC_DRBG standard, but is designed to not use Dual_EC_DRBG as its primary random number generator. ScreenOS uses it in a way that should not be vulnerable to the possible issue that has been brought to light. Instead of using the NIST recommended curve points it uses self-generated basis points and then takes the output as an input to FIPS/ANSI X.9.31 PRNG, which is the random number generator used in ScreenOS cryptographic operations.


We don't know if the RNG is intentionally seeded with another known constant, but I can't think of any other reason to implement such a convoluted generator, especially after vulns have been discovered, unless shenanigans are involved. But the fact that this text appears in their knowledge base makes me wonder how it was "discovered" during a code review.
posted by RobotVoodooPower at 1:17 PM on December 19, 2015 [1 favorite]


Considering that the NSA flat-out paid RSA to set Dual_EC_DRBG as the default in their security software, I think the assumption has to be that similar defaults are the result of NSA malfeasance.
posted by Joe in Australia at 2:59 PM on December 19, 2015 [1 favorite]


Geez, thinking about using "enterprise" devices from big name vendors. RSA, Juniper firewalls, Cisco routers... It seems very tenuous to continue with these vendors. It's one thing to have normal security bugs, but to have supposedly big players shipping hardware that was intentionally compromised in ways unknown to the customers is just devastating.

OpenBSD had accusations of the insertions of backdoors from FBI contractors, but if they were ever created the consensus seems to be that they never made it into primary tree. So it seems that open source software does have some protections here.

So what's left? What are the chances that Microsoft's stuff is secure? Palo Alto Networks? Can we trust OpenVPN?
posted by Llama-Lime at 5:00 PM on December 19, 2015


I still worry about OpenSSL. It's gotten a lot of attention recently, which helps, but it's still a huge mess.
posted by Nelson at 7:29 PM on December 19, 2015


The FBI has said they're investigating this hack. I figure this takes a bit of the stench of guilt off the NSA. I know Russia has a lot of Juniper gear, but I think it's the JunOS branch, not the ScreenOS stuff. I'm guessing either Israel or China. (If it was a more recent hack, I'd assume China but 3 years ago it seems like a tie between them from what I've picked up by infosec osmosis)
posted by rmd1023 at 10:19 AM on December 20, 2015


I can't think of a better position to be in than in charge of "investigating" misdeeds that you perpetrated yourself.

I still worry about OpenSSL. It's gotten a lot of attention recently, which helps, but it's still a huge mess.

I take solace in the fact that at least one fork is getting the OpenBSD treatment of throwing out all the design-by-committee crap that doesn't make sense.
posted by indubitable at 1:25 PM on December 20, 2015 [1 favorite]


ssh backdoor password identified. Found by reverse engineering a patch Juniper released. The ssh part of the backdoor was a very simple strcmp() against the hardcoded string <<< %s(un='%s') = %u. I can't overemphasize how simplistic this backdoor is. It's not hidden at all, nor encrypted, nor obfuscated; the only slight nod to being undetectable is the hardcoded password looks like format string gibberish. It's a surprising thing to find because it's so clumsily done.

I wonder if this backdoor code is in Juniper's source, or if the build toolchain was hacked to insert it.
posted by Nelson at 7:04 PM on December 20, 2015 [2 favorites]


It could be clumsy, or it could be very cleverly obfuscated source code to avoid detection over the course of three years. We won't know without more info.
posted by Llama-Lime at 7:23 PM on December 20, 2015


Wired has a good writeup of this with what I think are new developments. There have been multiple failures (as always) and what I now recognise as a pattern:
1) Backdoor (deliberate or accidental) exists;
2) Company implements plan to close the backdoor;
3) It turns out that their "fix" was sidestepped (deliberately or accidentally);
4) Backdoor persists ...

I've started to presume that these companies have been so thoroughly rooted that they are literally incapable of closing NSA-implemented backdoors. The NSA must be holding something over them - either financial (RSA was paid $10,000,000 by the NSA) or legal. Like, there may be some sort of court order we don't know about, or the executives may have been blackmailed. Anyway, here's the Wired story:
Researchers Solve Juniper Backdoor Mystery; Signs Point to NSA
posted by Joe in Australia at 3:23 AM on December 22, 2015 [4 favorites]


Woah that Wired article by Kim Zetter is great. It's mostly a civilian-friendly description of this research by Ralf-Philipp Weinmann. There's two shocking claims it makes.

1) Juniper's patch is not complete, their OS is still insecure.
2) The weakness comes from them choosing Dual EC, then either screwing up or including back doors. They claimed to secure it by using their own Q constant and mixing it with another RNG. But the mixing didn't work in a way that looks like a deliberate backdoor leaking Dual EC state. And someone, somehow, replaced the Q.

This case is the first I've heard about where the whole design of Dual EC has made American products less secure. Previously we thought the damage of NSA's malfeasance with Dual EC was limited, because only NSA knew how to use the backdoor. But the algorithm itself is compromised; any proprietary choice of the Q constant has a risk of there being a backdoor. That weakness was exploited. NSA has actively harmed American security products in a way that potentially anyone could have exploited.

So far I've read nothing connecting this Dual EC weakness to the SSH backdoor that was also found. This Dual EC attack was quite subtle and elegant and smells like NSA or another state actor. I still think that SSH backdoor is clumsy and oafish and may be a different penetration.

In a better world, no one would ever buy a Juniper product again. But most engineers making equipment decisions just assume all vendor equipment is equally insecure. Also RSA sets a terrible precedent; they literally sold out their customers to the NSA for $10M, and yet that has hardly harmed their business.
posted by Nelson at 8:08 AM on December 22, 2015 [4 favorites]


This case is the first I've heard about where the whole design of Dual EC has made American products less secure. Previously we thought the damage of NSA's malfeasance with Dual EC was limited, because only NSA knew how to use the backdoor. But the algorithm itself is compromised; any proprietary choice of the Q constant has a risk of there being a backdoor.

That's a really good point. The NSA's ability to read any particular person's mail is really beside the point; they wanted to read everybody's mail cheaply and consequently lots of people are using an algorithm that is intrinsically unsafe. This is a bit weird, because in other areas (intellectual property rights, for example) the USA has been on the side of big business against consumers; this is a case where it's against everybody but itself.
posted by Joe in Australia at 7:38 PM on December 22, 2015


NSA helped British spies find security holes in Juniper firewalls. A newly unearthed penetration by GCHQ, dating to 2011. It's not clear whether this attack on Juniper is related to these new security holes in Juniper products or not.
posted by Nelson at 3:34 PM on December 23, 2015


New Questions Swirl About Security Failure at Tech Giant Juniper Networks

Summary: Juniper made multiple changes to its encryption that had a cumulative weakening effect, after DUAL_EC was known to be unsafe. And it won't say why. Stinks like rotten fish, IMO.
posted by Joe in Australia at 1:33 AM on January 10, 2016


Has Juniper come clean about how the ssh backdoor got inserted? That's a separate attack from the crypto and much more clumsily done. I have a tiny bit of sympathy for a company being coerced by the NSA to insert tricksy backdoors like DUAL_EC into their products; it's safest to assume all American products are similarly compromised. But some other attacker set it up so all these Juniper products let anyone just log in as administrator with a known, fixed password. That's negligence.
posted by Nelson at 6:57 AM on January 10, 2016


To the extent they've said anything at all, they said they don't know. But I suppose they'd have said that anyway. I think the choice of fixed password implies that it was a malicious insert; a debugging password would be something in plain text, but this was a clever-ish one and a superficial observer (e.g. me) would have just passed it by.
posted by Joe in Australia at 6:11 PM on January 10, 2016 [1 favorite]


Fortinet also had an SSH backdoor that they used for management authentication. They even had the temerity to say:
After careful analysis and investigation, we were able to verify this issue was not due to any malicious activity by any party, internal or external.
Which tells me that I will do everything in my power from ever letting Fortinet onto networks. That they don't see a discoverable, hard-coded and unknown SSH password as a backdoor speaks volumes about them.

I sure hope nothing has been snuck into OpenVPN or pfsense like this.
posted by Llama-Lime at 12:01 PM on January 13, 2016


« Older Rossi Ruiz   |   It's All Relative Newer »


This thread has been archived and is closed to new comments