Spy chips found on server motherboards
October 4, 2018 5:46 AM   Subscribe

Bloomberg reports a major supply-chain hack. After Apple, Amazon and others started to see unusual behaviour in servers from SuperMicro, investigations found chips on the motherboards that shouldn't have been there. Disguised as other components, the chips reportedly intercepted and modified low-level code, creating back doors for remote exploitation. But who put them there?
posted by Devonian (103 comments total) 44 users marked this as a favorite
 
I bet the super rich have gone all analog. Like instead of a pocket computer, there’s an exquisitely trained eunuch who just follows them around giving them answers to questions and remembering their schedule. And no more email for them. Back to coded messages, wax seals, and a little bowl where you can watch them burn to ashes.

The rest of us have been pwned by Russia or China many times over.
posted by schadenfrau at 5:53 AM on October 4, 2018 [23 favorites]


Also this sounds extremely serious, no?

Are the big 5 just going to move to doing everything in house? Like this will be their argument against breaking up their monopolies?
posted by schadenfrau at 5:54 AM on October 4, 2018 [1 favorite]




schadenfrau, I believe you're looking for a Mentat.
posted by gwint at 6:08 AM on October 4, 2018 [29 favorites]


thatsthejoke.gif
posted by thelonius at 6:23 AM on October 4, 2018 [8 favorites]


It's kind of hard to believe that the military did not catch this with their own due diligence. Don't they use custom fabricated/hardened boards anyway?
posted by carter at 6:26 AM on October 4, 2018 [1 favorite]


Are the big 5 just going to move to doing everything in house? Like this will be their argument against breaking up their monopolies?

You have to move everything in house, though, including design tools, chip foundries and so on, and you can't afford to do that and stay competitive. This sort of thing has been going on for a very long time - the Soviets interdicted and put spy circuitry in Selectric typewriters in the 1970s - but the weak link is always the need to get the intercepted data out, which has to involve anomalous network activity. Apple spotted this in their Siri data centres, which one imagines have quite tightly characterised traffic in normal operation.

I'd build my secure system out of recycled Z80s from Sinclair ZX Spectrums. Same thinking as using pre-war steel for low-radiation equipment.
posted by Devonian at 6:27 AM on October 4, 2018 [20 favorites]


This is how governments will break (already have broken?) our encryption. Have whatever unbreakable method you want; the unremoveable chip is listening inside the realm where you've unencrypted your data. Firewall? No problem, it's just another motherboard, there's a cooperating chip there.

Puts a whole new spin on (in)security through obscurity.
posted by zaixfeep at 6:31 AM on October 4, 2018 [7 favorites]


There was a proposal to use an old Apple ][ as part of a nuclear weapon verification system, since you can visually inspect all of the chips involved, and they're well-documented.

This is why I get twitchy when people complain about the antique 8" floppy disks we still use in nuclear silos, suggesting we replace them with SD cards or something. Noooooooo!
posted by RobotVoodooPower at 6:32 AM on October 4, 2018 [34 favorites]


the weak link is always the need to get the intercepted data out, which has to involve anomalous network activity

If you design your network topology well, tune your alert messages, and train your operators properly, this stuff really stands out. Of course, it's hard to tune alert messages so the useful ones stand out, and operators get a bit of alert blindness and easily adjust to a "new normal" that could, in fact, be hiding anomalous activity, so this is a hard problem. But to catch anomalous activity at all you have to know what your outbound network traffic is supposed to be doing at all times.
posted by fedward at 6:37 AM on October 4, 2018 [8 favorites]


I remember a similar story hitting the news a few years ago, about Chinese chips in "Made in USA" circuit boards powering our military equipment.
posted by rebent at 6:45 AM on October 4, 2018 [1 favorite]


Basic security for digital files involves a trusted source (i.e., the organization offering the file) offering the MD5 hash of the file publicly, allowing anyone to generate the same hash on the file after downloading it and comparing the two. This prevents altering the file in transit or a malicious actor substituting their own binary at the source. Is anything comparable even possible for something as complex as a motherboard? One can imagine at the the chip level that a deterministic series of operations could generate a signature of some kind--that could even be specified during chip design. But for a whole system? Eesh.

Also, Bloomberg is getting sassy:
Elemental [video compression] servers sold for as much as $100,000 each, at profit margins of as high as 70 percent, according to a former adviser to the company. Two of Elemental’s biggest early clients were the Mormon church, which used the technology to beam sermons to congregations around the world, and the adult film industry, which did not.
Emphasis mine.
posted by fatbird at 6:50 AM on October 4, 2018 [36 favorites]


I don't think you'd use MD5. It's not that difficult to generate collisions with it.
posted by edd at 6:56 AM on October 4, 2018 [2 favorites]


Apple, Amazon deny this has happened.
posted by doctornemo at 6:57 AM on October 4, 2018 [6 favorites]


One HackerNews thread. Another.
posted by doctornemo at 7:01 AM on October 4, 2018


This is exactly why I keep telling people we should not network the Battlestars.
posted by Slarty Bartfast at 7:03 AM on October 4, 2018 [46 favorites]


> Apple, Amazon deny this has happened

Is that a case of them having to deny it, for legal reasons? i.e. either due to on-going investigations, or some convoluted "mustn't rock the boat lest our precious shareholders lose value!!~" thing?
posted by slater at 7:06 AM on October 4, 2018 [3 favorites]


I'm just going to say that some of the claims made in the article about what this hardware was purportedly able to do are absolutely breathtaking from a technical perspective (to the point where I'm somewhat incredulous that you can pull off this kind of widespread, sustained high-level exploit with a tiny microcontroller).
posted by schmod at 7:07 AM on October 4, 2018 [7 favorites]


Is that a case of them having to deny it, for legal reasons? i.e. either due to on-going investigations, or some convoluted "mustn't rock the boat lest our precious shareholders lose value!!~" thing?

I imagine they’d be exposed to a shit ton of liability for lying about whether or not it happened
posted by schadenfrau at 7:10 AM on October 4, 2018 [5 favorites]


But to catch anomalous activity at all you have to know what your outbound network traffic is supposed to be doing at all times.
Also, your networking monitoring system itself needs to be free from compromise — hope you didn’t use your standard server build… — and you need some pretty sophisticated tools to segregate traffic and inspect things to make sure that, say, data isn’t being exfiltrated in your mail server’s error responses to that spambot.
posted by adamsc at 7:17 AM on October 4, 2018


"I imagine they’d be exposed to a shit ton of liability for lying about whether or not it happened"

What does that even mean for megacompanies like these? If anything, there might be a small penalty fee they can shrug off or subsidize.

What strikes me is how small the chip is, I didn't know they went that size! Inspecting the board physically it's just a random speck, could be anything.
posted by GoblinHoney at 7:25 AM on October 4, 2018 [3 favorites]


Amazon’s security team conducted its own investigation into AWS’s Beijing facilities and found altered motherboards there as well, including more sophisticated designs than they’d previously encountered. In one case, the malicious chips were thin enough that they’d been embedded between the layers of fiberglass onto which the other components were attached, according to one person who saw pictures of the chips.

This is a thing.
posted by flabdablet at 7:28 AM on October 4, 2018 [25 favorites]


Interesting possibility that the Butlerian Jihad wasn't intelligence vs artificial intelligence, but intelligence vs digital devices being used against us by intelligence.
posted by Mr.Encyclopedia at 7:40 AM on October 4, 2018 [4 favorites]


I'm just going to say that some of the claims made in the article about what this hardware was purportedly able to do are absolutely breathtaking from a technical perspective (to the point where I'm somewhat incredulous that you can pull off this kind of widespread, sustained high-level exploit with a tiny microcontroller).

The stuff about modifying the OS from the system instruction outward seems hard but perhaps not impossible, given what we now know about structured attacks on speculative execution. "Linux" is no one thing, though, and there are other kernels or microkernels that could be in use, so even if you could put such an attack into a logic board you're depending a lot on the chance that the OS can be modified in the way you expect. You're not just looking for a particular instruction or set of instructions you can modify, you're relying on other features of the OS being somewhere you can get at them, and you also need the OS not to be self-validating.

In theory a sophisticated enough attack on the host OS could start from a tiny microcontroller as long as it had network access to load the rest of the exploit. In that case all you really need is something to inject a pretty small payload, detect the OS, and fetch the exploit for that particular OS before its startup completes. Pair it with an EFI vulnerability, then like an iPhone jailbreak, you just have to defeat the kernel validation to execute your OS-level payload. If you've compromised board firmware you may be able to execute that jailbreak at every startup in such a way the OS on disk never needs to be overwritten, making the whole thing much harder to detect.
posted by fedward at 7:55 AM on October 4, 2018 [7 favorites]



Is that a case of them having to deny it, for legal reasons?


Having to deny an intrusion onto systems holding credit card data? I'm having trouble coming up with a scenario that would have that happen. Having to decline to comment is one thing. Deny? That's a crime.
posted by ocschwar at 8:06 AM on October 4, 2018 [1 favorite]


You're not just looking for a particular instruction or set of instructions you can modify, you're relying on other features of the OS being somewhere you can get at them, and you also need the OS not to be self-validating.

If your attack is based on injecting a System Management Mode rootkit during cold boot, you don't need the OS to fail to validate its own code because you're never actually touching the OS code; SMM rootkits can do stuff like rewriting user IDs in process descriptor tables in order to confer root privileges on any process with no changes to kernel code whatsoever.
posted by flabdablet at 8:07 AM on October 4, 2018 [5 favorites]


I bet the super rich have gone all analog.

I can assure you they have not; mostly they use iPhones, or in the cases of the old technophobes, their assistants use iPhones. Here's Suzy Welch (wife of Jack Welch) gushing about her new iPhone X. Here's a whole article about what phones billionaire CEOs use. Spoiler: mostly iPhones, with a few people who are clearly not friends of Apple opting for Samsungs or other high-end Android handsets. A few privacy-conscious celebrities apparently use burner flipphones, but I kinda doubt that's their only phone or that they don't have an assistant covering their email with a modern smartphone.

You can start a hell of an argument among security people about the iPhone vs. Android thing; Samsung handsets are theoretically a cleaner supply chain, because Samsung is vertically-integrated and does most of its manufacturing in Korea, which might be a bit harder for the PLA to infiltrate than mainland PRC or Taiwan, although I still have my doubts, personally; the iPhone seems to have a better security posture in terms of its software and S/H integration, including how it uses encryption and the Secure Enclave. How they keep their Chinese and China-adjacent supply chain clean is a huge question, though—if someone slid a few extra transistors into the Secure Enclave (which is on the processor die, I think, and made by TSMC in Taiwan), you'd be fucked. Presumably they do a lot of random inspections, but it's a very hard problem.

Until you get up to military stuff used on classified networks—really eye-wateringly expensive gear—there's not really much of an alternative to the consumer products. Essentially, the market hasn't produced any middle ground. There's consumer gear, and then there's stuff produced in secure facilities and vetted by the NSA, which you can't buy as random Joe Citizen anyway. Most of the "secure" products marketed by other companies are just software loaded onto consumer handsets, which addresses software vulnerabilities but not hardware issues. They help against script kiddies and maybe corporate espionage but not nation state attacks, particularly if you are having your hardware made in the country you're trying to protect yourself from.

Why we thought it would be possible to have all our hardware made in China without the Chinese compromising it is beyond me.
posted by Kadin2048 at 8:27 AM on October 4, 2018 [33 favorites]


Bloomberg is getting sassy

That line popped out at me too! Quite pleasing writing overall - I may actually start dipping my toes into BW again occasionally.
posted by aspersioncast at 8:28 AM on October 4, 2018


If your attack is based on injecting a System Management Mode rootkit during cold boot, you don't need the OS to fail to validate its own code because you're never actually touching the OS code

With access to the hardware there are multiple avenues of attack. I suspect that the sort of attack you describe is a more likely sort of attack, but the Bloomberg article says it's modifying the OS. I deleted my speculation that the author might have misunderstood and/or conflated, because, well, it was speculation. That limited me to addressing the problem as stated (tiny chip -> [SOMETHING] -> modified OS), which seems difficult but not impossible. Modifying the OS seems the most difficult to pull off from such a tiny controller alone, but it's certainly possible to do that more robustly if the tiny controller is but one piece of the overall exploit.
posted by fedward at 8:32 AM on October 4, 2018


Why we thought it would be possible to have all our hardware made in China without the Chinese compromising it is beyond me.

MmmHmm. Texas Instruments should really consider making a smart phone.
posted by aspersioncast at 8:32 AM on October 4, 2018 [8 favorites]


Something about this denial doesn't feel right.

If Amazon and Apple are vehemently denying this story, what was Bloomberg's source?

Who leaked this story? How did Bloomberg corroborate it? Did they compromise an FBI investigation by publishing it?

[Puts on tinfoil hat] -- Did somebody within the Trump administration leak this in an attempt to escalate the trade war?
posted by schmod at 8:40 AM on October 4, 2018 [12 favorites]


But to catch anomalous activity at all you have to know what your outbound network traffic is supposed to be doing at all times.

No. It's not simple. That might catch some of the command-and-control (C2) traffic, but not if it's suitably well-disguised. And the number of ways it might be disguised, when you are talking about a datacenter's worth of traffic, hosting VMs for other people especially, is staggering. They didn't catch the traffic, and I don't think it's because Amazon wasn't looking for weird traffic; they didn't know what to look for until the chips were discovered and then they found the C2 messages.

Also, there are other attacks that wouldn't require a C2 channel. Keep in mind that a nation state actor doesn't need to exfiltrate unencrypted data like a run-of-the-mill hacker group stealing credit card numbers; they have their own resources to throw at the data, and abilities to intercept it outside the datacenter. The hardware is just one prong in the overall attack. It might be sufficient to just weaken the encryption in some way, to make a particular attack feasible. E.g. by doing something to the system's random number generator, whether to affect the network encryption directly, or maybe just to be able to predict memory addresses better for some other sidechannel attack. And that's just the tip of the iceberg, really.

There's fundamentally no good way to do trusted computing on untrusted devices. There's decades of failed attempts; it's time to stop trying. Security has to be built up from the silicon, you can't build it down from the software.
posted by Kadin2048 at 8:42 AM on October 4, 2018 [13 favorites]


I'm just going to say that some of the claims made in the article about what this hardware was purportedly able to do are absolutely breathtaking from a technical perspective (to the point where I'm somewhat incredulous that you can pull off this kind of widespread, sustained high-level exploit with a tiny microcontroller).

SuperMicro motherboards have extensive built in system management capability with remote access. I assume they were subverting the access control to that system rather than putting all that capability on the covert chip.
posted by atrazine at 8:59 AM on October 4, 2018 [4 favorites]


What does that even mean for megacompanies like these? If anything, there might be a small penalty fee they can shrug off or subsidize.

No, it'd be a king-hell shareholders' class action and a field day for the plaintiffs' bar, with punitive damages numbering in the eight figures entirely plausible. If anything like this did transpire — which, frankly, I'm still quite skeptical about — the targets'd have to be the fuckingest stupid people you've ever met to lie about it.
posted by adamgreenfield at 9:02 AM on October 4, 2018 [2 favorites]


There are no "trusted" computing devices. There is no perfect security, and there never will be. We need to stop telling ourselves this comforting lie.

In the 1940s, the Russians built a piece of artwork that functioned as a listening device, even though it contained no active electronics. There's always going to be clever shit like this, and there's nothing we can do about it.

Side-channel attacks will always be a vulnerability (and they're somehow getting easier to exploit). Designs in hardware and software can always have unintended side-effects, and it's impossible to tell whether or a particular design is secure, malicious, benign, or inert.
posted by schmod at 9:04 AM on October 4, 2018 [9 favorites]


I bet the super rich have gone all analog. Like instead of a pocket computer, there’s an exquisitely trained eunuch who just follows them around giving them answers to questions and remembering their schedule. And no more email for them. Back to coded messages, wax seals, and a little bowl where you can watch them burn to ashes.

They just use other human beings as their input devices. So the dark web will still get their precious risotto recipes.
posted by srboisvert at 9:06 AM on October 4, 2018 [1 favorite]


Imagine the botnet you could create with these chips installed in consumer electronics. Your own private 0-day, no way to stop it. It’s a nuke pointed right at the internet, one that can’t be routed around.
posted by bigbigdog at 9:23 AM on October 4, 2018 [2 favorites]


I have a friend who recently retired from the chip design business in the US. He spoke of working in the "Spook Fab" for a while, where chips were designed and created for the CIA, NSA, and others. They were very much aware of the risks associated with having others design and build their chips, as malicious content could be hidden deep within the chips.

Of course, they were also quite aware of the benefits of inserting a bit of something into the chips designed for others.
posted by Midnight Skulker at 9:28 AM on October 4, 2018 [3 favorites]


Wow... This has been the biggest year for fundamental security issues - bad silicon, you cannot get worse than that.

Don't forget about the hundreds of millions of "IoT" devices like digital security cameras that are also coming from countries known to spy.
posted by jkaczor at 9:34 AM on October 4, 2018 [3 favorites]


the targets'd have to be the fuckingest stupid people you've ever met to lie about it.

Exactly. There is no upside for them to blatantly lie like this. A simple, "we don't discuss possible investigations". This is the business equivalent of calling them a fucking liar. You don't do that without cause.
posted by zabuni at 9:47 AM on October 4, 2018 [1 favorite]


TPM: White House Begins the China Counter-Narrative

I'm sure the timing of this is all one huge coincidence.
posted by schmod at 10:03 AM on October 4, 2018 [12 favorites]


And the number of ways it might be disguised, when you are talking about a datacenter's worth of traffic, hosting VMs for other people especially, is staggering.

I didn't want to dig too much into that aspect of my comment but I've worked in a couple environments where all network egress was monitored. At the level of a whole datacenter, you're correct that this traffic isn't likely to be noticed. But at one old (datacenter) job in particular we had lots of controlled access where only the expected traffic was even allowed by the outbound link to the next concentric ring of security. For the servers within the tightest ring we could afford to do stateful inspection of the packets that weren't dropped by the egress filters. For the highest security devices terminal access was air gapped; for the next ring there was an out of band IPKVM on a completely separate network that had no upstream connection.

I'm not saying it would be impossible to exfiltrate data from the most tightly controlled security layer of such a network, but I'm saying it would be so hard to do without having inside knowledge of the network architecture and perhaps a way of bridging a network air gap.

SuperMicro motherboards have extensive built in system management capability with remote access. I assume they were subverting the access control to that system rather than putting all that capability on the covert chip.

I'd be curious if the lights out management went over the same physical interface as the public network, or if it's completely separated down to the traces on the logic board. It's been a while since I was hands on in a datacenter, but all our LOM stuff was patched into a private network. I wouldn't ever put LOM on a publicly accessible network, but I guess if there's a hardware way to subvert that isolation that could be effective.
posted by fedward at 10:13 AM on October 4, 2018 [5 favorites]


> > Why we thought it would be possible to have all our hardware made in China without the Chinese compromising it is beyond me.
> MmmHmm. Texas Instruments should really consider making a smart phone.


but then the official NSA Backdoor® will be installed. There's no escape. The manufacturing processes are too expensive. High-tech products will always be built at a few large, centralized shops that are easy to compromise.
posted by scose at 10:16 AM on October 4, 2018 [1 favorite]


> ...it'd be a king-hell shareholders' class action and a field day for the plaintiffs' bar, with punitive damages numbering in the eight figures entirely plausible.

Both Amazon and Apple are trillion-dollar companies, and eight figures would be easy to write off. So I agree with you but I think the net costs and penalties would be a hell of a lot more.

Amazon and Apple are both publicly held companies and, in the US at least, false statements can be treated as attempting market fraud. When you see a well-written PR rebuttal (and all corporate rebuttals like this go through legal and PR staff), you usually have to work to find the potential easy outs, something that could be followed up with, "We only said we weren't contacted by Bob, not Jim" or "We didn't say anything about green logic boards". You can search yourself for the easy outs in the official rebuttals.

Apple's is broad and doesn't leave much wiggle room: "Apple has never found malicious chips, 'hardware manipulations' or vulnerabilities purposely planted in any server. Apple never had any contact with the FBI or any other agency about such an incident. We are not aware of any investigation by the FBI, nor are our contacts in law enforcement."

Amazon's leaves openings, for example 'yeah but we didn't say the NSA hadn't contacted us': "It’s untrue that AWS knew about a supply chain compromise, an issue with malicious chips, or hardware modifications when acquiring Elemental. It’s also untrue that AWS knew about servers containing malicious chips or modifications in data centers based in China, or that AWS worked with the FBI to investigate or provide data about malicious hardware."
posted by at by at 10:23 AM on October 4, 2018 [4 favorites]


The article is great and pretty detailed about how this took place and how the investigation proceeded, and I don't doubt that this sort of thing (or a lesser or software variation) is being done by or attempted by various nations including ours. But ... given the current administration in the US I'm inclined to take this with a grain of yellowcake salt.
posted by freecellwizard at 10:37 AM on October 4, 2018 [2 favorites]


What I want to know is who Supermicro pissed off and why.
posted by nikaspark at 10:46 AM on October 4, 2018 [2 favorites]


This story is too tidy. It tickles all of our “just so” fancies of how hacking and interdiction works. I’m not buying it just yet.
posted by nikaspark at 10:47 AM on October 4, 2018 [8 favorites]


Indeed. Injecting fake news into the mass media is a lot cheaper and easier than injecting super duper spy chips into fibreglass motherboards and, given the predictable effect of a story like this on Supermicro's share price, possibly more lucrative as well.
posted by flabdablet at 10:55 AM on October 4, 2018 [2 favorites]


I suspect that the sort of attack you describe is a more likely sort of attack, but the Bloomberg article says it's modifying the OS.

It does indeed say that. It also says this:
The illicit chips could do all this because they were connected to the baseboard management controller, a kind of superchip that administrators use to remotely log in to problematic servers, giving them access to the most sensitive code even on machines that have crashed or are turned off.
To the best of my knowledge the BMC is able to interact with the CPU, if the latter is powered up, by generating interrupts that cause the CPU to run chunks of code in system management mode.

I don't know enough about the way Supermicro implements IPMI to say whether or not their BMCs are actually capable of injecting arbitrary SMM firmware into SMM RAM before CPU startup, but given how awful I know BIOS firmware to be in general, it would absolutely not surprise me to find that they could.
posted by flabdablet at 11:17 AM on October 4, 2018 [3 favorites]


Kadin2048: That might catch some of the command-and-control (C2) traffic, but not if it's suitably well-disguised. And the number of ways it might be disguised...is staggering

On this week's Risky Business podcast there is a bit how the Russians just ginned up a compatible client for the LoJack anti-theft software but swapped in the IP addresses of their own C&C servers. Because it's a well-known commercial product , it was white-listed by every AV vendor and firewall. Because it is totally unencrypted, they could see every detail required to build their own client. Because it delivers a firmware-resident, reboot-proof UEFI client, it was a perfect perch from which to attack target systems.

Here's the source article they were discussing: https://arstechnica.com/information-technology/2018/10/first-uefi-malware-discovered-in-wild-is-laptop-security-software-hijacked-by-russians/

Of course, because LoJack is so seldom used, the traffic kind of stood out for being unusual.
posted by wenestvedt at 11:25 AM on October 4, 2018 [3 favorites]


but then the official NSA Backdoor® will be installed.
[Adds additional tinfoil] -- What if this is the NSA backdoor?
posted by schmod at 11:35 AM on October 4, 2018


Then I'm having my monocle reground.
posted by clavdivs at 11:43 AM on October 4, 2018 [7 favorites]


That SMM injection proof of concept is wild.
posted by fedward at 11:53 AM on October 4, 2018


John Gruber of Apple- & Kubrick-phile blog "Daring Fireball" is quite blunt in a piece today:
Someone is either wrong or lying. This cannot all be true.
posted by wenestvedt at 12:27 PM on October 4, 2018 [15 favorites]


Two of Elemental’s biggest early clients were the Mormon church, which used the technology to beam sermons to congregations around the world, and the adult film industry, which did not.

I’m gonna guess it was the Mormons.
posted by bendy at 1:18 PM on October 4, 2018


flabdablet said "This is a thing."


yes but that is a very expensive thing, you don't just sneak a single embedded chip into a manufacturing process, it requires far to many extra expensive steps ... now if SuperMicro is already using embedded components then yes, you can sneak an extra component in (besides part of the process of making these boards involves xraying each and every one of them)
posted by mbo at 1:20 PM on October 4, 2018 [1 favorite]


Someone is either wrong or lying. This cannot all be true.

At a first guess, I would say Bloomberg got important details wrong, probably at the instigation of some of their sources. Because the alternative is Amazon deciding to exit the cloud services market as soon as the truth comes out. Nobody can stay at Amazon Web Services if they lied about this.
posted by ocschwar at 1:38 PM on October 4, 2018 [1 favorite]


Dang it fatbird.
posted by bendy at 1:40 PM on October 4, 2018


I'm not sure I buy all the tech side of the story. In particular, instruction and address buses in CPUs are massively parallel and you don't get to mess with that from a 'signal conditioning' chip which has at most four pins on it (I went looking online, because I'm not familiar with those being used in digital circuits, and the little ones have four pins max). And the article implies that off-chip CPU caches are still a thing, which i do not think they are. I could be wrong on both counts, but I do say that that story wasn't edited for technical accuracy by someone who knows servers down to chip level - the phrasing is too odd.
posted by Devonian at 1:42 PM on October 4, 2018 [2 favorites]


I'm going to ask the grossly naive question - is this the kind of thing that FPGAs would solve (if they were viable for modern computing requirements up to scale, which they're not) or is my hardware ignorance so profound that I'm on a completely different planet?
posted by eclectist at 1:51 PM on October 4, 2018


From TFA:
The companies’ denials are countered by six current and former senior national security officials, who—in conversations that began during the Obama administration and continued under the Trump administration—detailed the discovery of the chips and the government’s investigation. One of those officials and two people inside AWS provided extensive information on how the attack played out at Elemental and Amazon; the official and one of the insiders also described Amazon’s cooperation with the government investigation. In addition to the three Apple insiders, four of the six U.S. officials confirmed that Apple was a victim. In all, 17 people confirmed the manipulation of Supermicro’s hardware and other elements of the attacks. The sources were granted anonymity because of the sensitive, and in some cases classified, nature of the information.
The timing of the release might have been chosen by someone for its political value, but it sounds like this story has been cooking for a while, and Bloomberg sat on it. Maybe they ran it because they were afraid they were going to get scooped, or maybe they ran it because they were given the greenlight by their sources.

Bloomberg may not be everyone's favorite organization or the most feel-good news outlet, but they are taken pretty seriously; their business is basically built on having solid data.

I am surprised that people are so surprised about it. Lots of people have been waiting for something like this to show up for a while. It was going to happen eventually, the only question was when, and the particular form (on-die vs. on-board vs. microcode).

I'm going to guess that while there might be some minor clarifications—who-knew-what-when sort of stuff, and the allegation that Apple covered it up seems like the most likely parts to get "clarified" later—the core of the story is true.

I wouldn't put it past the current Administration to go public and burn an ongoing FBI counterintel investigation for political points, though, especially if it helps both the China counternarrative and distracts attention from the Kavanaugh debacle. Poking the FBI in the eye would be a bonus.

Wonder if this is part of why Mattis isn't going to China? Would have made the meetings a bit awkward.
posted by Kadin2048 at 1:54 PM on October 4, 2018 [2 favorites]



I'm going to ask the grossly naive question - is this the kind of thing that FPGAs would solve (if they were viable for modern computing requirements up to scale, which they're not


Depends on what you mean by solve.

I could see implants like this being disabled after detection by an update to the FPGA on that board. (In some cases, not all). But I could also see an FPGA update causing a security problem just as easily. WHen you make the hardware more malleable, you make it more malleable by all parties, good and bad.
posted by ocschwar at 2:00 PM on October 4, 2018


Nobody can stay at Amazon Web Services if they lied about this.

In other news, Amazon is in the midst of a very ugly fight with other major cloud providers (MSFT, IBM, etc.) over USG use of AWS as its preferred cloud platform. DoD wants AWS and only AWS, the other vendors obviously think they should get a piece of the action.

They would stand to gain a lot if it looks like Amazon can't keep its supply chain clean, and I'd imagine this will be a talking point going forward. Not that I suspect any of the other major players are any better, and are likely to be worse—Amazon can, due to its size, have all its own hardware manufactured if it wanted to. The smaller cloud providers might have a harder time doing that, and my guess is their farms are full of gear made by third parties. It'd be beyond stones-in-glass-houses if they helped leak the story (not impossible; there's a lot of revolving-door staffing in the space), but who knows.
posted by Kadin2048 at 2:02 PM on October 4, 2018


Google HAS had hardware custom made for their data centers. And with RISC V and related efforts, it's getting easier for smaller players to do it too.
posted by ocschwar at 2:17 PM on October 4, 2018


One way that comes to mind for a 4-terminal device to thoroughly subvert your whole computer is if it sits on the data line of an SPI bus. While I don't know the state of the art on Supermicro motherboards, many ThinkPad laptops have SPI BIOSes.

This hypothetical device's connections would be VCC, GND, and a pair of pins that interrupt the signal called "MISO", carrying data from the SPI EEPROM to the embedded microcontroller.

In normal operation, the hypothetical device internally connects the two "MISO" pins. But when activated, it interrupts the connection and substitutes its own malicious signal. This signal is reassembled into instructions and executed by the target microcontroller, leading to the next level of compromise.

The "activation" could be any number of things. It could be something external, such as based on power cycling, RF/magnetic field, or even based on analyzing the "MISO" data coming from the legitimate EEPROM device.

If you want to put your device on a parallel bus like main system RAM, you can still do interesting things. For example on x86 there's just one bit of difference between "Jump if zero" and "Jump if not zero"; Now, you just have to place your single-bit-subverting device on the right bit of the data bus, recognize a target instruction sequence by that single bit, and flip that bit at the right moment. Yes, this is totally hand-wavy and is greatly complicated by CPU caches, and will also take a big bite out of the margin for RAM signals which is always tight.

If this is real, my bet is on subverting a serial bus to an EEPROM, not one bit of a whole parallel bus.
posted by the antecedent of that pronoun at 3:02 PM on October 4, 2018 [2 favorites]


Ars Technica's take: Bloomberg: Super Micro motherboards used by Apple, Amazon contained Chinese spy chips -- subtitle: Super Micro, Amazon, and Apple deny everything in the report.

The article is framed as "Bloomberg claims" and "the report alleges," and includes this paragraph:
Super Micro, Apple, and Amazon all deny every part of the Bloomberg story (Bloomberg: The Big Hack: Statements From Amazon, Apple, Supermicro, and the Chinese Government). Amazon says that it's untrue that "[Amazon Web Services] worked with the FBI to investigate or provide data about malicious hardware;" Apple writes that it is "not aware of any investigation by the FBI," and Super Micro similarly is "not aware of any investigation regarding this topic." Apple suggests further that Bloomberg may be misunderstanding the 2016 incident in which a Super Micro server with malware-infected firmware was found in Apple's design lab.

Apple's denial in particular is unusually verbose, addressing several different parts of the Bloomberg report explicitly, and is a far cry from the kind of vague denial that one might expect if the company were subject to a government gag order preventing it from speaking freely about the alleged hack.
Apple's response opens with this paragraph:
The October 8, 2018 issue of Bloomberg Businessweek incorrectly reports that Apple found “malicious chips” in servers on its network in 2015. As Apple has repeatedly explained to Bloomberg reporters and editors over the past 12 months, there is no truth to these claims.
Meanwhile, the auto-playing Bloomberg audio in "The Big Hack" opens with one television personality saying "Jordan, great scoop, congratulations on it," and then I closed that tab.
posted by filthy light thief at 3:11 PM on October 4, 2018 [1 favorite]


Here's my apocryphal story that I heard once in the mid 90's:

During the first Gulf War on a particular night the Baghdad air defense system was down. Because they used a system bought from the French (or something, not a dig at the French just Countries buying military equipment from other Countries). The thing is that a short-ish time before, they had ordered a new printer. Either the US just knew this and had an exploit, or they managed to do something to that printer. This was a SCSI printer. I can come up with dozens of ways this is plausible.

I just wonder how to trigger it.

Anyways, that's pretty much how I think about bad electronics. It just needs to see a sequence of bits and it can jam the whole system into not-working. Who needs an EMP?
posted by zengargoyle at 3:41 PM on October 4, 2018


If this is real, my bet is on subverting a serial bus to an EEPROM, not one bit of a whole parallel bus.


Yessss... and if you have a serial BIOS then intercepting it does give you an awful lot of interesting playthings. But then that doesn't mesh with the rest of the story.

Given the unusually rigorous denials of the parties named and the fuzzy tech details, I'd be leaning heavily on the 'hoax' side of this story, but Bloomberg really should have the editorial mojo to spot that.

Fascinating, Captain.
posted by Devonian at 3:52 PM on October 4, 2018


Journalismwise was this a good article compared to similar events in the past? The reporters literally have no proof, when they could probably have obtained an existing server board and actually find some hackers willing to investigate and actually really show this magic chip exists. Instead the piece is a bunch of allegations of senior insiders and ex-officials so forth like in a Hollywood movie.

Meanwhile, the more strongly a party denies something… I think we've learned that volume has little correlation to the truth in either direction.
posted by polymodus at 3:54 PM on October 4, 2018 [1 favorite]


Some people buy old cars and keep them repaired, some people buy computers with old hardware and run highly-efficient software on them.

Some people drive their old cars off the freeway. Some people keep those old computers off the net.

Alternately, buy a RasPi and know *exactly* what hardware's on it. KISS.
posted by Twang at 4:33 PM on October 4, 2018 [1 favorite]


Meanwhile, the more strongly a party denies something… I think we've learned that volume has little correlation to the truth in either direction.

Corporations are not people, my friend.

If this happened, Amazon could say "we discovered X number of boards in Y data centers that had this implant. The incident began at D date of installation, until D2 when anomalous traffic alerted us to the implant, and D3 when all the boards were turned off and evicted from our premises. Z number of customers were affected, all of whom were alerted and took corrective action to avooid harm to their own stakeholders. We've severed our ties to SUperMicro and have instituted network monitoring policies to better spot such incidents in the future, as well as introduced a new policy of randomly x-raying motherboards prior to installation."

And all would be fine.

If it happened and they say it didn't, every cloud customer they have would be obliged to depart for reasons of due diligence. And the EU would love, love the opportunity to encourage European customers to avoid Amazon. THere is no way Amazon legal would sign off on a false denial.
posted by ocschwar at 4:35 PM on October 4, 2018 [1 favorite]


Either the US just knew this and had an exploit, or they managed to do something to that printer. This was a SCSI printer.

Was there such a thing? I though that pre-USB printers all connected to computers with simple unidirectional links (parallel or serial) that were physically unable to UPLOAD VIRUS to the target machine as described, and the story was either disinformation to protect sources or else a game of telephone.
posted by acb at 6:19 PM on October 4, 2018


Printer connections were always two directional.

That would let you know when a job was complete, or when a printer jammed, or (not a joke) if the printer caught fire. And SCSI devices can get VIP treatment on a motherboard (90's era desktops could sometimes put the RAM on the SCSI chain). Still probably false, but still plausible.
posted by ocschwar at 6:54 PM on October 4, 2018 [3 favorites]


I'd build my secure system out of recycled Z80s from Sinclair ZX Spectrums

I know this is jest, but where are you going to get the chips? The ones that aren't buried miles deep in landfills were shipped in containerloads of e-waste to China. Some smart folks there realized that if they separated the chips and tested them, someone, somewhere would want 'em badly enough to make it worthwhile. This strategy has allowed some really old chips to be still available — at a price.

Not all is rosy, though. The chips are described as “refreshed”, which means cleaned-up packaging, new bright-tinned pins and a new silkscreen on top. Sometimes these are not the right chips. Even obscure chips like the Harris HD6120 (basically a DEC PDP-8 on a CMOS chip in a chunky ceramic package) are showing up as physically perfect, but whatever silicon is inside the package, it sure isn't a PDP-8 on a chip.
posted by scruss at 7:30 PM on October 4, 2018 [2 favorites]


THere is no way Amazon legal would sign off on a false denial.

I think this assertion is simply amazing by any reasonable standard. I'd start by revising that more critically.
posted by polymodus at 9:09 PM on October 4, 2018 [1 favorite]


Besides analyzing hypothetical behaviors of companies and large groups of people, just look at Apple as a historical example. Either you believe it participated in PRISM or not. And Apple has "strongly" denied this.
posted by polymodus at 9:14 PM on October 4, 2018


you don't just sneak a single embedded chip into a manufacturing process, it requires far to many extra expensive steps

This is the main point that my own skepticism about this story currently rests on. If I were in charge of a spy agency project aimed at subverting commercial computers via their supply chain, I'd sure as shit not be organizing ways to add extra stuff to the boards; I'd be looking at pin-compatible substitutions for existing chips. Way easier to achieve and way less likely to be detected.
posted by flabdablet at 10:07 PM on October 4, 2018 [4 favorites]


Also, I wouldn't be putting too much credence in the fine technical detail even if the story is in some way vaguely accurate. I have enough experience with tech journalism to believe that teeth-grinding mishmashes of half-baked misunderstandings with heavy ladlings of Hollywood sauce are par for the course in tech-related stories published by non-primarily-tech-related media outlets.

BRB; need to go create a GUI in Visual Basic, see if I can track an IP address.
posted by flabdablet at 10:16 PM on October 4, 2018


That SMM injection proof of concept is wild.

Indeed it is.

I especially enjoyed the part where it can run arbitrary privileged non-preemptible code from incoming UDP packets deposited into RAM buffers via Ethernet controller DMA before the kernel's packet filter even knows that they've arrived.
posted by flabdablet at 10:33 PM on October 4, 2018


I was dubious when I read the story, but mostly about the magic chip. But after the row hammer example I will never question the ingenuity of some to exploit a vulnerability. So the jury is out for me.
What I find amusing is the people here confidently asserting the denials must be true because of $$$$ at risk.
When agent Smith and Agent Smith (no relation) cancel Jeff Bezos’ afternoon appointments and carefully explain to him what he has to do to avoid a waterboarding, and then they explain it in the same terms to any lawyer who won’t sign off on the statement, and they clarify that the law can be retrospectively changed to make refusal punishable by 30 years in Leavenworth, and SEC Regs are explicitly over ridden, why would you place any value in a corporate denial?
Of course, it doesn’t happen that way, because Bezos already knows when he gets the call from the State department, he just goes along with it, rather than all that unpleasantness.
National security is so many levels above “shareholder rights” I don’t know what to say.
posted by bystander at 5:36 AM on October 5, 2018 [1 favorite]


I'm sure the timing of this is all one huge coincidence (re: White House, anti-China narrative)

Good - because apparently, Canadian steel is enough of a security threat for severe tariffs...

(bitter? Yes, yes I am)
posted by jkaczor at 6:41 AM on October 5, 2018 [1 favorite]



When agent Smith and Agent Smith (no relation) cancel Jeff Bezos’ afternoon appointments and carefully explain to him what he has to do to avoid a waterboarding, and then they explain it in the same terms to any lawyer who won’t sign off on the statement, and they clarify that the law can be retrospectively changed to make refusal punishable by 30 years in Leavenworth,


Ex post facto law, a constitutional no-no, being used as a threat against a man who really can get the best lawyers money can buy. Sure.

THere's a reason I'm focusing on Amazon. Apple's customers are schmoes like you and me. Apple can lie to us and not suffer when caught.

But Amazon's profits hinge on telling other corporate customers that their data are safe even when they're running stuff on an Amazon motherboard that's being shared with other customers. If Amazon is shown to have lied on this make-or-break criterion, their corporate customer base has to migrate elsewhere or else face shareholder lawsuits and loss of their own customers for the same reason.

So Trump Cousin #22 lying to a Bloomberg reporter into order to mire Amazon into this story is a far more plausible explanation for this story than Agents Smith and Smith.
posted by ocschwar at 6:47 AM on October 5, 2018 [5 favorites]


El Reg weighs in
posted by flabdablet at 6:49 AM on October 5, 2018 [1 favorite]


When agent Smith and Agent Smith (no relation) cancel Jeff Bezos’ afternoon appointments and carefully explain to him what he has to do to avoid a waterboarding

Why'd you go and send Smith? I'd send agent Rogers, good friend of Jeff, and reassure him that there is no possibility of legal exposure, since any physical evidence of the events in question, which most certainly never happened, has been scrupulously removed to the inaccessible reaches of spookdom, and all 23 of the people with direct knowledge of these events that never happened can be relied upon to be sensible and in any case will be watched closely. It's the least we can do. After all, what's good for Amazon is good for America.
posted by sfenders at 5:00 PM on October 5, 2018


acb: that's the bit that needs to know about SCSI. You have a Bus and you are 0 and master and disks/printers/etc. are 1-7 but all plugged in and just assumes to be working as expected. But there's no reason why you printer can not impersonate the systems hard drive.... Does the string (and I'm making this up.) "Four hostiles in sector 2" get printed on your printers log... Start sending SCSI resets and start sending writes to all 0-7 devices that aren't yourself to overwrite the inital blocks of memory. Catch that read request to the HD array or that data from the data-collection device (aka radar) and intercept it with false data or wipe the disk or make the entire SCSI bus go insane and unusable just because some string got printed. Totally cromulent. Good f'*** hack triggered by a string of a few ASCII characters sent to a printer that cause all hell to brake loose.

Black Ice, a Basilisk... If you see it, you're already dead.

... A SCSI bus of devices is ripe for BAM!!!!

I can pretend to be the HD and feed you false information, I can pretend to be the host OS and corrupt your memory, I can jam the communications to make the whole system unworkable.

The seductive force of the rumor is strong. It's not that hard to imagine happening in the early-ish 90's. All you have to do is get a trigger and start screaming. Until somebody decides to take the printer off-line but your OS disk-array is already corrupted.
posted by zengargoyle at 6:46 PM on October 5, 2018


Alternately, buy a RasPi and know *exactly* what hardware's on it. KISS.

Lol, you think the SoC on a Pi couldn't be pwned in exactly the same way as any other chip. Hell, given the sourcing of most Pis, I'd say you've got a better chance than with most hardware of having something..unexpected lurking in there.
posted by wierdo at 12:41 AM on October 6, 2018 [4 favorites]


Ars Technica update. Nothing new, but Bloomberg isn't backing down from the story. Apple has specifically said that it's not under a gag order, for whatever that's worth.

As noted previously, Amazon very much wants to do business with Uncle Sam, so their cooperation with the FBI would be entirely plausible and consistent with their business objectives. Apple has generally had a chillier relationship with the Feds, so one would imagine they'd be more free to say what they want, but they also have a conflict of interest due to their manufacturing in China. So there are reasons to not take either denial at face value, but they aren't the usual non-denial-denials that you get from companies when they know the news is going to break eventually and they're just trying to play along temporarily. Anybody's guess at this point.

Ars points out that we should know for sure, one way or the other, quite soon, because the Bloomberg story was very specific and should allow someone to do a teardown of one of the allegedly-affected Supermicro boards and analyze it. If the Bloomberg team was good, they probably should have secured a board as proof before running with the story, but they don't seem to have done that. Finding out where the equipment that allegedly got pulled from Apple's and Amazon's datacenters would be very interesting.
posted by Kadin2048 at 11:29 AM on October 6, 2018


From the Ars comments:

1. Bloomberg is simply lying
2. Apple is simply lying
3. 17 different sources are lying to Bloomberg
4. 3 Apple "insiders" lied to Bloomberg in a way consistent with 4 government sources
5. 3 Apple "insiders" lied to Apple during Apple's internal investigations / discussions
6. 3 Apple "insiders" were not part of Apple's internal investigations / discussions
7. ???

Of the above, I consider all scenarios unlikely, but #5 or #6 are the "least unlikely" in my opinion.


Really curious to see how this plays out.
posted by aspersioncast at 6:02 PM on October 6, 2018 [1 favorite]


If the story were true you would think Bloomberg could produce a motherboard, an x-ray of a motherboard, a boot trace, or something. Without actual evidence it's just a cool story.
posted by Joe in Australia at 10:21 PM on October 6, 2018 [1 favorite]


This story is very strange. Taking Apple as an example, Apple denied the report in the original story, followed up with a public statement of denial, then published an even stronger rebuttal on their web site, then unnamed Apple executives leak that "this did not happen" according to Apple's "endoscopic" internal investigation into Bloomberg's accusations, and then Apple's recently retired general counsel said he called the FBI after being contacted by Bloomberg for this story and he was told that nobody at the FBI knew about it. And now the U.S. Department of Homeland Security have issued a statement saying they have no reason to doubt the denials.

Bloomberg is a reliable news organization, it is not like them to publish a story like this without solid sources (even if they are anonymous). Somebody has to be telling a huge lie. Given the extraordinary denaials from Apple, if they are lying about this their customers will never trust them again. On the other hand, if it turns out that Bloomberg's sources were lying and Bloomberg didn't catch them at it, Bloomberg's going to take a major hit to their reputation.
posted by RichardP at 1:09 AM on October 7, 2018 [1 favorite]


Somebody has to be telling a huge lie.

If by "telling a lie" you mean "speaking with intent to deceive", I disagree.

If by "telling a lie" you mean "claiming things that are not actually facts to be facts", I agree.

Technology is complicated, and there are a hell of a lot of people who are completely convinced that their grasp of it is far better than it actually is. If this is a whodunit, my money is on Dunning and Kruger.
posted by flabdablet at 7:03 AM on October 7, 2018 [2 favorites]


I'm completely convinced that the details of this don't make any sense, especially to the extent that this either isn't being caught in preproduction or the boards aren't being modified at a later stage in the way we know the CIA and others have done.

I can see how a strategically placed chip could monitor and potentially modify signalling to the SPI flash chip that holds the system firmware, to pick one example, but that should be caught in dev and or QA unless it's being done only to selected boards. It's not as if it would be terribly hard to make a modified "midnight run" and replace legit boards being shipped to select customers in transit.

However, absent a mole slipping in an extra component and modified board layout inside Supermicro, modifications made by the manufacturer should be obvious when a sample of the production run is put into test, photographed, and x-rayed to within an inch of its life and compared both manually and with software to the design specification. X-raying parts is pretty standard at companies large enough to afford the gear, so it would have to go beyond simply adding unexpected stuff to every production board.

I held off commenting hoping more information would become available since, as people here have already pointed out, journalists often get the details wrong for various reasons while still communicating an reasonably accurate overview of the overall subject.

It's not that it's impossible as described, it just seems unlikely given Supermicro's size that they don't X-ray boards as part of their QA process. If the claim was that some of Supermicro's particularly interesting/useful customers were receiving equipment that had been altered in distribution it would make a lot more sense.

If one manufactured a run of signal conditioners or normally passive ICs with some malicious circuits added to the IC that were externally identical to legit parts and were of similar construction and die size, it would be pretty easy to intercept shipments. Maybe you could pwn a whole run of boards by slipping your parts into the manufacturing process, but that would require a much larger conspiracy.
posted by wierdo at 10:29 PM on October 7, 2018 [1 favorite]


modified at a later stage in the way we know the CIA and others have done

I would have thought that modifying boards after they had already been populated would yield alterations much easier to spot physically than those resulting from a covertly modified manufacturing spec. On the other hand, inserting a malicious rev into a manufacturing spec would make it much easier to spot the ring-in after the fact just by auditing paperwork.

If the aim is to subvert boards en masse, supplying a few shipments of doctored chip-level components to an otherwise unmodified board manufacturing step looks much easier all round. Assuming your substitutes were fit for their original purpose as well as for your covert one, and didn't stick out like dogs' balls on physical examination, all you'd need is some kind of hold over as few as one components QA person at the board plant.
posted by flabdablet at 11:16 PM on October 7, 2018


I don't see the spooks having someone smuggle in a reel of chips. They're not exactly small. It could be done, but there's many opportunities for exposure. Better to intercept a shipment of parts if you're going that route. They'd have to be robust to X-ray examination either way.

I think the reason they've done it at the distribution level when they've implanted routers and other equipment in the past is that the risk of detection is much lower, as is manufacturing the malicious part since most customers don't X-ray their routers and white box servers, so visually similar and pin compatible is good enough. It's also more conservative of resources since it's targeted.

Still, the article specifically alleged that chips had been implanted in internal board layers. That obviously can only happen during PCB manufacture and implies a fairly broad conspiracy since it's both very unusual (doable in many PCB factories without new equipment, but still an unusual process) and likely to be detected by automated QA processes at both ends of the shipment. Getting away with that is some serious wizard level shit that would likely generate rumors, even in China because it requires enough collaboration (or the use of enough intrusion capability) in a single operation to make the shadowy types nervous.
posted by wierdo at 11:56 PM on October 7, 2018 [1 favorite]


I don't see the spooks having someone smuggle in a reel of chips.

Me either, but I can easily see them arranging for somebody not to look too closely at the results of whatever inspections are routinely done on incoming parts at a board assembly factory.
posted by flabdablet at 8:31 AM on October 8, 2018


I agree. What I find outlandish is the idea that major companies wouldn't notice an oddly opaque spot on a board where there isn't supposed to be anything but a few vias given that they use X-rays on the assembled boards as a matter of course and that QA is done here.
posted by wierdo at 9:14 AM on October 9, 2018 [1 favorite]


The common wisdom has been moving towards calling shenanigans on this story, but here's Bloomberg today doubling down with a new story: New Evidence of Hacked Supermicro Hardware Found in U.S. Telecom
A major U.S. telecommunications company discovered manipulated hardware from Super Micro Computer Inc. in its network and removed it in August, fresh evidence of tampering in China of critical technology components bound for the U.S., according to a security expert working for the telecom company.

The security expert, Yossi Appleboum, provided documents, analysis and other evidence of the discovery following the publication of an investigative report in Bloomberg Businessweek that detailed how China’s intelligence services had ordered subcontractors to plant malicious chips in Supermicro server motherboards over a two-year period ending in 2015.
posted by gwint at 12:08 PM on October 9, 2018


John Gruber at Daring Fireball excerpts from a podcast interview with one of the few named sources in the Bloomberg story:
[Interviewer]: OK, right. You find that a bit strange? That every single thing you seem to tell him, or a large proportion of what you told him, was then confirmed by his other sources.

[Hardware security researcher Joe] Fitzpatrick: Yeah, basically. Either I have excellent foresight or something else is going on.
posted by RedOrGreen at 1:10 PM on October 9, 2018


It's certainly not impossible that Fitzpatrick might have posited some possible hacks to Bloomberg that turned out to have already been done. Most of this stuff, on a conceptual level at least, is plainly obvious with any knowledge of how electronics are constructed.

However, if it was actually the case that each and every scenario outlined in the Bloomberg story was aligned perfectly with what he had told Bloomberg, that's a bit..fishy. That said, I could easily see a scenario where he had mentioned all of those same attacks in previous public talks which would, of course, have the usual TLA attendees there to write their memos about what is being presented and talked about at the conferences. Those memos, filtering their way through various government offices could then get mangled along the way and ended up being presented to some people as recitations of actual live hacks, sparking yet more rounds of increasingly outlandish exaggerations and eventually being related to some unfortunate reporter who couldn't or didn't make the connection.

That's one of many possibilities, of course, up to and including intentional media manipulation (whether to drumbeat on China or discredit news outlets) by the executive branch. The story is definitely getting more interesting now, not less, just (probably) not in the way whoever first whispered it in the reporter's ear probably intended. There's definitely some palace intrigue going on somewhere unless Bloomberg has no editorial process at all for their news reporting.
posted by wierdo at 5:53 PM on October 9, 2018 [1 favorite]


More from Ars Technica:
The complexity, sophistication, and surgical precision needed to pull off such attacks as reported are breathtaking, particularly at the reported scale. First, there’s the considerable logistics capability required to seed supply chains starting in China in a way the ensures backdoored equipment ships to specific US targets but not so widely to become discovered.
[…]
The articles don’t explain how attackers ensured the altered equipment shipped broadly enough to reach intended targets in a distant country without also going to other unintended companies. Nation-state hackers almost always endeavor to distribute their custom spyware as narrowly as possible to only chosen high-value targets, lest the spy tools spread widely and become discovered the way the Stuxnet worm that targeted Iran’s nuclear program became public when its creators lost control of it.

The other monumental effort required by the reported supply-chain attacks is the vast amount of engineering and reverse engineering. Based on Bloomberg’s descriptions, the attacks involved designing at least two custom implants (one that was no bigger than a grain of rice), modifying the motherboards to work with the custom implants, and ensuring the modified boards would work even when administrators installed new firmware on the boards. While the requirements are within the means of a determined nation, three hardware security experts interviewed for this story said the factory-seeded hardware implants are unnecessarily complex and cumbersome, particularly at the reported scale, which involved almost 30 targets.
The whole thing builds to the kicker that if you wanted to compromise a SuperMicro board, it would have been much easier, and just as effective, to use a known vulnerability to install your own firmware. No new hardware components required.
posted by fedward at 7:51 PM on October 11, 2018 [2 favorites]


here's Bloomberg today doubling down with a new story: New Evidence of Hacked Supermicro Hardware Found in U.S. Telecom

From that story:
Unusual communications from a Supermicro server and a subsequent physical inspection revealed an implant built into the server’s Ethernet connector, a component that's used to attach network cables to the computer, Appleboum said.
Importantly, this is not a doubling-down on the Extra Spy Chip Subverts BMC story, it's a description of a completely different hardware compromise.

And although their source says
he has seen similar manipulations of different vendors' computer hardware made by contractors in China, not just products from Supermicro. “Supermicro is a victim -- so is everyone else,” he said. Appleboum said his concern is that there are countless points in the supply chain in China where manipulations can be introduced, and deducing them can in many cases be impossible. “That's the problem with the Chinese supply chain,” he said.
the Bloomberg article then makes multiple mentions of Supermicro specifically, before delivering this final kick in the head:
“Manufacturers that overlook this concern are ignoring a potentially serious problem,” Kanuck said. “Capable cyber actors -- like the Chinese intelligence and security services -- can access the IT supply chain at multiple points to create advanced and persistent subversions.”
...the clear though plausibly deniable implication being that Supermicro is one such overlooker.

I don't think this news is necessarily fake in the sense of having been made up from whole cloth; I'm sure there do exist more vulnerabilities in any offshore supply chain than would exist for a local manufacturer. But I am even less inclined to assign much credibility to the original Spy Chips Found story after reading the followup.

Rather, I'd be inclined to follow the money. If Bloomberg still takes the attitude that the only journalism that matters is the kind that moves markets: well, these journalists have certainly moved this one. It would be interesting to know how Bloomberg itself and/or certain Bloomberg employees have benefited from that movement.
posted by flabdablet at 8:45 PM on October 11, 2018


Some of the accounts implied that there was already a pad for the (alleged) dodgy component; it just wasn't populated on most boards. But yes; if you're in a position to pull off something like this then you could achieve the same result in many other ways that are far less detectable. The recent cache attacks show that our hardware designs are still vulnerable even when they're working as specified. There are undoubtedly actual bugs in modern CPUs that we don't know about, and now that we're pushing the boundaries of circuit design it's entirely possible that there can be analog vulnerabilities (like Rowhammer, but for CPUs) showing up too. Given state-level resources thrown at a specific manufacturer I'm sure they could have found something.
posted by Joe in Australia at 8:51 PM on October 11, 2018


Bloomberg's source for the second story is not happy about their spin.
posted by flabdablet at 7:33 AM on October 12, 2018


« Older Go ahead and hate your neighbor, go ahead and...   |   Go home birds, you're drunk Newer »


This thread has been archived and is closed to new comments