Tar Trap Caught
March 30, 2024 2:21 PM Subscribe
A sophisticated backdoor in Linux's xz compression tool, apparently inserted by a long time contributor, was fortunately discovered yesterday in the latest unstable distribution of Debian, before it could be spread more widely.
blocked gzip and bz2 4 life
posted by They sucked his brains out! at 2:36 PM on March 30 [2 favorites]
posted by They sucked his brains out! at 2:36 PM on March 30 [2 favorites]
I had it on a non-internet-facing computer, it was already gone on another one running the same edition of a distribution into which this backdoored package was installed.
The tarball supplied differs from the GitHub download, containing the backdoor execution, even if the plumbing to set it up is in the main git history.
posted by k3ninho at 2:50 PM on March 30 [1 favorite]
The tarball supplied differs from the GitHub download, containing the backdoor execution, even if the plumbing to set it up is in the main git history.
posted by k3ninho at 2:50 PM on March 30 [1 favorite]
What's "exciting" about this is how close they were to being successful, despite a history that looks somewhat dodgy (at least in retrospect). It really makes you wonder how many times an attack like this has been successful in the past. I'm curious to see how this ends up impacting the OSS approval process at work.
posted by Slothrup at 2:55 PM on March 30 [5 favorites]
posted by Slothrup at 2:55 PM on March 30 [5 favorites]
Also, Andres Freund is a hero. Quoting from his Mastodon:
I was doing some micro-benchmarking at the time, needed to quiesce the system to reduce noise. Saw sshd processes were using a surprising amount of CPU, despite immediately failing because of wrong usernames etc. Profiled sshd, showing lots of cpu time in liblzma, with perf unable to attribute it to a symbol. Got suspicious. Recalled that I had seen an odd valgrind complaint in automated testing of postgres, a few weeks earlier, after package updates.posted by Slothrup at 2:57 PM on March 30 [18 favorites]
Isn't xz the one that they use to compress kernel modules nowadays? That seems like a good place to insert your backdoor.
(now off to read the articles)
posted by clawsoon at 3:00 PM on March 30
(now off to read the articles)
posted by clawsoon at 3:00 PM on March 30
It's a library that doesn't raise eyebrows for SSHD to load (compression of data in transit? or something).
posted by ocschwar at 3:01 PM on March 30
posted by ocschwar at 3:01 PM on March 30
Injected through a test suite, I see. Clearly my failure to ever keep a test suite in working condition over the long term has been justified.
posted by clawsoon at 3:12 PM on March 30 [18 favorites]
posted by clawsoon at 3:12 PM on March 30 [18 favorites]
The most interesting part of this analysis to me is how serious an effort this exploit was to place. Two years of social engineering and technical work, slowly worming its way in. We caught this one because some random developer happened to look closely at some odd system performance. Which other supply chain attacks have we not expected yet? And who's doing them? Easy to imagine a state actor at work but an exploit this powerful and widespread is valuable to lots of organizations.
The specific point to catch this particular attack, IMHO, is that the source tarball people were using didn't match what was on the GitHub repository. That could have been verified, probably with automated tools. More generally I'm a fan of using cryptographic signatures for high value code releases but it puts a high burden on the developers to do that. The Python package library has had years of resistance to this extra measure, for instance.
An added problem is it wasn't clear libxz was so security critical. OpenSSH doesn't include it. But Debian's version does via its use of libsystemd, which itself uses libxz. (Insert complaint about systemd bloat here.)
posted by Nelson at 3:19 PM on March 30 [16 favorites]
The specific point to catch this particular attack, IMHO, is that the source tarball people were using didn't match what was on the GitHub repository. That could have been verified, probably with automated tools. More generally I'm a fan of using cryptographic signatures for high value code releases but it puts a high burden on the developers to do that. The Python package library has had years of resistance to this extra measure, for instance.
An added problem is it wasn't clear libxz was so security critical. OpenSSH doesn't include it. But Debian's version does via its use of libsystemd, which itself uses libxz. (Insert complaint about systemd bloat here.)
posted by Nelson at 3:19 PM on March 30 [16 favorites]
The modus operandi of the attack is fascinating and, speaking as someone who has maintained open source software packages, likely to be effective in many cases.
1. The attackers identified a critical software component (liblzma is linked into hundreds of tools), maintained by a single individual (Lasse Collin) not affiliated with a big company and attempting to do this difficult job in his spare time without pay.
2. The attackers used social engineering techniques to persuade Lasse to take on Jia Tan (the pseudonym used by the attackers) as co-maintainer. I have no idea whether any of the participants in this thread on the xz mailing list were sock-puppets for the attackers or not, but they might as well have been. Jigar Kumar says, "Progress will not happen until there is new maintainer... Submitting patches here has no purpose these days. The current maintainer lost interest or doesn't care to maintain anymore. It is sad to see for a repo like this." Lasse replies that he is "fairly limited mostly due to longterm mental health issues" and already getting help from Jia Tan, but Jigar is scathing: "You ignore the many patches bit rotting away on this mailing list. Right now you choke your repo. Why wait until 5.4.0 to change maintainer? Why delay what your repo needs?"
3. Jia Tan made large numbers (more than 700) of mostly legitimate commits to the project before inserting the backdoor code. The exploit code was disguised in the form of a test case for handling a corrupted xz archive, and the code for getting the exploit into the release was added to the build script. Build scripts often consist of lengthy, repetitive and obscure bits of shell code, so they are hard to review. It is not surprising to me that the exploit was missed by reviewers (who were also working in their spare time and unpaid, of course).
We got very lucky here that the attackers made a coding mistake (their code for exploiting sshd was inefficient enough for humans to notice the slowdown) and that this was spotted by an engineer (Andres Freund) with the skills and determination to get to the bottom of the problem. It would not surprise me at all if there are other similar attacks that have not been discovered and which are lurking in widely deployed systems today.
Basically we cannot expect people maintaining software in their spare time for no money to resist these kinds of attacks. The software industry has to find some way to fund the maintenance of critical dependencies like xz.
posted by cyanistes at 3:23 PM on March 30 [59 favorites]
1. The attackers identified a critical software component (liblzma is linked into hundreds of tools), maintained by a single individual (Lasse Collin) not affiliated with a big company and attempting to do this difficult job in his spare time without pay.
2. The attackers used social engineering techniques to persuade Lasse to take on Jia Tan (the pseudonym used by the attackers) as co-maintainer. I have no idea whether any of the participants in this thread on the xz mailing list were sock-puppets for the attackers or not, but they might as well have been. Jigar Kumar says, "Progress will not happen until there is new maintainer... Submitting patches here has no purpose these days. The current maintainer lost interest or doesn't care to maintain anymore. It is sad to see for a repo like this." Lasse replies that he is "fairly limited mostly due to longterm mental health issues" and already getting help from Jia Tan, but Jigar is scathing: "You ignore the many patches bit rotting away on this mailing list. Right now you choke your repo. Why wait until 5.4.0 to change maintainer? Why delay what your repo needs?"
3. Jia Tan made large numbers (more than 700) of mostly legitimate commits to the project before inserting the backdoor code. The exploit code was disguised in the form of a test case for handling a corrupted xz archive, and the code for getting the exploit into the release was added to the build script. Build scripts often consist of lengthy, repetitive and obscure bits of shell code, so they are hard to review. It is not surprising to me that the exploit was missed by reviewers (who were also working in their spare time and unpaid, of course).
We got very lucky here that the attackers made a coding mistake (their code for exploiting sshd was inefficient enough for humans to notice the slowdown) and that this was spotted by an engineer (Andres Freund) with the skills and determination to get to the bottom of the problem. It would not surprise me at all if there are other similar attacks that have not been discovered and which are lurking in widely deployed systems today.
Basically we cannot expect people maintaining software in their spare time for no money to resist these kinds of attacks. The software industry has to find some way to fund the maintenance of critical dependencies like xz.
posted by cyanistes at 3:23 PM on March 30 [59 favorites]
Poor Lasse, I feel so bad for him. As someone noted on Mastodon, this kind of social engineering is so easy because so many people treat open source maintainers like shit. That pressuring from the Jigar sockpuppet are par for the course in OSS support discourse.
Lasse emerged yesterday as a bit of a hero, publishing a FAQ and submitting fixes for some of the damage. I can only imagine he wants nothing at all to do with any of this anymore.
posted by Nelson at 3:30 PM on March 30 [24 favorites]
Lasse emerged yesterday as a bit of a hero, publishing a FAQ and submitting fixes for some of the damage. I can only imagine he wants nothing at all to do with any of this anymore.
posted by Nelson at 3:30 PM on March 30 [24 favorites]
There are already people saying, "It must be the Chinese!" just based on the name, despite the extent of the social engineering attack showing that whoever's behind this is fully capable of multiple levels of misdirection.
posted by clawsoon at 3:32 PM on March 30 [10 favorites]
posted by clawsoon at 3:32 PM on March 30 [10 favorites]
First thought: "We need the government to fund projects like this so that situations like this don't happen."
Second thought: "Oh, wait a minute, it's probably a government doing the hack, nevermind."
posted by clawsoon at 3:41 PM on March 30 [2 favorites]
Second thought: "Oh, wait a minute, it's probably a government doing the hack, nevermind."
posted by clawsoon at 3:41 PM on March 30 [2 favorites]
For people who haven't spotted the problem with the diff linked by Klipspringer above,
posted by cyanistes at 3:42 PM on March 30 [12 favorites]
click here for my explanation...
There's a dot at the start of the middle line here (in CMakeLists.txt)+ #include <sys/prctl.h> +. + void my_sandbox(void)A dot is not allowed at this position in C code, so compilation of this program will fail. This means that the variable
HAVE_LINUX_LANDLOCK will always be set to false, regardless of whether the platform has Landlock (a Linux security system). This meant that xz never used Landlock, bypassing a security measure that (I assume) would have caught or prevented the exploit.posted by cyanistes at 3:42 PM on March 30 [12 favorites]
looks like the linux linking architecture needs more dolphins here
posted by torokunai at 4:14 PM on March 30 [12 favorites]
posted by torokunai at 4:14 PM on March 30 [12 favorites]
Or bollards
posted by Greg_Ace at 4:33 PM on March 30 [6 favorites]
posted by Greg_Ace at 4:33 PM on March 30 [6 favorites]
The folks at libarchive are currently investigating a number of commits by the same attacker.
posted by Johnny Lawn and Garden at 4:45 PM on March 30 [4 favorites]
posted by Johnny Lawn and Garden at 4:45 PM on March 30 [4 favorites]
Scary to think about the ambiguities of an investigation like this, but I wonder what would turn up if we looked into any contributors who:
posted by Riki tiki at 4:52 PM on March 30 [2 favorites]
- became active around the same time as "Jia Tan",
- in popular or transitively-popular open-source projects, and
- which were maintained by few volunteers or just a single one.
posted by Riki tiki at 4:52 PM on March 30 [2 favorites]
There are already people saying, "It must be the Chinese!" just based on the name, despite the extent of the social engineering attack showing that whoever's behind this is fully capable of multiple levels of misdirection.
I honestly wouldn't be surprised if it was the NSA.
posted by lock robster at 6:04 PM on March 30 [6 favorites]
I honestly wouldn't be surprised if it was the NSA.
posted by lock robster at 6:04 PM on March 30 [6 favorites]
There are already people saying, "It must be the Chinese!" just based on the name, despite the extent of the social engineering attack showing that whoever's behind this is fully capable of multiple levels of misdirection.
Sinophobia is very popular in America (and parts of Europe) lately. (I mean, xenophobia and rabid nationalism are always way too popular, 100% of the time, but it's been especially bad in this particular direction for a couple years now.)
posted by adrienneleigh at 6:09 PM on March 30 [1 favorite]
Sinophobia is very popular in America (and parts of Europe) lately. (I mean, xenophobia and rabid nationalism are always way too popular, 100% of the time, but it's been especially bad in this particular direction for a couple years now.)
posted by adrienneleigh at 6:09 PM on March 30 [1 favorite]
To be fair, the Chinese are very good at cyberwarfare and this kind of long game intrusion. But then so are the US, and Israel, and a couple of European countries. And Russia. And North Korea. Attribution of cyberattacks is difficult. And of course the choice of "Jia Tan" as the psuedonym for the attacker tells us nothing.
The usual way we attribute attacks to APTs is based on the tools they use. But this attack seems pretty unique, so far I've read no reports saying "oh yeah this technique is just like X and Y from these other attacks". Is it really a new class of public attack? I know supply chain stuff has been theorized plenty of times and there's some clumsy cryptoscam stuff that's happened. But nothing this subtle and convoluted.
One other interesting technical observation about the exploit:
The good news is this exploit seems to only be usable by whoever is in possession of the private keys. The NSA Equation Group should have learned that lesson awhile back, would have saved us a lot of hacks after they mishandled their infiltration tools and they got out in the wild.
posted by Nelson at 6:34 PM on March 30 [10 favorites]
The usual way we attribute attacks to APTs is based on the tools they use. But this attack seems pretty unique, so far I've read no reports saying "oh yeah this technique is just like X and Y from these other attacks". Is it really a new class of public attack? I know supply chain stuff has been theorized plenty of times and there's some clumsy cryptoscam stuff that's happened. But nothing this subtle and convoluted.
One other interesting technical observation about the exploit:
Apparently the backdoor reverts back to regular operation if the payload is malformed or the signature from the attacker's key doesn't verify.Subtle and somewhat stealthy. Nasty.
Unfortunately, this means that unless a bug is found, we can't write a reliable/reusable over-the-network scanner.
The good news is this exploit seems to only be usable by whoever is in possession of the private keys. The NSA Equation Group should have learned that lesson awhile back, would have saved us a lot of hacks after they mishandled their infiltration tools and they got out in the wild.
posted by Nelson at 6:34 PM on March 30 [10 favorites]
A list of affected distro versions:
Evidence shows that the packages are only present in Fedora 41 and Fedora Rawhide, and do not impact Red Hat Enterprise Linux (RHEL), Debian Stable, Amazon Linux, and SUSE Linux Enterprise and Leap...posted by clawsoon at 6:57 PM on March 30 [1 favorite]
Some of the other Linux distributions impacted by the supply chain attack are below -
Kali Linux (between March 26 and 29)
openSUSE Tumbleweed and openSUSE MicroOS (between March 7 and 28)
Debian testing, unstable, and experimental versions (from 5.5.1alpha-0.1 to 5.6.1-1)
It's a library that doesn't raise eyebrows for SSHD to load (compression of data in transit? or something).
Someone on Hacker News commented that xz and lzma are compression tools falling into a particular category: they're merely okay at compression, but they're very fast, so they're easy to add at each of a pipeline (compress, send decompress) to get a significant and nearly free reduction in the size of transmitted/written data. xz and lzma are widely used and if this hack hadn't been discovered, would be a great place for a backdoor. The significance of SSHD is that 1) it uses lzma and 2) it normally runs as root, so if you can trigger this exploit, you have root access to any computer on the Internet for which public access by SSH is allowed, if it uses the backdoored compression libraries.
posted by fatbird at 7:51 PM on March 30 [5 favorites]
Someone on Hacker News commented that xz and lzma are compression tools falling into a particular category: they're merely okay at compression, but they're very fast, so they're easy to add at each of a pipeline (compress, send decompress) to get a significant and nearly free reduction in the size of transmitted/written data. xz and lzma are widely used and if this hack hadn't been discovered, would be a great place for a backdoor. The significance of SSHD is that 1) it uses lzma and 2) it normally runs as root, so if you can trigger this exploit, you have root access to any computer on the Internet for which public access by SSH is allowed, if it uses the backdoored compression libraries.
posted by fatbird at 7:51 PM on March 30 [5 favorites]
A dot is not allowed at this position in C code, so compilation of this program will fail.
Would compiling with pedantic or all flags turned on have helped catch this (or future attacks of a similar stripe)?
posted by They sucked his brains out! at 8:07 PM on March 30
Would compiling with pedantic or all flags turned on have helped catch this (or future attacks of a similar stripe)?
posted by They sucked his brains out! at 8:07 PM on March 30
>Would compiling with pedantic or all flags turned
this was some weird-ass C function call with the C source passed in as a string, or I guess char * because again we're in wacky hold-my-beer C Land here.
posted by torokunai at 8:26 PM on March 30 [1 favorite]
this was some weird-ass C function call with the C source passed in as a string, or I guess char * because again we're in wacky hold-my-beer C Land here.
posted by torokunai at 8:26 PM on March 30 [1 favorite]
I just wondered if more stringent flagging could have prevented compilation of the dependency and drawn attention to a problem within the larger project (kernel, etc.), that's all.
posted by They sucked his brains out! at 9:57 PM on March 30
posted by They sucked his brains out! at 9:57 PM on March 30
Would compiling with pedantic or all flags turned on have helped catch this (or future attacks of a similar stripe)?
I can't say for this particular case, but the attacker's commit history introduces many small changes to other repositories that either introduce novel exploits or are small steps towards that. I seem to recall the changing of compilation flags somewhere; maybe that's a specific change you're referencing. If that's the case, then yes, that seems to align with their general strategy.
posted by Johnny Lawn and Garden at 10:12 PM on March 30
I can't say for this particular case, but the attacker's commit history introduces many small changes to other repositories that either introduce novel exploits or are small steps towards that. I seem to recall the changing of compilation flags somewhere; maybe that's a specific change you're referencing. If that's the case, then yes, that seems to align with their general strategy.
posted by Johnny Lawn and Garden at 10:12 PM on March 30
the deal with the syntax error introduce in the C code to prevent compilation was that this was in a string in the source code sent to check_c_source_compiles()
This is the typical hacky-ass crap found in C programming since time immemorial, the point of trying to compile this code was just to test if the sys calls in this function:
(I do hope to god the last line of C/C++ code of my programming career will remain well in my past)
posted by torokunai at 10:38 PM on March 30 [5 favorites]
This is the typical hacky-ass crap found in C programming since time immemorial, the point of trying to compile this code was just to test if the sys calls in this function:
+ void my_sandbox(void)
+ {
+ (void)prctl(PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0);
+ (void)SYS_landlock_create_ruleset;
+ (void)SYS_landlock_restrict_self;
+ (void)LANDLOCK_CREATE_RULESET_VERSION;
+ return;
+ }
were 'actually' linkable, because I guess this environment doesn't have an actual direct way of checking this at compile time. The badguy inserted a hard-to-catch syntax error in this string of C code to force this runtime check to fail, which turned off Linux's landlock thing, whatever that is.(I do hope to god the last line of C/C++ code of my programming career will remain well in my past)
posted by torokunai at 10:38 PM on March 30 [5 favorites]
Really it's not C specific here - this is a check to see if the system has the include files installed and the libraries for the kernel landlock support before it does a build, if it succeeds it adds that support - if it was written in java or rust it could equally have done it in java or rust to see if their support was present (and equally have failed because someone had tossed a period in to cause the java or rust compile to fail).
posted by mbo at 10:51 PM on March 30 [1 favorite]
posted by mbo at 10:51 PM on March 30 [1 favorite]
Would compiling with pedantic or all flags turned on have helped catch this (or future attacks of a similar stripe)?
No, at least not in the current state of compilers and build tools. The bad code discussed above is not product code, but build configuration code using the "autoconf" system. The idea is to discover what the system is capable of building, in order to configure the product to include or omit features (in this case, whether to use the Landlock access-control system on Linux). The autoconf approach is empirical—it attempts to compile and run lots of small C programs. Some of them succeed in compiling, meaning that the feature is available, and some don't, meaning that the feature is unavailable. This information is used to configure the product accordingly. So an autoconf script is expected to have a mixture of successes and failures—that's its job—and so it's hard to see how to use the compiler to determine whether each failure is legit (meaning that it has correctly detected that a feature is unavailable) or not (as in this case, where a deliberate bug had been introduced to prevent the feature from being used).
The autoconf approach is designed for a kind of free-for-all in which anyone can download the source code and attempt to build it on their own computer, whatever features it happens to have. This has benefits (people aren't dependent on third-party app stores or package repositories) but also costs, as demonstrated by this attack.
posted by cyanistes at 1:28 AM on March 31 [8 favorites]
No, at least not in the current state of compilers and build tools. The bad code discussed above is not product code, but build configuration code using the "autoconf" system. The idea is to discover what the system is capable of building, in order to configure the product to include or omit features (in this case, whether to use the Landlock access-control system on Linux). The autoconf approach is empirical—it attempts to compile and run lots of small C programs. Some of them succeed in compiling, meaning that the feature is available, and some don't, meaning that the feature is unavailable. This information is used to configure the product accordingly. So an autoconf script is expected to have a mixture of successes and failures—that's its job—and so it's hard to see how to use the compiler to determine whether each failure is legit (meaning that it has correctly detected that a feature is unavailable) or not (as in this case, where a deliberate bug had been introduced to prevent the feature from being used).
The autoconf approach is designed for a kind of free-for-all in which anyone can download the source code and attempt to build it on their own computer, whatever features it happens to have. This has benefits (people aren't dependent on third-party app stores or package repositories) but also costs, as demonstrated by this attack.
posted by cyanistes at 1:28 AM on March 31 [8 favorites]
It's worth noting that MeFi's-own joeyh has been re-building the Debian Developer toolchain from older sources as a path to rebuilding trust in the distro's build systems.
posted by rum-soaked space hobo at 3:25 AM on March 31 [12 favorites]
posted by rum-soaked space hobo at 3:25 AM on March 31 [12 favorites]
Keanu-whoa for the remote access, remote code execution ... plus also subterfuge that could replace parts when decompressing any .deb package installed by dpkg in your Debian-derived (i.e. Ubuntu and many more) GNU/Linux distribution.
Reflections on Trusting Trust.
posted by k3ninho at 3:50 AM on March 31 [1 favorite]
Reflections on Trusting Trust.
posted by k3ninho at 3:50 AM on March 31 [1 favorite]
Regarding sinophobia and interpretations of the motives of Jia Tan, a narrative might be emerging from timestamp analysis of the repository. The offending commits appear to be at an unusual time of day compared to historical patterns, indicating that it's possible a third party hacked Jia Tan's infrastructure somehow and used that to sock-puppet these release artefacts.
But all this is speculation atop more speculation. We'll be studying this for a good long while.
posted by rum-soaked space hobo at 4:16 AM on March 31 [2 favorites]
But all this is speculation atop more speculation. We'll be studying this for a good long while.
posted by rum-soaked space hobo at 4:16 AM on March 31 [2 favorites]
might be emerging from timestamp analysis of the repository
Looks like their usual workday starts at noon UTC, which might put them in... Greenland? Brazil?
posted by clawsoon at 4:31 AM on March 31
Looks like their usual workday starts at noon UTC, which might put them in... Greenland? Brazil?
posted by clawsoon at 4:31 AM on March 31
Somebody named Mike Sass has been putting together a xz/liblzma Compromise Link Roundup.
posted by clawsoon at 4:53 AM on March 31 [4 favorites]
posted by clawsoon at 4:53 AM on March 31 [4 favorites]
The offending commits appear to be at an unusual time of day compared to historical patterns
There are several other possible explanations for this. Maybe the team of attackers split the roles, so that attacker A was responsible for the good commits and other public communication, while attacker B was responsible for the exploit code. Or maybe internal security precautions within the team meant that good commits could be made from multiple locations, while the exploit code was only available on premises.
posted by cyanistes at 5:17 AM on March 31 [2 favorites]
There are several other possible explanations for this. Maybe the team of attackers split the roles, so that attacker A was responsible for the good commits and other public communication, while attacker B was responsible for the exploit code. Or maybe internal security precautions within the team meant that good commits could be made from multiple locations, while the exploit code was only available on premises.
posted by cyanistes at 5:17 AM on March 31 [2 favorites]
what was super-sneaky was the inclusion of tests that would execute non-existent binaries, so that these binaries could be added later and extend the hack.
Evan Boehs (the link in the "long time contributor") was part of the group responsible for the everything NPM hack over the new year. He says he was being funny, but caused actual harm.
posted by scruss at 8:12 AM on March 31 [1 favorite]
Evan Boehs (the link in the "long time contributor") was part of the group responsible for the everything NPM hack over the new year. He says he was being funny, but caused actual harm.
posted by scruss at 8:12 AM on March 31 [1 favorite]
Has anyone found any presence on the Internet for Jia Tan other than their various contributions? some game's Discord? Arguing something unrelated in forums somewhere? The absence isn't itself damning, but is consistent with a constructed identity used for malicious purposes. I think it's pretty rare, actually, for developers working on open source software to have zero footprint outside of pull requests.
posted by fatbird at 8:45 AM on March 31 [2 favorites]
posted by fatbird at 8:45 AM on March 31 [2 favorites]
The second link in the post includes a bunch of detail on what's known about "Jia Tan". It's complicated and quite thin, identity is such a slippery thing online. But it notes their presence on an xz-related IRC channel and a LinkedIn profile that may or may not be the same nym. Not at all clear whether the Jia Tan nym that was used for libxz was always the same actual person. Probably not, if it's an APT that did the exploit work.
posted by Nelson at 9:43 AM on March 31 [1 favorite]
posted by Nelson at 9:43 AM on March 31 [1 favorite]
This is the kind of post I come to MetaFilter to read. I can't make sense of 99% of it but nevertheless it is fascinating. I have never compiled anything, have no knowledge or experience of programming aside from some very basic BASIC in the late 1970s, and find it all very mysterious albeit at the very heart of our current existence.
I do love argot, though, especially when spoken by mystics, and this is like the purest: In July, a PR was opened in oss-fuzz to disable ifunc for fuzzing builds, due to issues introduced by the changes above. This appears to be deliberate to mask the malicious changes that will be introduced soon. Also, JiaT75 opened an issue about a warning in clang that, while indeed incorrect, drew attention to ifuncs.
posted by chavenet at 12:36 PM on March 31 [14 favorites]
I do love argot, though, especially when spoken by mystics, and this is like the purest: In July, a PR was opened in oss-fuzz to disable ifunc for fuzzing builds, due to issues introduced by the changes above. This appears to be deliberate to mask the malicious changes that will be introduced soon. Also, JiaT75 opened an issue about a warning in clang that, while indeed incorrect, drew attention to ifuncs.
posted by chavenet at 12:36 PM on March 31 [14 favorites]
Thinking about how a few years back we learned the NSA had a wonderful collection of exploits and tools... why couldn't this be a US or "Western" effort? If I were them I'd misdirect by picking names that suggest elsewhere too.
posted by i_am_joe's_spleen at 12:37 PM on March 31 [3 favorites]
posted by i_am_joe's_spleen at 12:37 PM on March 31 [3 favorites]
Looks like their usual workday starts at noon UTC, which might put them in... Greenland? Brazil?
I could be wrong, but I think the part of interest starts just after 00:00 which is midnight. Worth noting that China is UTC+8 and NK is UTC+9. Of course I would assume anyone doing this wouldn't be just working 9-5 but still something of note.
This is the kind of post I come to MetaFilter to read. I can't make sense of 99% of it but nevertheless it is fascinating
Agreed. I don't understand much of it either, I haven't written code in over a decade (and even then was mostly hacked-together C) and only have a basic understanding of Linux. But despite not having extensive software development experience, I have wondered about this kind of thing before - what's to keep someone from tossing in a backdoor or something else nefarious? Much like the original underpinnings of the Internet, it seems like the system for developing code is set up assuming everyone has good intentions.
posted by photo guy at 2:01 PM on March 31
I could be wrong, but I think the part of interest starts just after 00:00 which is midnight. Worth noting that China is UTC+8 and NK is UTC+9. Of course I would assume anyone doing this wouldn't be just working 9-5 but still something of note.
This is the kind of post I come to MetaFilter to read. I can't make sense of 99% of it but nevertheless it is fascinating
Agreed. I don't understand much of it either, I haven't written code in over a decade (and even then was mostly hacked-together C) and only have a basic understanding of Linux. But despite not having extensive software development experience, I have wondered about this kind of thing before - what's to keep someone from tossing in a backdoor or something else nefarious? Much like the original underpinnings of the Internet, it seems like the system for developing code is set up assuming everyone has good intentions.
posted by photo guy at 2:01 PM on March 31
I could be wrong, but I think the part of interest starts just after 00:00 which is midnight.
Indeed. I was looking at the two years of trust-building, where there seems to be a clear daily starting line of 12 noon UTC.
posted by clawsoon at 2:05 PM on March 31
Indeed. I was looking at the two years of trust-building, where there seems to be a clear daily starting line of 12 noon UTC.
posted by clawsoon at 2:05 PM on March 31
a clear daily starting line of 12 noon UTC
Which by the way is 8am US Eastern Time. Which is where both NYC and Washington DC are located. Although it could equally be someone in a different place time-shifting their day. Hard to get much info from the timing alone.
posted by Greg_Ace at 2:23 PM on March 31 [2 favorites]
Which by the way is 8am US Eastern Time. Which is where both NYC and Washington DC are located. Although it could equally be someone in a different place time-shifting their day. Hard to get much info from the timing alone.
posted by Greg_Ace at 2:23 PM on March 31 [2 favorites]
the everything NPM hack
I hadn't heard about this, but it seems like (with the best of intentions) npm built into their package-management system a pretty weird rule: if package A depends on package B, that has impact on the management of package B. I get why they did that (to prevent breaking changes to/removal of widely used packages), but from a social-philosophy point of view, it's kind of asking for trouble; if you have an open publishing environment, and anyone can contribute content, project A should only be able to have an effect on project B by project B's consent, either explicit (i.e. the maintainer signs off on such a relationship) or implicit (if project B depends on project A --- not the other way around --- that is intrinsic control project A's maintainers have over project B).
Like, having to trust your dependencies, while not ideal (as we can see here with xz) is a system that more or less makes sense, because a dependency is a choice the maintainer makes, and if it's a bad one, at least it's theirs. Having to trust things which depend on you is just crazy, though, because a project maintainer can't, in an open-source environment, control who chooses to depend on them.
posted by jackbishop at 2:47 PM on March 31 [3 favorites]
I hadn't heard about this, but it seems like (with the best of intentions) npm built into their package-management system a pretty weird rule: if package A depends on package B, that has impact on the management of package B. I get why they did that (to prevent breaking changes to/removal of widely used packages), but from a social-philosophy point of view, it's kind of asking for trouble; if you have an open publishing environment, and anyone can contribute content, project A should only be able to have an effect on project B by project B's consent, either explicit (i.e. the maintainer signs off on such a relationship) or implicit (if project B depends on project A --- not the other way around --- that is intrinsic control project A's maintainers have over project B).
Like, having to trust your dependencies, while not ideal (as we can see here with xz) is a system that more or less makes sense, because a dependency is a choice the maintainer makes, and if it's a bad one, at least it's theirs. Having to trust things which depend on you is just crazy, though, because a project maintainer can't, in an open-source environment, control who chooses to depend on them.
posted by jackbishop at 2:47 PM on March 31 [3 favorites]
Back in '16 the author of the "left-pad" npm package (all 11 lines of it) revoked it out of pique , and it broke everything for a few days.
posted by torokunai at 8:06 PM on March 31 [1 favorite]
posted by torokunai at 8:06 PM on March 31 [1 favorite]
I've seen the point made that, for all the sophistication of this attack, it would have been detected very quickly if it had made it onto production servers. The internet is full of SSH scanners. If you increase the CPU time of a failed connection from 0.3s to 0.8s, that's going to show up in load monitoring of high-traffic servers.
Equally, have also seen suggestions that moves afoot in libsystemd to remove xz as a static dependency prompted the attacker to rush the work to get this into distros and avoid losing two years' investment.
posted by Klipspringer at 2:16 AM on April 1
Equally, have also seen suggestions that moves afoot in libsystemd to remove xz as a static dependency prompted the attacker to rush the work to get this into distros and avoid losing two years' investment.
posted by Klipspringer at 2:16 AM on April 1
Klipspringer: Equally, have also seen suggestions that moves afoot in libsystemd to remove xz as a static dependency prompted the attacker to rush the work to get this into distros and avoid losing two years' investment.
Would it make more sense to remove the systemd dependency from sshd? It seems that systemd will always be a very large attack surface, and I presume (without knowing much about it) that one would want sshd to have the smallest attack surface possible.
posted by clawsoon at 6:47 AM on April 1 [2 favorites]
Would it make more sense to remove the systemd dependency from sshd? It seems that systemd will always be a very large attack surface, and I presume (without knowing much about it) that one would want sshd to have the smallest attack surface possible.
posted by clawsoon at 6:47 AM on April 1 [2 favorites]
A lot of conversation about this in the Fediverse.
posted by brainwane at 8:02 AM on April 1 [3 favorites]
posted by brainwane at 8:02 AM on April 1 [3 favorites]
yeah happy to see this community apparently having moved off of xitter to some extent.
Even has a happy Model Y owner I have a hate/hate relationship with that man.
posted by torokunai at 8:50 AM on April 1
Even has a happy Model Y owner I have a hate/hate relationship with that man.
posted by torokunai at 8:50 AM on April 1
A more detailed time analysis: XZ Backdoor: Times, damned times, and scams
The author points out that "Jia Tan" never took Chinese holidays off, but always took Christmas and New Year's off.
However, they've concluded that the usual working day for Jia Tan was in the +0300 time zone, i.e. Eastern Europe, rather than -0300. This seems wrong to me based on the commit time graph above, and I wonder if they've mixed up their time zone math.
posted by clawsoon at 10:09 AM on April 1
The author points out that "Jia Tan" never took Chinese holidays off, but always took Christmas and New Year's off.
However, they've concluded that the usual working day for Jia Tan was in the +0300 time zone, i.e. Eastern Europe, rather than -0300. This seems wrong to me based on the commit time graph above, and I wonder if they've mixed up their time zone math.
posted by clawsoon at 10:09 AM on April 1
Imagine you're a bicycle mechanic. You've always wanted to be a bicycle mechanic. THere's nothing you love more than watching a customer pedal away on a newly tuned up bike. But, owing to circumstances, you're forced to train and retrain on the finer points of choosing, maintaining, repairing, and removing rust from bike locks. And knowing which ones are no longer good because thieves are good at defeating them. The topic bores you to tears, but if you want your customers to come back, you have to be good at helping them maintain possession of their bikes.
This is what many of us software developers are feeling like right now.
posted by ocschwar at 12:21 PM on April 1 [6 favorites]
This is what many of us software developers are feeling like right now.
posted by ocschwar at 12:21 PM on April 1 [6 favorites]
Unix expert rsc (Russ Cox) has weighed in with an analysis: timeline, shell code. He's someone worth paying attention to if you want technical detail.
posted by Nelson at 10:09 AM on April 2 [2 favorites]
posted by Nelson at 10:09 AM on April 2 [2 favorites]
Unix expert rsc (Russ Cox) has weighed in with an analysis: timeline, shell code. He's someone worth paying attention to if you want technical detail.
Nice to see him covering all the psychological pressure put on Lasse, too. At this point I'm assuming that there's a spreadsheet somewhere containing a list of OSS maintainers with a column for "exploitable mental health issues".
posted by clawsoon at 3:01 PM on April 2 [1 favorite]
Nice to see him covering all the psychological pressure put on Lasse, too. At this point I'm assuming that there's a spreadsheet somewhere containing a list of OSS maintainers with a column for "exploitable mental health issues".
posted by clawsoon at 3:01 PM on April 2 [1 favorite]
Someone on Hacker News commented that xz and lzma are compression tools falling into a particular category: they're merely okay at compression, but they're very fast, so they're easy to add at each of a pipeline (compress, send decompress) to get a significant and nearly free reduction in the size of transmitted/written data. xz and lzma are widely used and if this hack hadn't been discovered, would be a great place for a backdoor. The significance of SSHD is that 1) it uses lzma and 2) it normally runs as root, so if you can trigger this exploit, you have root access to any computer on the Internet for which public access by SSH is allowed, if it uses the backdoored compression libraries.
No, xz/liblzma is usually used for its high compression ratios despite being slow.
SSH doesn’t directly use liblzma. SSH compression is done using the same algorithm as used in gzip (the zlib algorithm). It doesn’t even use zlib itself and re-implements the algorithm in its own code, and they’ve clearly put a lot of thought into how that sort of thing should be happening.
As already mentioned upthread its only versions of OpenSSH that linked to systemd, and systemd linking to liblzma that made it vulnerable and bog standard OpenSSH doesn’t do that.
posted by edd at 6:36 PM on April 2 [2 favorites]
No, xz/liblzma is usually used for its high compression ratios despite being slow.
SSH doesn’t directly use liblzma. SSH compression is done using the same algorithm as used in gzip (the zlib algorithm). It doesn’t even use zlib itself and re-implements the algorithm in its own code, and they’ve clearly put a lot of thought into how that sort of thing should be happening.
As already mentioned upthread its only versions of OpenSSH that linked to systemd, and systemd linking to liblzma that made it vulnerable and bog standard OpenSSH doesn’t do that.
posted by edd at 6:36 PM on April 2 [2 favorites]
I don't know if it was just to complete the illusion, or if "Jia Tan" was feeling creative one day, but they made a logo for the xz project.
posted by clawsoon at 4:34 AM on April 3
posted by clawsoon at 4:34 AM on April 3
From what I'm reading, the timestamps on Git commits would've been set on Jia Tan's machine, so there's no guarantee that they're not shifted (in addition to the timezone they applied).
That gave me the brilliant idea as I woke up this morning to look at the full headers of emails that they sent to xz-devel, since there'd be email server "Received:" header timestamps that couldn't be shifted on the client side.
But I've been thwarted, since the only archive I can find is the one at mail-archive.com, and it doesn't appear to have a way to show full headers.
Can anybody figure out a way to get at those email headers?
posted by clawsoon at 4:40 AM on April 3
That gave me the brilliant idea as I woke up this morning to look at the full headers of emails that they sent to xz-devel, since there'd be email server "Received:" header timestamps that couldn't be shifted on the client side.
But I've been thwarted, since the only archive I can find is the one at mail-archive.com, and it doesn't appear to have a way to show full headers.
Can anybody figure out a way to get at those email headers?
posted by clawsoon at 4:40 AM on April 3
Luis Villa and I are friends so it's not surprising that I pretty much agree with what he wrote about the xz incident in his capacity as co-founder of Tidelift. (Disclaimer: I sometimes do a little consulting for Tidelift.)
posted by brainwane at 4:46 AM on April 3 [2 favorites]
posted by brainwane at 4:46 AM on April 3 [2 favorites]
Regarding timestamps and location, here's a take about how that looks if you don't assume "Jia Tan" was doing this on the clock.
posted by trig at 5:05 AM on April 3
posted by trig at 5:05 AM on April 3
Wired: The Mystery of ‘Jia Tan,’ the XZ Backdoor Mastermind. No original reporting but a good summary of all the sleuthing going on, with a better big picture than I've seen anywhere.
In fact, the only real footprints Jia Tan appears to have left behind were their contributions to the open source development community, where they were a prolific contributor ... In total, Jia Tan made 6,000 code changes to at least seven projects between 2021 and February 2024 ...posted by Nelson at 4:20 PM on April 3 [1 favorite]
it’s unlikely that Jia Tan is a real person, or even one person working alone. Instead, it seems clear that the persona was the online embodiment of a new tactic from a new, well-organized organization
Kevin Roose in the NYT has a summary of the xz hack and discovery that may be useful if you're trying to explain it to your less technically minded friends.
posted by Nelson at 10:21 AM on April 4 [2 favorites]
posted by Nelson at 10:21 AM on April 4 [2 favorites]
the xz sshd backdoor rabbithole goes quite a bit deeper. I was just able to trigger some harder to reach functionality of the backdoor. there's still more to explore.. 1/n [X thread]
posted by chavenet at 3:52 AM on April 7 [2 favorites]
posted by chavenet at 3:52 AM on April 7 [2 favorites]
« Older A computer that could expand with the addition of... | Hallucination attack Newer »
posted by torokunai at 2:31 PM on March 30