xz and liblzma: Exhausted by the Destruction of Trust
XZ for Java toot - @glyph@mastodon.social
Like a lot of technical people, I spent a bunch of time scrolling yesterday, reading about CVE-2024-3094. The short story: someone by the name or alias of Jia Tan started committing code to the xz project (specifically the liblzma library) a couple years ago, and recently it was discovered that some of this was to do with disabling security to allow for remote access bypassing ssh. Pretty sophisticated stuff. The first thing I saw was a Mastodon toot, which took me to a few more, then the requisite Hacker News thread(s). And I despaired, because xz is used a lot (basically, everywhere), and even if the affected version was only being used by bleeding-edge distros (Fedora 40 and 41, Debian Sid, and a few others), the end-game was clearly to try to evade detection long enough to have a backdoor into a massive swath of Linux machines across the internet.
I'm so fucking tired. While this is a sophisticated attack, and while people are trying to figure out the deeper details (who is Jia Tan; IS there a Jia Tan; is the person involved with organized crime, a state, etc), mostly I'm just thinking about how this is another way that sociopaths have attempted to control and shape the use of what could've been a golden communication medium. I feel like every day my original hopes for the internet slip further into the distance, and while a lot of that is the naivety of my then-youth, I should point out that in the 90s, a lot of us thought that the internet, like the printing press before it (and it was always compared to the printing press!), would help shape the world in a genuinely good way. Openness, democratization.
The last couple of decades have seen the rise of Open Source, but like seemingly everything else in our society, it seems to be predicated on the free labour of an particular group of people, in this case, developers passionate about improving the software available to anybody who wants it. This is a vanishingly small proportion of the overall population, as most of us tap mindlessly on our phone, watching cats fail in inventive new ways, but there are still a lot of us who care about the software we use. Not that I'm perfect in any sense: most of the time I'm using Windows, because for the last decade and a bit, I've been working on a big software project using Visual Studio. With apologies to CLion et al, Visual Studio is still the best C++ IDE available, and Windows the most important platform for what I make. I don't know, maybe that'll change eventually. Certainly we've hit the era where so much of our software is either web-based, or web-technology-based. But I still use Windows day-to-day, even if, with the same frequency, I still log into rawtext.club and elsewhere and happily use Linux and the command line.
We've seen the rise of Open Source and we've also seen the rise of the harried Open Source maintainer, often a single, scruffy person, doggedly working on the software because they inexplicably love that mess of code, and if nothing else in this world belongs to them, that does, at least in a sense. And entire corporations and industries have been based using and repackaging other people's work - see Amazon Linux, the distro used by that corporation for its VMs and Lambda fuctions that's based of Fedora and CentOS and who knows what else. Amazon makes many, many billions of dollars a year, while most OSS maintainers just want to pay their bills and their mortgages. They'll sometimes list Paypal links or whatever on their project pages, and more depressingly, sometimes list the totals.
Message Manager (MQ-agnostic manager)
At my last gig, in the insurance industry, we used a tool called Message Manager. We used it for managing the Sonic Message Queues associated with Aurea's Sonic middleware product. This was a piece of software they acquired from Progress Software, and kind of left to rot. It never had great support for actually inspecting what was on the queues at any given point in time, which is nuts for a product that was heavily based around an implementation of JMS. But a Dutch programmer, Gerco Dries, created a open source program to allow working with Sonic's message queues, as well as those of ActiveMQ, and other JMS implementations. It's not a simple piece of software, but it's a simple concept. It filled a huge need. It was a vital part of our day-to-day operations. In the last seven or eight years, it's made $255 across 5 donations.
That won't even cover domain registration and hosting.
Technologist vs spy: the xz backdoor
This is the environment into whomever Jia Tan was entered. From what we know right now (and to be fair, it's day 2 of this exploit), it looks like Tan exploited Lasse Collin, the former maintainer of xz, who it appears was going through some health issues and was falling behind. Or, that's the story - according to lcamtuf's substack, several sock-puppet accounts popped up, pressuring a change in maintainer. It appears Collin relented. And this is what really bothers me about all this, that real people were exploited, and hurt, so that someone could try to work in a backdoor for reasons unknown.
I say reasons unknown, but it's also well-known at this point that states have kind of a list of secretly-guarded exploits and backdoors in case they're ever needed. I'm sure, if this is state-sponsored, that whomever did this did so in the belief they were doing something morally (eventually) good. An "ends justify the means" sort of thing. But that's just not the case: what they were doing, as they've ever been doing, is fucking things up for everybody else. Making decisions that were once easy (trusting someone who said they wanted to help), hard.
This is the sort of thing that has long-term consequences, not just on harried maintainers like Lasse Collin (who will, unfairly, always be tied to this), but on developers everywhere, when every interaction is under the pall of uncertainty, of "is this person who they say they are?"
It's awful that people are happy to hurt other people just to gain some sort of advantage.
It's never been the case that people online were uniformly well-intentioned. But the CVE process didn't even exist until 1999, and HTTPS wasn't a real standard until the 2000s. People used to literally just run Apache on Linux boxes in their basements, and serve over HTTP, and everyone was fine. It wasn't perfect, but it was good enough.
Over the last couple of decades, we've seen an increasing focus on security and scrutiny. Part of this is due to the way that software has become a part of the fabric of our lives, but just as much it's because of why it's needed: because there are people out there, constantly seeking to hack into systems, gain unauthorized access, use it for shady or shitty purposes. This was always the case, but it's more so, now. When governments have entire divisions devoted to what would in any other context would be labelled "cybercrime", it's a constant push and pull. Exploits are reported, classified, patched ASAP; but these are only the ones we know about, the ones that were found.
And that's the other part of this: just how many Jia Tans are there out there? How many people have found clever ways to work their way into the projects that comprise the infrastructure of open source, injecting malicious code to allow them to do, well, whatever? This, I think, is the real reckoning: not that one person was able to do so, using manipulative methods to cause a maintainer to acquiesce and step down, but that we, the collective we, people who give a shit about software and who should be distrustful of centralized control, didn't consider the possibility sooner, weren't paranoid enough. I hate that we have to be. We shouldn't have to assume distrust. And this incident, and surely others like it, will linger like a question mark over our own individual interactions going forward.