Hello, I'm Rob Hirschfeld, CEO and co founder of rockin and your host for the cloud 23 podcast. In this episode, we discuss the SSH intrusion attack that was introduced just as April 2 So was introduced into the Linux kernel. we dissect what the attack is, how it happened, what the social engineering was, and the pressure that involved to make that happen. It's about a 17 minute discussion. We this was our intro for our technical talk, but it was such an important concept captured that I want to set it up as a dedicated special cloud 2030 topic I hope you enjoyed the discussion
the elephant in the room is the x v or issue with delivery.
I am not following that the x x v. So
it oz sorry, the the compression library used by some part that was included and been some Linux packages. On in de Vienna it was also tied in into SSH and turns out that one of the XE maintainers introduced a backdoor. Oh, for SSH
consensus so far was that it was a really epic case of social engineering over a multi year period to get me to interaccess. Um, basically by social engineering the burned out maintainer of XE and then leveraging that to inject some really cleverly hidden backdoors that took advantage of some some kind of system wide, free link optimization steps. It was there have been a lot there's been a lot of stuff on Hacker News about it. And it's basically the first widely distributed supply chain attack on an open source project
with a CVS a score of 10. So it's the worst possible score. Yeah.
So he this, this pseudo maintainer, I guess, our hostile maintainer was able to inject actual code like, like takeover vulnerabilities.
Yeah, basically, he arranged things so that backdoored X backdoored, lib LZ Ma. So that it could take advantage of a piece of functionality called iPhone, which allows you to substitute functions for platforms is it civic optimized versions of them really early in link time. utilized a combination of that system D pulling in lib LZ MA for compressed log handling, and a common distro setup where they would patch open SSH to pull in lib system D so that they could have daemon notification, and chain also the chain that all together to hijack part of the key handshake to detect when someone was trying to log in with a private certificate that had an exploit payload into it. And if they did that, then SSH would just exec the payload directly. It was an incredibly sophisticated attack that was only foiled because he introduced a point five second delay in startup and someone was wondering, you know why that happened while troubleshooting something unrelated in Postgres, and
this was a trusted account to there was a co maintainer, one of two maintainers. Yeah, long form, like multiple years, he's been around,
well engineered himself into becoming one. So
he identified a vulnerability and then identified a social engineering approach to be able to explore wholesale.
So there's an ability this was a backdoor built into the library, that would be triggered when used under certain conditions by the SH T. Right. But that
person had to identify,
Oh, he didn't identify anything equated. He
did. But that sounds like a he identified a place where he could inject this weakness And then set out to find the font, you know, because it not only had to be a vulnerability that he could add this this backdoor payload into, but it also had to be a on a on a library that where he could social engineers way ahead, right? It was Yeah.
Wow, this was a maintainer who the original maintainer takes frequent breaks from the internet, because he has a lot of mental health issues that he was trying to work through and, and whatnot. And as part of that, there's constant pressure from customers in the community says, Hey, we've got all these things that need to be maintained, you're taking too long, what are you going to do about it, they actually are following some of this. And they believe it's actually the guy or the people, the actors creating that pressure, which then allowed him to become a co maintainer, basically said, Hey, I understand your problem. How about I come in and help because the community still needs this. I don't know how they all came to agree to that. But basically, he worked his way in, well,
that this wasn't a single person doing this, like that this has all of the hallmarks of being state actors.
Yep. Yep. That's the that's the consensus at this point is that he was targeted by some state, agency or
other incredible story,
absolutely incredible story. But it also shows you just how much of the infrastructure we rely on back to that XKCD cartoon of how much of the infrastructure we rely on has down to us one, two, or, you know, re maintainers for those libraries, especially for old and stable, you know, for long term stable libraries that don't need necessarily a ton of bandwidth. But they do need a ton of architecture and, and expertise. And
also goes to show how much trust is placed not just in the package containers, but in the source code and containers. Because both the Debian once a testing are unstable. And I've I think a state was to another effect that said, and underlays Fedora, they were just blindly pulling the source code and not really doing much verification there.
Well, they were verifying the source code. The problem is the piece that was there was a binary blob that was for testing. And what the guy had done or the actors had done was very subtly make some small changes. Days before that, basically disabled the check to make sure that it was a valid file so they they just pulled the file down. And added that in and as part of the build process. Instead of trying to rebuild everything from scratch. I just pulled down the binary and as part of their build process injected it into the code. So whether or not they they verified the code itself they did not know it was in the binary and had trust that it was okay.
I did was a very solid change to the central just add one dot that made the file and login violet. And
that's the key you have to know that invalidates that file. Yeah,
it was a very sophisticated attack.
That was like how long was it in in place before it was detected? Three days that's incredible how that it was caught that fast. Your luck? Even
the person who found it was like totally
it hadn't in a second delay and starting in SSH initializing whatever it was opening a new a new connection no one would have found it out for a while now.
But I was actually talking to the person on the found that on on mastodon. The other new serious thing about this is that it had a lot of sophisticated evasion techniques. Like for example, a, if it detected a term or a shell or debugger, it would not execute the vulnerable code path. So Oh, interesting. Yeah. Yeah.
Wow. Only it was only baked into the library if it was being built as part of a Deb or an RPM package. If you were running the test suite, it would deliberately disable some checks that are designed to catch the sorts of bugs or vulnerabilities rather, holy cow. Yeah, it was it was very sophisticated, and, you know, put together over the course of a couple of years, including social engineering the maintainer to accept that to begin with
But on on the flip side, this would not have been able to evade ebpf based detection techniques, because that's done direct and the current
fuzzing, that was one of the things he did was specifically that attack disabled fuzzing test.
The author, there's been a lot of chatter in the community about this. Also about, well, the value of bill of materials have come far back, you need to go to verify trust. It's
yeah, I've seen a lot of citations on the good old Reflections on Trusting Trust paper.
Yeah. Like my takeaway on this is a lot of people are have been saying, like, well, like software bill of materials will not have caught this and why would you even use overbroad materials? If this hasn't caught? My senses that the opposite that like it, the goal of the bill of materials, whatnot, was not to catch this? Is that now that you know it exists? Let it be so much more agile, determining which of your environments are affected, right. And software
bill of materials is as much about remediation as it is about reduction of harm that exactly. But yeah.
The papers coming out of this.
It's fascinating, and that is this mostly Debian how many distros to cut to the touch.
Fedora,
at least once were fedora. I think Tuesday, a couple of weed was also affected and unstable was affected. Yeah.
It targeted glitzy, so any missile based distro, like Alpine was not affected. Funnily enough, Nix OS, it did have the vulnerable source code. But because Nix uses different paths. The backdoor code would have never been executed.
Yeah. And was also the injection only happened if you were building for a dev or an RPM. I don't think it targeted any other package management solution. Interesting. If
you if you weren't using SSH, you wouldn't be vulnerable, right? This was wrote on the SSH interface. The
backdoor was specifically for SSH. Yes. But I mean, that's what do you really trust the library after
this? No, no, it's it just it's one of those things that to me is a little bit funny in that I'm just thinking about like Ansible versus other management tools. And one of the things ah, you know, the agentless agentless aspect of SSH that everybody talks about where it's really just running SSH as an agent, but versus an agent. You know, that's, that's actually specific to the platform. And so it's it's one of those things where that's writing on a general purpose configuration port, in a way. That's fascinating. And then, last question, before I turn the turn us to the topic, is this sign that that I heard also requires you to use a custom certificate for this? Or how did they inject the payload and the certificates,
but anything in this certificate? Sure.
But I mean, if if I'm generating my own certificates, I'm assuming I have some degree of trust on that.
You would only accept the payload from a specific
certificate. So so it would be a client certificate, not a source. Okay.
Yeah, so you would present a specifically crafted client certificate that would authenticate against a public key that I believe was embedded as part of the exploit payload, or as part of the payload and if that was present, then SSH would execute whatever the certificate Polartec
so this wasn't carrying on your logins. What it was doing it was basically making SSH somebody could present a certificate that would then bypass all the all that basically all the authentication and process because of the payload and the certificate. Yep. And I just realized why that's a 10. Mackerel, yeah. So you're basically knocking on your you're knocking on the door, you're knocking on the door and the pattern invokes the bat and you're on the front door and that invokes the backdoor.
I had almost installed Fedora 40. The day before all that happened. Thankfully, I did not I made it aidid obvious choice, work work stability is very, very important. So
yeah, another another benefit to not having your servers on the Internet. Because even if nobody can get there, there, you're still protected.
And it would have been easy to exploit because SSH servers, public internet have to fend off, you know, Port knocking into zeros and random SSH attacks all the
time. Yeah. Yeah. And you wouldn't even notice necessarily, because there's a flood of those knocks anyway.
Yeah, wow.
Wow, it's fantastic to have these weekly sessions, where, when news of the day occurs, we can come back and zero in on exactly what's going on what the vulnerability is how it happened. Really powerful to be part of a community that's able to cover this type of material. So I hope you enjoyed this conversation. If you want to be part of these dialogues, join us at 2030 dot cloud. We post our schedule you can come in, we always spend a couple minutes. They don't always make it in the recordings like today. But we always spend a couple minutes talking about what's going on in tech, and bring each other up to speed. I hope to see you there. Thanks. Thank you for listening to the cloud 2030 podcast. It is sponsored by rackin. Where we are really working to build a community of people who are using and thinking about infrastructure differently. Because that's what rec end does. We write software that helps put operators back in control of distributed infrastructure, really thinking about how things should be run, and building software that makes that possible. If this is interesting to you, please try out the software. We would love to get your opinion and hear how you think this could transform infrastructure more broadly, or just keep enjoying the podcast and coming to the discussions and laying out your thoughts and how you see the future unfolding. All part of building a better infrastructure operations community. Thank you