Hello, I'm Rob Hirschfeld, I'm the CEO and co founder of RAC and in your host for the cloud 2030 podcast. In this episode, we dive deeply into DCP Pixie, I picked the UEFI BIOS and all of the things necessary to do network installs of servers, and incidentally, thin clients and PCs and other network switches and things like that. But specifically, we talked about the process of having secure and robust network provisioning. For infrastructure, we get down down deep, we talked about all the pieces that you need to know how the processes work, both in legacy and in modern current systems, and know you will get a lot out of this discussion.
I'm gonna bring this if Sorry, I'll start a little bit less technical. And then I'd love to have one of our team jump in and walk through the process because it's not as simple as I'm going to make it sound. Right. So the pixie DHCP. Question here is all about how do you start a machine that doesn't have an operating system on it? And what is the sequence of steps that you go through to install an operating system, or an operating environment you want on an ultimate on a machine. And that's fundamentally the problem that we're trying to solve. And there's a couple of ways to solve it. That and we've worked through many of them, going all the way back to my early days, where you can walk up with a USB stick and install something. What we're trying to do here is do something that doesn't require you to physically access to the device. So the process we're talking about here is to be able to do it from having the machine plugged in, attached to a network, and then initiate a sequence of events that installs not the operating system. And for that reason, and one of the things I think people don't, don't think about that much is that pixie is really starts from the network topology, and the NIC card even from a from an onboarding perspective. Does somebody on the RackN team want to walk through the sequence? I know we have graphics, let me see if I can pull up a graphic. Although I'd rather us walk through it because I'm just because of the record the way we're recording this, anybody want to start? They can take a pass on the sequence. I was thinking starting
from DHCP. Okay, how detailed Do you want me to be pretty detailed,
I think it's helpful. Okay.
Um, so to start off with DHCP is where everything starts. DHCP is all about being able to configure hosts to talk to their local network. These days, that's pretty much all ipv4 with some ipv6 thrown in. But we don't handle that yet. And what DHCP is responsible for is allowing a host that doesn't have any IP address to get one along with all the other options that needs to communicate with the local network and the wider and beyond. So DHCP is responsible for handing out IP addresses. It's responsible for handing out various options that control what DNS server you're going to use, what router you're going to talk to, and other local service things. DHCP has a protocol is pretty old, it kind of is an extension to an even older protocol called boot P which we won't get into because it's only relevant to exactly one use case that we cover. And nothing else anymore.
Forgotten for I forget about boot pa You're right. Yeah,
yeah, um, that dependency is completely gone. And DHCP version six, because no one cares about boot anymore, except for a very large blue three letter company. But if you get on the internet, these days, you're using DHCP for pretty much everything on your local laptop and in an awful lot of server environments as well because no one wants to manually configure systems to get on the internet. And so where pixie comes into play, is DHCP. In addition to handing out the IP addresses, it has a pretty flexible space for passing for the clients to say what options they're interested in getting, and for DHCP servers to satisfy those As options, and pixie is essentially DHCP, plus a couple of us a couple of bonus options that tell the system where to download a file and how to execute it. In the graph that we're talking about, see sort of sits embedded in DHCP. With a little bit of the second stage for TFTP. One of the options that a server can ask for is give me a file to boot from. And that is where pixie comes into play. For both legacy BIOS systems, and for UEFI systems, Pixie is accomplished from a network and a kind of a, an end user standpoint, by downloading an executable, or the system that's trying to boot and then running it. And after that, it's up to you well, that gets downloaded to handle everything else in the process. If demo that are the kind of the graph that Rob is showing, is for a legacy BIOS boot because it's asking for L pixie Linux dot zero and that is only that particular bootloader only supports legacy and I'll Pixy Linux has its own flow for getting the rest of the boot config files. That's not going to be the same across all the other boot loaders.
So, I mean, part of the reason it's like this is because we're talking about starting from firmware, which historically is meant to be pretty lightweight and simple. So it's TFTP is, you know, it's a UDP based protocol. It's super minimal from control. Yeah.
You mentioned USB keys. This process is older than USB key.
Okay, fair enough. USB keys did not exist while I was in the first time that I was pixie booting your right is useless, I says, drives
all CD ROM drives or piles of floppy diskettes. That's what
Yeah, so there's even more like a USB would be like, Oh, that's not so bad. At some point, I want to talk through the the idea actually, let me do it this way. So what if you know, if you wanted, could you just put L pixie on a USB stick and say boot from this every time and leave it leave it installed? How would that work? Like skip this first TFTP step.
You couldn't put L pixie on a USB stick but you could put ISO Linux on a pixie stick which looks almost exactly like a pixie except it loads from except it will pull things from a localized instead of communicating, communicating over the network. It makes sense. That's LPC specific. If you use something else like I Pixie, which is another pretty popular bootloader that works for both that has various legacy boot UEFI not insecure mode for x86 and four arm 64 You can do that you can burn a tiny little UEFI executable onto a USB stick or even a CD ROM with an embedded config file to say go finish doing the rest of your stuff on this server over there.
How does it get the information it needs to fit into this? So that's just basically a kernel to get you started. How does it know what to do next? Like do you still then you still have to provide it some like get on the net, you still get it on the network and provide some options. Right.
So that's one of the key differences between how l Pixy Linux or you know isolinux would work in this case versus I pixie is that L pixie Linux uses hashed information that it gets from the DHCP stuff. It doesn't go off and do its own second DHCP handshake to get more information. It relies on its own little config thing. So if it wasn't booted from DHCP, it won't know it won't have the options there. It won't know where to go talk to a TFTP server with files. With I pixie. Whenever I pixie loads, it doesn't know the DHCP handshake with a different set of options that the DHCP server can recognize and say, Oh, I've already done I've already loaded the I pixie boot loader, and now it wants a config file. So I'm going to hand back the config file for instead of the pixie boot loader again and continue on my merry way with whatever config file it serves me. So
it's actually a different DHCP interaction from that perspective. It's new, doesn't have to get a new address but the DHCP server needs to understand that it's it's an environment
what's the memory and executing do whatever it wants. Okay, so that's how I think it would work in this scenario your friend.
Doesn't I pixie have like skip the whole TFTP and just go h2 HTTP or HTTPS?
Yes, because it includes its own network stack. I relying on the firmwares network stack to do all of its work. It has its own little HTTP client, it has its own TFTP client. In some cases, it even has its own drivers that you can use in place of whatever the firmware provides. And
is it still a specialized OS from that perspective? Like is basically a bootstrapping OS? Or is a, you
could call it that. It's very limited in what it can do. But yes, you could call it that.
And then let's not forget also the one of the major use cases of pixie back in the day was for 10 clients.
Mm hmm. Yeah. So they could download with download whatever firmware blob they needed to do their business and download whatever configuration they needed for their local situation and do whatever they need it. We don't really support them clients like that, mostly because it tends to get into hyper specialized emitters specific stuff real fast. But the standard server pixie pads are they've been standardized for, what, 25 years now?
What do you what do you mean by a specialized path?
Um, oh, going back to what I said earlier, the basic idea of pixie is, you know, DHCP tells you to download an executable that will do a thing that executable does and how it handles the next step is totally up to the executable. In that sense, xe kind of stops once the initial boot loader is loaded into memory of running at that point, you're doing whatever the bootloader wants. For the common server boot loaders, those behaviors are also standardized around, go get a config file and do whatever it says to do, which is usually to load a kernel loader and initial RAM disk blow through configuration and start a real operates. Okay.
So that's, that would be enough because the
blob that it download could be, and here's what you're going to run on the thin client app. Or a could be, you know, here's a config block for the operating system we already have configured so that we can, you know, do whatever it is that thin client needs to do. And
they might be much more hardware specific because of security and security implications. Yeah. And which, which brings up like the Secure Boot piece when we talk about secure boot. Does that by No, is that eliminating TFTP and HTTP and jumping straight into an HTTPS, where does the secure part of the Secure Boot start from
all secure boot does from the perspective of pixie booting? First, obviously, it can only work in UEFI, as legacy BIOS is just inherently not secure. And it can't be made to hear.
Make sense. But from
a secure boot standpoint, what that means is that the bootloader that you get sent back as can be signed by a firmware key or by a by a key that the firmware knows about, which is why we can't use I pixie or UEFI Secure Boot, because I pixie doesn't publish signed releases signed by a key that firmware that most firmware has embedded into it. So from a from a secure boot standpoint, the only difference being the only difference between what's in this graph and what happens is secure boot is apt to after these client gets the boot file, it checks to make sure that it's signed by a key that knows about and if it's not secure boot stop that won't execute.
And that's so when I think about like early Pixie, I always thought it was the firmware was mainly in the network card. But it must have been more in the BIOS and I was thinking about
Yeah, so until has an entire pixie spec. Yeah, it was through the interactions that the NIC firmware and the bias has to have to be able to accomplish or legacy bias. The overwhelming majority of that spec is not relevant to anyone who isn't writing firmware isn't writing a bias. But it is used as kind of the source of truth for the for the options that that DHCP has to know about in order to enable
but in the UEFI BIOS UEFI BIOS scenario, in that case, that's that is the thing that's bringing in this executable, the new bootloader and then it's it's it's checking the signatures before it before hands off execution code 60 and submits a UEFI BIOS piece and those keys are basically injected by the Whoever manufactures that gear the first time and once once you're in secure mode, then you can't. I mean, I guess you could go onto the machine and turn out a secure mode, but every kernel that you're booting in that chain now has to be a secure, read the kernels checking for the signatures, right?
It's a little isn't that when you ever hear in UEFI, secure mode, one thing you can do is add extra keys into the database of AES that the BIOS knows to trust. on a temporary basis, that is how every Linux distribution manages to be secure boot enabled out of the box, or those districts that manage it. Okay, they started with a very tiny little bootloader. That is small enough to be completely audited. And that is signed by a key that Microsoft maintains that is embedded in pretty much every x86 compatible firmware on the planet. And when that shim, one of the things that shim bootloader has it is it has an additional II embedded in it that the district maintainers provide, and that he is what is used to sign the kernel and init rd images that are sorry, not the kernel needed, our DM is directly but that is what is used to sign the copy of it's almost always grub that gets loaded afterwards to verify that the bribe that is loaded is also trusted by Jim. And therefore it's indirectly trusted by Microsoft. And then Rob is responsible for loading the kernel and needed rd. And in
each each one adds some more more keys so that the trust the trust basically expands as you as you reload each one of these these next level shims.
Yeah, that's pretty much how that works. It's all about building a chain of trust from basically all the way from the whoever is packaging up the kernel all the way back to Microsoft. So
it's not the firmware each time doing the validation the firmware does each each new bootloader has a has a has a way to trust what's going on.
Know the firmware does every time it pieces can add additional trusted keys provided they're signed by either the former already trust us fascinating. Okay. Yeah. But that is the difference between secure boot and not secure boot. Basically,
I know that we had to, like tweak our flows for secure boot. I know it's not it's it's not as simple as just checking secure boot, because you have to make sure that you've what's what's made secure boot hard from that perspective,
um, that we have to know which litter to use for every distro we load. Oh, that was one part. And that actually necessitated a pretty major re architecture to our DHCP system. And that's one and that was one of the driving reasons why we have our own DHCP server basically, is so that on every machine, whenever a machine boots, we can go and examine what it's supposed to boot into, see whether or not we have a bootloader that can manage secure boot for it. And if we do and the license entitlements are correct. We can direct our DHCP server to go use the secure boot loader for that distro being pixie or a pixie Linux or you know, our embedded grub or whatever.
But those end up being distro specific, you can't have a generic boot loader process if you're going to do secure because you have to have the ones that match the ones the chain that you're starting on or the chain that you're on. Yeah, that process, okay? Which means that you have to detect it. So the only way to do it without that would be to only have one distro basically, does this does this key chain change on these distros over time, like even if I was just in a red hat distro, then you know, if I switched from seven, eight or eight to nine, do I then have to the keys change to
I haven't examined that directly. I work under the assumption that they do change at least with every major release, possibly with minor release that there are critical security updates. So
I mean, they will if the key is compromised, then they're going to change too. So you better be prepared for key rotation at every level. Right? Sort of the way I usually think of that. From that perspective,
yeah, and kind of the re kind of kind of the logic that drove the way we implemented sport pre secure boot. Was that we need to be able to do this without having any interaction whatsoever. At the console, we need to do have a system B and secure boot mode, possibly from the factory and be able to go through our complete inventory, go to install an OS, and go to running a payload without ever having to log on to the console. The reason that comes into play is that this route spec allows people to inject their own keys into the firmware and say that they are trusted. The way the spec is implemented, though, is that you have to do that from the console. You can't do that via any kind of scripted interaction or API interaction, without going through a significant number of shenanigans that most of our customers really wouldn't be okay with. Right,
we use that to turn on secure boot and turn off secure boot, or either not
to turn on or turn off. That's usually just a virus Settings Update, okay, to inject a key that can be used, oh, okay, we want to provide our own pixie with our own key. In order to get the firmware to trust that we would have to inject our certificate into the the storage space, you have to ask for user certificate. Gotcha.
Right. Okay. So like I had heard of a military grade hardware service that was doing exactly that they were removing the vendor keys and adding their own custom keys in and then I guess they have to have their own signed boot loaders for all that to work.
Everything.
Okay. I guess, but that guarantees that you were the only person who can write an OS that would run on or can deliver an OS that would run on that system?
Yep. Very good for that. Okay, a lot of effort to maintain if you don't have a specific mission to do so. Right? And, yeah,
yeah. But then at that point, though, that machine is functionally wrecked. Well, and you could conceivably switch it back to legacy mode, I guess, if it had a legacy mode, I'm assuming you could get access to the firmware. Let's see me you
could but at that point, you have to, you'd have to be looking left to get either into the IPMI console or to the physical console. And once you're there, from a securities perspective, the game's over. Gotcha.
So we're covering we're covering a lot of the issues I wanted to cover in the discussion. And the security pieces, which I think is really useful, I would move to troubleshooting. But I want it but I want to open up for questions or comments before I shift over. I actually had another question about HTTP versus HTTPS. This is just the up UEFI basically being smart enough and having the root certificate situation to be AES for that initial initial download?
Ah, yes, that's part of where HTTPS gets sort of challenging from a UV perspective is that, you know, it relies on having the entire certificate infrastructure that makes HTTPS work. And for, you know, client operating systems, that sort of stuff usually provided by the OS and updated on a regular basis or provided by your web browser and updated on a regular basis. But those certificate chains can change pretty rapidly in some cases, and no one's going to embed that stuff directly in firmware.
Right? No, it makes sense. I mean, there's at some point, you could just do a HTTPS, so you get encryption, but not but just take whatever key whatever not worry about the chain of trusts on the keys. Yeah. But just probably sufficient for what most most people would, I mean, at least you're you're trusting it is there. I mean, I know that we get pushed back, at times, on, secure on on DHCP and pixie not being secure enough. What are I mean, are those what are those objections? Are they are they real? Like, do you people really worried about pixie?
They're very real. On the flip side, you know, they are the tool that we have, and in some cases, you know, they're cemented in place by 30 odd years of legacy use. A lot of these protocols are initially written back in the days when security meant be nice.
Security meant be nice. What? Yeah,
like the default implementation of TFTP would let literally anyone who could talk to the device upload, upload an image and Okay, whatever image they have, whatever image they could get the file name to, um, you know, um, you know, it has no method of authentication or authenticating that this that the DHCP server you're talking to, is the one that you should be talking to, or the one they should trust. You know, it can't guard against someone. And first, you know, anyone could go and set up a DHCP server on a subnet that has access to the network. And if
they're faster, if they're faster than you are, then you're in trouble. I know, we have like a DHCP. There's there's two things DHCP proxies and DHCP, relays. Relays. Pretty simple, because it's just server his you're just forwarding. But is how does DHCP proxy effectively the same, the same thing as what you just described? No.
Sort of, it's not a DHCP proxy, the it's specifically an extension to DHCP called proxy DHCP. One DHCP server that's just responsible for handing out addresses. And another one that is just responsible for handing out pixie information. And you'll know when you're in when you're using an environment that has that, because it'll take a minute or so for it to decide that it can boot. A lot of sites that had that setup they have it's for separation of concerns. Reason, right? Outside is proxy. DHCP is inherently a slow protocol, it really be made fast.
You've got to you've got to respond from two different DHCP servers isn't that bit of a conflict.
Um, you have to have one server that's configured just respond to pixie options and not respond to address handout. There are a lot of like I pixies that has specific flags to point out disable proxy DHCP. Because when it is, when it isn't disabled, a boot can take up to a minute to decide that it can't exceed. And that's actually what's responsible for a lot of the lag and firmware once it gets to trying to pixie boot off of multiple NICs. You know it can talk to the DHCP server can get a response back condition address real fast, but it has to sit there and wait for a minute see if another DHCP server is going to give it pixie information before giving up and moving up.
That actually brings up one of the most common troubleshooting errors that we end up with is just the pixie stuff of DHCP getting lost. So where does this stuff go wrong? I mean, it's there's a lot of handoffs, there's a lot of bits and pieces to last services to juggle.
Well, after we run things, it's all in one service, which makes troubleshooting.
So what so like, I know, I know, there's times when you're foreseeing somebody's not able to fix you is that is that, you know, I know nine times out of 10, it feels like it's a firewall problem on the on the server eating, eating, eating the the some part of the protocol that I've
seen misconfigured firewalls, eating DHCP acknowledgments caused the issue. I have seen if issues where people are trying to pixie boot servers that are on another continent, and just okay, sheer latency that you get in TFTP, because it is a very old protocol, it doesn't know about things like pipelining. Or you know how to mitigate latency or
loss for lost packets, right? If it's, if it's UDP,
it can handle lost packets, but it does so in the simplest possible way that just makes everything much slower. Um, I've seen a lot of that I have seen issues where switches duplicate or duplicate traffic due to asymmetric routing. I have seen firmware that decides that any process that takes longer than five minutes is failed, even if you know there's a continuous stream of packets going back and forth over TFTP. And I've seen all sorts of stuff that
makes sense does switching it to a I mean, if you're using UEFI and HTTP, is that more resilient? Is that just a better better approach?
Assuming that the firmware stack supports HDD in a non terrible way? Yes. Okay, in turn a boot process that takes several minutes into a boot process that takes you know, seconds in the aforementioned case of pixie booting, you know, a server that is on another continent, because he knows about things like pipelining and you know, intelligent back off and retry.
So I mean, why why does I know that we see a relatively slow adoption of UEFI BIOS killing people switching back to legacy mode. For that, I mean, it seems like that that's always confused me, I don't understand why the resistance, um, that perspective,
I don't see that as much these days. Also, because UEFI is the default thing that comes on all your servers, it must be specifically asked for legacy, right?
But then you have to your boot infrastructure environment has to be able to handle an actual HTTP handoff.
Or if I enable it here, year to two spheres of concerns, goodbye to get HDB for the most case. But yeah, if I were TFTP is still I think, the way the majority of our clients work. So
but I mean, what makes Why would you refer the, why were we disabling it? Or why were customers disabling it for all those years? Was it just buggy? Or were they worried about it as a protocol, it was usually slower.
boot up. That's largely since mitigated and now UEFI is every bit as speedy as legacy BIOS on modern servers. Because even when you're booting in legacy mode, you're still moving into by first, basically a legacy interface to talk to. So
they didn't make it faster. They just made it equally bad. Yes.
Hilarious.
That's just I think it took the industry about 15 years to fully adopt using UEFI firmware as the basis for their stuff instead of legacy mode. BIOS is the basis for all their stuff. Okay.
I know that when we were doing some switch automation, the switches were using, they have a different sort of a different strategy. And they there had only some guy invented a talk to the guy did it at for forget the name of the turtle, open source switch company, the only open network, I
think, sponsor some of that stuff. Yeah. I managed to laugh.
Well, I mean, the switches didn't, you're not loading in OS, you're just you're just burning a firmware image. So when, when, when I saw this, and when I looked at the only pieces, it was literally like, all they were trying to do with with onI was, you know, here's a new network OS load, but the guy hated Pixie, and something, he hated Pixie, or the DHCP says, And he switched it all to like, a chain of HTTP, like, he would look at a given address on the network. And then he would like just go fishing for a file, which strikes me is really actually worse than what we've described with, with options and DHCP.
It's the kind of isn't it? Isn't, I mean, the way things you know, there's easy it's easier to make it secure. Bypassing HTTPS along the way. You know, that's UEFI with secure boot kind of mitigates that because it download an exe, you can download a malicious payload, if it isn't already signed by IE that the firmware trusts. Well, you can download it, it just won't do anything. And, you know, he basically, it sounds like he just wanted to kind of reinvent a better wheel instead of leveraging technology.
Yes, I talked to him about it. And that was he was basically like, I don't like this I'll invent a new one. Yeah, there's, there's a significant amount of, of what came before me is, by default, bad, by definition, bad and I have to re re architect.
But it's just funny because it goes back to what he was trying to do is basically recapitulating what the boss was talking about within clients.
Hmm, yeah, that's true. Well, it's the switches are very similar, right? There is no, you're not trying to bootstrap and OS five, getting significant seed sequentially more impactful kernels. You're just saying, This is my gear, I need you to reimage reimage the firmware on it, which is very much like a thin client approach. Yeah, there's, I mean, from what you're describing, right, we're moving into basically a flexible the bootstrap process gives you a flexible kernel, the flexible kernel let you do you know, you could do almost anything at that point, which is good, but you have to control that
Yeah, it's also you know, before Secure Boot was bad because the colonel could literally do any UEFI I can literally do anything while running in a trust the context of you know, being essentially in the firmware execution space. Right.
So how did that make sense? Okay. Are we did we miss something in this process? I think we covered what I was hoping to for DHCP. I know from when we look at this, we've we've covered a lot of the pieces. And we wrote this graph almost, I'd have to look at let me say, only five years
ago, probably update it. So. Yeah, I think
we should, we should. And part of what I'm hoping we're accomplishing here is talking this through so people understand what what's happening with UEFI. And I mean, we're and we didn't touch on things like doing this process without without having a DHCP boot. I mean, you can't get it on the network without, well, you could pre wire. I mean, when we do immediate attach, are we pre loading those attached media's with enough bootstrap information that you can avoid the whole DHCP cycle and the options? No, that's a whole, that's a huge topic in itself. So ah,
or routing purposes? Yes. DHCP is generally still ECP at some point, is generally still involved, at least to get the out of band controller. That you're, unless you have someone that wants to do it all static, we can make it work that way too. But you know, we that wonder his way of our of anything, they really
well, I am really enjoying this tech ops series, where we take these core infrastructure concepts, and we dig deep in them towards the idea of building a curriculum, this educational material, so that we can help operators understand these advanced concepts. If you don't understand how these systems fit together, how these processes work, it is impossible to automate them in an effective way. And that's what we're doing with these tech, this tech ops series. And I'm super proud of the content that we're building and the discussions that we're having. If you find this valuable, I hope you do. Please check out our at the 23 dot cloud, join the conversations, let us know what topics that you want to hear more about. They are really fantastic, interesting topics. And we're going to be pulling all this together and creating ultimately a book, I think about what this is. So hope to see you there. Thank you for listening to the cloud 2030 podcast, it is sponsored by rockin where we are really working to build a community of people who are using and thinking about infrastructure differently. Because that's what rec end does. We write software that helps put operators back in control of distributed infrastructure, really thinking about how things should be run, and building software that makes that possible. If this is interesting to you, please try out the software. We would love to get your opinion and hear how you think this could transform infrastructure more broadly or just keep enjoying the podcast and coming to the discussions and laying out your thoughts and how you see the future unfolding. All part of building a better infrastructure operations community. Thank you