Download Otter for your meeting notes

20250819 TechOps Scaling Challenges: Otter Voice Meeting Notes

20250819 TechOps Scaling Challenges

Rob Hirschfeld56min

Rob Hirschfeld

00:06

Hello. I'm Rob Hirschfeld, CEO and co founder of RackN, and your host for the cloud 2030 podcast. In this episode of the tech ops series, we talk about scale, and one of the most fascinating things about scale is thinking through the impact of failures during scale, how to isolate yourself from them and how to plan for it. What impacts are when your systems fail, how you can build failure resistant systems. And then, if that wasn't enough, we dive off from there into thinking about other components of scale. So fantastic lessons learned in this really experience based podcast, so I know you'll enjoy it.

Rob Hirschfeld

00:53

Tom you and I discussed the topic for today, struggling to remember it. Do you remember

Speaker 1

00:59

we were talking about the frequency of improbable things at scale.

Rob Hirschfeld

01:07

Ah, perfect. Okay, good, we'll get we'll get to that. Thank you. That was, that was

Speaker 1

01:15

just a stats problem. Yeah, exactly. It's, it's, it's, it's, it's, I don't want to call it a human failing, but it's, it's really an inability to understand how, when you're doing very, very, very large numbers of things, those things that you think only happen one in a million or one in a billion times might be happening a couple times a day, if you're talking about things like database operations or, you know, really any sort of transactional work, when you're doing it really quickly.

Rob Hirschfeld

01:55

This is we were, we were talking about the AI pipelines that we're building, and the Thomas Tom has overseen for his previous employer, and just how important they are. And one of the challenges that we have in talking to especially the AI companies, is they have a tendency to think that if they set the gear up once, or get it right, and then the system's just going to keep working, and they don't. They don't need to worry about building, building additional automation, or reversing the automation.

Speaker 1

02:28

It reminds me of the remember space team from the beach trip. The opening line is, don't worry. The ship flies itself, and then you proceed to have 10 minutes of chaos, exactly.

Rob Hirschfeld

02:42

I, and this is, right, it's, it's one of those things that I find really frustrating when we, when we try to sell into companies where they're like, Well, why do I need to automate the gear setup? I only have to, you know, onboard it once or only have to, you know, that doesn't happen very often. I'll tell a story. I mean, we'll congregate, but I would love to just jump into it, and then if we wrap up early, we can wrap up early. But, you know, y'all, y'all know the story of one of our prospects, where they were onboarding new gear. I think this was just, I don't even, yeah, it was, it was, but somebody had misconfigured a switch, which happens all the friggin time. So like, a gateway was misconfigured, DHCP was misconfigured, and they onboarded a whole bunch of gear with this bad setting in it, and nobody noticed. And so they got further and further down the process until, until until somebody finally realized things weren't working right, and then they had to go and troubleshoot the problem, which took time. I don't, does anybody remember? I'm trying not to name the customer from a recording perspective, but which this was, this is the recent one. But, yeah, it took them, you know, take what took weeks, and then once, once they realized things were wrong, they had to go back and reset the whole environment and fix it. So you went from what should have been a routine setup into something that took not just days of of effort to troubleshoot and find the problem, probably, you know, multiple teams of people, overlapping of, you know, well, it's not my problem, it's your problem. It's not this, is not that. And then having to then reason, you know, recover the setup or go backwards and reset everything, which, depending on how your install scripts work, that can be also very hard. A lot of install scripts are not designed for a reset. Even more numbers. Yeah, yeah, it's so and the whole idea of that I'm talking about here is just being able to run a full diagnostic suite to hold this is a different it's actually a different class of problem Tom that's related to the frequency problem, but the. Idea that, you know, I don't have time to run diagnostics. I just want to move really quickly. Don't take the 10 minutes I'm trying to get this done an hour. Don't take 10 minutes running diagnostics when they people don't think about the well, but if you catch an error before you move forward, then that's going to save you time.

Speaker 1

05:21

Yeah, that's one of the things that I always recommend to people whenever they're re imaging machines, is not to skip the discovery phase, because you're going to get an updated machine record with all the updated hardware information that we get when we do discovery. And if you run your classifier against that you're going to find out any config difference that may or may not be expected, and you can count for that at that time. Instead of waiting, like you said, until it goes into production, you realize, oh, this machine's running like a dog when everything else is running the way it's supposed to, because it's got two bad sticks around in it

Rob Hirschfeld

06:02

and, and that's what it drives me nuts because, like, we have, you know, another customer who insists on, like, they want to speed up their boot process because, because they're running it during a service window, and they don't they, they're trying to fit a, you know, a re image in A it's not even as a reinstall inside of a service window. And we're like, you know, trying to skip a reboot because you're in a service window, just means stop doing the work in service windows. The reboot allows you to do the diagnostics, take the time to do that work, right and do the scan and do the double check make sure the BIOS change you thought was going in on your service window actually went in, because a lot of times they don't. And it's been a that's been a huge fight this. They're trying to optimize a reboot and not counting on automation.

Speaker 1

06:56

Yeah, what I always tell people, you can have scheduled downtime or you can have unscheduled downtime. It's your choice.

Rob Hirschfeld

07:06

And what's the secret to not having unscheduled downtime?

Speaker 1

07:09

You take the scheduled downtime when you can, ah, so and do the work. You set a window right? You say, Okay, the the second Tuesday of every month, we're going to have a four hour window where we do stuff, and even if you don't have anything planned, you still take that window, and you take those machines away from people so that they get used to the fact that that is a hard and fast thing, and they Don't expect, oh, wow, we didn't have a window this week. Maybe the machines are just up and running still.

Rob Hirschfeld

07:47

Couldn't you do it? I mean, we should have enough gear for, you know, the customers we're talking to to have enough gear to just take machines out of rotation and re image them. Like, do the like take, you know, having a window strikes me as unnecessary if you have sufficient infrastructure,

Speaker 1

08:06

yeah. I mean, yeah, typically, you're gonna save that for when you're rolling code, right?

Jon Aykroyd

08:15

Well, I mean, in theory, right? You should be able to do maintenance on anything at any time. But in practice, I've never seen that actually be employed, because executives are always concerned about, you know, outages during peak time and things like that. Yeah.

Speaker 1

08:36

And the other thing you need to make clear to people is that I forget the statistic, but it's something on the order of like 90 to 92% of all hardware failures come when you power cycle a machine. That's right, so checking that after you've done that power cycle is a smart thing to do.

Jon Aykroyd

09:00

Well, well, it takes, it takes more force to start something moving and keep something moving, right?

Speaker 1

09:07

Yeah. Well, you

Rob Hirschfeld

09:08

also have a thermal cycle, which, which will shake out problems in systems. I mean, but I it's funny because I'm hearing you say it, Tom. Hold on. Just a second, John, I hear you saying that Tom, and I think I hear that as the whole system failure. But your earlier comment about bad dims, or, you know, when you and I talked about this before the call, we were talking about Mellanox driver cards, just going belly up. Rob, you know, those are, those are things that are going to shake out during a reboot. That's part of that 92% the 92% is nine times, you know, 92% of the failures of our the whole machine. It's of a component and the machine. This is the danger that people don't realize the machine. Looks fine because it rebooted and the OS started, but something, some component, failed. That's what you're really saying, right?

Speaker 1

10:08

That's exactly what I'm saying. It's a It's silent. I mean, there might be a message that pops out on the console, but if you don't have logging that's looking for that, then you're not going to know what happened. You're only going to figure it out later when things don't work, or when they don't work the way you expect them to. It's a really important point with that, that that I

Rob Hirschfeld

10:32

think people misunderstand how complex these machines are and how many components they have that could go bad, that would cause the system to degrade. John, what were you going to say? I'm sorry, I cut you off.

Jon Aykroyd

10:44

I don't remember.

Rob Hirschfeld

10:47

Come back. So, so let's, let's dive into the law of large numbers with this then, because I think that's our starting point for this. I mean, what? What are we when we're talking about an AI system, and we can, we can focus on that, because I think there's a lot of interesting ground to cover in these AI systems. Because, one, they're very expensive, but they're, they they're hot, you know, they run very hot. They run a lot of power. They have a lot of components, sorry, GPUs, Nic cards, usually RAM modules or they're fully loaded systems. How do we How should we look at this as a from a failure perspective?

Speaker 1

11:33

You know, obviously you've got runtime failures, which, again, you should be monitoring and logging if they happen in real time. But you know, the other side of that is when these machines are cycled, typically in in most of your your large AI providers, they're not static machines. They don't set them up just one time, right? You get a cluster of machines that you build for a specific task for this customer who needs this large language model, right? And you build that for them, and then when they're done, they give it back, and you have to completely re image and reload that machine, because most of those models are proprietary. You don't want to take that and have it accidentally made available to somebody else when they get it. So they have to be really good about scrubbing the machine. And that scrubbing process can sometimes be trying on machines,

Rob Hirschfeld

12:35

okay, oh, the the decommissioning process itself. Yes, okay,

Speaker 1

12:39

especially when you're doing things like, you know, if you're doing a Blanco erase where it's overwriting things seven times and nine different patterns, and you know,

Rob Hirschfeld

12:52

your, your makes sense. Well, are they also changing the I mean, you're, you're putting the latest. There's a fair bit of bios and firmware updates. I'm assuming, in all this stuff too, where you've gotta you're gonna put the latest bios in, especially if it's been tuned better. You know, the NIC cards are, are very firmware dependent. I'm assuming new

Speaker 1

13:14

drivers, yeah, Mellanox releases new drivers about once every couple of weeks. It seems

Rob Hirschfeld

13:21

is it? Is it stressful on systems to to actually flash those the firmware infuriate failure.

Speaker 1

13:31

I mean, the firmware is, it's basically non volatile RAM, right? So you know you're going to update that they have a given number of write cycles, just like any other RAM, just like any SSD, there's only a certain amount of times that can happen before you start not being able to write to those pieces anymore. Now you know, in theory, that number is something huge, like, yeah, 5000 hours of operation.

Rob Hirschfeld

14:07

But, well, but it's that also those, those are electromechanical devices. Flash, RAM, right is, that's the reason it wears is because it's, you know, it's an electromechanical device where you're switching the polarity of bits. So I'm assuming, if you're running those devices hot, that's, that's going to create additional wear cycle, yeah, yeah, because, and they're not running these systems casually. They're, it's, not like a virtualization cluster from last decade where it was, you know, 10% utilized, 20% utilized. It's, you know, these systems are running flat out while they're running. Those wouldn't make sense to do anything else, yeah, I guess I'm trying to figure out if you were liquid cooling them. Yeah, which I, you know, we should assume. I know liquid cooling is really efficient. Does it have the same heat wear profile as a air cooled system?

Speaker 1

15:10

Would? I don't know the answer to that question. Honestly, it's

Rob Hirschfeld

15:15

interesting to me, because I know in some of the liquid cooling systems, and I've seen demos of this, that the air comes out colder on the on the exit side, than it return side, on the return side, well, because they're pumping, you know, chill liquid through it, and so you're but what I what I don't know, is if those are, are the running systems, or the, you know, I've heard people brag about that, but I suspect it's the you know, my friends at Dell who are in a lab walking behind the servers, not you know, somebody at Microsoft walking behind the, You know, open AI, inferencing engine systems. I suspect even with liquid cooling, they're still heating the environment up. Yeah, it's just what you're what you're able to cool. There's still a lot of other components on that system in addition to what you know, what you're putting, what you have in contact with the cooling, with the liquid cooling.

Speaker 1

16:19

Yeah, and realize too at the end of the day, it's still all about the airflow. I mean, having the liquid there is good, but that there's still the initial heat transfer that's got to occur between the liquid and the metal, the heat coming off the CPU, which you then have to pass the air across to move so that more heat can be transferred off the liquid, right?

Rob Hirschfeld

16:51

It's the thing. The systems I've seen this, there's two types of systems, hey, Martez, we're talking about the the accumulative effects of failures and systems and designing and designing and planning around, you know, basically the the accrual of failure modes. We're still getting to that, some of that, some of that more advanced point right now, we're talking about cooling, just, just general. The two things I've seen for cooling are either they're running a essentially a refrigerant like you would on a regular like a desktop CPU, where you you bait, you have a heat sink on the the hot chips, and then you're running refrigerant off from from an external source, through those, through the pipes attached to the heat sinks so they cool, they cool the major heating elements, and then they they remove the heat as a fluid. Or they have immersive tanks, where they put the whole motherboard in a non conductive tank and then circulate that circulate that fluid and cool that fluid. I was just at a conference where people were raving about the immersive cooling, which I get it. If you're worried about all the components on the board, it probably is a very productive way to do it, but handling the fluid just strikes me as as a nightmare. Yeah.

Speaker 1

18:32

I mean, it still becomes a problem of heat transfer, right? The fluid is more efficient than air, sure, but you still have to, at some point, get that heat out of there,

Rob Hirschfeld

18:45

correct, right. So some point you, I mean, you still have to circulate and cool the fluid, right to to distribute it. The the piped fluid to a heat sink requires you to have, you know, a way to attach and detach the fluid lines. And you know, you can only, you're only liquid cooling part, the parts that are actively have active heat dissipation. So you're still, you're still putting you still need some fans. You still need some airflow, the immersive approach, you still have to circulate the fluid and cool the fluid, but, and I haven't seen enough of that to know, you know how you know what, what that, that circulation looks like, but conceivably, you could fill a whole cabinet with this fluid and then, you know, just ducked out the tucked out the fluid.

Speaker 1

19:42

You could right and but you effectively, you just made a much larger thermal mass to absorb the heat coming off of your components. That thermal mass still has to eventually be cooled.

Rob Hirschfeld

19:58

Yeah, if you put a gigawatt of. Feet of electricity in. Then you're convert, you're, you know, converting it to heat. It's gonna have to come out. You're gonna have to release a gig of it, a gigawatt of heat on the Yeah, at some point.

Speaker 1

20:13

So, and there's, there's a lot of different ways that you can that you can say that one of the one of the things that Microsoft was doing in one of their experimental Labs was they, they were doing some immersion techniques like that, and then the the outflow went out, and then went down 200 feet into the sea floor and then back up again. And it was the trip through that 200 feet that cooled off the thermal mass. Now, admittedly, that's probably outside the range what most people are going to do, but it's all about the heat at the end of the day.

Rob Hirschfeld

21:05

A lot, a lot of and getting power to it, but, but I'm pulling us away from the conversation on thinking through how we, you know, better handle this. This the probability of error. I mean, if we're dealing with with these complex systems, a lot of components. I mean, how do we calculate the probability of a failure? And actually, let me back up if the most likely failure is through a planned power cycle. Does? Does that mean we should stretch out the power cycles? What's the consequence of ignoring it and letting the components fail? Otherwise, why not just run the system? You know, brag about your unlimited uptime on whatever core systems you have?

Speaker 1

21:56

Yeah, then you run into the issue of cumulative failures that just stack up until they eventually take the machine down? That goes back to that statement about plan versus unplanned.

Rob Hirschfeld

22:07

So if you have a dim that's, you know, getting to its end of life, and wouldn't, wouldn't recover if you booted it, but you leave it in place. I mean, I guess it's going to start having more read errors, or more more more errors, right? How would it take? How would it take down the whole system? Well, if you've got kernel code in that stick and it gets corrupted, you're going to cause a segmentation fault that will knock out the

Speaker 1

22:35

OS. Okay, so, yeah, it's, it's,

Rob Hirschfeld

22:39

that would be bad.

Unknown Speaker

22:42

Yeah, definitely. Um, and protect

Rob Hirschfeld

22:45

so you would have, you have a you have a crash, and you'd be, your systems would be dealing with a crash recovery from right, from an AI training perspective, that means that you started a run and the whole run is corrupted

Speaker 1

22:57

by that failure. Depending on how your cluster setup, it could be that you can tolerate the loss of, you know, n minus 1n, minus 2m, minus whatever, in your cluster, and still be able to keep going.

Rob Hirschfeld

23:10

So my understanding with the with the frontier model training was that it you had to check you it was not tolerant to a failure during the training and so they had to build training windows, and they would put checkpoints in, because if there was a failure of any system during the the run, during in between checkpoints, they lost all the work across the entire cluster for that checkpoint. Maybe it was just one one node, but, but you can restart checkpoint, and you can restart from the checkpoint, hopefully with, with, you know, one fewer machines, or with a backup machine, but,

Speaker 1

23:52

but still, we're talking about equipment failure. It's just, it just comes down to working the numbers. Do the math right if you've got, let's say, a 64 node cluster that you're using for AI training. And let's say each one of those 64 nodes has 32 sticks of RAM in it. And we've got what's that to the 11,024 No, that's 24 2000

Rob Hirschfeld

24:21

you don't have to work in base. In base 10, you could just say we've got 100 node cluster, each of them with 10 nodes, 10 sticks of RAM in it, if you want

Unknown Speaker

24:30

exactly. So I like to work.

Rob Hirschfeld

24:32

So you gotta, you got 1000 Yeah,

Speaker 1

24:35

but so, so let's say that we've got, we've got 2000 sticks of RAM across these, this cluster node, right? We note that the failure rate of these RAM sticks is one in every 5000 hours of operation. And then you do the numbers, and that says time between failures for that number of sticks of RAM. Works out to something or to your order of one every other day. So right point, you have to make a decision, right are you? Are you able to tolerate a failure of one every two days? Because it's going to happen, according to the math. Now, does it happen? No, not always. Of course not. Because what is math and what is real life are not always the same thing, but you still have to be prepared for that possibility. And worst case scenario, it could even happen more often than that, right? You could get three or four failures in a single day,

Rob Hirschfeld

25:41

right? You will occasionally, that's right, still, so go ahead.

Speaker 1

25:47

So you have to, you know, where's the trade off? Where's the trade off? Now you have to go, Okay, let's make the checkpoints happen a little more often, right? Let's, let's do a checkpoint once an hour, because that way, if something happens between one hour and the next, it's a lot easier for me to recover from that failure than it is for some random thing to pop out if I'm only doing a checkpoint once every four hours. Oh, that could be a whole lot worse, right, right? That's true so

Rob Hirschfeld

26:19

and so, right? And I know that, like GPUs, especially the latest and video ones, have significant challenges with heat, or they're just so finely cast, right? They're, they're, they're much more likely to have failures or budget bus issues, or the, you know, even the the PCI busses are being run very hot, and there you have failures with chips dropping off the bus. They're otherwise okay. They just drop off the bus. I'm not sure that that's something that we would be able to remediate, short of being more careful in how you're overloading the machines or when your checkpoints are, but maybe, yeah, well, but, but, yeah. So you're talking about a system that, you know, in a in a 64 node cluster. You're talking about systems that are having, you know, cross the cluster one failure a day. Do we mitigate that by by, you know, rebooting them more and hoping we catch them during the during the reboot cycle. Is that that? I guess that's better than a hot failure.

Speaker 1

27:31

Yeah, yeah. I mean, so in our post for our hardware, we're gonna, we're gonna detect a bad dip, and we can blacklist that dip as part of the post so that we know. Okay, look, that guy's out. He's done. I don't like the errors I'm getting off of him, but now you have, now you have a weaker node in the cluster. So this is where you're logging. This is where your monitoring comes in, and you recognize that this cluster is not in its optimal state. And then, you know, then you can plan your depth, as opposed to the unplanned downtime that we don't have.

Rob Hirschfeld

28:14

So if you're taking your cluster, if you're so, if you're taking the cluster, this is the interesting instead of a rolling upgrade. You're saying, Oh, wait, we're going to take the cluster down, you know, for a recycle in an hour. That might be handy, because you might say, All right, we're going to have P, you know, somebody with a crash cart there to remediate, you know, whatever the issues are in the moment, so we can go through and do that work.

Speaker 1

28:38

Yeah, exactly. And interesting,

Rob Hirschfeld

28:44

is there predictive like, could you actually start predicting that? Or, I guess you can with, I know, with with Ram. I don't know if it's as easy with a GPU, but if you have a consistent issue with one, one GPU card, I was assuming you would remove it and swap it.

Speaker 1

29:00

Yes, the NVIDIA tool part with a model. Do the NVIDIA tools will do some diagnostics even on working cards, and can give you predictive failure analysis. So that's that's, again, something you can definitely plan for. And even if you you know, again, depending on how your cluster set up and defined, you may be able to shoot that node like cattle and bring a new node in and integrate it without having to completely re, initialize the cluster, right?

Rob Hirschfeld

29:47

It's actually, it's an it comes back. I can't help but connect this to the immersive cooling versus piped cooling argument, just in, you know, depending on how an immersive cooling system. System is set up, removing one one. I can easily see somebody taking a node out of, you know, this node has a problem. We're going to pull it out for diagnostics, move it to a bench somewhere, because you don't, you're not going to want somebody, you know, cracking open. I don't know that. I would, hopefully I'd have enough extra stuff to have a to pull out the whole node and then take it to the bench and recondition it based on knowledge, and then put a new slide, a new node in to replace it.

Unknown Speaker

30:31

Yeah, that it becomes a much more involved

Rob Hirschfeld

30:33

process at that point. Yeah, yeah. To remove a specific it's an interesting I don't, I don't know. I haven't, I don't know. You know, I can. Can see the ongoing maintenance of pulling those out and running them through the through the process, and having spare commodities on hand, which, in itself, is a challenge, because you're RMA ing, the bad card. You need a replacement for it. You're going to send it back. It could be a significant amount of capacity at the failure rates you're describing. That's fascinating. I mean, all of this has to be part of your design. For the capacity you have, the snow, the network topology you have to, you have to build, you know, a degree of that into how your systems operate. Bulk tolerance, yeah, it's fascinating to me, because I think I feel like a lot of, I guess, VMware clusters were not run to the to the performance demands that you know these AI clusters are running at. Because I feel like, you know, the some of these AI clusters were basically, you know, performant bunkers. You put in, you buy a whole bunch of cards, get a five or 10, you put it up to a Stan array, and there wasn't a lot of maintenance, I guess they if you had a card failure or something fail on the card, you'd pop out the card, that was the whole rationale, and then slide in a new card.

Speaker 1

32:01

Yeah, well, you got to remember, in a lot of the older clusters, too, were first doing VM designs, and, you know, we would typically over engineer them and account for redundancy so that we weren't really over subscribed as much, right? But with with the GPUs, you have fixed numbers, right? And you want to get your money's worth out of them, so you're gonna run them at 100% all the time to get everything you can, because there's so much data that's got to be processed, right? To train these language models. So the the utilization is a much different picture for these than it was, you know, things like our older virtual machine clusters.

Rob Hirschfeld

32:59

One of the things that you and I talked about here was the redundancy. Some of it is scale. Right when we're talking about a fiber channel network and maxing out, you're not just maxing out the individual machine. We're running the fiber channel fabrics, and it will do it'll have the same problem with Ethernet fabrics to their extreme limits, also right from a span and control perspective, for the type of East West switching that we're trying to do, you don't, you can't have too many switches involved. So part of the other redundancy problem is that you're not. You know when we would design, I remember this very clearly, are the when we would design VMware clusters, we would design redundant paths to redundant switches to it was incredibly expensive, yeah, but the consequence was that if you lost a switch or you lost a card, the systems would keep Running. There was those cars would would not die. But, you know, or I don't feel like the AI systems, because you're trying to put so many nodes on a switch, they're they're scaling out. They're not worried about the they're compromising the redundancy.

Speaker 1

34:15

It depends. I mean, like the clusters we're building for Microsoft, let's take a 64 node cluster. For example. Each node in the cluster had eight InfiniBand cards, eight right cards, or ports, cards, ports. They were but, but they were single port cards at, okay, so, so they were. They were beasts, right? So, the way that these nodes, they they had them segregated into nibbles to minimize the. East, West traversal, right? Because you wouldn't want, you wouldn't want a chunk of data from node one to have to traverse all the way over to node 34 to get a piece of data. So, so they, they would, they would segregate them into chunks of four, right? So these four would be, they would have, let's call it layer one would be, those four would all be interconnected. And then on the odd chance that we did need to get a piece of data for somewhere else, there was a layer two that was 16 nodes connected. So four of the layer one switches would talk to each other, and then, you know, there was a layer three above that that would have the layer twos all connected together. And it was very intricate mesh of data. But again, it was, it was done to minimize the amount of traffic that would be traversing the network, because you would try to keep that as local as

Rob Hirschfeld

36:17

possible, possible, because for those, eight alerts would so you would have like, would, would the switches do that the top level aggregation. So you would have one server connected to eight different switch, you know, level one switches, which then could talk to. I mean, those are usually eight port switches or 16 port switches, so they lose a couple of ports for the uplinks. But that would mean that you would have, you know, immediate neighbors on the same switch, and then a ring of a couple of switches. Or, go ahead, yeah.

Speaker 1

36:56

So, like I said, you'd have the cluster of four, like node one would have node one would have four data lines going to each pair because they repaired switches for that layer one right? So four nodes going to two switches. So four ports on each node would go to each switch, and the traffic would be aggregated by the InfiniBand network. Okay? And then any traffic that had to leave that four node chunk would go from layer one to layer two and get propagated across the layer two switches for the sixteens and then propagated up to layer three to go across the whole cluster node.

Rob Hirschfeld

37:55

What were the other four ports? More than in the nodes?

Speaker 1

38:00

Redundant path, right? So four, four paths

Rob Hirschfeld

38:06

switch A and another to switch a, and

Speaker 1

38:09

paths would go to switch B. Okay. The idea was that if there was a card failure, it would be tolerant,

Rob Hirschfeld

38:17

um, right, because the cost of losing a node is very high, yes, but so when, when you start getting into that design, then you're on the other side of so now you've you've isolated a card failure or a port failure on the switch or a switch, because now you have redundant switches. But you can't just route around a node if a node goes out, then you've got a three, a three legged chair doing work you're gonna write. You a degraded node is probably problematic in that, in that design, yeah, I guess a degraded node is better if it can fit, help you finish a job or something like that, get limp, limp along. But that's fascinating. So it's, it's like the design includes, like, Okay, we were the mallanox cards might fail, so we're going to have four way redundancy. They're probably using the the four way to have parallel bandwidth. So probably a bandwidth multiplier to the Yeah, it was

Speaker 1

39:23

an aggregate throughput of something on the order of like, 300 and some odd gigabits per second.

Rob Hirschfeld

39:29

Okay, right? That makes sense. And if most of the traffic you're trying to east west to the other machines in that cluster, right? That then, you know, they, they need that ability to send traffic. You're I saw some Nvidia stuff where, fundamentally they are, they were sharing RAM. That's why they needed so much bandwidth. They were they're functionally sharing RAM between the machines,

Speaker 1

39:55

which is why the InfiniBand was necessary. Because that is. It's not just it's not just data that they're going after. It's memory locations as well

Rob Hirschfeld

40:08

direction, right? That was, that was the thing that was in the new designs they're working at, they ended up with these functionally shared memory environments. I don't know how many of those have gotten into production there. The networking is incredibly

Speaker 1

40:26

it's still a network bottleneck, though, because PCI bus is way faster than your network is going to be, right?

Rob Hirschfeld

40:32

Well, that was part of what they were talking about doing, is using the what's the PCI bus? There's a liquid uses this stuff back first, sorry, CXL as DXL or CXL, C as in Charlie, yeah, CXL, yeah. CXL stuff is the interconnect, yeah. Martez, Hey. Do you have, do you have any any thoughts on how people should think about dealing with, you know, the failure rates on clusters and designing around that?

Speaker 2

41:08

Yeah, I mean it. For me, it all comes down to financials in the sense of, what's the value back to the business. So, to the point that was being made earlier, the difference between a AI cluster and some of the the VMware design clusters. And this has always been one of the things where it's interesting when you look at different markets as it relates to how they approach this problem, whether it be service providers, cloud service providers, enterprises, small businesses, research institutes in terms of where they focus their efforts in because, for certain situations, I don't need redundancy, because there's not enough justifiable revenue being generated from this infrastructure. To say, Yes, I need full redundancy, five nines of uptime, availability to run my print server in that sits in the closet.

Rob Hirschfeld

42:07

That's true. I was, I thought you were gonna go to patch jobs, patch UPS also have some tolerance from that perspective, yeah, yeah.

Speaker 2

42:13

And that's where it gets into while, from a pure technical standpoint, it's the idea of having fully redundant systems and everything being highly available makes sense, but from a business standpoint, becomes a situation of what makes the most sense for the particular workload that I'm running, and where Are my tolerances in terms of unavailability. Go ahead, right?

Speaker 1

42:42

Ahead, right. And what's the cost to the business for that downtime?

Speaker 2

42:47

Yeah, well, and even the inverse, I would say sometimes, is even if that cost to my business is more than I would like to bear, is the investment of me putting resources into it also going to be worth it. On the flip side of is it somehow driving more revenue? Because one could argue that, you know what, even if it's costly to the business, if I were to get to a situation where I essentially almost offset that potential possibility of a loss, then I might say, You know what? It's an acceptable risk. We might lose a million dollars, but it might cost us half a million dollars to develop that level of redundancy. And if we feel you know what, maybe that outage happens once every three years, maybe once every five years, this where it gets back into that prediction

Rob Hirschfeld

43:44

that, well, this is, this is part of what the what we're talking through is that a lot of times it feels like people do a really bad job calculating how often that failure is going to occur, because they assume they're like, Well, it's, you know, this, these, You know, these systems aren't likely to fail, and therefore we're not going to worry about the contingency planning. And you're right. That cuts both ways. They might be overbuilding. I know we, you know, they certainly seen people over build on redundancies and pay a lot of money for overbuilt redundancies. Do you think that they need to factor in the time to recover for this? Right? They could say like, yeah, I don't need all the redundancies. But then net put in place a way to, you know what we were talking about before you joined, was scan and check and, you know, cycle systems through and have standby components to absorb the failures.

Speaker 2

44:48

I think, I think that makes sense for service providers, people whose money is literally derived from their infrastructure. Or now, when we start to get into enterprises, I think that ends up running the gamut of, this is the same with all things, whether it be, do you have a good DR plan? Do you have a good this? Do you have a good that? And it comes a situation of, is it worth that investment for them? Yeah.

Speaker 1

45:20

I mean, back when I was working at AT T we had a tolerance downtime of literally less than five minutes per year, right?

Rob Hirschfeld

45:30

And so for that, you're going to have to build redundancy into the system, because you can't recover, yeah.

Speaker 2

45:35

And for them at that point, I mean, literally, their direct dollars are derived from the technology. It's not as though we're talking, so to speak, let's say an insurance company, where, in theory, I could keep getting by without necessarily the systems being highly, highly, highly available in some form or fashion. And that's where even you look at like the linkedins, the metas, the Facebooks, all those very technology centric companies. There's a reason why they invest, because they can literally say this server helped generate this amount of money. Challenges on the enterprise side, I would argue for most, it's very difficult to quantify that I bought a new server. How much revenue did that server generate for the business?

Rob Hirschfeld

46:30

I agree with you. It's an interesting component, though, because I think that we watch companies like say, All right, I'm gonna spend you know, $100,000 buying a new server. You know, usually they buy, you know, rack or some units a server. But if that server takes three, six months to get into production because they don't have processes and automation to manage it, then, you know, I they that is, that actually is a real cost, or they can only buy it from Dell, because that's all their automation can do. The thing that I find fascinating is a lot of times we'll see customers or prospects who started with like, Oh, I'm using Dell, everything's great, or I'm using HP or super, doesn't matter which vendor insert, vendor choice, and they get, they have some error, like we've been talking about, right? A, you know, they need to flash, you know, a nick card, or they have a dim error, a storage error, rate card error, the machine itself and the tooling has around that system has changed, but because they don't use it very often, they they don't have a quick path to fix it. And so they might, they might have, you know, weeks or months before they can actually fix whatever the issue is, or at least in an automated way, because they don't do it very often. Yeah. I mean, maybe that feels like a little bit of a tangent, but it relates back to this idea of, hey, you know, reboot your systems often, run diagnostics and scans. Be ready. You know, be aware that things are going to fail, sort of shake out failures as fast as you can, rather than wait for them to accumulate or catch you when you don't have time for it. Sorry, go ahead. Yes, no.

Speaker 2

48:32

But I mean, I would say it goes back to it's the same line of thinking. The reason why I don't do that is because there's not as much of an emphasis placed on the actual technology and infrastructure itself. If you compare, let's say, a enterprise, your average enterprise, to a service provider, the service provider commonly knows down to the sense how much that server costs them everything that runs on it, as well as how much they make per physical node in their environment. So for them, there's a much greater focus on those sorts of processes, the automation, the just general thinking that makes me invest time in ensuring that I've got things lined up and being essentially proactive as opposed to reactive, whereas oftentimes on the enterprise side, it's the infrastructure is there. We know it all. We all know it helps support the business. But there's not that level of emphasis on, so to speak, dotting your I's and crossing your T's, for lack of a better phrase,

Speaker 1

49:48

yeah, it's because it with the service providers. It is a profit center with enterprise. It's a cost center, and you have to work. The numbers as to what's going to be best for the company overall. I mean, yeah, can you take a downtime of an hour in a service provider? No, you're losing a whole lot of money if you do that. Can you take an hour of downtime in an enterprise where you can just restart the processes once you get systems back online, probably will you lose money? Maybe a little, not much.

Speaker 2

50:27

Yeah. And I think the part that sticks out the most is, I think few have a great grasp of how much money that would be.

Unknown Speaker

50:38

Yeah, it's mean,

Speaker 2

50:39

it the thought pattern makes sense. And for me, this was also one of the things I thought about several years ago when event driven automation started become a or event driven remediation. So let's say, became a popularized thought process topic at companies like the linkedins, obviously, the CSPs of the world, hyperscalers have adopted those same sort of paradigms. The challenge I always had was, Does this make sense for the enterprise to have in effectively, an engine that looks at the scanning the environment, looks at the monitoring, the logging all those things, and attempts to take action in in an automated fashion to help remediate it, and the challenge becomes similar to a lot of security things and various other things there. There just isn't going to be the investment in those sorts of processes because of the way in which the infrastructure is viewed.

Rob Hirschfeld

51:39

Does some of that change with with AI infrastructure,

Speaker 2

51:45

I would say, No, my personal thought, because, I mean, like I would argue, we've had quote, unquote, AI ops for what, the last 10 years.

Rob Hirschfeld

51:56

Ai ai ops, as far as I can tell, has never been about running the AI infrastructure. It's always been about feeding the AI platforms on top of it. Yeah, just, I was literally just doing research about because I keep looking for somebody to talk to about AI and operations. And, you know, I was just searching Gartner for their AI ops materials, which are 100% about feeding the AI engines or not about running the actual AI infrastructure. Well, absolutely nuts.

Speaker 2

52:33

Yeah. So in regards to the AI infrastructure, I would argue that's follows the sort of the separate track that HPC has been on for what the last 2030, years plus, in which in many situations it's it's its own sort of space that operates under very specific rules, and has always had even back to The conversation earlier around InfiniBand. How many times have you seen InfiniBand in regular production networking environments? You don't because it has certain properties and capabilities that lend itself well in that HPC AI space. And so there are certain paradigms and methods of managing infrastructure that I think are just applicable to that space. Do you

Rob Hirschfeld

53:28

think the HPC space does a better job accounting for error rates on these systems? Maybe, or maybe they just don't care, because it's really batch systems. They're not, they're not really, you know, whole cluster operations, ah, I mean, you see what I'm saying is, yes, I mean, I know they brag about having, oh, I've got 1000 servers running whatever in this cluster. And, you know, by all rights, that would that's going to put them in daily failure mode and and all sorts of stuff. But they usually treat those clusters as very monolithic stacks. Yeah.

Speaker 2

54:12

I mean, I think there's a couple of pieces there intrude fashion to take advantage of the processing power. And I think it's a situation of there's a more often than not, a very prescriptive set of architectures and applications that will run on those environments, which I think helps to simplify the overall process. When you start to talk about all the way up the stack, as it relates to from the lowest level to the application and being designed in a fashion that makes the most sense from a resiliency perspective, that I think just will never apply to your general purpose infrastructure stack.

Rob Hirschfeld

54:56

That's been my experience, too. The HPC people are. In their own world. Yeah.

Speaker 2

55:02

And the other thing, I would argue, goes back to that sort of delineation of who you are and what you do. From the standpoint of a lot of HPC AI has been universities labs, a slightly different space to where not to say that money doesn't matter, but oftentimes you might get a grant that affords you the opportunity to spend money possibly more than a large enterprise would on certain

Rob Hirschfeld

55:33

things. There's so much I love about doing these advanced series tech op sessions, and I hope they're helping you. We are putting all of these things together to help people find and learn from our experience in the field. And we want to hear about your experiences. How do you react to what we're talking about? Please come in. We want to hear your thoughts, and we want to make operations better. That's the goal of the whole series. And thank you for listening. Thank you for listening to the cloud 2030 podcast. It is sponsored by RackN, where we are really working to build a community of people who are using and thinking about infrastructure differently, because that's what RackN does. We write software that helps put operators back in control of distributed infrastructure, really thinking about how things should be run, and building software that makes that possible. If this is interesting to you, please try out the software. We would love to get your opinion and hear how you think this could transform infrastructure more broadly, or just keep enjoying the podcast and coming to the discussions and laying out your thoughts and how you see the future unfolding, all part of building a better infrastructure operations community. Thank you. You.

00:0000:00