Rob, Hello, I'm Rob Hirschfeld, CEO and co founder of RackN and your host for the cloud 2030 podcast. In this episode, we go deep into this concept of building small language models or taking large language models and then training them to be smaller. There's a lot of different processes and ways to do this. There's a lot of benefits in how that work, and we'll discuss those. There's also risks and dangers companies that are involved in sort of doing this work and training. It's really fascinating. Deep dive into the emerging AI ml, large language model, discussion about how we're going to make these models really useful for inference, where we're going to do the training to create those small models, and what are their benefits? And then, of course, how would we govern them? Govern them. We're going to cover all of that and more in this great episode. I know you will enjoy
it. There's an interesting commercial thing I'll talk about a second, but they are the and this lines up with what the veil conversations were. There is not enough power or cooling capacity for companies to build their own training systems. Right? Even if it was more cost effective, they they might not be able to do it because their their power budgets don't allow them to buy those systems. Um, and so they're willing to pay the overhead to let to do it in the cloud.
And by the way, the cloud, the CSPs, they can't even run whole. You know, they cannot. They cannot place the training efforts in one spot. They have to segment and Ted spread them out because they, they, they will say, if we tried to run the whole thing, you know, it, you know, Amazon center, yeah, it would, you know, it would, it would, you know, take the power grid down, yeah.
Well, it's interesting that you're talking about this, and go check out composable without the E at the end, composed bowl. It's just BL the end, it is they just launched. They're along the same lines as what rich you and I have been talking about, except it's not as advertised, because they make API calls to all the language models, and they allow you to create and use they have SDKs that are in open source, and what they are allowing you to do is to basically train your own models based on make the API calls to lambda or whatever, whatever. Choose the model that you want, assign your tasks, tell it what you want it to do. There's a script writer, a prompt writer, and all of that kind of stuff. And then they're the way they're charging. If you look at the the rates. It's going to be very, very, very expensive based on how many times you're running each training model, how you're refining it, how you're trying to put in, I put it in the chat, perception, learning, strategy, forward planning and deduction. Those are the those are the agent the traits and characteristics. To make a simple generative model into something agentic, right? You have to have degrees of each of those things in terms of perception, so it becomes contextually or situationally aware. And the situational awareness is the key element of the small language model, as I see it. That's its main claim to fame, because if you have a Just hear me out for a second, an industry specific or or enterprise specific data dictionary, then you could cut down on the refinement of your small language model significantly, because this is the lingua franca of your enterprise and or industry. So the terms of reference become smaller and smaller, therefore the number of parameters become more limited, and you can fine tune based on the task you want the AI to do, to execute based on that set of terminology. So a workflow definition in it may be very different from one industry to another. Insurance claims are not filed the same. Way as something on a shop floor, right? So the the intrinsic, no pun intended definitions of the from the data dictionary would allow you to start honing and refining, but in composable case, they're allowing you to do X number of runs against each language model that you want to look at to do the Compare and contrast of which language model for which task does a better job in terms of re returning more accuracy, and you can mix and match over the course of a workflow, individual tasks in that workflow that's unique and interesting, but I think the cost of doing that is going to be very, very high, because once the CSPs, or whoever, the creators of the Large language models start to figure out their business model in a more significant way. Those charges are going to start racking up based on the use of which model. How much is going to be GTP, how much is going to be lambda, how much is going to be whatever, and that's going to be super expensive for an enterprise to run. But check it out. It's, it's very interesting in the way that they've created the studio, and how they're giving you some of the capability to create an agent. Interesting. Okay, yeah, it is. I was looking at the SDK in the open source side to see if it would make sense to leverage it and it, it is interesting. I'm not quite sure I can I get all of it, because I think there's nuances all the way through this. The if you look at the website and the information that they're providing, you really have to think through what they're saying. And it's kind of like I'm going to give you the APIs to every engine that you want, and I'm going to ask you to set up your tasks in such a way that they are prompted to see what the returns are from each model, but to then train it, teach it, and then move it to your real environment. There's a gap there that I can't quite get my head around. How do you take something that's done in simulation and well trained and well refined, and you're getting the right answer 90% of the time, or 98% of the time, and then operationalize that unless you're feeding composable all of that data dictionary, and that's just an example, right? The parameters that are germane to your specific workflow or your specific enterprise. How do you get away from doing that
in a by comparison, by comparison, feeding a data dictionary, as opposed to, you know, the the entirety of of a Reddit or Twitter or so, I mean, by comparison, it's, it's, It's tiny. So feeding the feeding the the data dictionary, which in way you're doing, but they're more involved with feeding taxonomies and ontologies that are right, that are appropriate to the task and and by comparison, they're they're quite small. What it what answer? What kind of goes to your question, Joanne, is, once you build this, what are the, what are the models that you're actually playing with? And they have, almost have to be hybrids. They're, they are not just completely built on doing in, you know, an embedding and doing kind of conventional similarity there, there. There are a couple ways that, when you have agents like this, you can do re ranking. You can incorporate a network, a graph network, for example, or you can incorporate something that looks more like a relational database or an ontology to it. So that's what they're they're banking on. But the the weak spot when I took a look at composable was, as far as I could tell, there was not a they kind of wave their hands when they talk about. About evaluation, how do you how do you build the tests that are thorough enough that kind of say, every time I run with the same parameters, I'm getting an accurate answer, or an answer that satisfies some kind of parameters. That's and, yeah, they're going to be, they're going to be expensive to build in some of these cases, but the the benefit, the return on that investment, is probably going to be pretty good. And the way that composables building them, I think kind of makes a lot of sense. And to your other point, it's basically going to this big, you know, CSPs, the big, the big players with the this super large models, and basically piggybacking on their investment, getting the return from it. So
yeah, and by the way, my use of, my use of, you know, data dictionaries, I don't know what I don't know about, whether it's a catalog, an ontology, a taxonomy, or whatever. It's just something germane to the organization where you could conceivably and and my question to you, riches and and as well, to you, Rob, is, couldn't you, would it make any sense? And it was just an idea that I, I was walking around in the in the garden last night, and I'm like thinking about this, and I'm saying, Could you mind the metadata in such a way that you could relatively quickly create that ontology or taxonomy from an enterprise.
Actually, you don't do it so much from the metadata, because, once again, how are you deciding what metadata gets, you know, gets captured, and what doesn't? The best way, the best, the best answer to your question is when you're doing knowledge graphs. For example, there are ways of taking your raw materials. Just say your procedure documents, all of the all the rules and regulations, all of the description of what the steps are, and you run them through what effectively is taking. Kind of open text and creating a an entity, you know, basically an entity, you know, identifier there. There are a whole series of of applications that basically go through and collect all the proper nouns, collect all of the the relationships and and collapse them. And so that you know, you basically have a list of all the proper nouns, all of the relationships that are of value. You deep dedupe that, and get rid of all of the the overlaps and kind of and that's a that's pretty automated process now. So for example, Neo, for j does exactly that and builds this very nice network graph. Then what you want to do is add, as you just pointed out, metadata to the nodes, classic embeddings to the nodes, so you use it like you would a normal vector database. That's when you get what you're looking for. And yes the answer is, you can do that, and it is automated. It's it's automatable.
So if you take that into the environment of a composable for example, would that give you like, I'm just thinking from an enterprise point of view, right? Like, and then how much? How would you if their agent How would you run it if, if
they're, if they're, if they're SDKs, if they're agentic, if their agents can be adapted to work with hybrid data sources, as opposed to pure play vector, data, data sources. Yeah, yeah, that you have the then Bob's your uncle. That is absolutely the way to do it.
That would be an interesting play. I don't know that they're going to go down that the key to is going to go down that road. It. But it's, I mean, his whole thing is, teach the machine or machine teaching, not machine learning. And I think it's, it's an interesting paradigm. I, like I said, I have a gap trying to figure out how you're going to operationalize this for an enterprise, and without doing a tremendous amount of pre work.
Well, the learning probably doesn't want to be incorporated, actually, in the LLM or into in the language model. It might be in the fine tuning if it once you kind of get something set up the way you want it. It's going to look more like you'll start out with rag okay. And over time, you'll, you know, get evaluation so that the combination of content and inquiry the way you set it up, and then the use of these other helper tools to to learn about the process. Then once you get to the point where you run evaluations, and you're hitting all the check marks every time, yeah, that's when you start to All right, let's, let's incorporate it into the, you know, longer term knowledge of of whatever the the agent or the the model is that we're using. Yeah, it's a it's a refinement process. It is a learning process.
It's, it's going to be interesting. Rob, somebody asked me the other day, I guess, was Tuesday. How do you take bare metal and make it ready for, like, literally out of the box? AI ready? Yeah, sorry. I told them to call you.
Thank you. What I know, what they meant by a ray riff, what did they mean by that joy?
Well, that's what I asked. What do you mean by that? Well, you know, we want to put we're doing a lot of work with Gen AI and I. Which model do you know, what kind of compute you need? Like, whoa, whoa, whoa, whoa, nobody's configuring these machines. We think we have to, just like, go to an outsource and say, Okay, we want to do X, give us y pre configure everything, and they want. And the interesting part is, they're trying to build some form of hardware guardrail into this, like if you go over a certain amount of shared memory, or if you go over a certain amount of not GPU. He said something else. It was just a bizarre, a bizarre question. And I said, you know, there's only a few people in the world I know that know this call rod, so yeah, we're, and somebody uses my name, you'll know this
is always take that call. All right. Yes, thank you. We're, actually, this is this. We're, we are, I'm looking at building a offering for RackN around strategic hardware consult like, do it? Do a we're seeing enough people asking the same question, and our customer like we're even inside of our customers, they're really struggling to think about how they can manage hardware strategically. What, you know, the Broadcom exit, the architectural changes between inferencing, this idea of, you know, like, we, like, I just described, where they're outsourcing training, and likely to keep outsourcing the integrating the power consumption and utilization refresh cycle, you know, AMD versus Intel, you know, maybe more art, like, like, there's a there's a lot of strategy stuff, and then the things that you're describing where you can totally turn on governance capabilities. It's a matter of, you know, having systems and controls to do it. We did. We the Tuesday group talked about this. The there's a the signing certificates for the Secure Boot operating system, boot chain. Okay, got, got they, they have some certificates in that chain that are shouldn't have been allowed. Meaning, does it? It's a huge it's a huge potential compromise where somebody could build a could sign an OS, a malicious OS with a very minimal key that is trusted inside of most BIOSIS, because they left these basically test keys in there and in there, and then there's a question as to whether or not they honor revocation lists, or if the revocation lists are updated, or things like that. So there's some, some very easy to use keys that you could use to sign an OS, to make it, keep it as a trusted, trusted OS. And so there's actually a significant repatriation discussion going on at the moment, but it's bracketed by, you know, when am I going to do a hardware refresh to exit VMware? How am I going to find enough power to do this. Which workloads should it be? How, you know, what? What am I going to buy? What is it? What does the next generation machine look like for, you know? And so, yeah, so
we're,
we're entertaining this idea of taking this very unique industry knowledge that we have
and making it a sort of a consultative survey surface. I
That's a great idea. I think you're going to find a lot, a big market in that, particularly around and because this is a question that I get a lot now, is, if I want to really secure my AI, shouldn't I bring it back into the enterprise, as opposed to keep it anywhere in the cloud, in a public cloud?
That's it's a very real question. Yeah, the interesting thing is, and straight back to our small model, but the question I had on the on the training here is because there's a lot of fear about how much data you're going to leak for the training exercise, which which is a risk. But instead of leak, like, how much, how much data are you going to have to potentially put at risk for that training exercise, if you're giving it over, let's say somebody else to do the training. How much of my data is going to composable, passing through their systems with potentially
with unknown that is security that shoot, that's where you get very, very, yeah, you're, you're right. And then you and they are very,
yeah. Now you, now you actually have corporate secrets that are, are, USB size, USB size.
You can you This is why you're going to do a lot of the first iterations of this in rag locally, not very performant. It's going to be a lot of testing at that point. Then if we're, if we're, if you're really solid on this, and it's mostly operational data that you're using, you're going to use synthetic data. You're going to generate synthetic data to do the training, right?
Oh, I see that makes sense. And this, these are all new skills that companies need to have. I do expect that. And this was the the training a small model is not a one and done. Thing was, what I what I don't know is what the half life is of a small model,
of a train, of a trained version on a small model,
right? So you go through a small model, you train it. Yay. I'm happy. Is that a one week thing is a I mean, I'm sure it's going to be highly variable, but am I going to be a quarter? You know, I can't imagine it being longer than three they're short, you know, longer than three months. But, you know, hopefully you're not distributing a new trained, small model every
which, again, is why you you you have a trained model, fine tuned and on. And using that model, you build a rag, which is very, you know, easily changed. You can find stuff that that's in error. You can rearrange things. You can slice it and dice it in different ways. You can you can experiment with it before you commit to the fine tuning process, and that. Uh, observe is observability evaluation, governance on that data is what you're going to do locally in house, you're going to, you're going to have a lot of those kinds of to that kind of tooling. And the result of that, will then go and say, All right, we're ready for our semi annual update, you know, find, you know, revision of our fine tuned model. That's likely to happen.
That makes sense. So, so I have a question about that, which has been niggling at me for a long time. The world is dynamic. Well, okay, be like mentally Be my guest. Must be Thursday. Everything is dynamic. When you build your small model, and you're constantly iterating it, and you get it to a point of, let's say, you know, 99% fine tuned, and then the world shifts. Things happen, the augmentation of new scenarios into that small language model. Does that is that actually the end of the shelf life of the small model? Or No? Does it
that that's when you that's when you add to its behavior the specified knowledge that comes from a rag. It's like saying I've got this kind of inefficient and and it's a little clunky overlay on my fine tuned model that I did you know, on January 1, right, and I'm, I'm refining it, I'm filtering it, I'm adding, I'm adding content or kind of new piece parts that are going to show up and do a better job to a question or to a to a synthesis. And that's also where you implement things like network graphs or relational databases in concert with and all of that is built first on the rag. That's where you you absolutely, kind of do all of the the testing and evaluation, and you get to use the fine tuned model again that you've adjusted in for some period of time, until you decide, I know enough from having done evaluation. I have a governance approach to the data that's going to go into the fine tuning. Now I'm ready to go and generate a new fine tuned model. You do have lifetime. It's just it gets it gets slower. It gets a little bit more. There's a, there's a different level of effort, and it's a, there's a lot of it's local. It's capable of being secured, and you don't waste your money doing a fine tuning before it's time. No no wine before it's time. Right,
right, right. And you know, I'm looking at it from the point of view of and I know this is going to sound strange, but the event trigger that would make you go through that process. You can have a frequency issue of, you know, this happened, then something else happened, then something else happened, and you end up with an SLM that then becomes noisy, for lack of a better word, even though you've done a lot of the work in the rag,
yeah, you have to account for there are things that you know in any model, machine learning models, you know, with regression or Gen EIA, there is such a thing called drift. And the models drift the context changes to your point, and that's why you kind of have these fine tunings, these adjustments that you can do using rag without going and constantly. Doing a fine tuning of this small language model that that, and this is the question that that Rob is asking, and yeah, you have to think of them as kind of layers, and that's probably the best way to think about it. But there is there is a pipeline and
clearly, but then the answer is the answer to Rob's question, one of the small language model may have a longer shelf life than you anticipate by the fact that you can use the rag and do all of that. So, okay, so you really, it's like putting oil in the engine. It's, it's, you can do it without changing the engine.
Yeah, okay, here's the here's the thing you then have to invest in how to test and evaluate the response of this combined Small language model in Reg, and that's going to be very situational. I mean, you made the point, you know what? What would be a an acceptable answer in one company for a particular process would be completely unusable in another one. So it's like setting up a big set of questions and and check boxes so that you can run a test multiple times, ask the question in different ways, make the requests in different ways, and have basically a separate, a separate language model that basically is the you know, the greater you know, kind of saying, yeah, it hit that one. Check, check, check, check, no, didn't make it on that question. Didn't make it on that aspect which says, Go back, change the rag, modify it, until you can get it to that level of accuracy that you're that you see as acceptable. So
then, how do you would you use the same process in a situation where, you know, we all have different points of view. We all have different perceptions of or or intrinsic understanding of,
no, we don't. There's only mine. Joanne, sorry, I had to chime in. We
were all thinking it to him, it's so good, you're good,
you're good. Go on.
Thanks, Tim. Really appreciate that.
I had to, like,
I had to lighten it up a bit. I mean, you guys are going
super deep. I get that. I get that. And she who thinks of the question, it's only my insight that actually counts. However That being said, you get where I'm coming from, we all have different ways of interpreting a question or a or a piece of data based on our intrinsic and or tacit knowledge of a topic. So how do you so my next question with regard to what you were saying is, it's not about bias, it's about
it is about personalization.
Of you know, here's your basic slim, and now you want to what if it were personal? Like a recommendations engine, but personalize it based on your level of expertise or your level of experience doing a particular thing.
If you have a process that you it's an industrial process or an office process or something, and you don't have automatons doing the work. You have people, their associates, their others. You provide them with guidance, process descriptions, and as you're training them to into the job, you check it, you have somebody who is designated as, or some group that's designated as, have they done an acceptable job on whatever the process is, and the. They basically come to an agreement. And, yeah, there's going to be those parts of the thing that is going to be very much a subjective issue. You know what? The The only thing that you can do is, is try to model the model the you know, the the persons or the group of people who are you know, seem to do the best job of of, you know, culling the acceptable from the unacceptable and making them the kind of, the arbiter of of what passes evaluation and what does not?
Yeah, I think so, because I heard something earlier this morning, and then it'll be move, allow you to move off the topic where they were talking about using AI in the workforce, and, you know, a mean training time for certain operators in certain industries, and this would be like anywhere between insurance and an actuary who has 30 years experience versus a newbie who's got two and doesn't and is not considered to be qualified until they have five or 10 years of experience. How do you ate after that knowledge from that arbiter of 30 years. And how do you fast track the learning and experiential expertise of the newbie? How do you fast track their education of like what you've actually done for that period of time, and that's why I asked the
question, can I ask can I ask you a clarifying, clarifying question, and that is, does the answer have to be a completely automated process? Because what we're talking about here is actually a situation that the banks are finding themselves in right now to validate and document for the purposes of compliance their risk models, right? This is an example I'm into in big depth right now, and up until recently, the way in which you validated a risk model. That is the kind of process that you use to decide whether you're going to give someone a loan, credit card, how big a loan so forth is to take a freshly minted MBA, sit them down with a big manual and say Your job is to run through this drill, validate, you know, learn how to validate these models and document them and the models that were that are in the bank. You know, some of these banks have 1000 different risk models, so when the law changes and they have to validate them at least once a year, if not every or every other year, 1000 of those models, that's a lot of that's a lot of people, because each one of those takes six to eight weeks to do one model. It turns out that you can use machine learning and some Gen AI to reduce that six to eight weeks to 10 days. Okay, but it's 10 days including people time. There's 10 days of people time involved in there. And so what you're doing is you're giving somebody like these newly minted MBA some superpowers. That's where you go with stuff like what you're what you're addressing here over time, maybe you get better with the models and do a better job of validating.
So what happens? Go ahead,
Tim, well, I was
something I was thinking about as I was listening to the two of you talk, and I'm sorry for coming in kind of midstream here, but something that that isn't being discussed as widely, but I think is, is going to be a real issue if we don't address these, these problems, is when you get into model proliferation, so you get. Smaller models, and you get to more of them. And then the other piece, and this came up, this is something I had raised in a couple conversations when Rob and I were in Vail, was what about when you start pushing models to the edge? You know, there's this assumption that models run in the public cloud or run on big infrastructure at least. That's the that's the big conversation, right? But the reality is, and one of the companies that I've been on the advisory board of now for for a number of years, is about moving models literally to the edge edge, right? So the use case that that they're using, and it's a great example of this is, imagine a soldier in theater completely disconnected from the network because they don't want to give away their position. They have a small Android device on their chest, and they're able to do threat detection, which is based off generative AI running on that device. So, yeah, the edge, edge. But you have the same thing when you talk about manufacturing lines and anomaly detection and things like that, right?
I'm sorry, like quite a semi automated car, autonomous vehicle,
right? Vehicle, right, right, right. But, but, if, but going back to my the core of my question, you almost have to have some degree of automation, and I don't know, I don't want to say policy, but some means in which to manage and avoid model sprawl and Model proliferation to the point that now you've got these models that may not represent the changing of the data. You know, we talk about how you have model drift. Well, model drift can be a good thing too, right? It isn't. It isn't just bad. But then the other piece is, as you start to push this to the edge, it's a quantity, and it's a, it's a management piece. So anyway, I was just curious, kind of, how you thought about that too, in the context of this conversation,
it's gotta be it's gotta be incorporated. You're absolutely right.
It has to be incorporated. But the other thing is also, and I and again, this is part of a similar discussion that I had yesterday was the contextualization around the model, almost a semantic layer, and how that gets infused, and how that data stream starts to change the model, because you may be missing an attribute, or you may be missing something else that later on would be very important. So to me, it has to be as automated as possible, so that you can call in those and the discussion was just as an example, Thomson Reuters, or Reuters wants to take a third dimension, if you will, into the small language model and add real time data on things like weather or commodity pricing to the small language model that's being used or that's about to be talked about by some of the big equipment manufacturers, like a caterpillar or a John Deere or whatever, where they don't manufacture tractors or equipment, they sell product as an experience. So how many more data feeds may come in? Could be anywhere from the commodity markets on greens to the weather forecast for, you know, a longer range period of
time, it's going to result in the kind of question, yeah, and that's it's going to result in two, in two things, Tim, one is, there's going to be self testing, you know, Basically it's like anything that you know, if you think about them as, as any piece of automation, there's going to be measurement, you know, is it, is it still working within the parameters that was set? Is it, you know, is it failing? Is it failing slowly? Is it failing more now, those types of things, so figure that every one of those models is going to have to be subject to, you know, local co resident testing for the most, you know, catastrophic kinds of failures.
I hear what
go ahead, and then bringing it, bringing it back to the shop, every you know, you know, every night, every week, every month, whatever the right time period is. So
here's the here's the piece that I get concerned with. So I completely agree with you. Um. Right? And I think that's that's a really kind of smart approach to take with this. I just worry that, in reality, this plays out very differently based on past experience, so maybe not the best example. But let's, let's look at this from a report standpoint, or even from a virtualization standpoint or resource consumption standpoint, there should have been some form of management to ensure that I'm using management in a very little m you know, broad management is a term, but there should be some degree of maturity around being able to do exactly what you outlined. However, what I get concerned with is someone does it, and then they move on to the next thing, and there's no reevaluation of it. Kind of to Joanne's point. You brought in weather data, or you brought in some data set. Weather data is probably not a good example, just because it's highly dynamic, but you brought in some data set, and it doesn't change that much, but it does change over time, and someone doesn't go back and refactor or update the model, these small models, especially at the edge. And so, as you get to that proliferation point where you see this model sprawl, and I'm, I'm just projecting here. I'm, I don't have a good use case for this, but I anticipate this will be an issue that, unless you have that degree of discipline that you outline rich, we're gonna, we're going to have these, these data sets that are sitting out there. I mean, we see this in Excel, right in finance. You know, I took a copy of my Excel spreadsheet, gave it to you. You started working with it, massaging the data. Now you have some core data, but you also have an old set of data that I have since moved on with.
You are addressing exactly one of the three major problems that I've been working on for far years with provident, and that is, how do you create a think of it as a Pub Sub kind of an approach to those data sets, and it's it is literally to create, logically, a master which, if there's any change, Prop, can propagate the change to All of the subscribe the legitimate subscribers know when the legitimate subscriber has actually flipped the switch and said yes, incorporate it or not and do exactly what you're talking about. And by the way, if somebody fudges it or does a cut and paste of a of an exec spreadsheet, got Excel spreadsheet and use that and start to throw it in there be able to detect that someone has violated the right, the the information, the data, use rights, or the license, if you want to think of it that way, that is basically what we do with GitHub and code. This is what and that in conjunction with with, you know, the managers that that look at dependencies, that is what you should be doing with data sets.
So you said something, and I know we're, we're coming up on the top of the hour. Unfortunately, I'm going to have to break here. But something you said Rich, I think is is really key, and that is in the conversations that I've been in, when they're talking about smaller models, they're not talking about tying it back to the original work, but what they're really talking about my words, not theirs, is an orphaned situation, right, where I've taken a copy and moved on with it, and now I have an orphan i that is not tied back to the original. So if you update the original, my orphan or my subordinate, or what, how, you know, I'm sure there's a better term to use, but my subordinate, of that should get that update, or at least be alerted that, hey, there's an update. But that isn't that is not the conversation that I hear a lot of folks should be for SLM. If
somebody shows up with a, you know, with a model that needs to get compared to the the master, it's. Be able to say, You know what? That's not a legitimate model, that's not up to date. Here's why I know that and so forth. But yeah, great
conversation. I got a jam yet again. All right, great to see you all have a good one. A
week. Okay,
we missed the V we knew we missed venture capital models. Yeah, it's
not that exciting a thing so well, yeah, more interesting next, next week is software defined, devices and edge. But by the way, I'm a new event showed up on my potential calendar called yada. It's an, it's an infrastructure event. There's, there's I'm tracking and trying to decide about travel on three events, y
O T, T a,
y O T, T, A.
Is this related to the the the big data stuff that was being done out of Canada years ago. I
don't know could be it looks I'm trying to figure out, like the sponsor. It's an event sponsors. It's an event in Vegas, October 7, the ninth
ampere cable. I've heard about this.
Is there a
lot of sponsors, including shell Oracle Alliance. Wow,
yeah. Oh, that it's
related to DCD. I think it's managed by the DC the data center dynamics people.
It is,
yeah, the reason, I think the reason that shell is part of it and some other people in oil and gas, is because this is about, it's, kind of framed around sustainability, I guess, is the best way to put it, okay, and the power consumption being used by AI, and how you're going to really deal with this. And it should be an interesting event, but I don't know, I haven't seen a detailed agenda. This is not like big debt in big data Toronto, which goes on every year, which is a huge conference. But I was toying with this. I was thinking about doing a speaking,
yeah, I hadn't heard of
it until this week, and it looks interesting.
So yeah, does. I'm really kind of fascinated though, that we got on this primarily because the topics of conversation that you were having in Vail in film were so directly, you know, kind of in this, what are the pragmatics and doing using AI, where do you use it? How do you distribute it? What are all the other parts of it? Yeah, it was
surprising, you know, surprisingly practical for that.
Well, I have a question, though, when Tim was speaking, and he was talking about sprawl, is his definition of sprawl? The broadening of the model, or no, the proliferation of
proliferation of distinct models that that, okay, that kind of they might all have had the initial, the same origin, but they their child or derivative models and, and, you know, or somebody tweaks It, you know, I'm going to add this little spell. My conditions, my context, demand that I add this little, little thing to the to the recipe, and now suddenly it's distinct from the others. And that's, I think, what he's talking about by sprawl, like VM, sprawl,
yeah, and so that would be, hmm, wouldn't you have governance around stuff like that? Yeah, yeah, that.
I think that's what the question Rishi was asking. What is what constitutes governance that would keep stuff like that from happening and keep it in check,
by the way, a question about that, that bank example that you were doing, yeah, can you not use, I mean, is it a CDC issue? Is it a deal, you know, come. Parent contrast the delta so you can reduce the amount of time, because when legislation and compliance rules change, they can do it down to the, you know, minutia. You can put a lot of stuff in that people never realize or hear about in, in an appendix or something like that. Can it not? Can you not scan between year one and year two? Pick up what those changes? What are,
what they're, what they are, what they are looking at are validating the models that are being used to make loan decisions. For example, you're right, right? And what they're there, they have a whole kind of checklist, you know, are there, you know? Is it devoid of bias on some basis that we can identify? Could be, you know, so that has to be at, does the model check for that? Is it biased in any way? How is that decision made? That's what validating the the risk model is all about, and documenting what it does. It's it's it and the other part is documenting it. It's documenting it for internal use, but also for an outside auditor to come in, take a look and say, You have done a tremendously great job. Yes, there are some failures in the scheme of things. No model is perfect. Certainly wasn't perfect. When we add human beings doing it, it's better. Now you've you've taken enough care to eliminate bias, to adjust to changes, things like that, and that's what the auditor is going to basically sign off on for the bank. Wow. But in order to get there is for a human being to do it. It's a hell of a lot of work. If you can reduce that by 80%
sure it's a big deal. Well, well,
and, and, you know, the I don't want to take up too much time, I know I do
need to ride to wrap up. Yes,
oh, sorry. Bias is one thing. Nuance is separate. How we use words, the way words change in regulatory documents, can be a nightmare, because the perception of what you're reading is different. And that's really the question that I was asking of Could you, could you cut it down to even less number of people days you can,
and it's over time, and part of it is getting terminology straight, uniformity of terminology and identification of relationships and and kind of using some of these other other Tools, besides pure Gen AI, by which I mean vector. You know, using vectors and similarity metrics to to generate synthesis and doing the inference. In other words, you're, you're throwing multiple ways of of doing it, and that's, that's what we're talking about. Yeah, right, these things change and will happen over time. Yeah,
early days. Thank you. Sorry. I sorry.
That's okay. This is what I love about this, these conversations, they're, you know, impromptu, and they, they follow where we want to go, just the best,
absolutely. All right, thank
you. Take care. See ya. Bye, bye.
This type of conversation is one of the most amazing aspects of what we've created with the cloud. 2030, groups. If you're listening to me do these howtros, then you are enjoying it. Also, always want to encourage you to come in be part of the conversation. You can find out our list and schedule at the 2030 dot cloud, and I will tell you like today's conversation. We went deep because this was a topic of interest. We had planned to talk about VCs and do an update in how the investment industry is working and influencing things, and we'll get to that. But I love when we can take a topic and then explore it deeply, conversationally. I hope you're enjoying it, too. Thanks. Thank you for listening to the cloud 20. 30 podcast. It is sponsored by RackN, where we are really working to build a community of people who are using and thinking about infrastructure differently, because that's what RackN does. We write software that helps put operators back in control of distributed infrastructure, really thinking about how things should be run and building software that makes that possible. If this is interesting to you, please try out the software. We would love to get your opinion and hear how you think this could transform infrastructure more broadly, or just keep enjoying the podcast and coming to the discussions and laying out your thoughts and how you see the future unfolding. It's all part of building a better infrastructure, operations community. Thank you.