Loading...
you. Thank you. So I can't set up our next speaker is Sally Ann Bucha, and she's going to explore the use of guard rails and experiments to enhance AI application performance and reliability. This talk is going to provide specific insights into optimizing the development processes to achieve cutting edge results once applications hit production, as I mentioned, Sally Ann is the product manager at Rise AI, which is a leader in LLM observability and evaluation. She's a passionate machine learning enthusiast and generative AI specialist. She holds a Master's Degree in Applied data science. She combines a creative outlet with a dedication to developing solutions that are not only technically sound, but also socially responsible. Please join me in welcoming Sally and Delicia.
chatgpt, etc, okay, almost 55 a little bit over 55% actually, of those same users report that they're using unapproved generative AI tools at work anyway. They're using chatgpt on their phones. They're emailing things to themselves. They're off corporate networks. They're getting around the guardrails because they know the productivity game. These are not malicious people. These are people who are trying to get their job done. Now here's the big 160. 4% of those individuals, however, are using that generative AI powered content or work, they are representing it as their own when they go to their managers and their leadership teams won't touch that one too much. But there's issues here. In terms of organizations are developing policies, procedures, processes for generative AI employees aren't following it. And again, there's not a whole lot of malice here, folks, right? The typical healthcare practitioner, a front office person in a doctor's office, the typical insurance underwriter, the typical Wealth Management advisor, the typical individual working in these regulated industries is buried under work. They're trying to get their job done. They want to do their job well, because don't we all and how could they not be attracted to technology that says, I can help you do that better, faster, higher quality. So that is why we came into the breach. Steve and I came from FIS. We saw firsthand what it was like for a regular
ensure that interactions with large language models are properly cleansed of this information in a way that doesn't destroy the overall flow of communication. So how do we do this? The core of our platform is very fortunate. I'm very fortunate to have a head of machine learning who is a multi competition winning model designer, we have an algorithm in place that is incredibly good at understanding where sensitive data is likely to exist in a prompt. Don't have a big database of every bit of sensitive information on the sun. We don't have every single regular expression or rules engine out there that handle sensitive data. We've got an algorithm that understands what are the ways in which these individuals are likely to talk about sensitive data, what modes, what contexts, etc. So what liminal does is it acts as that secure gateway for the employees of that hospital that ensure that wealth management firm, that life sciences company, in a way where we don't put the burden on the end user. So slide of AI. We say, do your job. That's what you're trying to do here. We'll take care of the security side of things for you. Now, crucial to this, it's easy, actually, really, to redact sensitive information, but as we know, large language models thrive on context. If I give a bunch of redactions to a large language model, I'm going to get a bunch of garbage back and won't do what I've asked it to do. So that latter half of this whole exchange is the ability to retain all the rich context, the intention of the user's attitude and that prompt, so that even though the sensitive data is taken out. It's no longer non compliant for that particular organization. We can ensure that the response to that end user, again, that front office staff at a doctor's office, that insurer, that underwriter, whoever that is, that they get the interaction that they're expecting from generative AI. And we do this across the three big modalities. I think we've all seen this. There are probably three big categories of use around gender debate.
a bit more fun to show you how we do that. So I'm going to take a look today at the actual liminal platform, and I'm going to show you kind of two different angles. Remember those two kind of primary masters, if you will, that we have to serve in a regulated industry. Number one, that end user, that person at the doctor's office, that underwriter, that wealth management, advisor, whoever that is, the individual is trying to get their job done, but also the cyber security organization whose job it is to ensure that these interactions are safe, secure and compliant. So I'm going to start here. This is the I'm in dashboard. I'm in cyber security. I like dashboards. So this is where I start my day. One of the things I love to highlight liminal we have the pleasure and the honor of not having a particular course in the generative AI race. Today, we support about 13 different providers, all the names, you know, perplexity, open AI Azure, open area and drop it, etc. We do that because our earliest hypothesis is proving true in these industries. As you all know, there is such an arms race right now, with all of these providers coming out every single day, one upping each other with models. Everyone in these industries is looking for how do I get my job done better? And they recognize that there are different models trained for different purposes across these different modalities and functionalities. Luminal allows you to connect as many instances of every single model under those 13 providers that you like. Crucially, as an administrator in these organizations, I can say whether that model is licensed or available to the entire organization, or whether it's available to a particular team. You can imagine Aaron the sales guy in CC. I'm not a sales guy, but if I was a sales guy, you can imagine Aaron the sales guy that in one particular model with very restrictive policies, but maybe a research group gets access to six or seven models with looser policies. This is the first part of observability and governance over generative. Ai, these individuals in cyber security are used to being able to provide fine grained control over the experiences based on who you are and what your job function is. We allow them to do that. Regenerative. AI, I'm gonna spend just a moment here too in our policy controls. This is where from the simple little screen, but this is where the magic happens. And what do we do when we find generative, excuse me, when we find sensitive data inside a generative AI prompt, liminal comes out of the box knowing a lot about categories of sensitive information, PII, medical information, financial information, also allow organizations to define their own custom terminology, as you might imagine, just because you're a hospital and you're under HIPAA regulation, that doesn't mean you might not have data that is specifically sensitive to your healthcare organization, regardless of what that is, we allow the administrators the ability to say, based on either globally or across all of the models that have been connected to my organization, what Do I want to do when I find sensitive data? I mentioned this before. The core of our algorithm is a set of models arrayed in an ensemble that we have trained for the purpose of looking for sensitive information in context. That doesn't mean we know every name under the sun. We know every location or medical facility or occupation on the sun. The Model understands. What are the diverse manners in which people talk about these concepts? Sounds obvious, right? Regulated industries is full of expressions and huge databases and Rules Engines. It's kind of a novel approach for them to believe that an algorithm can understand what you're trying to represent, even if we don't happen to know the specific manifestation of what you're representing. Okay? So we detect a lot of different sensitive data. Sensitive data, the policies that we apply very straightforward. Again, we want this to be a very clean user experience. A lot of security tooling. If you've been in cybersecurity like I have, user experience is not high on the list. We allow users the ability to say, what do I want to do when I find this data? The two I'm going to highlight here redaction is exactly what it sounds like, and replace Aaron Bach with person zero. It's like I never existed. I don't care if the downstream model provider says they're not training their model on my data. I don't care you're going to learn about person zero at Aaron Bach. Intelligent masking is a little bit different, where we're going to apply a heuristic to that particular category that eliminates the sensitivity, but leaves behind context that's going to be useful for the LLM to be able to produce a response. So an easy one. If I were to masks, mask ages, that's an easy 138, years old. Under the HIPAA guidelines, that's identifiable. You cannot use that information because that could identify me. But late 30s, as long as there's no other identifying information that promptly surveys is okay. So intelligent asking is a way that we apply heuristics to different concepts of data such that it
removes the identifiability leaves behind context. A lot of other stuff we could touch in here, but you get an understanding now of empowering the, excuse me, cybersecurity organization with the ability to enable safe, security, generative AI, I want to show you what this looks like from an end user standpoint, merci for chat experience. Remember chat old hat to all of us, to individuals in the regulated industries, this is the most common modality through which they are engaging, generally not what they're used to. Now I'm going to come to my prompt here, or paste in one that I created for this event. Want to show you something here. I pasted this prompt and notice right away some sensitive data. Wanted to prove to you database under the sun, that's Elon Musk kid's actual son's name. I don't know how to pronounce it. It's not a name I've seen before, but our algorithm understands the context in which a name is likely to occur. We understand it. Crucially, know what we're doing here. If you've been in cyber security at all, been around data loss protection, data loss prevention software, when you put sensitive information into a
platform, you get your hand slapped. Don't do that. Aaron, rewrite your prompt. Get rid of the sensitive data. Well, we know in generative AI, that's not going to work, because that's the whole point of me writing this prompt. We don't do that. We say the user, hey, we identified a couple things here, some blue things, some yellow things. I don't need you to worry about that right now. Go ahead and submit your prompt. In this particular case, I'm gonna pick GPT four open AI, little behind the scenes. It's going to cleanse that information on the fly to open AI, as that response gets streamed back to us, chunk by chunk, we're going to rehydrate it on the fly. Show you what that means here in a moment. But moment. But crucially, you can see if I'm writing here by creating a comprehensive insurance proposal. I'm an underwriter of some sort. I'm getting what I expect. I got a client X, whatever I should write his name, client. He's not.
got my job done here. This individual needs me to provide advice for them on insurance, I was able to plug in some data and get that information to be quick and easily. I can copy paste this massage and move on with my day. Now, again, that serves one end user, but we got to make sure that we serve both. And if I really quickly go back here to our admin dashboard, if I'm an IT administrator in this organization, you'll notice here is a log entry that shows the prompt as the user input it. But crucially, here's what we actually sent to open AI, very straightforward. You see us, based on the policies of this model for this team inside this organization, redacted that individual's name. You see an example of intelligent masking of their age. You see an example of intelligent masking of a street address, HIPAA says that a state is okay as long as there's no other identifiable information, maybe the large language model will write differently to somebody in their mid 20s who lives in California. I don't but nevertheless, we enabled that interaction for that individual, and did it in such a manner that we retained a sense of our data security, sovereignty, governance and observability in a manner that I can show my auditors. Yes, I am compliant. I've done what you've asked me to do, and I've ensured the highest standard while enabling this technology that can impact so much of my workforce. Now, again, I could talk all day about how we do this, maybe a little bit of time back there, but want to just highlight one thing as I kind of talk about the founding journey of liminal. I was recently part of a accelerator program where we got invited by a very, very large bank, a name you would know, to talk about the idea of responsible deployment of AI, generative AI, intrinsically a bit AI at large. And of course, everybody in the audience talked about important topics, removing bias, handling hallucinations, data sovereignty, all that type of stuff. One that has become particularly pertinent to me when we think about the responsible deployment of AI, regardless of our industries, regardless of our experience of craft. I'm thinking about those individuals in those industries. I'm thinking about the story that I heard of a doctor and the network of one of our employees, a doctor whose PTO was docked because they didn't complete their patient notes on time. They didn't complete their patient notes on time because they have been working around the clock for the last year, making up for staffing shortages within their hospital. We're trying to harm anybody. But they got deal. They're what they need more than anything, is time off. And they got that taken away from them because their organization had a policy, maybe followed policy. I'm thinking about a large wealth management firm that I spoke to recently, and watching individuals talk about the long 550, 100 year plan of generative AI, and watching all these wealth managers just fall asleep, watching them lower their eyes, and watching them perk up. When we start to talk about, how can we help you serve your clients better, your clients who are relying on you for their retirement. When we talk about the responsible deployment of AI at large, among all the other topics that we're already used to, I want to encourage you to think about what is on my heart really, which is the legions and legions and legions of everyday people around the world and these organizations, they are just trying to get their job done. They're trying to do a good job. They're trying to meet their performance reviews. They're trying to get their boss to be happy with their work. They're trying to go home and be with their families at the end of the day, we have an opportunity to unlock the capabilities of generative AI in a safe and secure and compliant fashion to ease the burden on those individuals. That's sort of what we carry forward when we think about the individuals in the regulated industries, legions of them just mired in endless busy work. And that part of the ultimate dream and vision of generative AI and its responsible deployment is pulling that load off their back by enabling a safe and secure and compliant experience. Thanks so much.