Rob, Hello, I'm Rob Hirschfeld, CEO and co founder of RackN and your host for the cloud 23 podcast. In this episode, we talk about SSH, secure shell, and one of those topics that people take for granted because it is a ubiquitous way to log in and access systems, but true to form for the tech ops series, we break that down into much more detailed and granular components. We talk about how to secure it, what best practices are, how to use it for tunneling, or, more specifically, not use it for tunneling, and why all of this matters to your operations environment and what new things we're doing that avoid having to have network access at all fascinating and interesting conversation that I know you will enjoy.
I thought that they were trying to eliminate VPNs by using browsers like you would attach, you could attach through a gateway to an internal network or a secure network. And they were, they were trying to take on the VPN network, the VPN, which is that's actually gonna tie in. I think this will tie into the topic of the day, but none
definitely that sounds like they're like Identity Aware of proxies, and not just your trust or kind of implementations. Yeah, those exist. Do
Okay, are they? Are they useful as an alternative to a VPN? Oh,
absolutely, that they're. They're arguably even, even safer than VPNs, because with a VPN, it's, it's all or nothing. Once you're connected to the end, you have access to every network that the VPN has with Identity Aware Proxy, essentially you browser connects to the proxy. Proxy authenticates you and authenticates the specific and authorizes the specific request that you're making. So you can be very, very granular in what you want to allow certain users to access,
but never look past the cynical of Rob. I think some of that's also trying to get people to skip region aware content blocking and content direction. So if you can do that without your VPN, because that's what a lot of people are using VPNs on a non commercial, private, residential, whatever you want to call it. Methodology is, I just don't want to look like I'm in America, or I don't look like I'm coming from blah, so I can take advantage of this or that, or whatever. So there's an element of some of that. And I can see a browser trying to or a browser service effectively saying, we're going to provide that. You know by
it, but I mean that means that browser is literally somewhere there, there's okay. That's what the Identity Aware Proxy is, okay. So they're sending your traffic through another route, rather than directly to the website. Well,
I think Identity Aware Proxy is solving a real business issue, which is, I want Greg to have different opt out, access and controls than Rob. That's potentially a real business use case. So I'm talking about, I want to skip ads, or I want to look like I'm from Sweden so I can get access to this product, piece versus right?
Yeah, word, process, identity, where proxy is a server side solution. It is for you as a company to allow your team access anywhere in the world without a VPN, as long as they provide their identity and essentially like it. It's not so different from, really, from OAuth, like you can access Gmail anywhere in the world without a VPN.
But that's Gmail is a public service, I guess. I mean, I'm usually a VPN is to connect traffic behind your firewall. You're, you're maybe there's two ways to think about it, right. You're you have inbound and outbound, but
it there's a public service. But before you can can access your own inbox, you need to authenticate what Google serves. Yes, and that's essentially that the process that the identity world proxy does is just like, Okay, who are you? And you'll respond back say, Okay, I'm this person, and that is typically an OAuth type authorization. And then, based on who you are, the proxy knows what you're about to access, and then that passes that information on to the back end, saying, Okay, this is the person that's sending this request, and the back end can then survey your own personal data.
Think the confusion they end up causing in the market is, I don't think they clearly express, from the get go, where that service sits. Is it behind the firewall? Is it in front of the firewall? I think that's the the thing they don't properly articulate typically, when talking about zero trust or the Identity Aware proxies that I've seen.
Yeah, I mean, an Identity Aware Proxy is like, if you boil it down, it can be nothing more than a reverse proxy with an authorization component. So it okay, for example, if you use an English gateway in coordinates like English and Gen X or whatever, it's a reverse proxy. It has routes, and you can optionally add authentication to it. So it is a poor man's IDP. IDP. Now the value added by these Identity Aware proxies is that they give you all the management console. They give you the auditing capabilities to give you the additional security policies to say, okay, like so many failed authentications, block the user and so on. I mean, I
mean, at some point you're what you're describing, I would describe as a service mesh.
That's what, as we were building up service meshes, that was a lot of what you're describing, you know, this would was, you know, that type of traffic routing was, you know, if you include the fact that you could go to multiple, there's multiple endpoints to service your traffic, and something's making making aware of that that's very service mesh.
Yeah, you're not wrong. Service mesh is essentially another implementation of zero trust authentication on the difference is that service mesh is service IEP is the user to service, makes sense?
Well, let me. Let me do this, because I want to. I think what we're talking about is not that dissimilar than what the topic of the day was, which was SSH, which, to me, in some cases, can sound like the most boring topic ever, because it's just, you know, getting, you know, on the surface is just get a terminal to a from one computer to another. But there's a huge but with this, because it's been used so dramatically for so much stuff, it's, I think it's worth talking about SSH more broadly, and by so much stuff, I'm meaning actually building VPN, like tunnels to and through different networks, and using SSH gateway as an entry point for a system, or even something like using Ansible, where, you know, you're not, you don't have to install an agent, but I'm Eric putting no agent. But what you can do is you can SSH into a box and then have, you know, run a whole bunch of commands through Ansible. You know, SSH has really morphed control into a control plane, and I'm interested in people's first thoughts on that. But you know, what do we need to know to do it? Well, I. And I'll put in my my the topics I had as primer, so that you can see some of what I'm thinking towards. Am I wrong? Is that is this? Am I? Am I giving SSH more credit than it deserves, or too little?
I wouldn't go as far as calling SSH a control plane, but it is a very convenient and reasonably efficient communications platform. I'm going to say reasonably efficient, because there's clue, there's clearly some protocols that are much more efficient, like gRPC, which one like, but really for like, given its its history, SSH has withstood the test of time, sure, and I think one of the key things is that it made the authentication and authorization part of communicating with the remote system relatively unobtrusive.
Having said that the SSH that we have today is significantly different from the SSH that we've had 10 or 20 years ago. There's been a lot of improvements, and we'll probably be spending a lot of time talking about those.
Well, how do you how did, how is it different? How's it better? Um, I hadn't thought about it having changed, changed much at all
the when SSH did keep up with modern authentication methods, like nowadays, you can, for example, use on an SK type SSH key, which means that when you SSH in the system, you not only have your SSH Key, you not only have your client present the SSH key, but you also need to use a security key, which is what the SK stands for, QA, and this is baked into the SSH key when you're generating it, the deciphers evolved significantly as well. Like RSA, is by far now the least preferred one, and it's still available, particularly in older systems. But like the recommended one is the ED five, five digit string I keep forgetting. Um,
does that defeat like, I mean, if I was edcsa, sorry. Ed,
no, no, that's a different one. That's ECB, DSA, but I'm talking about Ed 250, 519,
well, they're both the new elliptic curve kind of fast processing version, right? Yeah. So, yeah, there may be a Yeah, that's
cool. I'm sorry, Rob, we talked over you. No,
it was fun. You guys were clarifying something important. I mean, some of that the performance, just how fast you can log in, or is it, you know, or is it actually, I'm assuming it's also more secure from actually what it's encrypting, okay? And then the performance in the encryption matters, because if it's over, there's there's going to be overhead in whatever you're encrypting.
Um, there's a number of things on this, on a lot of them, like cryptographers spent a lot of time thinking about this. So there's a lot of clever things happening here. One is that, yes, you can, you can verify a much more complex key in less amount of time, which means that you can use more secure keys, like higher
or establishing the secure. Or for actually logging into the systems. Yeah. The
other one also is that the these new types of keys are much more resistant to timing attacks, and that is with some hashing algorithms, you can start guessing what the expected key should be based on how long it takes the system to respond to you. Okay with this, with the newer elliptic curve, once the response time is much more normalized, so you're not able to do that undersh server itself. That's also so capabilities there, but, but that's not the only thing. Also, like nowadays, with on SSH servers, you can even use TLS keys to authenticate, which is incredibly convenient, because then you can tie that into an identity provider, which just gives you an MTLS client key, a short lived one, and you can use that to log in,
so you could bounce through like an OAuth provider, is what you're thinking, Yeah,
speaking of OAuth, or even LDAP integration with Active Directory and other kinds of providers, is much more streamlined these days, like it, it's now like incredibly easy to have a central authentication system where you either push out via the central system your your authorized keys, or you have you really just keep them in your central system on the SSH server knows to connect to that one and say, like, Okay, give me the the authorized keys for for this user. Or, again, with TLS, you don't even need to do that. You just validate the client server and see the certificate authority. Yeah, go ahead, and then it's also easy to revoke as well.
Yeah, I was gonna, I was gonna say the whole idea of having authorized keys on a system, which is, you know, still goes back to my, my, my normal day, day to day is, if you know, login, add your key, you know, keep going, strikes me as hideously risky for from an enterprise perspective, right? You need you shouldn't. You should be using a third, you know, an external validation, although, you know, some point you actually do need to have some you know, better to have keys than passwords from a from a login perspective. So you're going to need some, some back, backup way to get into the system would be my, my thought. Or, I guess you could just with immutable, with immutable boot. Maybe you don't even worry about it's like you don't have any keys except a third party auth. And if you can't, if you could get locked out of the system, you just re image it, reinstall it. How, how crazy am I? Am I getting?
I mean, the only way you would get locked out of the system is if the server is not able to communicate with your identity provider, and in that case, yeah, you have bigger problems. But for the most part, you shouldn't need to provide that. There's still going to be some systems, particularly edge systems, which are going to have less capabilities and many state require, like fixed keys. But again, even then, like with security type, security key type, all the keys, the risk of using those is minimized as well.
Greg, you were gonna make a comment I'm making. I'm curious about what what practice, what best practice is? I
Yes, yeah, I'm not. I don't know, right? Because in some regards, we're mixing a couple things, right? There's, how do you know what you're going to do, right? Who's going to do it, right? So you have certain identity, authentication and how. So then the question becomes, like a lot of what class is talking about, right? Is, in some regards, represented in the whole subsystem of Linux, right? And it's integration with SSH, so that you can get to all those different kind of things, and then that allows you to drive that process. I think the challenge, at least from our perspective around provisioning and control, is a lot of times those configurations aren't in place yet, right? And so the question is, how do you bootstrap those in a secure and undrivable way? And then one of the things we're beginning to see from certain vendors and appliance like providers, if you think of ESXi like that, is that a lot of them believe that SSH is a security vector for attack problem, and so they're just getting rid of it all together. And okay, so in some regards, that policy path becomes a different question of, okay, so, and really, as Klaus points out, it's not SSH that's the problem, right? As with most security these days, it's not necessarily the protocol of the person or the tooling. It's the actual individual that makes sense. Well, it's, it's, I'm lazy, so I create one key that I put on all my systems that my administrators can then access and control. Well, okay, yeah, SSH is still as secure as ever. It's doing the right things. But if that key is compromised, right now, how are you going to fix it. Well, if I did one RSA key right for everything, now I got a problem, right? So the question is, are you building tooling right? And this is one of the challenges of like, Ansible as a control plane, which is where I think you were headed, is a lot of times in a lot of the Ansible setups, right? They're assuming everybody has the same SSH key, though, obviously that's not required, but it makes it easier to use. So everybody says, well, just make sure my SSH keys in place, and then I can manage everything. And I think the challenge with some of the SSH as a control plane discussion I think you were headed towards is that the ease of use, and the quote, air quotes lazy, because operators aren't lazy. It's just they're busy, right? Is I just put one SSH key in my Linode account or my Amazon account, and now it's everywhere, and that lets me do whatever I need to do, whenever I need to do it, right, okay, well, that's right. The problem is we haven't necessarily made better paths for those kind of situations, right?
I can tell you, when we generate an essay, we have automation that generates an essay. This is H key for every, you know, every cluster, every interaction, and that ends up causing a ton of orphaned garbage keys that are that are very difficult to collect out. So, yeah, there's, I the hygiene on this is really hard, right? If you, if you're propagating the same key, or propagating a key off of a standard list in a cloud, I guess at some point you're going to, you should assume that this, those machines are going to get hacked at least, at least if a machine is hacked, you're not going to, you're not all the other machines aren't compromised. I'm assuming this the key the public private keys, or protector, you know, at least that safe enough for that. But if somebody social engineers your account, or gets into your account, they could, if you're using the same keys everywhere, you're going to have provided access to all your machines.
That is where, where where the use of SSL certificates instead of sh keys really shines, because you can have short lived certificates, let's say half an hour an hour, just have a lifetime, just long enough for you to do your task, but then if it gets leaked, the time left for an attacker to abuse it is solely small. It's but it's not zero, but small, but
that would, that would still require a centralized authority to validate the login,
right? Yes, yeah. Which,
which people should be doing, from, from that perspective, turning off, turning off, you know, a key, a key distribution, in favor of still using SSH, but letting it be authenticated by a some, some external system.
I. It looks like it's actually an easy pattern to have something like an OTP based like additional password for multi factor SSH set up, and even ashgor vault will automate it too, and Ubuntu has some tutorials for that. So do you think that adding more layers of op might help with that kind of potential leak, even if you're wrangling many keys, you could still have like a shared OTP that requires other sign in before you actually get that SSH off.
Definitely a valid mitigation.
The challenge I've seen is the administrator experience is, how shall I say? Not the greatest. And so unfortunately, like most things, you have run into the battle of, do I do? It's easy. Do I do? It's most secure. And I think again and again, we haven't figured out a great way as an industry to bridge that gap.
There's one more automate OTP there, and you could have OTP both be easy and secure. You just have it generate OTPs, and you can disable people's OTP generation,
right? And that's the third circle in the Venn diagram, or, really, the pyramid there. Do I do what's cheap, or do I do what's
more?
We're back to it not being the default patterns, right? But yeah.
Now the strength again, of SSH is that it has a very low entry barrier. It's incredibly accessible, but there is a large gap between using SSH for personal use in an environment that's where private keys are just good enough, and using SSH in the enterprise, where you have significantly tougher requirements. The good news is that, again, for enterprise, there's off the shelf integrations with identity providers, with Okta, with with dual or with like even with Asher. So you spend a little bit of money on you get significantly more secure solution with very little effort. I
but you have to take the time to set it up and leverage it is on the along the same lines, how? What about monitoring to actually just know if people are accessing your systems when they're not supposed to, like, I know whenever I log in, tells me the last time I never read these messages, tells me the last time somebody logged in using that that account. You know, half of it is one half, which is securing the login and the access. The other half of it is is actually being able to survey that something, somebody's using that account that shouldn't be using it, and then taking protective action. That is, I feel like that's even more rarely. That's that's done even more rarely. Yeah, that
is arguably outside of the scope of SSH itself. SSH might provide information for this, but really what you want to do is run an audit system and have that collected centrally, and have alerting rules based on what you collect centrally to say, hey, Taiwan just locked into The system, and it's suspicious. But again, this goes way beyond just SSH, got system accounting and yeah, like intrusion, that
takes some some degree of right scan. Interesting.
It's part of the broader, I think security as an industry that I often wonder about even to the point of the auditing part. I think as an industry, the goal has been to move it more and more to sort of the automagic detection or the the blanket rules, which I think we haven't really updated, to a larger degree, in that if I as an engineer, I like to work at two o'clock in the morning, and that's when I get my my best work done, so to speak. Does the rule flag me, because I work at two o'clock in the morning? Right? Do we need to move to more of a almost like the fraud detection, so to speak, even going back to the automatic detection of, hey, Martez, you logged in at two in the morning. Is this really you doesn't have to automatically flag me out of bounds? And it to me, it almost need, honestly, particularly needs to be pushed back onto the administrators to say, Hey, Rob, did you log in at two o'clock in the afternoon? Was that really you? This is a production system. This is mission critical, and almost to inform the administrator of certain activities, as opposed to the system to try to magically detect it.
Well, that's, but that's, to me, in some of the cases, that's a two factor auth pass key to to be able to be able to come back and say, you know, that's an auth system saying, Hey, I'm great. I'm about to grant you access here. I need a secondary validation, or for it to
be you. Well, it's not just that. I mean two factors. Two factor can be compromised as well. So there's the it's a combination of multiple, multi factor authentication and machine learning as well. Like if you set a pattern of constantly working on systems at 3am in the morning, then you logging in at 2am on a Saturday. Might not be suspicious, because you've set the pattern before, but if you constantly log in between 995 and one day logging at 2am well that should definitely raise a flag with instant response which should like, the team should reach out to you saying, like, hey, is this actually you? Now, having said that, I do disagree with Martis that about this being automated or not? I think it has to be automated because the sheer volume of attacks that systems get these days, or the sheer volume of unusual data that you see these days, it's just too much for a single person to or even just a single team to handle. You need the machine learning to filter out the or essentially remove the chaff and leave you with the list of events that you really need to follow up on, and yes, that part can be done by human operator.
I'm not opposed to the automation. The thing I think through is how much should actually be pushed down to the actual operator or administrator, as opposed to having to go through security to perform their initial validation or inspection. Because, let's say I'm the I'm the owner of a server, and somebody logs into that server. Does that need to go through security all the time, or is there an opportunity for it to go right to me that says, hey, somebody logged into this server. Was that legit? Should that be verified? I think mine is. I like the idea of the automation and all that comes with that, I think, to a degree Now granted, like, if you're talking public facing, Internet facing servers, those are going to get hammered in a different way than an internal let's say ERP system might from a an a known attacker standpoint. So I think to me, some things can can be a simply a validation of, Hey, Rob somebody logged into your server or to this production system. Is that legit login?
Absolutely yes. Thank you. We can probably take a page out of like margin volume services, like, for example, valves, Steam, like they're really great at their author authentication process once you have two FA enabled, particularly if you have, like, their, their, their steam. I think what do we call a steam card? I think the app, but by essentially, yeah, well,
that's going to come back and ask you
if you're signing from a new address. Let's say, because you're you're traveling, or because your ISP, rotate your address. It will, it will require you to use MFA. You to validate that, yes, this is you, or even from a new system. So, so yeah, the the processes, I think, are fairly well established. It's just that not every platform implements them to the same degree. And that's where the the Robbie points start.
There's there's serious challenges with this. I had a question along the lines of going another step down, sort of back towards the VPN piece of using SSH as a network bridge, right? So we've been talking about off and letting you in, right? But there's a big piece of SSH which is also, or can also be used from a network tunneling perspective, where you're using you know where you're forwarding traffic from you through through another machine to get to another network. And one of the reasons why all this off matters so much is is it's not just logging what you're doing on that machine and access you have that machine if you have access to the right machine, you can use that as a gateway to access other machines on other you know in the network you can go through
if you have the appropriate privileges, yes, like SSH, by default, will not let you port forward traffic to a destination that is not the same machine. You can enable it an SSH server. But let's say, if you wanted to use an SSH server as a core man's proxy, and let's say a cessation to server a, on forwarded traffic to server B. You need to enable that capability on the on a.
I didn't realize it was off by default. I thought. I thought it was on by default.
Nope, no, you come forward, forward non privileged ports from the from the same server to your client to the client machine, on vice versa. But if you if you wanted to use the SSH server as let's see, like like a jump host, then you need to enable the additional capabilities. And in SSHD, I
that makes, that makes me feel better. I'd assume that if you could SSH into a box, you could always board forward. I've never, I guess the stuff I've done has never hit restrictions that perspective, because it, you know, if we're you're watching a box and somebody logs in and then jumps right from that to another box. You might not even it might not trigger all the alarms that we're talking about.
Well, that would get to even a more fundamental aspect of network design is micro segmentation, which is unbelievably difficult to actually implement at scale for most networks i You
mean,
can you elaborate? What do you like, like spending traffic from crossing like that, preventing
one box from jumping to another, if it isn't explicitly authorized from a network access standpoint? So as an example, do all of my servers need to be able to talk to one another from a network traffic standpoint? Or is it my job post or my management box needs to talk to all the servers, or portion of the servers, and then there's some interconnectivity between some of the servers for the explicit business purposes? I Yeah,
most, most people aren't, because it's so hard to have a network design that has restrictions in it. People are not thinking that way. We're back. We're back to the challenge of better designs are much harder to maintain and support and troubleshoot, right?
Yeah, the one thing I think, from a like an app standpoint, is usually you get a list of, hopefully, when you deploy an app, an off the shelf app. That is that you get a list of ports and protocols that the systems need to communicate with one another in and unfortunately, as an industry, it's still very much a it's in an HTML table or maybe a markdown file. One would have thought after all this time, we'd have a JSON or machine readable format that will be widely and consistently utilized to when you go and purchase or leverage this certain app, you're going to be able to easily ingest a list of ports and protocols to then be utilized to generate your firewall rules. I
It's a safer bet just to turn I guess you would just disable port forwarding or an sh it's, I don't think a lot of people think of SSH from a port there or that like I don't think it's a feature. People think about that often. I know I do whenever I have to channel through something, but, you know, it's not, it's not top of mind, not perspective, yeah,
I think that's one of the challenges with SSH today, is it's become a kitchen sink or a Swiss army knife to a certain degree, for partial solutions to all sorts of problems that get you quote, just enough to get you past a threshold, right, be it an arbitrary security threshold, right, even though there's right Martez and Klaus right, listed off all sorts of other ways to do things that are probably better, more effective, more secure things, right? But you're saying, how many times you just say, Well, I just want to port forward this through this jump box so I can access this one thing, so I can get this done, right, right? And all of a sudden, you're now open and running things, right?
That's the trick, right? You went, you went from I needed a quick tool to get in there, to I have a security violation, or somebody's built a whole, you know, access system based on top of it
well, and I think that's the challenge is, as one of the things that we've been thinking about, right? Is, why do you have SSH, right? What is it bringing into your your system and environment, and what's its purpose? And I mentioned the VMware potential changes in the future of things like SSH going away. I mean, they've already turned it off by default to make it a horrific thing to turn back on. Yeah, right. It throws all sorts of warnings, and ESXi eight saying, Hey, you just turn this on, we're not going to, like, validate your warranties and all that stuff, right? Even though most customers do it because they want to be able to access it just in case. And I think that's the challenge is, what are you what are using SSH for? Right, especially as we look at things like cloud native applications and other things like that, running in containers and stuff like that, right? They don't have SSH. So how are you choosing to debug and manage and control these things, and what was the purpose that SSH was bringing to your story? And how is that useful or not? And I'm not sure organizations are taking that discussion around, right? I mean, if you think about it, if you're looking at from this perspective, right, in AWS, if you go to s3 not s3 but EC two, right? What are you getting? You're getting a machine, and how do you access it, through an SSH key. The point is the delivery product, in that case, is something that you're going to SSH into to do other things. Okay? So that's, that's the end point. That's the end destination. But in a lot of organizations, the SSH is there for what, failover, for failure, determination, corrective action, those kind of things. And I think a valid question becomes, is that the proper path for provisioning now, is that the proper path for control and auditing? Right? Some rigor is mentioning things like, hey, I want my audit logs going somewhere, and I want a separate entry method, okay, those may facilitate other kind of services or other actions. I'm not sure. As an industry, we've made the leap beyond to Okay. What's the next way we're going to go about managing these things in a secure and a tractable way, right?
No, and that's you, are you thinking about containers or systems in general? Because I think there's, there's both, well, challenges, yeah, both right,
in some regards, especially as we start looking at more and more of these platforms trying to close themselves off. And say, Well, you don't get access to our system, like, Well, how do we know what's going on? Right? Some regards, even the even looking into some of the OpenShift stuff, right? OpenShift with some of its core OS based, Red Hat core OS, or whatever you want to call it these days, right? They're trying to lock that in where you don't even log into it anymore, and they don't want you to have access to it, because they want you to treat it like an appliance that's just running your platform.
Immutable systems, right? Have no well
or
but like, if you look at OpenShift from Red Hat's perspective, it's make sure you have a TLS based identity key in place so that you can use that to access the exposed API endpoints that we're going to let you drive and configure enough of the core OS to do the changes you need. We're not even, I mean, while they still have it, the default for a lot of the OpenShift stuff doesn't actually put accessible methods into the system. And so they're either going to say, like, what you were referencing, reboot it, or use our quote, vetted, approved API, where it looks more like a switch, if you will, traditional switch, where you're using a specific API to only expose certain control elements,
right? And so it's a locked, it's a locked box, except through very prescriptive control mechanisms. Excellent, well, and if we're talking about immutable pieces, and yeah, you don't even you would change the immutable description of SS and create a new immutable image, and then cycle, cycle the images out on the idea that if nobody can access the box, they're more secure, right? I mean, that's, it's at the end of the day. Sort of what we're saying is SSH by, you know, having something that can access the box is fundamentally, maybe we're being overly simplistic, but is, is fundamentally a security issue. I don't believe that because accessing a box or accessing something and being able to say, I need the logs, I need the information, I need to do, you know, post mortem, or I need to do forensics on a system, if the only way you can get into it is through, you know, there is no way to get into it that seems like a risk to me.
Well, I mean, that's all. It's a little bit more news than that. The So fundamentally, at least in an enterprise context, locking down the box is it's not a bad idea, like, definitely the box is allowed to do all the certain things, which also makes securities job a lot easier, because if you, if you have, then a sensor on the box that says, hey, a new shell was launched here. That's an extraordinary event, and it's suspicious. That's right, pretty much just the same as security on trueness. Now when it comes to debugging or or like getting a stack trace or logs and whatnot. You also got to keep in mind that these are these systems are not independent systems anymore. They're part of an ecosystem, so the logs are invariably going to be sent somewhere else by a native process, so that you don't need to log into the box to get the logs anymore in order to do certain operations, you might not even need a shell anymore either. All you would need is a privileged container that is able to do these tests for you. So you're it's not like you are removing the capabilities that you used to have with SSH on a shell. It's just that you're shifting them to a more compartmentalized process,
which makes sense, right? So we're consistently, we don't want people accessing these systems. It's what you're describing, very, more compartmentalized, yeah,
or it's not that we don't want people accessing the systems. Is that we want to make the access an exception if we make the process of doing every task that you used to do by accessing the system easy. Are via other methods, then the list of reasons for needing that access is significantly cut down, and then it's also much easier to audit.
I have an interesting question. I know we're out of time, but it's, it's an IT. There's times like for us, where we bounce people through a discovery OS, like we have, we can put machines in special modes that allow us to do certain tasks or specific access. Greg was making a comment about this, like with SSH and VMware, we're like, we need SSH to set the machine up, but it's only for a very small window of time, and then you can turn it off, disable it and move on. The same thing would be, if you have a locked OS where you don't have any tools, we would put a discovery OS on, do the work we need, and then switch back into the imtable system. Those are sort of like, like you were describing. They're very narrow windows where you can say, I am in an exception mode, doing my exceptions, and then I'm back into normal operations, where you know the same anything like that happening. Would have, would should fire alarm bells. I think that's a reasonable pattern. Rather, rather than taking like we've done traditionally, and taking a server and letting it do anything at any time, and then it becomes your monitoring of that system becomes really, really hard and confusing, because it could, it's general purpose at that point. I
a something to consider.
I like the patterns that are emerging. Thank you all. I suspected SSH wouldn't would not be a five minute. Yes, there it is topic. So I feel vindicated and knowing that this was a more complex topic than than you would think on the surface. So thank you all appreciate the time. Thank you. It's cheers, cheers, cheers.
Wow. One of the things that really impresses me about the tech ops series is just how much detail is necessary to be a skilled operator in today's environments. So when you take what most people assume to be is a simple take it for granted type of technology like SSH, and really look at it from an operational perspective, it explodes into something very complex and nuanced and a lot of multifaceted technology that you need to understand to build good operational environments. I hope this was helpful. If it is, please check out our other tech ops series components. You can find the whole series and a whole bunch more at the 2030 cloud. I'll see you there. Thank you for listening to the cloud 2030 podcast. It is sponsored by RackN, where we are really working to build a community of people who are using and thinking about infrastructure differently, because that's what RackN does. We write software that helps put operators back in control of distributed infrastructure, really thinking about how things should be run, and building software that makes that possible. If this is interesting to you, please try out the software. We would love to get your opinion and hear how you think this could transform infrastructure more broadly, or just keep enjoying the podcast and coming to the discussions and laying out your thoughts and how you see the future unfolding. It's all part of building a better infrastructure operations community.
