Hello, I'm Rob Hirschfeld, CEO and co founder of RackN and your host for the cloud 2030 podcast, this DevOps Lunch and Learn was about security practices. Specifically, we started building an outline of topics insecurity that we think are necessary for developers and operators to build secure applications. And so we chose to go broad, and try to identify topics and information that people should know. And we basically built a week long course curriculum. And we will, over time go through what this course curriculum is. But today, it's really fascinating to hear us walk through the process of understanding and knowing and following the thread of if I need this information, I need that information. This was typed and look in the show notes for a link to this these resources. There will be some places where we're typing and obviously looking at the screen, and I apologize in advance. So if you want to see all of the detail here, please pull up the document in the show notes. Thanks. And enjoy.
I guess we can start with the obvious place like ci components or as people like to call it these days that cyber Ops, which is typical. Lending fast, fast.
glentworth so fast
as a St. On da. T
isn't an acronym? I don't think it is I think it was
I mean, I guess I guess under lending, you could also do a logo, style enforcement and whatnot. But but it's in the end the same rudimentary
word Oh, d d is dependencies.
No. dynamic and
dynamic and static. Alright. So static analysis and dynamic analysis. Boy, spelling brain, am I taking pregnant kind of guy? Which is a problem.
Yeah, that was great. Greg and I were talking about this this morning. About the the value would that catch dependency because I'm in to me, there's a dependency problem.
Yeah, so SAS will usually cover the dependency part. I mean, he will, he will rely on our Vulnerability Database. But basically, if you run sass against your code, and your code imports a library that has a vulnerable version, and you pull the full version, then SAS should tell you that the Hey, just read from that your folders will know.
Right, SAS will also do
licensing issues and or can also be licensing issues and other things as well. Right? Right. Is it going to give you like, best practice like Okay, wait a second, you've exposed these ports or you have this cutting style that would generate could potentially generate a security issue have to have it resolved or be after mark that it was appropriate that
the style maybe like depending on what the particular malpractices there for exposed sports not really so much since you can't be sure that that that port is like expose publicly versus internally versus only do a VLAN versus only localhost. So on. Did they are trying to like generally, trial practices for port scanning and so on, I would typically put that more into cis benchmarks, then analysis then call the license.
So it's not so much a development practice, but CS, but benchmarks should probably be included. Hear us up
in the discussion topic in Yeah. And just as a top level thing. Yeah. cs being cis, cis.
It's Center for Internet Security.
That's That's okay.
But basically, those are typically the host hardening guidelines, as opposed to application hardening guidelines.
We could do a whole there's a whole operational.
All right. We could we could do a whole section on secure optimal secure deployment practices as a separate.
Okay, so since benchmarks,
yeah, to what extent do we need to cover how do we want to cover relying on system system security or platform security versus application security?
I mean, you usually want both. But as a platform operator, you you don't always control the application code. Either because, because it's not in house or because your teams are siloed. So So system hardening typically starts with assumption, your application might be vulnerable. How can you reduce the fall out from a vulnerability? Oh, I
like okay.
All right, our blast radius.
So from that perspective, should we be talking about data storage, partitioning? secrets?
encryption, yes.
On the side of secure development, using appropriate encryption as an like, partner, like poly D developed not in house encryption, making sure you using the up to date algorithms also know and defy. Solving your keys. All the fun stuff.
Yeah, it's one of the I remember we got one of the things that we got asked to do was actually not not not enforce an algorithm, but limit algorithms that you can use set systems you could you could force people into more secure algorithms. There's something else in the Oh, salting. Yeah.
It's actually a really hard one. To find the fault behaviors and defaults and safe.
Sorry, I'm typing in not talking for People are watching.
Don't worry, they're
secure false. All right, yeah. Known Achilles heel of
publications. It's a Well, one of the ones that That to me is the eliminating, there's a secure system hardening, I'm in here the route dealing with elevated and root permissions.
Are we gonna say sorry?
I'm just basically the data security false are many cases, they are the Achilles heel of many applications. I mean, we've all heard about the back in the days the, the issues with with expose the databases, whether it's Elastic Search or readers or more with your or whatever, just because the default was running with other passwords.
Yeah, time to live on exceptions. tokens. Yeah. So like, like, I'm thinking through the idea of, you know, any, any exposed surface area understanding the, you know, the blast radius, but also understanding how long and and who has what permissions.
And to a degree, like you don't see this often far, I think it shouldn't be part of the usual development practices, not just for security, but that is including sufficiently verbose observability whether Doug's met, but just metrics or metrics and logs, or metrics, logs and traces. But basically, my opinion on this is that if your system is observable enough, then you don't need to rely on direct access, which means is that you can close several doors that can be used for escalation.
That makes sense. So can the substitute for direct access? I have actually the opposite problem, which is leaking sensitive data into into logs.
That's another one. Yeah. And anonymization of logs as well, it's, it's not as easy as you would think. The chance of leaking PII is quite high.
incredibly high. Yeah. And there's some places where it's just completely avoidable. From that perspective. Like, I know, there's times when we generate, you know, information on a very narrow time window, but sensitive information that's necessary to configure systems. And then it's, I mean, you have to have it. But then it can leak into logs and stuff like that. In the background, I was adding some time to live, elevated permission working with elevated permissions, time to time token revocation her all are all useful. To trust.
There's actually a whole section on on trust. I mean, we're gonna be talking about external secrets in a couple of weeks. And maybe I should just hold off on external, external secrets. Oh, you know, what else we're missing is tracking, tracking user behaviors.
So if I'm sitting down What about things like? Forget logs? API access, sir, https nowadays, you're We're less concerned about people reverse engineering URLs. Do we have URL token? Like it's back in the days when these rails friggin cookies and secure token cookies and everything?
So actually this session, there's a session. I mean, I'm not even sure where user session should go for this stuff user session.
That's probably another topic. Yeah.
caching sensitive information.
In tokens or cookies.
It's actually worth having a password management and processes.
Drink. Boy, there's a whole odds z off. And that's its own topic.
Don't forget the third day accounting.
And accounting
was a
All right, I'll pick up that role based off. So and actually doing this well. Really, really? that's a that's a we could have whole hours on. Its topics.
MFA? Yeah.
I guess I usually think about delegating it all the way. That's just that's delegated back to an authorization system, but heard off authentications.
But rolling your eyes.
And it's warning me that I might have internet problems. Alright. This one, this one actually, to me is fascinating. Because every time I've started a new platform, or an application, I've always been like, let's not do off. Let's make that 100%. delegated. And you can never that drives me nuts. You have to have some your own auth for something. Right? The system has to have something.
Yeah, I mean, at the very least you you need our back so that you can decide who has access to what. And then yes, you can delegate, like, just the user management to turn off or oidc provider. Yeah. But you still need to interface with them.
You still you still have to build all that stuff yourself. I've never, that's just never been an easy. I haven't seen an easy button for it. Really simple pattern. What are we missing? It's a good list. I feel like we're flying flying through it. So we're not we're not thinking that as deeply when we can go back and look at training outline.
And the next time the system is designed, they start with security. And they put the framework in place. And then it's like you have to like plugin on top of this security framework. I
the challenge that I've seen with with like, when we started rebar, we started with a secured like tokens and did I think a good job building a tokenized system at the base because we've done this multiple times. But as we went forward, there were more and more cases where we had to deal with secure, you know, exceptions or elevated permissions for certain things or it was it was having a having a basic system was critical but the exceptions We're, we're equally important from that perspective.
Yeah, Rocky, right, that's got to be gotta be designed in. That's partly why I want to try and,
and that's to what you were saying when you started with it. When you hit that, with the exceptions, you were dealing with the exceptions with the knowledge of a secure system, as opposed to dealing with the assumption that all these insecure things are just the way life is. So the the, the foundation would allowed you to be more secure, even when you didn't secure things.
Right, right. So you can make the you could you could open things up or make exceptions on it.
We go and find out, I look for an outline, and there's a flow that are not a git lab.
I'm looking at I'm looking at our pull up the link, so people can see it.
Oh, cool. Ad class posted. Can I share this? Yeah, absolutely. Okay. This is another cybersecurity meetup that we could collaborate with on the discussion. That might be fun.
But the organizers are largely out of Arctic, both in terms of them, okay. Yeah, they might double now. Yeah.
So the other thing about taking security into care. And how we're dealing with this is that it really goes against all these people tried to be agile, because you can't be agile and think about all these things and consider just the elevated authorities as normal because they're not but agile doesn't look that way. So security's that this is to agile in many ways.
I wouldn't go as far as saying that. I'm minded there is a place in Agile for concerning security, I would say more. The security tends to be the, or inversely, marketing and sales tends to be the Nemesis of security is like, Okay, show me a proof of concept. We have a proof of concept. Great. Let's sell it. And then they're like,
no. Yeah.
Get the context that my right when you were saying the specific thing.
Zoom is running at the same time.
Yeah, mine slides choppy, too. I think it might be a zoom. The I did I don't usually have internet issues. I mean, one of the things that this is part of part of this to me, it's like alright, how do we know you have a product? Seems gonna make this hard.
Maybe it's me.
Oh, can people hear each other? Or do we lose? Everybody
can still hear you. Okay, good.
I got
me this time.
Got a bit of compression noise on the audio every once in a while, but I would say 90% tolerable.
Then keeps telling me my Internet's unstable. So yeah, maybe motion machine just needs to reboot. But because of fiber.
It doesn't tell See that on my phone even though it is unstable on the phone?
It's a that's a given in the design at that point, right?
That's funny. All right,
I'm thinking through. We've got How valuable is the reporting and tracking vulnerabilities piece in this feels like? It's, it's, I feel like it's worth this discussing this, like, you report vulnerabilities. How do you report about what's what's like?
Are we talking internal process? One, one, say our pipeline rejects a commit because of a vulnerability? Or are we talking external reporting?
I'm thinking external reporting. Because this is this is where it gets, it gets trained, you could be in a situation where you find right, if you find a vulnerability for somebody you want to you want to hit you want to tell them? Right? But it's not always clear that what you're finding is a vulnerability. And so it's, it's useful to say, Alright, how do we evaluate? How do we know?
How do we know right? In some ways, that's what the whole discussion is supposed to be about. Cuz I mean, my experience has been that sometimes it's sometimes it's super clear. It's like, when you navigate here, it exposes sensitive information. And sometimes it's like, when you configure things in this way, you expose information. It's like, Well, yeah, but that's not a valid configuration, or it's not a secure configuration. So
I mean, that is kind of why you have the difference between the static analysis and dynamic analysis, as well as like things like coal was top down scanning is that you are, you're tracking your your own vulnerabilities at different levels. And then going to reporting topic, typically, you'll fall into two categories. One is a new vulnerability, the data just found out in which case, you're you're typically, top of the line security researcher, I'm not going to even cover that, because I'm not too familiar with that whole process. But then there's also the revert process of just people's scanning go the whole network on finding common known vulnerabilities, because they asked us, he mentioned this, you need to be misconfigured. Like, like, even using common tools, like show them.
But does that then create a you know, sometimes that's an error? Sometimes that's a Yeah, we know, it's just that's, that's what it's got to be.
I would say, most of the time. Those falls into two categories, either the person running the site or application doesn't care, or did they care to have that implemented to fix it? So it's either ignorance and just unwillingness to fix it. Those who are willing to fix their issues typically are friendly on top of their vulnerabilities. And the typically have like, like a contact that is publicly announced to say like, hey, if you find a vulnerability with our site, send it to this address. And we'll deal with it that they may even have like a bug bounty
software. Yeah. Yes, it's it's these reporting in bug bounties and things like that, I think are it's it's something like I I always assume I know it. And when I get into conversations about it, like how you know what notification requirements are and how you should notify and what you should do and things like that, it's
it's always sensitive. What is typically more controversial is, what happens if you find that we're not belty, you've reported, you've gotten the acknowledgement that it is a vulnerability. Nothing is changed. Let's say three months later, Maggie, the vulnerability system plays you suspect as it's being actively exploited. Do you go public with it? Do you pressure them? Yeah, so there's several articles over the years of various security researchers taking different stances on this.
And also, does it open you up for criminal charges if you report it out beyond the security community. So there's that out in that thing to where you could be legal trouble for reporting. Beyond the local stuff, though, there's also something just in Microsoft Exchange. It allows and fails to note, multiple languages within a domain name, so you can put Cyrillic letters in there, unless it's kerlick Cyrillic letters in that look just like English letters, and it doesn't identify. And the real bug was when you use that domain, and it will report back as if there were all English letters. So the contact information is shown from the contact, the domain your your spoofing, not at the, the one you've created, so as it applies to, and Microsoft update actually said, yeah, we know about it, we're not gonna fix it. Well, they supposedly fixed it, but they said we know about, we're not going to fix it. And that's just for fishing purposes really bad. But, you know, Microsoft initially said, not an issue when it was reported. So how do you deal with that?
Yeah, I remember that, that that thing would destroy particularly one. One, the, the restriction on on characters and domain names got lifted. There's one particular character that that's basically a non printable whitespace unique code that you could use an entire domain name. So you could create a domain that was at least to the to the end user, identical to an official one.
Well, and the letter I have, here's both there's the Cyrillic guy, and there's the Latin. And that's what they were showing today. And this was something that just happened recently, with the showing the contact of the person at the domain with the Latin eye, even though that domain actually had the Cyrillic eye in it. I only worked on exchange, it doesn't work on Outlook on over the over in browsers, but it's like, you know, not an issue. I guess. Somebody figured out it was an issue and they fixed it really quickly, but they didn't tell anybody they fixed it. So it's not clear the deck you got pushed out to any new new versions or patches.
homograph attack, that's the turtle category of skis kind of issues.
Yeah, this is there's that there's a whole security component of input validation, and making sure that the things that you're you're receiving are things that you expect, right, the classic Bobby drop tables.
Yeah, it's especially hard on loosely typed. I'm going to include type type validation in this thanks for
the report. It might actually be reasonable to put in there a section on responding to reports, because it's your code. So how should you respond to reports? And how should you handle reports addressed to you? Not just reporting out to other issues, issues with third party software.
And there's tracking.
This is tracking and incorporating fixes. Yeah, there's a whole if you could, you could spend the whole day talking about how do you manage other, you know, code that you depend on having vulnerabilities and what you do to manage that and how you make it go forward? Oh, you know, it's missing in this. I'll put it under process and security, which is
tracking channels. Channel issues. Yeah.
Yeah, that's I was going in different place. But let me get that one.
Oh, okay. Sorry. No,
it's good. I'm glad to have X. Toy ideas, get all the stuff. I was thinking about.
Processes safety, oh, release, backwards, back porting to previous releases back porting.
One of the things about all this stuff that is always interesting is like, Hey, I found a security vulnerability in some some library that we just hit this with something that we had a Ws s library that we depended on, gorilla, Greg wood, Craig would know offhand. And so there was a known that there was a posted vulnerability to a library that we depended on to generate sockets, or string or WebSocket events. And it's a question of how far back if we're, if we're patching that, how far back do we go in previous releases? It's like super easy to incorporate the new fix, hey, here's how the code we're working on now includes the new fixes for the libraries and dependencies. But propagating a release of the fixer release can be a pain in the toolkits. And then what's your what's your reporting obligation when you fix other you incorporate security vulnerabilities from other people's stuff? That's a good
point that Cisco had to do a security patch. I mean, this was back in the bots. And they had 256 releases they had to patch. Because of it. Wow. Yeah. I know that because that was the responsibility of a friend of mine. Another thing that comes up that I've been trying to get a friend to come on and talk about release. And he's one of those big companies that starts with the Wii. And he's talking about tracking down, he's released management, he's talking about tracking down every single source file that is out there, both in use and not. One is getting rid of all the things that aren't really being used, removing them from the system so that other people don't start using them. The other thing is going through and finding everything that's out there. Especially with these older versions that are antiquated and possibly security holes, getting those removed and replaced with something that that's more modern and maintainable.
Good luck tearing down Stack Overflow.
But there's an element of Hey, wait a second, we need to build you know, it's we just updated our whole build infrastructure use, get lab. Exciting, that probably be another backup complete our topic. But, you know, being able to go back and say, Alright, well, how do we, you know, is is a build from two years ago, you know, how do we know that the Git lab build from two years ago is that patchable do we need to care about it, how do we want to deal with it? We have we have we have contractual, you know, limits on on that type of repair. But certainly can be tricky, especially if you've changed the build system out.
Pour last Yeah. And this guy's he's released for the operating system, side of things, virtual machines and stuff like that, and the company. And he's kind of sort of a single source, central thing. And he's retiring in January. And so they're sucking information and building dashboards and stuff. And he's sitting there telling him, Well, this dashboard doesn't really show what the important stuff is, or, yeah, show something, gives you more information on how to find it or fix it. So they're tapping him for a lot more now that somebody cares about release management is in charge of it. But for up until last year, he was pretty much Yeah, they were actually cutting back on their release management group and whatnot. And they're open source, a lot of things were done via open source. And they knew he was going to be going soon, but they weren't doing anything to grab his knowledge. Yeah. So any big company with a history really have some hard problems in there when it comes to release management? And you're learning that to rob?
Well, it's, to me, this isn't just a big company problem. It's not a commercial software problem. It's it's a open source issue. It's it's a small company, a small companies. I'm actually I'm really proud of the the hygiene that we have on our build processes, especially with with this migration. But these are really hard, hard things. And they're security. There's huge security implications.
And it's even harder finding, sometimes finding the right talent to implement this. Yeah, it's a big problem. But smaller companies,
this is not usually considered the premium work.
Yes. And it's also getting the people with the right personality and mindset, because even if you hire someone to do the implementation, and they might have the experience, they might not have the personality to do the detailed bits in the essentially, what a lot of people consider grunt work, because it's very, very detail and repetitious in lots of ways in other ways. It's not, so it's just developers, developer attitudes, and, and personalities don't map to a lot of the security stuff. And, you know, just like QA people, testing QA people, they could develop, but their strength, it's like, how do you actually compensate people for the stuff that makes a difference between good product and just code whether even if it's good code, the bits and pieces attached to it, couldn't make it poor cup poor product. It's kind of the essential workers. In COVID, everybody had the folks that had to be there had to be the nurses, the doctors, the food delivery, the cooks and whatnot, but they're the folks who get paid shit because nobody wants to do that work. Yeah. off my soapbox,
no, this is part of what I'm trying to capture here is to raise, right, when we talk about security, and in some of those is our is our, our own background and our biases with this. But, you know, when we're talking security, in this case, we're looking at it very holistically. And we're looking at the billing infrastructure, we're looking at the data infrastructure, logging. It's funny because I, you know, part of what I was thinking we would talk about would be, you know, some, like string formatting behaviors and stuff like that, but I it's, it's not the place to start the place to start is understanding the environment the code is running in.
It would also be useful perhaps to, to consider the ultimate, ultimately the end goals of, of vulnerability, exploitation, whether whether that's system takeover or data exfiltration I
like it, what is it? What is it on your system that you have that is beautiful.
I like this slide. data extraction, type it all chaos. So I'm typing in you know, what's what's the the risk of if somebody is attacking you right? Are you compromise? Yeah, that's true.
Because lots of times, it's they don't have to do any of that stuff. All they have to do is so enough confusion that they can profit from it.
Yeah. Profit Ranson.
In fact, that was what happened to my husband, just this weekend thing popped up that said, your iPhone has been compromised. hackers have all your information this popped up on his iPad. Right? It was a piece of adware, malware that got somewhere in a Google search. So he went, he was do a Google search of pictures. And this popped up. And he's totally confused. I sat there and it's like, okay, turn off all the history on this, do this. Because it says, you get this pop up, and it says, okay, and this was the Google app on iPad, not a browser. So the everything they talk about repair is in browsers, not the app, yep, doesn't have a way to repair this, or remove this particularly much. So it's a pain in the butt. But it was just so in confusion. It just sent you off to a malware site to load all this ad where it's supposed to fix it off.
profit.
It's interesting, because your story is making me think we actually got a an email from the bank, relating to the long story, but but the bank sent us a count alert for an account we don't use much at all, don't even think about. And so we got a legitimate alert from the bank. My wife was like, Oh, no, she clicks on it starts opening it up, and doesn't recognize the account number. And then and you know, all hell and she's we're assuming it was a malware attack when it was actually a legitimate bank email. And it took us literally hours to track down, that it was legitimate, that we hadn't compromised our accounts and done all these other things. And there's an element in insecurity here, which is actually making sure that that people can trust the messages that you're sending them.
Conversely, there's the security training aspect as well. So not not just protecting your code, but protecting the users on their accounts. Or your developers.
Oh,
wow. Yeah, hold on, where
am I gonna put that in? We're out of we're out of time for for this processes security. Supply Chain attack, which we haven't even mentioned yet.
Well, we kind of sort of did in just third party software and, and maintenance and stuff like that. But no, it's still it's still got more stuff in it than that. So yes.
Yeah, that's alright. So
we've touched on it, but we haven't explored it.
Well, this is this is a good start. What What I would suggest we need to look at the outline as a whole and then see if we can break things down into this could literally be in topics that we just come back to as a as a topic list and pick a couple out shortage for topics.
Well, the only thing worth doing is a little bit of online research to see if other people have created the same thing. or something similar. And then. Yeah, curating and editing to the point where it's, like, best of kind of thing.
I mean, there's literally entire businesses around trying to address this.
Yeah. Yeah, that's why I pulled up this Git lab to make money. There's just so much to cover. Yeah. And yeah, these are these are paywalled, the additional resources, I'm looking to repair well, cool. I this was helpful. I actually, like getting the bird's eye view of security topics. And then we should come back and and do these this training, see if we can get especially we can get somebody to come in? Who wants to talk more in depth on any one of these topics? Because they're potentially along?
Yeah, that would be good.
Excellent. Thank you. Appreciate the time. And I actually feel a little smarter about security. A little more scared and a little bit smarter at the same time. Already, right. Thank you talk to you soon.
Well, as I was saying, security is one of those topics where there's so much it's so interconnected. And you really need to think about the system here. And I appreciate that in this conversation. We did think of it as a system, not just how do I write a line of secure code, we will keep going into security topics, secure configuration, operations development, in DevOps, lunch and learn. And if this is a place where you want to get smarter, or you have an expertise that you want to lend to the group, please join us at one of these lunch and learns, just let us know we'll put you on the schedule. And we will dive as deep as you want into your topic. I'm looking forward to hearing from you at the 2030 Club. Thanks. Thank you for listening to the cloud 2030 podcast is sponsored by rockin where we are really working to build a community of people who are using and thinking about infrastructure differently. Because that's what rec n does. We write software that helps put operators back in control of distributed infrastructure, really thinking about how things should be run, and building software that makes that possible. If this is interesting to you, please try out the software. We would love to get your opinion and hear how you think this could transform infrastructure more broadly, or just keep enjoying the podcast and coming to the discussions and laying out your thoughts and how you see the future unfolding. It's all part of building a better infrastructure operations community. Thank you
