Download Otter for your meeting notes

FINAL 20230314 Decentralized Platform Engineering: Otter Voice Meeting Notes

FINAL 20230314 Decentralized Platform Engineering

Rob Hirschfeld56min

Rob Hirschfeld

00:06

Hello, I'm Rob Hirschfeld CEO and co founder of RackN and your host for the cloud 2030 podcast. Today's conversation is about centralized versus decentralized platform engineering, or as came up in the conversation about platform engineering, it's the opposite of Java Enterprise, version and platform. We have a really thoughtful conversation about the human and management factors that go into building great platform engineering, and the efforts of control having too much control or too much flexibility, not enough collaboration, not creating space for innovation, and changing inside what's inside these platform engineering efforts. It was really a fantastic discussion overall, that went through the process. And really something that as you're doing this type of work interacting with platform teams about or thinking about platform teams should influence how you design and authorize the effort to make that work. What type of slack you need to put in the system, what type of authority needs to be given to the platform of engineering team, overall a fantastic conversation, and I know you're going to enjoy it.

Claus Strommer

01:21

Unfortunately, the glamour and stuff be as applying on the on the tools or not so much on the process. Again, in my opinion, the like the absolute The first step is like you need to know systems architecture and design. It doesn't matter what system you do either. That knowledge is portable. But if you enter it without that, you're going to be trading deepwater soon.

Rob Hirschfeld

01:51

It's interesting to me, because we had, we were doing a series of interviews for people. And they were using technology, but not understanding technology. In that sense. So some of what you're describing, was like, it was a little bit resume driven development, like the resume talked about get lab and Docker containers and TerraForm and stuff like that, and Ansible. But when we're like, how did you build that plan? And then they're like, well, in it somebody else for it the plan for me, I just pushed the button to run it, Mike Okay, well, how do you troubleshoot it when it breaks? And they hadn't taken the time to figure that stuff out. So but it's but cost? I mean, to me, it's like they, they're right there. They're just not asking asking the question.

Claus Strommer

02:47

Yeah, I get that too. Like, for example, with C ICD. It is not uncommon for for companies that don't have a team with a basic understanding of the ecosystem to wants to develop tools in house. And then you end up with a with an ecosystem that does something that is like, yeah, what you're looking to do, but it will close the door to something other capabilities that off the shelf tools give you and yeah, by not having knowledge than off these other tools means that you're like, if you don't go out and seek knowledge, then you're not going to do well in DevOps.

Rob Hirschfeld

03:51

Interesting point in that perspective. I can you elaborate, because I fundamentally agree. But

Claus Strommer

04:04

like, this is going back to going back to what I said before, like, if all the office is a hammer, everything starts to look like a nail. You need to understand what tools are out there. What what scenario those tools are better at. And, for example, like if you know Jenkins, like great, you'll have a very basic understanding of CI CD. But Jenkins is by nature, a push based configuration management model that you push changes out. If you don't understand that there's other models. You you end up trying to shoehorn your solutions into a few number of tools that you do know Did you end up you know, creating more work for yourself. And in many cases, it's worth while to consider a new tool, spend a week trying to play with it, and see how it fits into, what do you want to do than to stick with the devil that, you know,

Rob Hirschfeld

05:29

I think a lot of people would would have trouble thinking they have the weak, it's is part of the false premise that that we have from this is that a lot of times, people don't even ask to do the exploration, or their management team. And therefore don't give themselves permission to do that, to do that type of exploration.

Claus Strommer

05:54

I agree very much with that last part. DevOps is not friendly to us, man, you need to have a level of independence, that allows you to push back and say, This is not the right tool for this we like if we do this, it will work initially, but we're gonna have to we're going to be running into these problems later on. I was there before, like, years ago, and I understand how challenging it can feel, initially to push back. But you need to be opinionated at some point.

Rob Hirschfeld

06:41

I realized, it's, it's amazing to me just how many times people assume assume an answer, and then don't, don't try. It's it's, it's great. It's, it's crazy. And And it's funny, because on the other side of it, right managers are like, I want people to come to me with ideas, I would say yes, if they wanted to explore that I, I don't want to push people to do that. Right. I want it to be more organic. And so it definitely is a funny communication gap. It's interesting, because what we're talking about, I think is intimate, intimately linked to the topic of the day, maybe we should just share, which is centralized versus decentralized infrastructure and management. And this idea of like, what does it take to have centralized control for the types of stuff we're talking about, you know, you shall use Jenkins, you shall use Ansible? Or do you even better shall use these playbooks versus, you know, every team doing doing their own, you know, doing their own variations on it, making their own decisions? Neither extreme is good. But, I mean, how do we how do we balance that type of process?

Claus Strommer

08:07

Well, this also ties into previous conversations that we had and not only talks about, well, what what does it mean to do platform engineering, that I see a common thread there, and that is to definitely to produce a solution that is flexible enough to meet the needs of your development teams. But at the same time, not nothing better be a wild west? I mean, sure, you have other opinions as well?

Rob Hirschfeld

08:40

Well, it's a got to me, there's a there, I go back to the, you know, a lot of the tooling that we use, was not designed with a collaborative approach in mind. And so there's a whole there's a layer in here, that's interesting to me to get to, which is, you know, how to how do we make it if if people want to collaborate, how do we make it easier for people to collaborate? But I think there's where you are, is the first thing to talk about, it's like, you know, a platform engineering team. And I like the emerging definition that I've been, there's, there's so many definitions of platform engineering, my my preferred one talks about this drive for enterprises to you know, consolidate and reuse and have some centralized management of the these processes and stop the sprawl but that's got to start somewhere. I mean, to me, this is the question it's like, this that start at the management layer, is it is it you shall not sprawl or is it a cost issue? What's you'll have a thought on, like where people would would look at the cost.

Speaker 1

10:02

I think mine is figuring that out for your business. So one of the things I thought about the last number of weeks and months is the common response that people have to certain things, whether it's certain tasks, or certain things that should be done is does that bring business value to the business? Is ethics sometimes we use that in a weird sense that like, similar to the argument for your CES, the quote unquote, getting rid of ops people as if that they're not driving direct value to the business, they just not necessary. Sometimes it goes too far to those points. But the other thing then becomes is, even if we take the sprawl example, does that also translate to clouds? In the sense that if you're running AWS, Azure GCP, you should just pare all the way down to a single cloud? Or is three enough? Should you be running in five? Should you be running in 10? And I think that's where it gets starts to get really interesting. When we talk about things like tools, rationalization and centralizing on a single platform or capability, is I think it definitely comes down to a organization, but organizational basis of how much is enough? And

Rob Hirschfeld

11:17

I think that's good. Yeah, I was gonna, I was gonna spin that out, I was gonna use the multicloud example, as a concrete one, because let's assume that you got there, because a lot of companies are because individual teams made individual choices, right? It wasn't, I'm assuming it wasn't a calculated decision for companies. And there are companies where it is, and that's great. But assuming it isn't calculated, then you've got teams that made a choice that have experience of people have investment in the cloud that they're using. And for you to consolidate, somebody's going to be the loser. Right, and they might even lose functionality in that migration. Well, I

Speaker 1

12:06

wouldn't say all of them were technically intentional, is that I had options. And I picked one for a certain reason. And I think, when we start trying to rationalize that, I think it becomes certainly a difficult challenge to figure out, okay, yes, instead of you running in GCP, you could be running in AWS or Azure. And so then one thing that you touched on was the tools and the, the capability standpoint. So I think unfortunately, in many ways, we're just headed towards a more complex world. And we have to except that and figure out how best to adjust and adapt to

Rob Hirschfeld

12:45

that. Oh, my God, yes. But I mean, to me, the right answer becomes, you would change it, because you have to have a benefit, right? We're back to business benefits. So if you're like, alright, you know, we're on three clouds, we need to be on two clouds. Sorry, if you're on, you know, Azure, Amazon, pick the leader as a we're going to we're going to vacate Amazon. Because we're a pharmacy and we now compete with them. We're not there yet. There's a there's a rationale for you. But if you're going to do that, I you know, yay, that's a good, you know, that's a pretty shallow business rationalization, frankly. I mean, I think the, it's better to talk through it's like, well, if we have your team come out of Amazon, and we're part of what we're doing is now you might be able to lean on another team or share experience with another team, right? Or the platform team can help you be successful in Azure GCP. Because the, you know, we're we're building expertise, or we have training materials, right, where we're investing in the people using those platforms. So that's, to me that's giving somebody the rationale to pull back or to change course,

Speaker 1

14:07

the question then becomes what was the thing very tactically? How much experience did I lose by making that change, though? So you're telling me you've got these models, you've got these, this training, but I spent the last year and a half, recruiting AWS experts who have spent the better part of five or whatever years deep in AWS now,

Rob Hirschfeld

14:30

what do I do? Yeah, no, that's, and that's, and they've and they're commingled knowledge about you, and your product and what you have to do with the cloud expertise, and so you're going to be asking them to reskill, depending on how deeply that skill is? That's, that could be a really, that could be a really hard. I mean, I think it's one of the reasons why We see so much resistance around platform teams and the centralized controls is that, you know, I think the individuals involved here are not, you know, they, it's hard to get somebody to work against their own skill set. Maybe, although we see the same, the same argument with people resisting cloud from on prem. So it's not all resume driven development. But it

Speaker 1

15:30

goes to a very interesting business use case point there, even though the on prem versus cloud standpoint, and some would argue that even then it could still be a resume resume situation or scenario because something is not on my resume. I'm going to push the to keep it the way that things are on my resume that I continue to show and prove

Rob Hirschfeld

15:50

value. Yeah, it's, it's, at some point, I would hope that businesses are able to say, you know, we will, you know, help you move into this area, give you opportunities, keep notes. I mean, if we have people who are that restricted, and I've met people like this in career, it's like, no, I understand how to do this. And I'm not interested in that. That skill I have is in demand. And and we'll move on. I mean, we started the conversation talking about Kubernetes, everything. If somebody showed up and said, You know, it's better for our business to use Nomad or fargate, then, you know, would would people leave? That think that's a real question, right? If, if you had a business, and they're like, We don't need everything that Kubernetes has, we're going to switch to nomad. We've got good commercial reasons for it, are they going to lose team members?

Speaker 1

16:59

Possibly, this is one of the things that I've talked about recently is the players, we start talking about that the companies that want to attract top talent, it's the IT professionals that want to to be paid well and do things they find interesting. It's the vendors that are trying to continue to grow market and look at new different opportunities. So I would say, a business making a conscious decision to say, You know what, we're not gonna go Kubernetes we're gonna go a nomad or a fargate, or whatever they decide it, you know, stick sticking with VMware or just VMs is what's best specifically for the business, then your opportunity to potentially attract top talent certainly does go down in or even retaining talent in the sense that when it's time to talk about compensation, what a vSphere admin might make is oftentimes much different than what a an SRE or a DevOps engineer is going to make. So there's, there's those tough conversations.

Rob Hirschfeld

18:07

That's an interesting. No, I totally agree with you. It's, it's a skills demand question. So does that then make it? I mean, everything we're talking about keeps to me driving back to teams having to build their own skill sets. Right, ideally, a lot of the infrastructure stuff or Kubernetes pieces, all that that would come back to, you know, a team should want to give those up. And maybe vary. Let me let me pivot the question to this right. What does it take it? We're talking about individuals instead of individuals, what does it take in class to deal with this? I'm certain in your role, what does it take for a team? Right, the management of a team and the other engineers on the team to say, You know what, we want to rely on corporate platform engineering team for this rather than hiring people or having the headcount and responsibility Cecil SRE, like, frankly, but how do we, how do we get that your individual team and Senate to give up some of that, that responsibility control?

Claus Strommer

19:25

Um, I mean, it's partially a value proposition. I like to think that the way I set up our platform is something that that makes it easy to present the value. Sometimes it's regulatory compliance. So like, you need to get off this desktop platform because of XYZ series. But like I said before, on that case, it will help you With the transition. But yeah, by and large, the carrot that I use is it's about a proposition. So like, this platform is integrated, it means decent decent, these checkmarks. If you run your workloads on it, you don't have to worry about these because we take care of them for you. And for the large part, it is a sufficiently good incentive. Typically, the majority of the hesitation comes from a perceived loss of control. Particularly, because I am keen on locking down access to platforms but but what I offer in return on this was largely facilitated by get ups is full control by a declarative configuration. They're the owners of their configuration there, they decide which version do they push out, and when, and what flags they said. That just, it just that, after they make that decision is handed over to the platform, and the platform takes care of that. It's, in a sense, the same kind of argument, that part of the same kind of experience that I had with coordinates, I was, for a long time, skeptical of coordinates. Because I liked being in control, I was an old style sysadmin were like, Okay, I choose what gets deployed on the servers. One, I orchestrate which upgrade. It took me it was quite a pilot and shift to to try to coordinate as I have to give credit to my then manager, manager at the time, who said, You know what, I'm going to send you to a training course on coordinates. But then I got some hands on experience. And I was like, well, this actually does what I wanted to. I just don't need to put my finger in there. And I tried to replicate that same experience for for internal users.

Rob Hirschfeld

22:39

Really, I mean, it's funny, not, in some ways, not surprising, it's comes back to leadership, making time for people to learn those skills. Yeah, it does that. I think, you know, it's funny, because we're talking about platform engineering, in the industry very generally. And we're talking about IDPs. And stuff like that. I haven't seen this part of the conversation discussed much, which is platform engineering is actually going to require people to work on shared shared infrastructure, or shared. Ideally, a lot of the conversations I see basically are like, empower your dev team with an IDP, and they don't have to talk to anybody ever again. Which drives me insane. And I get, I get a little angry, and sad. But, right, I mean, I think that if we were saying, hey, look, you know, this is making it easier for a manager to have the conversation of, you're going to delegate part of this work to another team, so you can collaborate, that, that that's what we're talking about, right? It's it's giving managers the ability to coach developers to actually delegate and collaborate.

Claus Strommer

24:00

Yeah, it's, um, again, is, in addition to that, I said, like you, you need to make room for. For personal growth, you need to allow your team to spend a certain amount of time just work on working on a POC and figure out like, Okay, this is a new tool. I can like, we have so many people or so many teams, internally that can benefit from it. That's just give it a try. Like a big time box. Yes. But other it should be sufficient to say Yeah, well, if experience so far has met the value proposition. This can make our life easier.

Rob Hirschfeld

24:57

It's interesting because the raving I always read is developer productivity developer productivity. It's not, you know, it's not the collaboration message. It's not, you know, allowing giving developers time to do more exploration or learn new things. You know, it's funny, because in some ways, when I read it like that, it ends up feeling like, we're trying to change developers to keyboards, you know, more hours of the day, because that's right. I've seen this narrative all the time. It's like, my developers spent 20% of their time doing non development tasks or,

Claus Strommer

25:38

you know, that's,

Rob Hirschfeld

25:41

that part of that should be learning and understanding this infrastructure, you know, and the things they interact with.

Claus Strommer

25:49

On this, I forgot who said it, it may have been Chris Nova, because she, too, has had several comments about capitalism, but because of that, so the there is a there's a divide between what productivity means for developers and what productivity means for the company. Like, yeah, what should happen is that if you make a developer more productive, they can do the same amount of work with less time. By but what capitalism tends to drive companies towards is if I make the developers more productive, they can do more work with the same amount of time. And that shouldn't be like that, you because then you deprive your team, from the ability to, to expand your horizons, and your unique having your teams.

Rob Hirschfeld

26:59

This is this is actually I mean, you're making me think of, you know, one of my favorite all time books, which is business books, which is called Rat, the goal and Theory of Constraints, right, and, and, you know, he has in that book, it's, it all applies to what we're talking about. And that book, it's very vivid, right? They have like, you know, the most expensive equipment they have on the floor, the people looked at it, and we're like, this is the most expensive equipment, we're going to run it as much as we possibly can. And it would build up inventory behind it that would flood the system and caused all sorts of issues. And they made that expensive resource productive without thinking through, where are they fuming, distributed resources.

Claus Strommer

27:52

And it's hard to justify sometimes.

Rob Hirschfeld

27:56

Ideally, the platform engineering stuff, to me is the system thinking theme. And that's another framing that I don't, I don't see as much as I would like to see. It's like, somewhere somebody's like, looking at is our system well optimized. And that's something I feel like companies don't do well at all, they back to the decentralized versus centralized thing. I mean, you don't want a centralized control system that's making all your choices and being a bottleneck. But at the same time, if you're only looking at your route or your department, then you're you're definitely creating sub optimal.

Claus Strommer

28:40

Yeah. I mean, optimization is that which sort of if you're too late, you're wasteful. If you do it too early, you optimize yourself into a corner.

Speaker 1

28:54

Let's get that. If you look at it, it's, let's say across the business, if that specific business unit is driving the most revenue, and they've optimized specifically for their group, and that's where the money is, and even in a scenario where let's say you optimize every other group, but the fine tuned optimization that group was able to get is never outweighed by optimizing across the board, then you went to a business challenge of, you know, what, we optimized across the board, but we're still making $15 million dollars less revenue or whatever it

Unknown Speaker

29:29

might be.

Rob Hirschfeld

29:33

So you're thinking that if they optimized, like a system optimization, but they're still they're still losing opportunity? I mean, in theory,

Speaker 1

29:44

I don't know that it is because to some degree, I would imagine more often than not, you're they're going to have to give something up that potentially in theory, could further optimize their specific use case. As

Rob Hirschfeld

30:01

I said, I mean, when I think about the Goldratt piece, what you the whole idea of Theory of Constraints was that you look at the system, you find the constraint, and then you focus your optimization work on the constraint. And so, and that constraint could actually be, you know, could be defined in a couple of different ways. But, you know, to me, what you just described is, is, you've got a system systemic problem that has a bottleneck in it. And you need to you need to go fix it. To unlock that $15 million. You're reinventing numbers, but that that lost revenue.

Speaker 1

30:43

Yeah, if you say enterprise organization, in theory, there might be a group that let's say a specifically focused on AI and ML. And they've decided that you know, what, let's see, GCP is the best place for our workloads versus the rest of the organization running an Azure, maybe that honestly is the best place for their workloads to take advantage of that feature, differentiate any capability. But maybe you decide, you know, what, we actually want you to run on Azure. And ultimately, they ended up being less productive, less able to take advantage of those specific nuanced speeches and capabilities. Not say that that's actually the case. But I mean, I think,

Rob Hirschfeld

31:23

I think that's a great example, keep going to

Unknown Speaker

31:24

depth that goes into it.

Rob Hirschfeld

31:28

But to me, that's where the platform team is also about the system. And so what you really want to be able to do is have that team come back and say, we're thinking of Google, it's off, its off, its that doesn't match our Azure mandate. This is our justification for it. And you want the platform team to have the leadership and the mandate to be able to say, you know, alright, if you go to Google, you have valid reason. But it's going to cost our, we're not going to be able to help you, we're not going to be able to do the security, we're not going to have you're going to have to reskill on this like date, this is, to me the missing piece. What I love about this conversation is that is concrete, tangible stuff the platform team needs to be doing besides just handing you a portal, it should be saying, Hey, here's the business impact for this. And, you know, actually come back and saying there is a 20% premium or a 30% premium on you making that one off Choice versus coming back. And using the platform that we've been building and being able to help you for I think the premium might be much more thinking about through this rationalization. Help and help with that definition. Go ahead.

Claus Strommer

32:48

The platform team also needs to be championing for the developers. If you have three developer teams, and each each one individually goes to management and says, like, we need this feature that's on on on GCP, because Azure doesn't have an equivalent. While management might say like, well, you're, you're just one team. Like, that just doesn't make sense. The platform beyond the app, on the other hand, because we have the bird's eye view and saying like, look, we have an increased demand for this feature. It and as a result, say, if we if we enable, or if we had the capability to use this to our platform, if we go multi platform and say like Okay, now we're Asher and GCP, well, then we we can implement XYZ. And as a result, become a market leader on this, or that looking at from a different perspective. Management has a habit of saying we've used this tool all the time, and it's an add on we've, we've gone fine with it. Why do we need to switch?

Rob Hirschfeld

34:17

It becomes the Department of No, right? It's like do it our way or go go away. And that's not going to work. Especially not in today's environment. You have to have to be able to Naval optionality

Claus Strommer

34:33

here. Yeah. And that's why platform engineering is in a perfect position to do, as I said, be a champion for the developers and say, yes, we've used this so far. But you're spending some this much time this much effort into in making it work versus switching to a new platform. that meets our demands out of the box. And then you can frame it in terms of CapEx or OPEX, he can frame it in terms of revenue loss, which is something that is, is, at least in my experience difficult for those who do.

Rob Hirschfeld

35:20

It's, it's amazing to me because in some ways, this is full back to the idea that we had as an individual, stopping, you know, asking for permission to try something new or experiment or explore, you know, in some ways, the, the idea of the platform team, in defining the platform team is to have overhead just like SRE in this case, right? You part of the overhead of a platform team should be invested in that exploration work. So if you're building it, and not saying, you know, like, Oh, we're going I see this too much. We have a platform team, we built the platform, migrate to the platform, and what you just described is very different. You're like, yeah, you're gonna, we have our preferred stuff, we support that, that's great. But we're always going to listen to the other teams to see and test if we can justify expanding what we can support or looking for business just gotta give ourselves permission to ask.

Rob Hirschfeld

36:25

It's a really, I mean, it's a really important distinction from that perspective, because it lets the platform team be a source of innovation. Which I don't think people are, you know, discussing like that.

Speaker 1

36:45

I think the other tricky part becomes like you do have a platform team, there are still adjacent teams that have to provide assistance and support. So even to the scenario walkthrough of adding support or capability for GCP. Even then, from an operational perspective, it's not just the platform team, and the downstream development team on the hook for adding capability around GCP.

Rob Hirschfeld

37:19

Would Who else do you pull

Speaker 1

37:20

in? So you still got the security team that needs to either cook in their tools are developed processes around those things from that standpoint? question then becomes who's still on the hook from an operational standpoint, to manage and maintain what's been deployed into those environments? Or even build out those cloud environments? Billing?

Rob Hirschfeld

37:45

does? Does it make go head cloth if you're gonna add?

Claus Strommer

37:48

No, that's just like I was gonna say, like, billing is also an important one.

Speaker 1

37:54

The whole finance piece of getting hooked into systems cost centers and rolling up budget, Andrew there to counting.

Rob Hirschfeld

38:03

Does it make sense as part of that conversation to, you know, provide a path to retreat as an element to this, right. I mean, it's a little bit be more investment building something more portably is an investment, but it does give you some flexibility from that perspective.

Speaker 1

38:27

Okay, class, I get some thoughts on portability across class.

Rob Hirschfeld

38:31

Ah, can I tee that up? i Because I I mean, it costs. Actually, let me let my my expectation is that portability for clouds, adds across clouds, adds cost and adds effort and adds sustainability pieces. Yeah. Is it worth it? matters? What are you thinking?

Speaker 1

39:05

So I think the challenge I see is that people think it's much easier than it actually is in terms of the technical depth and new arts. And one of the things that was often touted even a few years back, was we'll just switch your workload from one cloud to the other cloud based upon the current price and, and everything will be good, the challenge becomes even spin up. Honestly, even if you take example, if you're an AWS, you're in two regions in AWS. spinning up a nether region can still be fairly difficult unless you've got like a fully automated process of just standing up the entire region of getting everything in place from the permissions and the the access and images and all those things, let alone trying to do that across an entirely different cloud. And one of the things that people have often brought up is Kubernetes as a means by Which to enhance portability? Certainly, that would be a capability. To that point, the question then becomes, well, what distribution are you using? Are you using third party distribution? Like in OpenShift? Or something like that? Are you looking to use Cloud managed? There's nuance difference between the cloud managed Kubernetes offerings? And are you how, how much? Can you abstract away some of those differences, without them adding just a massive amount of overhead at times? I mean,

Rob Hirschfeld

40:26

is there an 8020 rule for this where, you know, you could do you could walk forward in it without burning the bridges behind you, but, you know, knowing that, if you were going to retreat you, you, the, you're gonna, you're still gonna have work to do, like, you know, acknowledge, hey, I don't want to be totally locked in. But, you know, the cost of not being locked in basically means I'm gonna have to do the work on both clouds on an ongoing basis, just to ensure I'm not locked in.

Claus Strommer

40:59

There is but but it's backwards. If you go multi cloud, you're going to be doing 80% more effort and get 20% of the features. Sure, and being like if you're into be 100%, portable.

Rob Hirschfeld

41:20

To me, the only way you can be 100% portable is if you're actively doing simultaneously multiple clouds. There's right it's the effort to do it is high. And if you're not actively testing it, it's like backups that are being run there. They don't exist,

Claus Strommer

41:39

right? There are strong differences between where it starts. I'm not saying it's not possible, because I've done it in the past. But it requires a specific kind of workload to work properly. Like, for example, when I worked for a blockchain company, like, yeah, you start to define to be more declarative. In fact, we had the mandate to be multi cloud, in order to say like, yeah, this platform is truly decentralized. So we did spin up machines in AWS, and Azure, and GCP, in Digital Ocean, whatever we could find, just to be able to prove, like, hey, this works. And we're not tied to any one. Any one jurisdiction. But when you do that, you're also giving up some of the tight integrations, which, with a lot of publications, you end up using that. So for example, if, if you want to use rds, then you're tied into using RDS like I am for a RDS access. There's some requirements there. If you move to Azure, or GCP, there's gonna be different access methods for that you can choose to run your workloads entirely and coordinate is then say, like, I'm going to run my own database, but then you have to maintain that database.

Rob Hirschfeld

43:12

Is this potentially a place where platform engineering is a value? Like, could you have a multi cloud platform engineering team that then says, Look, we're gonna do the work to abstract out some of this stuff? But if your app requires rds, then use rds? That's cool. We'll we'll you know, provide a place for you to add that. Right. I mean, this is this is platform engineering design, which nobody has thought through. So we're in this this is the centralized decentralized, can you centralize as part of it and say, Hey, all this stuff we're going to own. And as a platform engineering team, we're multicloud. But as a streamlined team, you you don't have to be multicloud. But if you build on our base, then you only have to handle the the the the the Add arts, the specifics, the 20 20%. So is that work? Is that reasonable?

Claus Strommer

44:18

It should certainly be applied for engineering responsibility to help make it feasible. And, personally, I don't think the stopper is the platform engineering team. I did. The plumbing engineering team would want to make the features available. Again, you're supposed to be the champion for the developers. But if there are restrictions, compliance or financial or whatever that prevents you from being effectively more to cloud, you need to also be able to say, like, hey, we tried this does this the cost? And it's a management decision to say like, is this viable or not? It's, it's, it's kind of the same kind of situation us with security audits, you can say, this needs to be hardened. We need to take this approach to secure our, our data at rest, or we need to have this kind of audit capability in case something does happen. Whether who, who makes the decision to actually implement it, that's not secured. It's, again to manage it. So can you convince them? That I don't know?

Rob Hirschfeld

46:06

It's interesting to me, we keep coming back to the soft sense, but I would consider the soft side the non tech, technical tooling side of platform engineering. As as, as important.

Claus Strommer

46:21

It is, until you replace every developer with chat GPD, which we know is not going to happen.

Rob Hirschfeld

46:29

It's I mean, this is this is the whole idea, right? This is we're trying to get from a place where each team has made their own decisions, which from a corporate governance perspective is is not a sustainable pattern, to, you know, having some collaboration, shared governance between it and it's too much to expect. The key is to go through and, you know, learn what other teams do and compare notes and figure that out and say, oh, you know, if only we took time out of our schedule, and use your whatever to then write, alright, let me actually, I'll be very explicit, because this is a scenario that doesn't happen. And that's the open source dream scenario, then I think it's very hard. And that also is very hard to achieve. Here's the scenario. Team, Team A is doing something with Ansible templates and all that. Let's do TerraForm and pulumi. Yeah, TerraForm, and pull is Altera formed out and they're doing all this stuff, and it's cool, but they're like, now we have to hire TerraForm. Engineers, Team B over here is like we didn't do TerraForm. We did plumie. And we think it's awesome. And that's it, you know, they're bragging about that. And so what you'd like to see is some ability for Team A to be like, oh, you know, we'll just use the team B's pulumi stuff. But to do that, they're going to have to make the investment in switching. And then Team B is going to end up being a platform team functionally for Team A, because now you're they're using the stuff they wrote, and they're gonna have to give out time and support for that. And so what you've got is, or operationally, yay, we have two teams sharing work potentially with skills overlaps. And both the team A is no longer doing all this work and doesn't have to hire TerraForm engineers. But now, they've got to do the migration. And teams be now as to take time away from what they do to do the support. Neither teams is going to do that. It has to be pulled back into a into a corporate governance thing. That's sort of that's that's the way I keep seeing it. There aren't incentives, right?

Speaker 1

48:39

Yeah, I go one level different cities speak. What programming language? Are both the teams using? Are they using the exact same programming language?

Rob Hirschfeld

48:47

I, in some companies, there are strong incentives to a line like that. But no, I would assume they're not.

Speaker 1

48:57

And so if they're not sit there be an argument for both those teams use the same programming language to get the same benefits. We talked about with infrastructure set.

Rob Hirschfeld

49:07

Yep. And companies, a lot of times, we'll try and come through and impose a degree of standardization on that so that they're not right, that this was We saw this with Java for a long time. But I think that backfired. I think companies found that Java didn't create the Uniform Benefit. And maybe maybe we're looking for platform engineering to create a Uniform Benefit. That's not there along you know, is our test is platform engineering, the next Java Enterprise.

Speaker 1

49:40

It's a it's that same sort of rationalization conversation that we often go through with, with tooling and people's skill sets, if it gets a little more nuanced that we want to actually allude to.

Claus Strommer

49:57

I don't see that conflict But then again, my view on buffer engineering is, is taught it's it's claimed first, there should be planning first. Okay. I agree with like, if you do plan for engineering and you say like, you must use this tool. I mean, you're doing it wrong, you're setting us up to fail.

Rob Hirschfeld

50:20

No, I like to describe it as service oriented operations, as in providing service providing service prefer, right? If you have to have a service mentality, which is to me what you've come back to that multiple times, it's like, your job here is to enable your customers to be successful. Don't get don't don't think it's to control your customers.

Claus Strommer

50:49

And as a result, like if, if TMA is standardized on TerraForm, and timbi, is on call me? Well, yeah, we're going to see if we can meet both of their, the team needs with one solution. But if the answer is no, then we shouldn't force TerraForm or pulumi on either one of them.

Rob Hirschfeld

51:17

This, this is where I think infrastructure is different than than programming language choice. Because my my experience in this is that unless there's something very specific, both teams, if you if you went up to both teams and said, You shouldn't have to do either, we're going to give you an API that does that work for you, and we'll manage the infrastructure on the back end, both teams would would shout for joy and be excited. If you were if you were still, if you were gonna keep listening to them, if you stopped listening and said, you know, you know, you can have it, you can have a model, any color car you want, as long as it's black, back to the Ford Model T. Optimization, then I think that conversation doesn't go as well. But I, most teams do not want to do that. If that that work, they would if they felt they could they I think they would hand it off.

Claus Strommer

52:19

You don't want to be doing this. Or at least not as you give them the freedom to work within their environment. It was strictly

Rob Hirschfeld

52:30

they don't want ownership. Yeah. Right. And that funny that to me is the success equation for platform engineering that we've been describing is can you give up ownership of part of the work, but keep the influence keep the collaboration?

Claus Strommer

52:50

Yeah. Okay, if you if you make your platform too restrictive, then you end up with shadow IT. And then of course, you you feel that making an internal platform.

Rob Hirschfeld

53:02

That's actually part of what made Java the Java Enterprise stuff not work, by the way was when the Java Enterprise, the corporation was like, you have to use this version of Java and these libraries, and they became really slow than Java, which was probably it was pretty functional. It became a bottleneck where the company was moving too slowly and adopting the new standards. They became the bottleneck again.

Claus Strommer

53:34

The JVM wasn't success. Like it did allow horrible obligations to a very large degree. Java language, it, it did falter. And I, I mean, that's probably a topic for another time, but personally, I think I'll add it all over managed?

Rob Hirschfeld

54:03

Yeah, no, it's it's I haven't thought about Java in a long time. So. And we're at the top of the heap from giving myself Java nightmares. Let us wrap up. This was cool. I actually think there's a blog post in here about the management side of platform engineering and ensuring that people think about it carefully. And more broadly, so thank you. That was fun. Appreciate your time. Have a good day everybody. Cheers.

Rob Hirschfeld

54:53

I really love when we have these thoughtful conversations that go way beyond the tech and think through the human component. It's about building great systems, of working on how teams interact and work together. And it really comes from our experience and the ability to ask questions of each other that makes these conversations great. And I hope you enjoy them. If you do enjoy them, please come in, join us be part of the roundtable. Bring your thoughts and ideas. We have an agenda of topics that we post at the 20 theory dot cloud, and you can see what they are come in, bring your own questions and comments and topics and shape how the cloud 2030 goes forward from here. I'm looking forward to hearing from

Unknown Speaker

55:32

you. Thanks.

Rob Hirschfeld

55:36

Thank you for listening to the cloud 2030 podcast. It is sponsored by rockin where we are really working to build a community of people who are using and thinking about infrastructure differently. Because that's what rec end does. We write software that helps put operators back in control of distributed infrastructure, really thinking about how things should be run, and building software that makes that possible. If this is interesting to you, please try out the software. We would love to get your opinion and hear how you think this could transform infrastructure more broadly or just keep enjoying the podcast and coming to the discussions and laying out your thoughts and how you see the future unfolding. All part of building a better infrastructure operations community. Thank you

00:0000:00