PODCAST: Edge Control Planes

    9:23PM Aug 10, 2021

    Speakers:

    Rob Hirschfeld

    Rocky

    Mark Thiele

    Keywords:

    edge

    plane

    data

    people

    control

    infrastructure

    cloud

    design

    network

    point

    industry

    pipeline

    piece

    vendors

    compute

    talking

    rural areas

    store

    cases

    thinking

    Hello, I'm Rob Hirschfeld CEO and co founder of rockin and your host for the cloud 2030 podcast. The July 1 session in cloud 2013 was about the edge control plane. And we had a robust conversation about the challenges of building edge, and even really understanding what was available. And a lot of it came back to, as always, data, data pipelines or data orchestration, or data choreography, and how influential that is in all edge. typographies. So, during the conversation, we cover a lot of ground. The topic for the day is is edge an edge control plane. And we actually had sort of what I thought we would do as a stepping off point into edge from the work anywhere story, because work anywhere is great until you're realize that you're supposed to be connecting to a cloud and you don't have connectivity, right? And so, the, this, this idea of us being able to work anywhere, implies that we have compute everywhere, or compute. close to us, I think, especially as our daily work becomes more compute intensive. Well,

    compute everywhere is easier than network everywhere. And

    I mean, so So,

    and in fact, at&t got caught? Well, I don't know if you can say got caught. But at&t is telling the US government that the rural areas don't need anything more than then 10 megabits per second upload at most. And they're telling Yeah, because they don't want to replace their, their copper with fiber. And they're telling their own investors, that fiber is the way to go with symmetric up and down. And that they're rolling it out to specific neighborhoods, so the people who can pay, but they don't want competition in the rural areas, and they want Well, they've read fiber in my neighborhood for a year, but you can't get it from them. But Comcast just put in fiber.

    I Rocky, in Palo Alto, we couldn't get fiber in many of our places. And I didn't have it until moving here to San Mateo. From at&t, I've got this symmetric. And I, you know, I had to, I had to adjust my expectations on downloads, and you know, you'd started being download and turn off and go, you know, walk down down the hall to get a cup of coffee or something. Now, it's right there. It's too it's it's pretty, pretty amazing. Like, gee, what the first world really is like, when you when you have broadband. But, yeah, this the notion that you know, people in rural areas, no, I'm done the more than 10 megabits megabytes up. It's just crazy. Um, what about the

    they're just saying that because they don't want the government to subsidize fiber, that would be community owned instead of better, because it's also been demonstrated that they charge more in areas where they don't have competition.

    Yeah, exactly. But, but would, who is network, ubiquity, quality network ubiquity. Something that we should plan into our designs are, you know, are we do? Do we, This to me is one of the questions for edge. No, will we need to have more, you know, distributed infrastructure? Because we're not, you know, either the networks are never going to have enough bandwidth or the networks or the latency is always going to be a problem, you know, that, that we're, the applications that we build are going to require more local compute infrastructure, which I tend to believe is yes. Or is there a hope that we could have these stretched control planes where I've got a region and everything's actually managed out of a big data center somewhere.

    So where does 5g fit into this? For where they do have 5g makes a what? Well, 5g is also still it It's, it's still city stuff because you have to actually put your towers closer together because the frequencies are higher. So you need repeaters more frequently. So again, unless you get it, unless there's somebody who wants to feed rural, you don't get rural on 5g. And so they fall back to 4g, or 3g or sometimes even 2g. And so you really need to plan your edge control plane for a lossy environment where things drop off. And for multiple generations of technology, the IoT in farm area has been was one of the things that back in 2006, I was hearing hearing about and we were doing a, we had a control plane for edge that had both 2g and satellite. And we would fall back to satellite, if there was no 2g and satellite, you get 50, you get a 2k, every 15 minutes back then. So you had to design your communications. So they were really, really efficient, because you couldn't get any more than a certain amount of calm. And it really hasn't changed for the rural areas. You're still relying on satellite. There's more satellite out there, but it's still extremely pricey. I don't think

    that's true. I mean, I actually have a customer and he's doing a tech. Right? we're deploying mesh networks into the fields, right? We're interconnecting that back over into the Wi Fi components that they've got out there. And quite frankly, the data we're collecting isn't very much. I mean, in actual, you're not just saying because there's not there's not a need to collect a lot of data.

    But there is a need for control, for instance, well, they've they've designed it so that the control is local. They were doing GPS based robotic farming back in 2006. With the big combines following GPS along their land, they really could have used 3g, GPS, but they're relying on the edge being smart enough that they don't have to connect or send very much data. And so it's already been designed to deal with a very efficient, low data rates to the control center outside of their local area. But yeah, and mesh and stash. That's why fast on that.

    This wasn't no one approach, designing the solution saying bandwidth is limited. They have to have connectivity within the field.

    Right. Back when they first started it. So this was the state of affairs, it's the Sure. So I started with the Brownfield aspect.

    Yeah, exactly. They don't think it's true any longer. But I think the two of the things you're kind of missing, when you start talking about these conversations, right? When I was doing a project with LG, and we're looking at broadband distribution, and what percentage people that actually uptaken into it. Even though high speed broadband was available, like 90% of the people elected to go with a $10 a month plan. They weren't willing to pay more to get high speed bandwidth. And so there's a consumer economic piece to this. And then there's also this a business economic piece to that, right. If you're going in the US and you're building out data centers, and you're trying to figure out where you're going to turn a profit on these things. Only 50 of the major Mets matter. That's that's where like 98% of the revenue is generated from. And so now you're saying we have to distributed to get to pick a number last 10% 20% 5%. Right, we're going to basically make a massive infrastructure spend when that's not where the revenue flow comes from.

    That's why the US has the loss for for carriers as opposed to common carriers, as opposed to regular data. And we don't have common carriers for data as yet. But the phone company, it was subsidized by the US and the government and the government wanted it everywhere. And it was considered national security. And hold on though, to sing with data. Yeah,

    I made that I were getting on on a on a network perspective. We have two hands. We have three hands raised. And I think we're a little off on us. Some of the edge pieces. Joanne, I want to throw it back to you. And then mark and then rich.

    Actually rich rich before mark. Oh, sorry.

    Okay, well, ladies before gentlemen, um, I guess I'll go down the two routes. One is, um, I've been involved in projects, not only from an enterprise point of view, but also from the analyst point of view where the last mile is always the issue. And like, I'll give you a for instance, there's some very large us headquartered OEMs, that have a lot of factories in Monterey. And Monterey is known for two things, very bad comms because of the mountains around it. And the issue of not being able to connect, so you adopt the five nines failover tactic of if I can do it, T one, you know, t three t one, satellite, wireless, whatever means of communication, I can use, even running wire between factories, if you have two halves of the factory that are in a valley, and they're on two sides of the mountain, you have massive issues with satellite communications and everything else. But that said, from the edge perspective of the control plane side, the 5g only gets you to the telephone pole outside, it doesn't get you in. So that becomes part of the issue to be resolved. But from the control plane side where and, folks what I heard yesterday, from Mesa, which is about 790,000 companies around the world, all in the supply chain side, is that their issue with the control plane is they haven't figured out how to identify. And this is a gap in the market where the compute versus the storage side of edge have to be and what they're looking for in the control plane is, okay, how do I make best use of my edge computing devices across multiple types of factories, and in some very remote locations, like one of the members is a gentleman from leg lakes, right. So they're agribusiness. And they're also the pet food, business and Purina and all of that kind of stuff. So they deal with a lot of remote locations. And what they've been doing in the past is using GPS coordinates to try and partner with like Caterpillar and other companies to figure out that edge plane, I look at it from another point of view, I think that we have a better way of doing that, for them both of the design side of the software, and also the hardware configurations that we put in whether we do compute in one location and storage elsewhere, or portable storage, we can design that in based on the flow of data, which is more likely what we're going to need to use for insight than anything else. That's just my two cents on how that control plane might take a design premise, not how it's actually designed. But certainly a

    very strong nods going on all around.

    I mean, part of part of what you're saying to me is that we get distracted talking about the network. And what we really should be talking about is the data flow, the network is the data flow, but it's but we were as easy to talk about, the network's what we're really talking about is where the data needs to be. And when.

    Okay, when I started, didn't Trump but when I speak of data flow, I mean, the flow of data between machine one and machine two or machine one and human two, or any permutation there of where the design point needs to be, what is the fastest, most efficient way to take the data and turn it into information, which we can then turn into insight. So it's a choreography, or an orchestration, I prefer choreography, simply because it has to be flexible. And there's a change because patients

    and you've got data that's real time near real time. And you've got data that can sit and wait till a point where there's more bandwidth, like during the transfer and stuff like that. And so you need to understand all the different types of data and when it needs to be in the place that needs to be for analysis.

    Correct. And you also need to understand that there has to be some level of, um, for lack of a better phrase ETL that has to happen because at the protocol level, you've got 13 different protocols that you're working with, for various types of machinery and all Also those used by humans, so has to be human readable as well. And this has become a huge issue in terms of that, whether it's design patterns for cloud, whether it's design patterns for edge or the aggregation of both. And that's what I refer to as data flow. And the choreography around that, to me, that's the most important part of the data plan. And it could be a very narrow lens that I'm using. But it seems to be the one that I'm hearing the most about, from the supply chain communities from the manufacturing communities. And that to me, also sets the stage for a B test to trickle down to the consumer side. Because very quickly, we're getting in automotive and in retail, my IoT has to now incorporate the IoT for feedback loops from the consumer. And that implies security connectivity, and everything else. And so to me, those are the key factors, at the highest level of abstraction is the flow of data, simply because of so many variants that are involved.

    And along those lines, again, it's the storyboard versus what needs to be real time and needs to be more along the lines of mesh and whatnot. And it's a totally different design from store and forward where you need the history, the logs and the information to aggregate beyond what you need right now to make a decision. Correct. There's strategic versus tactical, and the tactical stuff is right now and the strategic stuff. In some ways, it's not really that division, but it's Yeah, and this is what fascinates some, what fascinates me about edge is getting this point, right.

    Well, I think there's right now and there's almost right now, because in some cases, the right now decision impacts the almost right now. And so you may only be storing for a millisecond before that data is actually being used or shipped. Yes. Whereas the rest is I need it now. And something's going to happen if I don't have it, whether it's on the predictive analytic side, the continuous intelligence side, or the Will I be subject to 6 million unit recall? Because I'm offset by a tolerance of point 000 1%. And by the way, the next iteration of laptops will have battery problems, as they did several years ago, we're gonna see massive recalls, the same is about to happen in 2021 22.

    Great, thank you, my batteries are already swelling and breaking the my laptop from the swelling and this is the old one.

    Well, newer, all the new ones will will be overheating in a way that nobody ever expected. What fun? Next hand?

    There it should you you know, you have something to add or take some new direction?

    Well, two things. I think when we were talking about the considerations. And you you posited the kind of one extreme. Rob, that do we assume that network is always there, it's one of those things where I don't think you can, for purposes of either business continuity, or safety, and including safety kind of security and privacy kind of combined there. there's every reason to think that you have to design edge to some degree, the way

    cloud native applications have to be designed for you no failure will happen. Something will something will break. there's a there's a fail, fail over there's a failed back, that has to has to be incorporated in the in the infrastructure in the design, the basic infrastructure. The second point that I wanted, so I wanted to put those two as high order considerations, business continuity, whether it's an emergency, you know, that's a natural emergency or some mal some bad actor that's that's doing something. The other issue is one of the considerations and going to these questions is to recall, what are the scarce resources. And I think Joanne and rocky have both honed in on the notion of data, I mean, we think about, you know, categorizing infrastructure with regard to compute, network storage. But there's a fourth, and that is the data that is being moved across this, the networks, and then and that is actually actually occupying storage. And when thinking about control, when thinking about the layering, we've got more thought being given to differentiating a data plane from a control plane. But I'm, I'm not sure that this is what Joanne was referring to, or not, but I almost think that there's a there's something that's like a control plane, but it's for a different, it's there for a different purpose. And it's, it's more like a, I'll call it an API plane, it's, it's some of the transformation and filtering and control that sits in front of and is used in stream. So those are, those are the kinds of approaches that I would kind of start out with, you know, what are your basic assumptions? What are our presuppositions and axioms about edge, and then move from there?

    It's in the API control plane comment that you make is interesting to me, because I think, you know, when I think about the Kubernetes pieces, in these, right, that that ends up being an API control plane, we've been, we've been talking, you know, from our perspective on the infra what we're starting to call an infrastructure pipeline, I'm watching the data pipeline pieces, right, you have a code pipeline, you have an infrastructure pipeline, you have a data pipeline. And it's, it's interesting to me to think of the API, an API control plane separate from the infrastructure pipeline, but that might be the decoupling that we need to or infrastructure control plane that we need to have,

    you know, when, when we set up things for data centers themselves, there are, you know, there are there are levels of control, there are, you know, kind of categories of control, and, and, and both administrative control management control of the data center. Um, it starts with the, you know, basic architecture and, and, you know, it usually, you know, in the old days, locking certain things down, like VPN gateways, so forth. So, there's a, there's a authorization, granted to a certain community of processes and people, the individual making use of that of that data center, and that infrastructure has a different set of controls that, you know, sometimes dips down into the network, but very often, it used to, at least, you know, you didn't get a lot, you didn't get a lot of didn't get a lot of control of what was going on in the network, it was pretty much locked down by whatever the organization was that is providing the resource to you. But there, there is truly a distinction there. And then the applications themselves, you know, being concerned with what is actually being accomplished, what the data is that's being moved across the networks. And I think your point about Kubernetes is exactly where I'm where I'm taking this kind of splitting this notion of a control plane. And then I think your your notion of a pipeline may be the right way to think about it. I think it makes a great deal of sense.

    That's, we're about to start doing a whole bunch of work explaining infrastructure pipelines as a concept is beyond workflows. Because they're turning out to be different in our instruction. But there's some thing that you said that I would connect back to what Joanne said that I think is material. When we think about cloud, we don't data, the data. choreography is not something we have to worry about too much. It's not a first order concern, because we assume no storage and network are basically elastic assumption or elastic consumption in the cloud. And in in edge cases, it's a bit you know, moving moving information about and processing information is not a automatic assumption.

    It's you're looking at, it's automatic than you might think, in cloud. I mean, when you've got multiple regions, when you've got when you when you're, when the demands are for date, intelligent data replication, and data replication, that must be done not just for the purpose of transporting one app or one stack, move it to somebody else, move it to some other cloud, but actually ongoing. That starts to that starts to address some of the things that you're saying. So it's not existential. It hasn't been existential into Cloud cloud design of application. But it's getting

    interesting.

    Yeah, pipelines is a is a coming analogy, I think more and more. Mark, you if you want to take the next Yeah, I am.

    I started thinking about an answer to a question that is now you know, 20 minutes old. But I will, I will add that that's, you know, that's how it works. I'm not complaining. It's just it may not be worth much as much as much now. But I first one is, is just a comment on something john said earlier about the study with LG. And I would argue that the I would ask if that study was done pre COVID? Because I think the answers would be very different today from the potential clients. And I would argue that many 5g rollouts is at least many that I've seen and where I've talked to people involved have indicated that, that where performance can be something close to guaranteed, there are a significant number of people that are willing to pay more for that low latency and availability. Korea was one of the first examples. But beyond that, you know, I, I love the topic, and I love the discussion around control plane and about data movement. I think, you know, the localization of network traffic is really the key long term answer towards the ability to provide some level of guarantee for edge and EDGE PERFORMANCE and data resiliency or or access resiliency, it's a lot easier to put three or four connections to the same space in a city than it is to put three or four broadband connections from a town in Iowa back to a town in California or Ashford. And so figuring out the right way to manage to, you know, the establishment of local networks, whether that's with 5g or not, seems to be that the control plane needs to allow for some level of localization. But I also I also think that we're, we're, and I don't know if there's a better way to talk about it. But I feel like we're talking about this edge problem. As if we're in a position to solve it for a single deployment of edge near the world. And we're not right, it doesn't matter if we could dream about it. I'm fine with dreaming about it. But we're not. And so what the problem that john talked about earlier about, you know, low data requirements is going to be true for 1000s. The question we don't understand is, how do those 1000s add up on top of each other from a backhaul standpoint, we also don't really have a firm grip on, you know, what it looks like when every every house and every building and every city is considered smart. We don't have an idea of what that looks like from a total bandwidth requirement, or from a performance requirement or from an emergency response requirement. We have to recognize that that localization may become a necessity not even for some of the reasons that we care about, but for safety reasons, because, you know, I've been talking about the notion for a couple years now, but the notion of you provide a service To someone, once they get used to it, it's not a matter of just saying, Oh, go back the way you're doing it before, right. And a simple analogy would be if you go into a store that used to have checkers at every counter, and could handle, you know, 100 patrons, every hour, you can't just say, Oh, the automatic checkout isn't working. But patrons can keep coming, you've effectively got to shut the store down. Because you can't handle it with the one or two people that are still in the store. Even if they had the provisions to try and do it, they wouldn't be able to. And you would lose your you'd lose your tracking for your inventory, and you'd lose your your security tracking, and all the other stuff that go along with it. So all of a sudden, the store is closed because of some sort of network traffic problem potentially, right now, what happens when, you know, three out of 10, or five out of 10 of those things actually relate to providing for some form of safety, even assumed safety, right? I mean, I'm going to give something really graphically stupid. But in the United States, it's fairly common in most cities now, to find paper to put on the toilet seat before you sit down. Right now, there's never actually been any evidence anywhere that I've seen that said, that paper actually provided any benefit, because there was actually no risk, other than getting your butt dirty, there was no risk to your health. Right. And but how many people I would argue I'm one of them, I'll be the first to admit I'm one of them. Once I started using those things, I can't fucking use a public toilet without putting something on there on the seat. So safety

    becomes something of opinion, something of psychology and a reality. So the services around how to how to cross the street, how to drive a car in a town, how to use a building. All of those things with each passing day, those activities become safety issues, because humans begin to forget their natural habit of assessing the environment before they use it. You know, where most people get killed on the road? I shouldn't say most people, I don't know if that's true anymore. But I looked at this about five or six years ago, actually a little bit longer than that. We're still in Union City. So it's more like 15 years ago, and I looked at it because some people in neighborhood got run over in a crosswalk. No, what I found out more people die crossing the street when they know it's no, it's safe. What happens when those safety guidelines when everybody believes everything is safe, it's safe to get in the car, it's safe to get on the train, it's safe to use the elevator, it's safe to to walk in a crowded space, it's safe to do whatever what happens when those things become unreliable or unavailable? Right. So edge to me, for a lot of near term obvious reasons requires its own ability to manage the flow and availability of services through data. Right. And over time, whether we like it or not, we will become more what are the British call it the British have a term for this? Where you know, the the society has become so protective of the of the humans? You know, nobody can get about a nanny state? Yes, yes. I mean, that's right. Every city will become the equivalent of a nanny state. And, and we won't be able to live when the nanny state isn't available.

    Quick, quick point. You know, what you when you look at some of the issues that you're addressing them, Mark. couple months ago, I had a deep dive with some folks in infrastructure at Walmart. And Walmart's a good example of an organization that establishes it's kind of service metrics around availability. And what they do is have in addition to generation power generators, that will take care of the refrigeration and limited lights. And also, the availability of checkout counters, the checkout and they also have a limited time set up such that they can actually do transactions, take cash swipe credit cards. Granted, there's a you know, they're delaying the connectivity for A lot of them, but they can operate for 48, maybe even a little bit longer than that, without electric power coming in from the grid, or data communications. And the whole point there is they become, you know, they become a lifeline, they become a become a, an emergency resource that, and by they can only do that by virtue of the fact that they have a, an ability to operate independently in an in a disconnected mode for some length of time.

    Yeah, no, absolutely, absolutely. Right. I mean, that's everything that I heard when I talked to them as well. Um, and, you know, so And when you think about, of course, Walmart has the advantage, as well of you know, that this may be a benefit in an earthquake, or maybe you benefit in a flood or a hurricane or something like that. But it's also that similar to the way large scale data center campuses can afford to do things that have, you know, a five megawatt or a two megawatt facility can't is that they make so much money at the site on an individual site, that having it down for two days, is way more costly than whether or not they have a generator, or, you know, some localized IT infrastructure.

    But, and we've talked about things like this in the past, you know, and it's funny, because I was thinking about the camper, the CTO advisor camper thing, right, you put in an electric self driving car in front of that camper. And then all of a sudden, that the computational facility of the car becomes your office computer, when you're camped. The battery of that camper becomes your, your, your house, our, when you're camped, right, we're actually talking your edge has the potential This to me is and this is where autonomy really gets interesting. We could be on the cusp with electrification, storage and compute of actually having real, you know, reuse of local resources. Right? You know, I think we've had conversations, the password of your store, you know, to maintain the freezers might actually have battery power and generation backup, they could be selling to charge cars during normal business hours, right? There's a whole bunch of opportunities that we're moving into, especially with with with compute being generic and storage being generic that we could actually see some convergence from that perspective. And it's not edge computing, it's just, it's just in situ computing, it's just in situ environments. But but this is where I was actually thinking we would go with the control plane is, in those cases, you have enough power and processing to be autonomous, what we haven't worked out is the control planes that allow those systems to operate autonomously, we've sort of gotten used to moving all of this stuff to the cloud, letting service providers be the pros, and letting them then handle all those situations. And they're very eager to do that for commercial reasons. The thing to me that is really hard with the edge control planes is you have to have an autonomous control plane has to be local, hit count on the network, it can't, right, I don't care how good we get with our networking years, that store for all the reasons we just mentioned, has to be autonomous, which means the management of it has to be repeatable without IT professionals running it. And so the paradox for edge that I think is really hard is that we have to have a distributed control plane that allows us to have but it's not a stretched control plane. It's distributed autonomous, distributed federated control planes. To me, that's the breakthrough that's missing from an edge edge perspective.

    The only Oregon the only group that seems to be taking that on Rob, is the smart grid, people in electric power. Smart Grid, smart grid work. It's not done yet. But there's a there's a lot of work being done in what you almost might think of as kind of a modular Smart Grid where there is a kind of a smallest unit that can be self self managed, at least for some period of time without being connected to the rest of the grid.

    I think the autonomous vehicle community is also doing that. It's just you don't see as much of it. Yeah. Oh, actually.

    Sorry, just just in to add on ASR g is the group and there's a lot of work going on that's very similar to the smart grid group. This is for a tongue autonomous vehicles and vehicles in general on security on autonomy. I'm part of the ASR G, and I'm about to become an industry advisor for the Sentinel security, for cybersecurity, for vehicles. So the automotive industry has taken this up. And there are actually two ISO standards that you might want to look at with regard to that, and how they're defined. One is predominantly around cybersecurity and malevolent actors. And it goes back to ties into this notion of pipeline. My comment on pipeline is doesn't control plan have to come before the pipeline, so that you could catch them a level and send any bad actor or anything else from the security point of view. But also, from the autonomy point of view, I think you'll find that there's a lot of learning that you could take and reuse of leverage from those two groups in automotive for the control plane for edge.

    That's a convergence, we're sorry for the interruption. Great. Do you want to we're about five minutes from the top, you want to give a topic for us to close this out?

    I don't know I got server thought. I think you kind of said one piece to it is we always talked about these near edge compute for control planes into it. And the reality check is, every time we got into it, if it was missing, critical, it was on site. Right, no one's going to tolerate a network component to it. And so I think a lot of these things that talk about near edge compute use cases are wishful thinking. Because mission critical applications will be on site, no one is going to basically run your airplane from a network on ground. Right. So I think that's that's one piece committed, I think the second piece comes into when you contrast Walmart, right? Walmart has scale, Walmart can afford to do those things. And so I think they're different than most of the other use cases we've looked at. Because every time we looked at a use case, it wound up being constrained by the economics, there wasn't enough business revenue, to justify the deployment of the IT infrastructure into it. So I think you've got to look at the economics of these things, because we talk about these things as if it's ubiquitous, but there is an economic component to it. When it comes to edge control plane, right. The problem we had with edge control plane is the cost of deploying the control plane often exceeded the cost of deploying the application. Yeah, right. So it's not a Can we do it, it's another one of those things of the cost just didn't add up to solving the problem with it. So I think that it gets got to be really a nice conversation to be grounded in the real world. And then I guess the last piece that is I would say, you know, when it when it comes to is that is the data pipeline, part of the the witch hunts? first answer is it's all part of the system's architecture. If I had to manually provision my data pipeline, I do it right. So there's several components to the system architecture. And which comes first is it really can be a question of how do we automate it. And I guess the last one, I'll just close on I look at a lot of use cases, we talked about the day and I've got like, they're really not painful today. And most of the ones we found with our pain, were volumetric, you know, and, and, you know, some Cupertino phone company had problems, because they use HTTP long poles, will change your system architecture, change your software architecture, and that problem goes away. However, having said that, I think what you should really be thinking about is let's talk, you know, five years down the road, 10 years down the road, we're instead of having a couple of billion devices connected, we have a couple trillion, right? That That, to me, is where these problems really start to take root. So I don't think it's an immediate how do we solve these things? Yeah, there's fringe use cases, right? Like like rural manufacturing? Are there pieces to that? And I'm sure those are real problems for people today. But I don't think they're going to become systemic enough for society until we start getting to critical mass and devices.

    I'll leave it there. I agree with you on how do we manage these things out? I don't know that there are simple solutions on all the pieces we're seeing like we're some of this is me very self inflicted pain. Where the idea here is that vendors just aren't collaborating together. So we're working with some telcos and helping them build and they're insisting on stretch control planes, because they don't want to put to your point the control plane overhead on the on a tower, they want to, they're there. They're trying to keep the they're like, okay, I can do a regional control plane and stretch it. When we get it. That's fine. But the like, they're talking to us to control the compute. But then the people who are selling the switches are like, Well, we'd have a solution. You can't You can't mix those. The switches are its own control plane in the computer. It's it's uncontrollable, and we're talking about two devices. And so the that part of the problem that stands in our way for a lot of these things, really comes back to, you know, the fact that our vendors are building their solution selling their solution. And there, we're still siloing these things together from an edge perspective. I mean, gentlemen, I know you're helping people overcome this and built integrated solutions. But when we're looking at the market, it's, you know, network storage and compute being completely different isolated vendors, if you bring in three vendor, and multi, multi multiple vendors in any one of those categories, now, you're totally off the reservation from building a real control plane.

    Um, I think the way I'm seeing it play out, I'm not saying you're wrong, I do agree with your point. I think where I'm seeing it play out in a more collaborative sort of format, is within industry groups. And I know that that's not necessarily the best place or the worst place. But industry itself is coming together in different verticals, to look at these problems and say, we need a solution period. And I think that's a better route that the vendor is trying to do it on their own. Because you all see a problem of a corporation from a different lens, where your piece of it fits in. If it's a group of companies in the same industry, let's not call it the Walmarts of the world, but rather their supply chain or their distribution chain. That's where you're going to see the most bang for the buck in terms of getting things to a point of being able to look at industry standard or look at a company together of all the different pieces of pie from each of the vendors. I've seen it in automotive, I'm seeing it in electronics, I'm also seeing it retail. And in smart cities, to Rich's point about the grid grid is leading a lot of the work with the smart city centers. And, you know, with no disrespect to my American friends, we have initiatives here that are geared towards pushing people together through the university systems of, you know, the private public collaborations, where it's pushing industry vendors to come together at one level with government, not necessarily around funding, but around new projects. And new industries. Like we're doing it for automotive, we have five now hubs of AI simply designed for smart cities. These are like Montreal, Ottawa, Toronto, Vancouver, those are hubs where people are throwing money into the arena to say, go build a solution that can be ubiquitous across the country. So you know, I just put on the thing, take a look at the global edge x platform, in terms of control planet in terms of this conversation only, not that it's the best or the brightest, it's a vision of what I think you're articulating, and I think could be leveraged into fuller design. I also don't disagree with John's point. I mean, some of the cases that I get happened to be real life, mostly from previous clients or places that I've had roles. But they're big companies with big problems. And I think the difference is also geographically, where is the problem coming from? And that also goes into the design of the solution plane, because when you look at control plane, you need to take that by the blinders off of geographics, like, for example, murky, you should probably I'll get, I'll send you a link to the spacer group, because this is companies from around the world. And what I heard yesterday, as just one example, and I'm sorry, that we're over time is up in parts of Asia that have not yet adopted cloud, they're much more prone to adopting edge because in certain regions edge would the cannot activity is better. And that would give them a better rate, though. So these are these are more industrial based, they're not all manufacturers, some of them are NRF. companies as well, but it's a hub where you can find this kind of information that is real world that people are actually dealing with. And I think that that informs the that control plane design could give a much broader way.

    Yeah. How how participation is there from the significant vertical, the players and the customers themselves. The the potential beneficiaries of this design, not the manufacturers, not the service providers of the technology, but the consumers on it.

    Well, I think the manufacturers and the suppliers are the consumers of it, and driving it, or driving it, because these are solutions that they needed.

    Right. And, you know, I, I completely agree that you have to incorporate some of the biggest consumers of a technology to get the standardization you need, it rarely works without that kind of pull from that kind of demand.

    I think there are other groups that are looking at this kind of standardization that run anywhere from the industrial internet Consortium, which is a global player to the threat group, which is a massive group now of, you know, very arm related, etc, etc. But if you look at these ecosystems as they're being developed, they're they're quite robust, in some cases, and quite lacking in others. And where people like me come in as we go, No, wait a second, why are you reinventing the wheel when some of this other stuff already exists, you should be able to leverage it and build upon it. But the consumer rich to your point is actually going forward. You and I, we are the consumers of this technology.

    I'm I'm hardware. I'm harkening back to the days when electronic mail was new. So yeah, it myself, but one of the big things was we had to two groups of protocols, that standards being developed. But in point of fact, all of the value added networks had their own email services, but they were trying to, you know, sell to big companies at that point in time. And they did not interconnect with when they did not interoperate. And one of the things that we did in the with the creation of electronic mail Association in the 80s was purposefully incorporated into the group, the big consuming industries, American Electronics Association, American, the American aviation, and those communities. And they came in and they basically basically, use them as the lever, we they basically got on board and said, Unless you utilize the standards that are already being written up for interoperability, interconnectivity, we will not use your services if you do. And it was amazing how quickly at&t MCI. And the small folks as well got on board with interoperability mostly around addressing and directory. That's no, that's what you're,

    well, you know, I care, I call it the carrot and stick play. And it worked extremely well for IBM and five shahadi, with Rosetta net way back when, since 2003 2004. And it literally got to the point where the big gulyas of I realized that's an oxymoron. The gulyas turned around and said, You either do it my way where you no longer have my business period. And I've seen the same thing unfolding with cloud with edge with certain types of technology, certain varieties within those categories of technologies. And that's why part of my issue with the notion of data pipeline and with choreography, is because unless the industry that it's designed for or multiple industries that it's designed for heavily participate, the characteristic will no longer work, because they will have clouded edge are two areas of technology that are far more specific than b2b was.

    Exactly and they are they're becoming more existential for these. Yes, yes, absolutely. Folks, are here. We

    need to have a conversation. Great.

    I'm going back to Canada Day you all have a great Fourth of July. I'm gonna go celebrate my holiday now.

    Canada Day, happy. No, no meeting next week. I took

    the There's no meeting next week, john. Okay,

    what were you gonna say? Oh,

    no, I was gonna say just put in people's mind leave for next one is is I argue edge compute. We talk about it this way argue the battles already over. It's it's Microsoft, it's AWS, they're deep enough in their data centers to cover most use cases, to talk about standardizing stacks, I think you've got three or four major players that are the stack.

    That is saying talk about for next time. Yep. All right, everybody, have a good one. ruminate on that on your resume. Also,

    I hope you enjoyed this session about edge control planes, we spend a lot of time thinking and talking about edge and edge control planes, and how edge is going to get rolled out. And this was a good topic. I think we summarized a lot of a lot of insights. And you know, john really left us on this topic we need to get into more, which is how the hyperscale cloud vendors are going to try to own the edge. And if that's what we want, and how are you going to listen to the cloud 2030 podcast is sponsored by Rankin, where we are really working to build a community of people who are using and thinking about infrastructure differently. Because that's what rec end does. We write software that helps put operators back in control of distributed infrastructure, really thinking about how things should be run, and building software that makes that possible. If this is interesting to you, please try out the software. We would love to get your opinion and hear how you think this could transform infrastructure more broadly, or just keep enjoying the podcast and coming to the discussions and laying out your thoughts and how you see the future unfolding. All part of building a better infrastructure operations community. Thank you.