All CI all deployments and for all clients, okay? Yes. And now I'm trying to use GitLab Kubernetes shared runner to, to build MFPs. Yeah and trying to to find the full policy to to build the base image and then to get the latest, latest, fresh, base image to continue to build building images. Yeah, and the default default policy is if not present, and it can cause some subtle difficulties for everybody is that, for example, I just right now I just try and I'm trying the button, maybe you familiar with it? In you know, dialect, no utility to replace MPM. So we experimented with experimenting with PMPM and now I'm trying to compare it to this bomb. It declares that it ran 17 times faster on install, in time trying to build our MFE which was it Yeah. And my current case is about that. I just switched from being pm to one. But this default policy proposal is late. Not the latest from our registry. It's tricky
but I'm curious to know how you make a lift button. I've always found an interesting project. I I use Dino a lot which is another alternative. And I love it. But it's not as node compatible as buttons that you typically get. But button looks really cool. But it's also made in a really interesting programming language called St goods I never heard of until I saw but but it's neat. Hey, everyone, hopefully be enjoying
it you guys
I was muted sorry. I was asking if you're ready. And I don't know if someone will be joining from here today. But other than that, the other organizations are represented.
With the layoffs sent to you unfortunately, Jeremy and Bowman is no longer going to be working there and are able to join us and think, what can you say? Yeah, last week, Adam told us that Matt Hughes would be replacing Adam but Matt was also laid off. So another way is I don't know who from two years is going to be joining this meeting if anyone.
Right. Unfortunate. i That's a lot of complicated patterns panorama. But I suppose they still need to operate and where it ends up doing the operation which techniques invite him or invite her to his meeting. Special. So if it turns out that it's someone new that maybe doesn't have that much experience with the stock. I think they might find this cool product value.
Yeah, I don't think there's any new people. I think they just laid off a bunch of people but it will probably be someone who's worked there a while. That's just doing more stuff.
I would definitely hope so. Because they have a lot of Excel
we can do our original date, and then we can raise the topics
genuinely like doing from our side
nothing big happening from from our site. We were discussing Cavitt with Mike says about the future of the harmony project. A couple of interesting points. We're going to put in the in the show we created for the purpose Yeah, that's it.
Maybe I'd like to mention two more things happening. One is that probably because of some meetings military moment into contact but my sister and I were contacted by a university team. Team from to you and they wanted to make sure that the implementation that they're doing on code JL was working and tried to see the open at the community implementation that's maintained by us and see how those two match and apparently it wasn't working at all and they were worried that this was not peaceful. But in that conversation. Maybe Moises pointed to something or, or just by the conversation alone team found out what was what was failing, and now they have something at least a proof of concept of the rain, about using co J in a similar manner. And what we're really seeing, I think one of the one of the key differences is that they didn't want to use a flask service, but instead how janissaries which is completely unreasonable. And if there was interest, we couldn't migrate, it was just easier to put up a flask service because flask is like super slim. So we're not also like hardwired to do some testing. It's just what we've been maintaining so far, and it'll work with almost knowing stripe on our site. Why why change it if it's not broken next year. But anyways, this worked. So depending on what goes on into you, I will expect that they also start considering moving the LMS CMS into Kubernetes sooner or later because so
because that was a big blocker for them, right was code Yeah,
that was that was the last blocker so that's important, and from internally at Nginx. Also incubation that we have a customer that was keen on starting working on aspects even though it wasn't final, but the progress was enough that they decided to jump in the pool and try it. So we've been trying that and I don't know the specifics of it. And you need to know whatever it is, please let us know. But to my understanding this was installed and it was running. So this is the customer that's running and running out of beta and has aspects connected to I'm going to say an ollie based installation, because the changes are pretty open so they aren't much parabolic to winter latest releases. So as far as we know this went well which is encouraging us for the operations of the.
Venue let me continue maybe.
Yeah. I just I just wanted to ask regarding the coetail topic if you guys perform some low tech testing on on that tool since we knew that code Jane was using code JL was generating a couple of performance issues in terms of CPU and memory consumption.
I don't know because that conversation was interrupted by the rails. So I know that this was advancing and progressing until that point and then things said like I don't know what's going to happen and give me a few weeks to figure it out. That's the point we're okay. Got it
sorry, Max, you can go ahead.
Yeah, yeah. Yeah.
Okay, so regarding our work I've asked our DevOps team to start looking for some issues for some pull requests in the harmony project. Currently, the what I can see the majority server dwarfs tried to propose some enhancement for velero pull request, I can send it here for a second seeing Gabor and material had some discussion about maybe some concerns and let's look how it will go on. What else maybe regarding aspects, we discussed, we talked some time, some time ago about maybe this investigation part of aspects from from our site. Currently, we've integrated aspects in one of our client installation, testing will be gone on stage and hopefully in a week on production. Oh, right now, everything is great. And I will, I will monitor the performance may be some needs, how it operates, maybe some production concerns. Only one thing that is strange for me right now. I tried to combine in my mind, all the data flow as it currently implemented in aspects and I can see that in LMS, we have two plugins for X VI and for some sync events. And also we have a vector vector tools that can scrub all all logs from from pods in Kubernetes cluster and send it to somewhere. I just can't can't understand what it actually do. So I will look. I will look at the earliest date, maybe I will look a bit closer to this architecture. But right now I can see that everything is scrubbed from std out of all containers. They are interested in in but it is not saved anywhere and not used anywhere. So I I think it's just some some point in the middle when it was legacy architecture, maybe some kind of new architecture and the aspects team maybe changed in Samsung. So it just my my observations and maybe a misunderstanding. So yeah, if anybody will have also some experience in terminal experience, how to configure it. What about bottlenecks in performance we can have in our Coburn Kubernetes cluster? Yeah, let's share our thoughts. So that's all about aspects. So some work in progress. And also, I have maybe question or request for help a bit. But maybe a little let's move to eat a bit later.
You also mentioned that you're investigating, building MFPs with a button or something like that. Oh yeah.
I'm not sure it's not related to harmony, probably. But to the tutor. Yeah. Let me repeat myself. We are trying to speed up as MFE builds for anything for single installation for Cooper installation doesn't matter and we to get around to to do it in some clever way to do some base built for MSPs and then to to build some kind of some type of maybe overrides. So some sense that can be added on top on specific projects. As an MF your rights and right now, what I can see it can be done in some some like 10 or maybe 20 minutes with bomb Yeah, so I'm experimenting with PMPM and with bomb Yeah, we will see what will be in final final release of our internal sales sales in the process. And I will share it
that is PNP N and 171
B and D U N.
Okay. Thanks.
Well, interesting enough, we have a so the way we are organizing ourselves for the last few months is that there is a team that normally comes here and there by Johnny and he says that is looking at the operation of after after images are built, how to operate that in production. And the other one is looking at how to package the platform into the images. And I know that that team has suffered quite a lot because of the slowness to wield NFS. So I will point them in this direction and closely tell them to contact you if they have something to add in that in that conversation. Yeah, so
I don't know if you guys remember but I mentioned on a previous call that for another project, I did this thing where I was tired of having to configure the MFE and then build it and then configure another one and then build the same MFE so what I did was I just put unique random values in for each configuration setting in The MFE then built it with Webpack and produced an image. And on startup it reads the variables from the environment and then just uses like said to go through the compiled JavaScript and replace all those unique strings with the configuration variables at the time that container boots up. So that lets you have a transpiled minified built MFE with one container image but you can reconfigure it at runtime and it starts up instantly or there's like a very slight delay while that runs said on this like 10 megabyte file or something but it worked really well. So
Moises can correct me but I think we took that idea for the path of the CDN and we use that in this plugin that we use to extend the MFIs to you see ants.
Yeah, yeah, I remember that. It was a few minutes ago were bleeding commented that and I tried. I tried to change the polling time via an environment variable, and it actually worked flawlessly. And it's quite quick I tested the for the I run that setting in all the compiled JS files for the for the chairman fee and it does in one second and it would go and it's only had container startup time. So it was actually quite good for that. I'm not an expert in aspects. I only have to wait. One of our team may study the one that is heavily involved in the project. But I remember that the the vector wasn't actually it really needed thing for that part for the transformation was was 12 that actually made a few transformation for forex API and eventually, it a it put a put the wensing click house that was the so the Germans really needs vector, like the vector was like, how vestige from the initial kind implementation but seems like love can do or what better vector was supposed to.
Yeah, you're quite quite right. I understand it's required. I should debarked a lot of its internals and we actually change it to be just configuration normal for vector but I just can't understand I don't understand. What's the output because I can see this click house final stage in vector tomo and click house X CPI. Sinco how to say it. But output of click house X IP X API is zero. And the target table where current version of vector in tutor contrib prospects plugin tried trying to save everything in click house is absent in database. So they actually grab transform and trying to push it to non existent table and the second part, the IG CPI part is not pushing anything. I can see it in vector top status table and we can see it in real time but what actually happens, so I can see that it turns transform a lot of logs, but it's not saving it actually not saving it to click house and I can't understand why maybe I missed something maybe our tutor country aspects plan is not final yet. But actually all x epi events, right now comes from comes from event wrote in backend plugins that we have to install in LMS and it actually works but not we're not via vector transformation and say, Oh, it's really bothering me. A vi vi I actually disabled vector I actually shut it down vector and nothing change it and they can have updated statistics and data and dashboards. So it's interesting. Maybe I missed Samsung and some subtle some some Hidin dashboards actually presented. I just didn't find it
okay, I will say it's always an issue in the plugin but as far as I know, it is still in development. So yeah.
Yes, it is that the pilot Mozilla's not be changing much at this point. And right now most of the most of the focus is on how we transformed our unceasing click house. The reports that we can get out of this data. So we said if you if you can please relay that message to PCM. And perhaps we'll invite Chris into the next meeting. Or if Max you have some comments that you can give in an issue in harmony, we can ask Christian to review those and see
an issue in aspects repository. It can be in the repository
or if you want in the harmony project is to say I'm trying to make this work. We'll try to pull some some focus from the song okay. Like having the greatest. I am foreseeing that at least some of that knowledge of how to run aspects sort of permeates the operators and OpenGraph that trucking gang at me and whatnot. We have like a very big spot and I'm happy that the guy that knows this the most in our team, but I would be equally worried if it was in someone else's team. Like if he decided to leave the company, then the loss would be tremendous and to mitigate that by hiring more people that's appointed with the system in indicato. Now in any case, the the focus of one of the focus from Redwood or in the redwood trees, it's going to be aspects I will need to iron out some of the details before these goes into like marketing mode. Because if the people in this call control aspects though, we're in the WorldCat. So
that's running fine.
But until it isn't like recently with NFS, I've had this situation where I'm supposed to be like one of the top open edX experts. And if the MFE isn't running, I just sit there crossing my arms and I say I have no idea how to fix this. So a new technology in this time definitely requires some new information.
Yeah. It's got a lot of that it's off the shelf software, though. a huge change from the insights
Yeah, yeah. And so far, I mean, to the extent that I've been involved in it, it is good so far. So when I looked at what's not working, how I can make Why is this not doing what I wanted? As long as data is getting through all this time into click house and being super safe, then I can take myself out of the hole without calling question is mostly the first parts where we transfer that identically cars that are consumed.
Braden delectable. Sure.
I'm not exactly sure what Gábor has been working on this sprint but basically we're continuing our move towards moving our internal posting logic out of our growth project and into harmony and replacing chunks of it with harmony. So you saw some of the pull requests saw it that he has opened up and other than that, I think it's been a quiet sprint for us. Not sure of any interesting stories about our hostings
Well, I've noticed it's working then. I suppose not. If it stops working on if you find a problem, something that that means a wants go then definitely is.
Anything else on your end?
Um No, I
don't think so. I'd like to raise something that came from the from the Grotto of of axion from yesterday. There was different reasons to mention Elasticsearch and alien search, because some of the functionalities that are coming along for tribal require that it searches working correctly. And one of the things that was proposed is to deploy that will let someone install Open, open searches, the fork of Elasticsearch and sort of taking a stand on what the open X project will do independently what do you have been doing? So you indeed stay in OpenSearch, but not everyone else. And there is a push to make Elasticsearch or the search mechanism again, some what official report that I wanted to raise that here and say, Let's keep an eye out on what that will be. Because if it isn't a global game, we should try to match that up with the work that we've already done in harmony so that we are not maintaining more than one standard if to use reducing the amount of support that they will give to the software. We live community will be stretched thin. And that is even if we don't do like suboptimal stuff like maintaining two times the same thing. So yeah,
for harmony, it would be nice if we can just pick one and have it you know, if there's one that everyone is happy with, then we don't have to maintain two separate things. As far as the code goes, I think the search functionality that my team is going to be working on and stuff, it should work fine with either Elastic Search or open search. So I think it's more of an operations decision. Maybe in the future that will change and there will be at least features that one has that the other doesn't or something but for just basic building an index and doing a filter and search keyword search experience for users. The API is pretty similar. I think it's the same
which is but I think we use a search for your cases. I don't think that we have ever been exploiting were specific, super secret features of one of the providers.
From ti told me that there are several different ways that edX searches used in each of them instead of incompatible let's see how their end users slightly different approaches. We'll have to be a bit careful.
All right, that's good to know. That
I don't know if using copper open searching a single installation in a cluster. Using open searcher to do a plugin is easier than going to the harmonic project and InStyle hidden charges to use the OpenSearch it's inside edition if I use OpenSearch as a plugin, which forces me to have different OpenSearch for different installations or use our global OpenSearch which makes sense for multi tenant and multi tenant clusters. But for for a single installation. Maybe open searches or Twitter blogging is easier so I think that both approach over two different possibilities.
What I didn't quite understand was how much can we for example with a pandemic search do the community sites with elastic search and access? You'll have to use open search because licensing issues. So you will have to have a code separate that is maintained by the community for the search capabilities.
Oh,
sorry, why is that?
So FedEx Do you have the limitation that they have to use? OpenSearch because licensing issues right.
on AWS.
I think the change on the license for Elasticsearch from 7.1 onwards, I remember Jeremy explained that a little bit ago, so they're forced to use OpenSearch. So we have to make the decision if we we use OpenSearch or if we use Elasticsearch um, they use some so maybe as only available for the concert you think we have to make out on our own ledger for searching like for COVID etc or something like that.
I think I'm hoping that will be necessary but yeah.
Is anyone interested in investigating this and doing some research for us about what what the difference is I I might be able to take it on in the future since I'm doing some work on search for the tagging project.
I was going to say that we could be you I won't be offended anyone to call yesterday from the roadmapping investigating that and deciding what are the projects going I suppose I might be paraphrasing and giving him more responsibility that he actually took on but but from an operator standpoint, I would say whatever the project chooses, we should put that in terms of support for any HMI project. If someone has a specific requirements and that don't necessarily which is not, I mean, at least from from preparing perspectives. We will move everything behind and that's successful and Saracen and David once and not continue maintaining something that's different.
Isn't recycled. So I will say that the solution for family will be like, what does the community support? So at the moment, it seems like Elasticsearch is the default. From some time we will have to to keep using an exception the particular API that they use for open search. We make this work then harmony swaps to open search I will say
that makes sense to me.
What do you mean by souped up switch? So where they currently have support for Elastic Search and open search in harmony? And you're implying that we will have some some default? Some default search capabilities? Yeah.
Right. We don't have kind of the factory follies Elastic Search because most most people, I guess, deploy Elasticsearch, I will say, like, from the the old playbooks in in configuration, we are all used to to Elasticsearch so the community has this de facto the full of Elasticsearch as as a search layer. So if we eventually all start using our concerts, so it will make sense to to make all our tests or or the fall for harmony to be a concert.
Yeah, okay. I got
right. So we do have a follow up step of investigating this further. And journey analysis, I'll invite you to take a look at this. We need to filter through priorities of what what's coming up for the next few sprints, but definitely we need to have an opinion a stronger opinion on these if that is to push for quicker switch and the community or more maintenance world. And we can discuss that later. And just since I put the link to the MFE stations to plug in for a CDN, if I think we just publish it because because why not but if there is interest from any of you guys, and you see value in that and would like to, I don't know collaborate or see the take it for a spin and see limitations. I should I say that. We started this with like a bigger appetite that we could actually manage. And we thought like we're going to support these for all the different releases that we have out there. And then during the work, we actually have to scale it down and focus on not being on early because that's where customers were at out of time. But we are moving into Queens real soon in the next few sprints to keep it up to date. So if it's outdated, just give us a sprint or two and it should be less
anything else? Anyone? Yes?
Yeah, from my side if you have, if you have we have time, I want to ask maybe have some knowledge about implementing Redis cluster in our installation. Maybe it was related to harmony or maybe some large installation. So we are in our Kubernetes AWS installation, we are trying to combine AWS elastic cash or radius as a cluster. So in cluster mode, not a single node but sharded multi node cluster, and it doesn't work. I'm not sure. I am sure it doesn't work and it should not work. But I want to ask if anybody had maybe some experience is it because for Django and for Open edX, to use ranges as a cluster we have to change the change configuration to change the client class. So it simply doesn't work. Ssh. In our tutor is in our tutor. Any in any tutor configuration if we switch self hosted register to AWS cluster in Gloucester mode, I can post it I couldn't post it. A simple not simple but actual error that I I've got actually this big ground oh sorry. Sorry. And didn't didn't cope it Oh
yeah.
In couple ports, we are trying to use it in this mode and the default registered client can't use it. Because they just chickens at different different houses, different items can be a hazard for different knots and failed to even start with application. So just wondering, maybe, yeah, you tried it. Maybe in some projects and no, maybe open edX, as huge Oh edx.org Sorry, as a huge AWS customer. For this huge info installation tried to use ranges as a cluster, not as a single not installation. And
I was going to inform which is actually to say something but recently had we wanted to turn on Ricci for some customers, and they ended up using or we ended up finding some issues with arrays because of the way that he manages those. And I remember later on I can't remember exactly which is the final solution and I was looking at the code but I couldn't find it quickly enough. Remember that we experimented with Ray, Sentinel and junk arrays as ways to us ways to manage it from my room from from my memory I think I couldn't make tricentennial work. But just break Django res. I can configure it and it ended up working. But it might be so that the way the issue as the tricentennial wanted me to specify the nodes of the multi cluster and for that I only have one so there's gonna be an easier way if you're facing issues with the cluster and which switch to yellow to reset it and then it will be much easier for you to configure the cluster since you're right happened.
Oh, yeah, okay. Thank you. I will look at it.
We know that we haven't really we have use elastic actually for but we inserted up in in cluster amounts. When we never faced that issue is based on a few different issues. Visually we the case that I'm not directing so then the memory fills up. But besides that, I think we never we never actually use it in in in in cluster mode. With with the shards. I think I don't remember what I think our trial ended off but I think they they pull back the changes. I think that was a while ago. So they're not really that much experience with cluster.
Yeah, okay. It's also available.
I can see we had Helm chart deployed Redis cluster and girls at one point like you just said that it seems like it's not there anymore. So I'm not sure what we're doing for that now.
Okay, okay. Is it interesting for us as Garmin new project to move? Maybe in some far, far future and in this direction, if it will share? Maybe already? Yeah, I think courageous for our future installation for different tenants and they will have to come up with this comes some like, first of all, maybe a coupon offers a solutions or maybe the Sentinel but maybe it's real Redis cluster mode, and now
Yang will save you time that actually really fits our interests as always something that we will be looking
okay, okay. So this will be available if we will come up with some real production usage cluster
or projects
All right. Any final points? If not, we will get like 10 minutes back to this.
From what I can tell what we're doing right now is just letting tutor deploy a separate Redis for each instance instead of having a big shared cluster. I'll follow up with gobbler and see if he knows anything about Redis clusters. It seems like we tried them in two different ways and then switch to a separate Redis instance.
Will Okay. Would you mind also, asking can be faced with an issue with the MetLife cluster failing up after a Why sure. Yeah. Thank you.
Now when I can remember and also can remember when tres decided to talk all the memory across AWS zones, and it charges like crazy for data transfer.
Like but that was that lesson with the open data
wasn't even race yet but everything race already has like a cost tag associated in my head. Because that one was quite expensive. Just for ladies to send the entire crossings
right and it is where the the permitting Syrian coming
