Hi, I'm Chelsea. I'm a newish Product Manager at Axim. And at the moment I'm currently working most closely with Brian Messick on defining those product requirements for the aspects release. And I will pass it to Peter.
Let me try again. I'm Peter pinch. I'm with MIT open learning. I've been on BTR since before there was a BTR. And I'm here mostly because I got invited, which I think was because I used to write the release notes. In the last release. One of my colleagues wrote it who I thought was going to be here but he is not so it's a good thing. I am I'm so I'm gonna help out with the communication around the redwood release. And then also, you know, we are very interested in what's in redwoods. So that's also a reason why I'm here. Oh, I guess I have to pick somebody
or hey went, Oh, it must have been late at night at Milligan hasn't Katie.
Good morning. I am the product manager, Western Governors University for Open edX. And I have never been a part of the community releases. So I'm interested to learn more about this, but also wanted to learn what was included in the red book release. And I have no idea who hasn't gone.
Lacey hasn't. I'm here. My
name is Lacey Nesbitt. I also work with Katie Milligan. I'm with Western Governors University. I'm a senior system admin for our ODX platform. And I'm here for a couple of reasons. One, I'm really interested in building a stronger relationship with the edX community and learning more about how we can you know, establish feedback and really just kind of strengthen our relationship with Debbie Gu. I'm also shadowing Katie a bit so she invited me here I think as kind of the lens of a product manager. So I'm hoping just to learn about redwood and and yeah, learn more about it.
Max of yoga.
Yeah. Yeah. Hello, everyone. My name is Max. I'm a CTO at record again, and I'm contributing to BGR as release manager for Tom coincident, the next release or read or at least, yeah, and let's go to net.
Hi, I'm Ned I work at to you on open source logistics. I've been a part of BTR maybe since before Peter, maybe after I'm not sure. A long time but I'm here mostly because I'm interested in the dynamics of the community and how we are going to move forward on all sorts of things, including redwood. And I think that's it unless we're losing the autopilots though the AI the AI market.
Okay, if I record this meeting, as well
alright, so for the sake of the recording, this is the first to Redwood planning and I'm going to share my screen here.
Right, can everybody see the agenda? Yes. Cool. So there's, I think, I mean, from my perspective, from a product perspective, I have a couple of goals for this meeting. And I'm wondering if this may actually become like a kind of a reoccurring standing meeting. I think there's been a lot of conversation across the community about how we can centralize release planning moving forward, because there's a lot of disparate groups who kind of work on the releases from different perspectives. Right so product is working on it from the perspective of which features do we include BTR is working on it from the perspective of how do we test and how do we get the release builds? There's a lot of conversations going on on the MFT MFE side that intersect with product about which MFPs are included which features attached and these are included? And then I think there's a whole nother realm that we haven't even explored yet, which is like centralized product marketing around the releases, like how do we give a voice and a narrative that describes the value and the impact of the releases to folks who would be interested in upgrading or interested in adopting the platform? And so, I think, I think as like, as a community, we're kind of at the right place now where we can bring all of these different stakeholders together, and really come up with some sort of like organizational infrastructure where we can do release planning in a way that is centralized and that is organized all of these different stakeholders together. And so my goal for this particular meet this first meeting really is first I wanted to do an exercise as a group, because I think it'd be helpful for us to understand what the pain points are in the community rooms process from everybody's different perspectives. And if this exercise, does anything, I'm hoping that'll give us at least a sink like a singular space where we can compile all of the pain points, so that we can start talking about how we overcome those pain points and put together an organizational infrastructure that works for redwood. Um, the second thing which is tied to what came up in a lot of folks introductions is I know there's a lot of interest in like what's included in the releases in particular in Redwood. So I've got a kind of like a work in progress product Gantt chart that I'm hoping can become the centralized document that we use for planning for releases that will communicate what features are included in the release the timelines, for those features, so that BTR has a better sense of what's up and coming when things are going to be ready for testing. And then there's a lot of open questions that I have around like, what are the needs what is BTR? What is The MFE Working Group need from product in terms of release notes in terms of testing instructions. In terms of like, you know, technical configuration documentation, so, you know, can we come to a place where we can meet those needs and come up with a standard workflow that works for everybody so that everybody is working out the same standards. So questions for later in the meeting. Um, does anybody else have anything that you would like to add to the agenda or anything else you'd like to see covered? Cool. And then I'm at the end of the meeting, I think, the question is, like, do we meet regularly is this is this a space where this can become like a regular touch? point as we plan for releases, not just Redwood but thinking further out to sumac to teak and so forth. But in terms of kind of a little icebreaker exercise, I'd like to do what I what I've seen called a pain storming exercise, which I really like that term pain storming. Instead of brainstorming. So I've created this jam board in Google. Is everybody able to access this board? Through the link? I'll drop it in the chat as well.
I'm not for example,
probably Yeah, neither private net allowed to see it. Yeah.
Let me see if I can share permissions here.
Thank you.
All right. Try now.
Yeah, we're in. Awesome.
So this is pretty straightforward. We're at let's take five minutes of just kind of quiet heads downtime. Um, you can see here on the side, there's the ability to add a sticky note. So if you click on little sticky note, you can type in the sticky note you can make it whatever color you want. One, everybody take a moment to write like one pain point per sticky notes per sticky note. And what are the biggest pain points that you yourself have experienced around the Open edX community releases? And in the sticky note, just include your name and your role so that we can start to get a sense of which pain points are coming from different different groups of stakeholders across the community. So I'm going to set a timer for five minutes. Let's take five minutes to do this and then we'll come back and do a little bit of a like categorization and grouping exercise. With the pain points.
So Janna just said at least one read you're gonna have more than one. Oh,
yeah, please add as many as you can, in five minutes. As many as you'd like. But just one one per sticky note.
take maybe two more minutes.
Right? Now let's take a minute or two and read through some of the other sticky notes on here. And I'd like to see if we can come up with some general themes are kind of categories that these fall into. So for example, I'm starting to group a couple of these up here that seems to be more product related, right? There's themes around product documentation, release notes, not knowing what's included in releases from a product perspective. I see a number of themes around testing requirements. So let's just take two minutes individually read through these and if you see commonalities, start to group them. Together.
Okay, so let's let's kind of talk through some of these categories and see if we can maybe synthesize some common trends here. So it looks like there's kind of a emerging bucket up here on the upper left hand corner. About having a hard time knowing what new features are included in the release. Release Notes aren't always required. has been a definite problem in the past.
Product impact of release is not clear. Engineering asking what should be in a release. So if we sort of synthesize these into like product direction, lack of products, like intuition.
Your decision making and
lack of value communicated maybe.
Is this for the top left group? Yeah.
Group kind of up here.
I think perhaps another theme among those is that it is ambiguous what it even means for a feature to be in the release. Like does it mean it's intuitive, does it mean it's an intuitive plugin? It's murky conversation on VTR.
It doesn't mean it's enabled by default.
And then if we kind of move down into this lower left hand corner so centralized place to look at backporting. Bug triaging. So these are sort of issues around bug fixes and backporting is that the right way? To synthesize this?
Many how many of those posts you're trying to summarize.
It looks like there's a group here around. bug fixing and bug triaging, right.
Yeah.
So maybe lack of direction and coordination around saying, triaging which sort of ties to the, if there's no clear product direction, or there's not going to be a clear direction around bug fixing either. That makes sense.
And if you can't decide if a feature is in the release, then how do you know whether it fix it? Right.
Yeah, there's a definite connection there. Right? We need if there's no clear set of your sense of which features are included in a release, much less clear product documentation about the what those features are, how can you subsequently prioritize and triage bugs that makes sense
all right, there's a there's a set over here around tutor comm actually want to talk a little bit through these two here, these two blue ones?
I think mine the upper one is covered by what constitutes release? Yeah, although comes next has one that's more around. Think insurance claims get upgraded and handling and I don't know if you want to say anything else on that Max.
Yeah, it's basically about the timeline of testing. So just to not waste of time, when we have release or when the preparation for release and we are waiting for plugins, maybe it's just to calculate plan to test and maybe shift a bit.
And yeah, so maybe I'll move that down here. It seems like this little look down here in the lower right hand corner is more more focused on testing issues.
Yeah, it's about testing yeah. Can be contributing, contributing to testing
Yeah. So Adolfo has a card around meeting retesting. What do you mean by that? adulto retesting.
Kind of what another card there mentioned. Possible regressions after feature backporting but not just feature backward and simply the act of fixing a bug that was identified in the initial test phase. And you fix it and you you merge it and you backwards to fix there is no scheduled retesting or for that but so when I say retesting, I mean, like coming up with a complete testing strategy or, or release right which we have a part of it, but we don't have the whole picture. Figured out.
So like lack of an end to end testing strategy. Yeah. Which would encompass bug fixing country hashing.
Um, does that sort of fit loosely the rest of these testing cards here as well? Lack of involvement in state different stakeholders in the testing process? Regression issues
testing timelines kind of falls into that as well.
What about these ones up here in the upper right hand corner? Hard to get the attention developers who don't use the named releases. That is that from the perspective of again, testing or
it could be for any of these questions. I know, hey, there's a new FM MFP. What does it even do? Or problem
as well, right? Like this is exactly why do we have this feature? Right?
Or why? We have a bug, can you help us fix it or there's a back part you're going to merge? It like at all levels? Yeah.
features not developed with releases in minds. That's also a bit more generalized. Can you expand a little bit on that, Adolfo, what do you mean? Yeah,
so show I'm I did the initial grouping. On the top right there. And I see this as a lack of synchronization between the developers and the people responsible for the release in general, right. So and that is, due to this lack of, of communication, I guess. You end up with several different problems and details such as edx.org specific strings. In the end, the MFPs are in the codebase right, but this is all this the same problem, which is the developers are not developing the features thinking of them being released with Open edX, right.
I see Yeah. I see what you mean.
And it can also come in it can also show up in timeline issues. Like getting a release today. I was about to do the second half of the feature next week or something. Yeah. Or we're going to drop a database column. Well, you got to remember that in the name release world, that's not going to happen for six months. So you have to you know, yeah, that's a big one. That hasn't been done us in a couple of releases. But yeah, I think I think we've gotten better about that. But that's simple of the kind of thing that can happen. Yeah, but I remember a couple of releases being a bit complicated because of that. Yes.
And like from like, sort of building on that and from up sort of a product perspective, you know, like, hey, phase one of this was kind of buggy, but we're gonna move forward with phase two. But if we were actually prioritizing for the release, we'd probably actually make that feature Phase One less buggy than moving forward. Phase two. I'm kind of thinking about like the way that some to you features have been incrementally developed. Like they're just, they're not thinking from a product perspective about delivering a sort of a finished partial product in a given named release. It's just not in their timelines. Right.
If you're, if you can push every day you do it differently than if you have to think about it's going to appear in five months. And those two ways can clash. I also just sort of Cluj then the last note there. And it's also related to it's more than just single pictures, like large initiatives as well. When the whole moved to a piece started about now, four years ago, right? It was always meant to be an incremental thing. But if so, multiple releases, right? But if you don't have a target release in mind, you can have this go on forever and never ends. Right. So it's just one other thing that if you have if you're developing with releases in mind like just looking at generous community release plan, you start to think oh, wait. So if we do this for this release for the next one. We'll have time to finish I don't know The MFE migration, right which is actually some something that we're we're thinking about doing. But it's all what I'm trying to say is, if you think about development, with releases in mind, many of the problems we've seen so far sort of disappear, right? Or at least it's, it's, they're left troublesome.
And I would extend that to to include, like product development, as well as technical development and implementation and yeah.
So
finding what this means
and documenting it
and then we've got one one other category down here, in the lower right hand corner. PRs don't get enough information, not enough documentations documented documentation engineers to understands potential breaking changes. So this looks more like an implementation or technical implementation documentation issue. Does that sound right? Right we're clear
documentation configuration
All right, does that does that seem to encompass everything so we've got a category over here for sort of products, product direction, product documentation, product decision making. Adjacent to that is even defining what it means to be included in a release. So again, decision making documentation I think this one is probably second to that end to end testing strategy that encompasses bug fixing bug triaging and then there's this bigger category of kind of developing from the outset with releases in mind and then technical documentation, configuration documentation. Any other outliers folksy or
there's the categories and metal about build time and like size of the platform and I would move mine over here, because docks wouldn't really fix it. Okay. I think this this middle thing is not something that this group of people would fix. I think on an engineering level, we just need, like the platform is very complex. And we can do all these things to work hard to release it and manage that complexity. But there is an underlying complexity that could be tackled, that could make everything easier. Yep.
All right. What else any other outliers folksy or any other things that don't quite fit into this area, these categories?
I just want to say one thing, but like this, it looks like we're doing a bad job with releases. I just think we're not. We this is about improving, right. But like for instance, testing and bug triaging Maria did an excellent job. This time around. It just didn't make that clear. For example, it doesn't mean that, though, that we can't improve or make Mario's job easier, for example, or for the next release.
100% And I think one of the exciting things about Redwood is that for the first time when I'm looking maybe this is a good segue into the planning document which will kind of address I think this at least this upper left hand corner as a starting point. The Redwood release is I think the first maybe the first release where like 80% of the features are actually coming from various different stakeholders across the community rather than a single organization, which is really significant and I think worth calling out because it means we have the opportunity to get ahead of those releases or get ahead of those features and actually like to address the one of the problems in the upper right hand corner. We can synchronize development of those features so that they do align with the release, which is why I wanted to have this first planning meeting so early so that we can start thinking ahead to defining that feature set and what's included in Redwood and what that means for coordinating testing and coordinating bug fixes and, you know, coming up with some sort of end to end testing strategy. So yeah, to total to kind of echo Adolfo, this is meant to improve on and build on a workflow that is already pretty solid to begin with. And Redwood is kind of I see it as like a turning point because of the nature of the feature set. That's going to be included. We have insight into it from the ground up, which is really significant.
Um, let's just take one more minute. I'm curious, did you like a voting exercise? And it was going to be kind of skewed because the folks are going to be interested in fixing problems that you know, are going to be directly tied to your own your own workflows. But if you had to choose one category, so like these, there's 123456 Pink categories, one category to incrementally improve on for Redwood, which one would you choose? Everybody gets one vote and choose one of these pink, one of these pink categories. And actually, the question is how do we vote
everybody use this little pen tool? Maybe just
I think we can just click like boosted stick note and then we'll have our pen on it.
Okay, do that?
Can you see when multiple people have selected the same thing?
I don't think I can.
Oh yeah, this works for us. So I can see Maxime is grading like a like a poster with names so that could work. Well.
Let's do this. Let's copy it. Let's copy max. So everybody copies, copy Max's put your name on. It and then vote
final call for voting going once going twice.
I think Kyle may have just done a tiebreaker between the top two
we still saving as well.
Or did you
really decide? Yeah.
Well, it's close regardless, which is good. It sounds like we're all kind of aligned. So if we have to make if we had to make like choose one incremental improvement to make for Redwood it would be around maybe product decision making product documentation product division, followed closely by frameworks for developing features out the gate with releases in mind. And I think those two things are very closely tied together anyway. If we can get ahead of developing product features. Before they're well if we can get ahead of defining product features before they're developed, then we can take developing on for the releases in mind in terms of actually defining the specs for them and how they're how they're developed out the gate so awesome. Okay, thanks everybody for doing this this exercise. I think this has been I hope you found this helpful. I found this actually quite helpful. And it's something I'd like to revisit maybe with each release, like we can start to do release planning and retros and then look at how we can make incremental improvements for sumac and for teak that that follows it. Any other comments? Questions?
Jenna, you shared a link in the chat earlier that looked like it was a spreadsheet of expected features. Yep,
that's that's actually a good a good segue because in thinking about like, how, what what kind of concrete steps can we take in this kind of product definition space, or how can we make the releases more product driven? That's the second thing I wanted to touch on with this agenda is to introduce the spreadsheets. Let me reshare my screen because I'm only sharing one
and here's a link as well.
So I think I hope the spreadsheet actually ties into a lot of the post it notes in that product product category. So a couple of a kind of walk everybody through this spreadsheet and one of the things I'd like to do from this group is get a sense of whether this is helpful. How can this be more helpful? Is this something that we can use start using as one centralizing centralizing planning document that can bring together product and The MFE group and the BTR group and anybody who's doing release notes? So the way this document is structured is so kind of tied tied back to the one of the sticky notes was around like, releases don't even have a clear product vision, right or a clear product narrative to kind of tie the release together. There's not no curation involved in the features. That's something that I'd like to start changing and that's releases need to have a clear, unique value proposition. We need to have decisions made around which investment areas help us fortify our strengths and which ones help fill parody gaps. And so this ice is our sort of platform strategy and then we have features that will support each of those different strategies. So there is an overarching strategic alignment in which features we decide go go into each release. I've got the kind of meat of this document is by initiative and by Epic, and the the idea is that I would like to have a clear sense by the end of January, right? So like five months out from when we release right would have which exact initiatives and which epics from those initiatives are going to be included in the release? And why right like what's the strategic alignment for including that particular set of epics in the release? And so to give an example, because I think that's maybe the best way to illustrates so tagging and taxonomies is one of the big new initiatives that open craft and XM have been working on and we've gotten tons of input from Western Governors University, among other other users. So tagging taxonomies is part of the unique value proposition its support for competency based education, which is one of the directions that we'd like to evolve the platform. It's a major initiative. There's many epics that we've got currently in refinements all the way to, you know, just to kind of barely have product specs for if we filter for simply redwood. I believe that we can get these epics into Redwood for tagging and taxonomies and the way I've got the structure it is kind of thinking through like a Gantt chart approach. So these are just generic sprints. Everybody works on different sprint cycles, but I figured dividing the month into two would be an easy way to do it. And so looking like each each sprint quote unquote, whereas any given epic in terms of its status, and in terms of the status definitions, I've got it divided into product definition. So with product definition, the specs are being defined and UX is being designed. Technical discovery approaches are being tested and refined implementation and refinement. Means work is in development and undergoing user testing. And then ready for community testing basically means it's ready to be built into the next community release and it's ready to undergo testing. I think there's a lot of questions that we have that are like really hazy around this handoff, right like what what needs to be included? For a feature to be ready for community testing, put to what testing instructions need to be included? What product release notes need to be included, like we can define that together as a group. But there's got to be some sort of handoff and the idea is that anybody should be able to look at any given epic, understand where it is in the Status cycle. And at some point, it'll be ready for community testing. And the idea is that we want to get ahead of the release, right? So community testing isn't all just being done like at the code cut off on April 15. But rather, maybe we can start to do more iterative iterative testing, or at least make sure that everything is set up in advance so that when testing starts, everything is ready to go. So that Peters not digging through PRs to understand like, which features are going to be included. And we're not getting surprised features two days before the release is cut of a feature that we have no idea what it means. So that's that's kind of an overview of what I'm hoping this document can provide. I'm curious for folks feedback, like is this? Does this seem like it would be helpful does this seem like it's the right way forward? What would be more helpful, you know, happy to take feedback now or we can do it asynchronously as well. This is very much a working a working documents. It's very much draft form and I'd like to iterate it to the point where it's helpful for everybody in this group. Go ahead and release.
Yeah, I think this is a great, great document. I'm really happy to see this. And I'm wondering if there's if this documents tracks all projects currently underway at to you.
So this is this is part of the overall product decision making. It includes features that I know about based on my communication with product managers that to you, but it's going to we're going to the dynamic here and say that it's actually the product working group that's going to make decisions about which features are included in the release. In other words, we don't know a feature is being developed to you, we don't have the documentation for it, then it's we can't consider it for the release because it's got to go through the same product review process that every other feature is going through from other people in the community. And so it does include some features that have been like kind of gone through that that product review process but there might be others and if there are we'll, we'll have to punch them to the next release or we'll have to figure out how to coordinate with the product managers, so that we get the right documentation. An example of that are the dama fee inversion projects. So to you has been driving this to a certain degree although X and that's becoming more involved. Record game has been obviously involved open craft has been involved. These are two you driven projects that I've I have great insight into and I have the right documentation for so and then they align with the strategic direction of the platform so they are included. But that's not to say that every single feature coming from too you would be included in the future.
Be clear, we are aware this is a huge change in how open edX is developed. From the very top and and this is me. Speaking from my own experience, this might not always work. But it's then pension. So it might be that a feature or others still filters through that is not listed. Here. But we don't want that to happen anymore. Right. So and this will require communication with all developers, right. So two, I mean, one of Ned's posted in the in the brainstorm session was about how do you, you know, basically, in essence was how do you communicate stuff to the developer, the themes that are developing? Right? So this is a communication challenge, as much as anything else. So Jen is doing all her all she can then of course, everybody that's developing stuff, but if you or your team are developing a feature, or want to develop a feature, please do talk to Jenna. First, right? I've I've seen a post posted on Slack today or yesterday, and on the front end working groups, about somebody doing just that. And the response was, oh, wait, we're already doing this. Right. So we also want to avoid rework, not just release coordination. We want to make sure we are optimizing community comm no resources for this stuff. I see many hands up. I think Katie was first.
Yeah, I was just wondering if this would be replacing the roadmap on GitHub, or is it meant to be supplemental
it's meant to be supplemental. It's it will reflect what's in GitHub, but the idea is, like, centralize the actual like a Gantt chart, which is not so easy to create and GitHub when we're managing like, there's like 16 Different GitHub boards that feed into the spreadsheet. It was just an easier way Okay, good.
Yeah, I just wanted to say that I did. I, I totally agree with a vote for we I mean, since since here in this meeting, there are several members of of the largest providers of Open edX and important members of the community. I was thinking after the meeting, I will run to my team and let them know we have to put everything we are doing. For for, for Open edX here in this list, but after that, I was wondering, is there a criteria to establish or to determine whether a specific contribution is a feature? So so?
Yeah, and so the product working group has developed a whole product review process, that nginx has actually been like Central and helping helping to put together and drive and that would actually be the first step and maybe that's maybe that's like an agenda topic for the next meeting is how the product review process sort of kicks off this whole process in terms of product decision making, what's included what's not included? Maybe that's a good segue into like a bigger topic for how we organize a workflow around what's included in releases.
I'm so aware of time we've got two minutes left. Does it does it seem like this group is is a worthwhile thing to continue meeting on a regular cadence, right? Like I'd like to see this group kind of evolve into an actual release planning touchpoint with everybody who's involved. Does it seem like a helpful use of time? What are folks thoughts in terms of moving forward with this and next steps?
Reduce
Yeah, I'm very much looking forward to having these meetings again. I think it would be very valuable to like, track the progress of individual features. And then so this would allow us to do the testing even earlier than the actual release. So we could test in, in what is called nightly interior and master and open index. That will be extremely valuable.
Yeah. And maybe that could be like one of the standing items on the agenda is just running through the list of anticipated features for the release and doing a status check to make sure everyone's on the same page about the status and where it is. I think a couple of the things I'd like to potentially do with this group is get a better understanding of what needs to happen in that handoff like what does BTR need, how can products create some of those standards and operations around documentation? So I think there's quite a bit that we could cover. If this time works for everybody, maybe we make this like a monthly standing meeting at this time.
If that works.
I'll take no distance as a yes. Awesome. Well, thank you everybody for your time. This has been really, really helpful and I hope that this is the right step in terms of coordinating around the release planning.
