Hey go handle where you click on one and expands information, quote from a paper and references that that talked about it which really helps consolidate and be more precise and kind of make these connections across papers and across terms that mean the same thing. And I think that in itself is useful beyond the context of this class. I think that elevates to at least a blog post, or something that kind of tries to say, listen, we're talking about these things, and it's kind of these are the things that people are talking about and these are the terms that I use. I think even that extends beyond that is just taking these connections. Seriously. So the dictionary is just kind of a flattened list of these things and definitions, right? And maybe readings that we can refer people to and that in itself is very useful, I think, things to share with with with others. But taking these arrows seriously and being thoughtful about it, I think adds another layer of conceptual conceptualizing the problem space, which makes it connect more directly to solutions, right, which is like if this is the structure that we believe of how things relate to one another, then it's very clear or becomes clearer when we look at a solution like preregistration, which nodes of the network its target targeting, and what kind of problems will it elevate even in theory, and how does that contribute to solving this kind of hairy mess? of a problem? And I think that would be something that definitely can elevate this project into something that's much more than a blog post, but perhaps a prospective piece in something like nature. Human behavior or something like that.
One thing I'm just thinking about what I just see this, I know it's a product map, but just this snapshot, right? It was like there's a lot of mentions of chance uncertainty, of measurement, reliability, and validity. I'm wondering if there's kind of like a formalism that might be useful to write for, for this, right. Like it's, there are a lot of different pathways here. But, for example, randomness seems to be kind of involved in a few different pathways. And so finding the formalism that kind of sets the stage for these different pathways might create use for it, and it doesn't have to be a strict formalism right? I think there's still value in like, symbolically describing a system even if it's kind of a bit vague, but it just helps people kind of there's a lot to absorb, but they have that starting formalism right, and then they can expand into this as they as they
started trying to do something to Eric's irreducible prediction error. Versus like, instead of convexity, reducible.
And I think another thing and here maybe Jason, you could, what you had in mind is is going to be helpful here, which is how to make this to be kind of fun, or useful framework and a presentation for the goals that we mentioned. And at the same time, maintaining its marketability for communication and, and things like that. So it's like and also if there are tools or frameworks or ideas on how we can make this thing commutative such that different people can contribute to it. Add to it so for example, I would have a suggestion for missing between problematic exhilarate ease that comes from high cause and density. So it's not only a measurement thing, but the fact that actually everything causes everything makes these auxiliary theories more problematic in the social sciences, then how would I be able to other like guidelines or rules that one can follow such that we can maintain the cleanliness and the order and at the same time, be able to add to it in a more distributed fashion. And maybe Jana has been thinking about this with another thing, for example, I think, will you expand the note and you'll see things like a reference, a court, maybe also another thing that we want to add is an example a concrete example of the manifestation of this problem, which I think would be extremely helpful. And writing this down. And talking about these things are for example, and we can just borrow the examples from the original paper or like multiple papers provide different examples and we can just like pick one to communicate that idea another thing is, do we need something like or is it going to be complicating things that we have like super nodes? So for example, to Hamlet's point there is a bunch of nodes in this network on this graph that are related to randomness. With many sometimes Divergent Paths, but sometimes common paths to just like an abstract problem for which these are special cases or instances of that more abstract problem. Is that a good idea? To have like I'm calling them super nodes, but like grouping nodes or maybe color coding and then at the same time not to be overly rigid, and then not be able to kind of move things to move things around. So I think there is, conceptually, this is this is a great progress so far. I think this is amazing. I think there are more substantive things to be done here but also more organizational logistics, grammar, things to be imposed. On. On top of this to make it easy to contribute to and communicate with us.
As part of the class next week, it's ng you're sure it's you? Yeah. Okay, yeah, we definitely can workshop this during class. We can
talk more about it next time, but
I'm worried that if there's too many competing goals for the project I know until I laid out some some of the First Order goals and then the kind of a second order goal of mapping that
the dependencies, dependencies
and interrelations, which to me it would be my first order goal.
So as long as it's clear, like, what the goals are
because it's very easy to get runaway complexity.
Can you like scroll to the right a bit for this illusion side
of things
and what is the integrity of experiments?
Yeah, just like that slide that that you have there, like I've been trying to as we read more of the solutions, papers, and a map that into there, so I'm happy to continue that conversation.
I put the link on the syllabus on the very last page where the project ideas are listed.
It's good to think about solutions. For the mapping exercise, I would try to keep that independent of the solution exercise for the mapping of the problem,
yeah, no, I agree that one could, I mean, how what, what could what I'm envisioning here is that if one could just like map that problem space, and I think that in itself has all of the benefits that we talked about, and it could be like, something like revisiting I think one of the papers where I went where the title was, like, what's wrong with psychology anyway? And kind of tried to list some of these things redundantly, and like, overlapping and maybe less precise than some other papers that some of these problems so it's like, it's like that. What's the problem? What's the problem with this slow progress in social and behavioral science? Well, these are the problems. And these are how they are conclusion and I think that's in itself is useful. I think one thing that would make it to add kind of more of a positive spin to it is to say that oh, there are current solutions and there are current reforms. But actually, once you contextualize them into this space into the problem space, it becomes slightly clearer on what aspects of the problem are they targeting and that will tell you how much which which again, I mean, contextualizing there will tell you that oh, what to expect in an ideal world for this problem for this solution, to actually have an impact on the progress of science. Is it treating the root cause or is it more than the symptoms are they there, and then the solutions have their own dimensions of like feasibility, maybe something is just treating the symptom but it's very easy to do and that's maybe worth it. And that's the separate kind of analysis to be done there. But I think putting them together is going to make for us stronger. Yes,
eventually. Yeah.
John, I had a great idea. So they let's first focus on the desirability of the solution then think about the feasibility. feasibility. Yeah. So like, it is meaningless. That is doable, but no one needs it right.
And then that word has to be like a solution that doesn't target any or that target something that's really upstream or downstream of the root causes. And it's just solving a very small manifestation of something much more complicated. And thinking about these two dimensions, I think is
Oh, I'm just focusing on the desirability I thought the three root causes which was pace, high causal density, and time ever changing rules. Temporal Validity, and measurement.
And I have to say, I mean, we've brought this up before and I don't want to we don't we should spend more time talking about the slide now, which is how to think about the problem of time. It's just I still see it as just a manifestation of the high costs and density. It's not a separate dimension. And what happy to have it to be to have a better grasp on the distinction that's
worth discussing and really, yeah, it sounds like
you Okay, good, who wants to go next?
So I agree that it will be important to maintain or degrade the complexity by like, choosing which registered or not, but what I was thinking is, of course, space and time could be massive, not supported involving people down we also mentioned humans white, and I was wondering that if this is specific to social and behavioral sciences, more like because we have limitation even we're doing physics, but maybe there's not much of a problem because of physics, this lower part also density. So if this is, if this limited cognitive capacity is something that becomes a problem with the interaction of microservices is something that could reduce maybe the the density of the web
Yeah, I I agree that actually and I'm not sure if it's not captured already here. But it's there is the interaction between the root causes can be problem, not their existence and isolation. As you said, kind of, I think the example of cognitive limitation, another one I added in the other graph, but maybe it was Did you share this link with with the rest of the class or at least with me?
I put it on the syllabus.
Okay. Because that was adding to the old one that we worked on, I think, but there is from the readings of this week, there is like there is confirmation bias. There is the desire for explanation, like the human intrinsic longing for having explained something's the subjective sense of having understood something. I think there are a bunch of, of things that fall under humans that are also root causes in their interaction with the high cause and density that leads to some of these symptoms. I showed a snapshot of what we did last week to Duncan Watts, who's teaching a PhD seminar this semester, on explanations, so it's different. There is some overlap and the readings, but it's more the course name is explaining explanations, which is mostly focused on the nodes. Under humans and how those cause a lot of the issues and the social and behavioral science, but you can see like in the in the way that they are thinking about that there is really they are like focused on the humans part and the academic system spot and how that leads to similar issues that we are talking about, but not in this kind of framework. So I showed him this and he's like, oh, you know what, I'm gonna do an exercise in class to map out the the same thing, and he's gonna share it with us tomorrow.
So that is modularized and we're outsourcing.
So there might be like an additional.
The other module that I'm working on is the very bottom one that kept them evaluated by Yeah, how the academic
system works,
or rather well, I'll talk about
that, ya know, through the academic systems, kind of the sociology of science is something that's it's part of someone can create a whole career just like under one over there, okay, I want to go next.
So probably put, I'm just sending this question of how generative AI, specifically MLMs are interacting with social sciences enterprise so there are some opportunities that alums pose, for example, in the field of behavioral design, but I think in the context of this class ellos have a potential to both help and harm with some of the goals that we have regarding the cumulative science. So I want to take kind of an in depth look at what are the opportunities, where are the risks, explicitly mapping them to some of the problems that are in this, this, this mind map, because I think this is useful for me to kind of look at it and say, Well, how can LLM make this better or worse? So that's, that's the angle that I'm taking with this process. And that
kind of for me, it's kind of at the solution side, potentially. And of course, I actually and of course of problems as he said, If Canvas can play in both worlds, for example, in the humans nodes here, we have limited cognitive capacity can we use we can use LLM to expand the cognitive capacity of theorizing but then what are the downstream effects of that one? And then which one is going to target by doing that and you can like think about it like going left to right, how does LLM change any of these nodes and what the consequences are going to be? positively and negatively and I think that's potentially a cleaned up version of that can be put on the solution side, at least the positive spin off it with giving the caveats of the problems that it can create. So
this is your first citation
Okay, cool. for what's next
we're going to talk about the academic system. Yeah, I was gonna go next but I think I think next
I'm happy to go.
Over, I'm interested in describing or rather understanding the persistence of known bad practices. So we talked about a lot of things that lead to bad social science that are contributing to lack of toughness. The importance of one sided null hypothesis testing, for example, ignoring heterogeneity, all these problems that we've gone through over the semester, and there's solutions that exist or proposed solutions that, you know, could be implemented but aren't. So the question is why? Even though we've known about many of these issues for decades, and they're known solutions, why are these problems so persistent? And to understand that I'm going to use organizational theory theories, to see how we can explain our institutions and practices and kind of why they're so hard to displace. That's the idea. And
one thing to consider again, I think you can take it as high level as you want or as narrow as you think is going to be useful. But like for me, it's even like, if we take one practice, like misuse of or the use of null hypothesis testing to evaluate theories, right like already in this class, we've read papers from me in the 50s. And then there are papers and papers from the 90s and papers from last year that criticize that practice. Yeah. How come? That like what's what, how can we use potentially these theories, sociological theories to explain the resistance of this practice, so maybe different practices, there are like different lenses that are used for something like that, but even picking one
they might help focus it and you know, really understand one thing well, as opposed to a bunch of things without really digging it.
Yeah. And I think again, like taking maybe starting from a node in this problem space, it's like, oh, this is an interesting, you know, like, why does this thing stay and that kind of, and thinking about it, but then also kind of as you as you understand it, kind of you go back and end this graph to see if actually, again, like it's a problem that actually cannot be resolved. We can see why it's possessed, but you can kind of solve until you deal with the upstream deal without without steam causes of it and then maybe there is like, a subgraph that you are studying this and the related issues that cause
Yeah, no. So that's part of the challenge and already thinking about this starting outline is you have to have a decent mental image of how these things are connected. This is more support for why this is important exercise. The causal map unquote. But I think the social theory isn't, we didn't talk a ton about like, the social theory of organizing, and I'm in another class that that does that. So I think there's some overlap. yield some interesting insight.
Yeah. And it's not only me, perhaps focusing on a sub graph that to explain by looking at the upstream nodes, but also potentially that would expose some missing nodes within that sub graph, things that are interacting with these things that is causing them to be persistent that
that probably will happen. As you as I go through the process. I don't have a clear picture yet. That's why it's helpful to like go through this medical
thing they can be like, if you talk to him or give them a system and possibly want him to feel better about how it happens outside the academic system. Like I think there are two ways to think about this. Like what one is, like something the argument that I'm doing the same thing that the rest of the world is doing. But there is something that we shouldn't be doing as scientists and another is that we're doing something differently in ways that on the outside world is not doing and that is what makes science so
so you've described the first one is institutional theory, which is institutions becomes apologized and people adopt them into inappropriate contexts because they must have figured out that this is a practice so we're gonna do it here. And then the other one is trying to explain heterogeneity of practices. That one's a little trickier, but there's like network theory and other kinds of ways to think about it that
could help it might be really interesting. It's
a helpful organizing framework sooner rather than later.
You want to go wrong. So I was also interested in the instance and exploring the economic system part why scientists focus on individual things in ways and I decided to focus now on my
it's very nice.
I think you already have like two so yeah, I mean, can we to focus on predictions and explanations more specifically, like why social scientists are thinking or at least pretending that they're doing predictions? Because for example, if you mean policy conditions, intervention, but in reality, y could be closer to just the market conditions similar to culture markets. Like for example, you, you have many bricks about the causality X causes Y distribution z. When z changes, this is how it changes and you have all the specs and people whether they're scientists or like lay people or managers, one wants some kind of story of how the world works, how the organization works. They just choose them, as you know, similar to how you choose which movies to watch and what to get out of it, but um, this will arguably correspond to how predictive things are. And I am going to start by I think I should use the pay like you mentioned, like expenditure and what why people like explanations and why it is more prevalent in the social sciences than to simply natural sciences. It could be because of our peoples, as Margaret said, people some, you know, we are within the social system. So we think that I'll be understand things that are happening around us. So maybe there's more incentive to get the validation of all these explanations, but it could also be that um, yeah, like, the these explanations. I think that just as an excuse for people to make decisions that they know, in this hypothesis space is going to be hard anyway, so I'm just going to make a decision that I can rationalize myself into making that decision. And the way I want to kind of and the direction that I want to take with this kind of framework and review is something related to the coin project that I'm doing with a filmer, which is kind of on escalation, which models to make direct recommendations about PGG like effective punishment in various experimental design settings. And what we found that is very good at generating explanations for why that may be thought which is going to punishment is going to be effective or ineffective, put on the ballot during the actual prediction. And they can make like all different variations of explanation. So what I'm interested in how humans respond to this, like, they could be some kind of, like similar to call to market the features that make some kind of explanations more plausible and more appealing to humans were designed to sort of lead people. But if we look at the music market result, it could very well be the case that they are the feature. It's like they are kind of dominated by, like how many people endorse that expedition. In the first place, or like from which scientists or which institution? Then the most important point that I'm interested in which I think that's why like classical music is that these publics punishes ID in any way. Like we'll leave it to how to like how correct that explanation is, because this motivation was partly because of like some social scientists like assumptions or implicit bias towards thinking that more explicit explainable theories are more likely to be correct. But um, I think we haven't really had like systematic way to self esteem that just because I'm controlling with expenditures generally. I got a challenging and we haven't been there's been like gone. I think we did more work on prediction itself like people's ability to predict similar to devoid
of activity. So yeah, I think so I got I'm not going to be running the experiment this semester. Rather, this is gonna be a starting point for me to motivate like this experiment, and to design it so that I could leave it in the background and put it something like that.
Yeah, that's one thing I'm gonna do with me actually.
I just asked, like, Can He send me the link to his to the syllabus for his clock class, which is explained explaining explanations which I think you will find a lot of the readings they're directly relevant to that part, particular of your project. So let me just put it at the hand. Okay, who wants to go next?
So I think I first want to like to watch Project parts. First one is like an extension or in the early part class between the critical utility paper and we work. It was saying that I bought this SES as a way to separate our
actually, we have the theory and that's why it's always has to be past. So these are like two contradictory explanations of some of the problems
with psycho psychological science. There's like a science. So that like, looking at a lot of the problems that we've covered and see what the implications of these two ways to get the problem worse those those problems, the solutions.
Secondarily, I was interested in like, a summer. So Jason, actually, I talked about this a little bit last week, which was like we've had this like, problems with scientists have adopted distributions to more rational choice model where like this incentive structure, physical block was announced
last year, the sensors but
the contents of the problem from the security model
a model of the scientific community
I mean I would like to use slightly to the President a second one because I think the first idea of like the tool explanations, Jason is going to tell you that activity sourcing because it goes in both directions, they are crossing each other, which i i would buy as a possibility. I think there is something to be done there. But I think it might be less impactful than the second project. The second project. The thing is, I don't I wouldn't be surprised if economist hadn't done something. Those lines.
Yeah. So there's, so I ended up like, trying to build this model. And then I was like, this is I built this like, really complicated way. And there's like the basic idea is like, better captured. There's better captures like global games. I used to be a global game, coordinating government. So I've been looking at that literature.
Yes, I think being exposed to that potential and then applying the approach or framework or extending some of the ideas there too to some of the problems in the problem space, right all in the solution, that is solutions that that we have and you know, try to see whether the model can be insightful about challenge of implementing a particular solution or some of some insights about connecting that solution to particular problems, I think, taking some sort of a toy model that already people have evolved to explain something within this domain, and then try to walk away and extended for something else within the same solution and we have a different kind of for you to get the practice but also I think it's a perspective that's the missing from class and kind of, for us to be aware of it, how to use it, and what insights can be derived from it would be if my paper were already done, it would be a good input into yours for policy analysis. Understanding how altering the incentive structure to alter behavior have you read the the folly of coping for a while incentivizing? Okay, I'll start with you. It's
a great short little paper about hoping for one thing but incentivizing some
candidates right yeah
okay, I only have a PDF
so previously was drawn us project and like, building on up to last comment about we should design as modular kind of or accumulative like this is irrelevant to my project, but I think having a seminar every semester for each root cause might be a good way. You have collaborators across us. We can do parallel across like C and you owe your collaborators right? You should personally and your friends serious. Okay, how can I help? Okay, so this one I think is in general related to the evaluation piece in the board, like how evaluation happens and how that is related to an agent's prediction and a world. But for specificity I gave came up with an example with a startup that looks for product in a market, but evaluation logic is still there. So I decomposed how an agent learns from the valuation, which is this gap between predict evaluation and sample the valuation and how it updates its belief about its solution and a neat outcome. So there are three components mainly in here. So first is the word is changing. So that's related to the time components the dinette market is very dynamic. And also we have a higher expectation of my product would work well in the market in general. So that's optimism. And also we have uncertainty on market over uncertainty on product. So this two goes hand in hand so they can be combined in one parameter for belief up for parameter update. So what it does in the end is given the dynamics of a market and optimist optimism and uncertainty on market over product, it is a simulation model that it goes through several time steps and visualize how this entrepreneurs or startup agents belief is updated. And that is observed in historical data as startup pivots. So you can think if Yeah, this so this is an example of a table that I have come up with. It's a little bit small, but I think the core Insight was up there last integrative design, where different parameters all interact with each other. So each has some conflicting hypothesis on whether the market becomes more dynamic. This startup should pivot in terms of product or in terms of market. So the example is, they start from a Evie company in a rural rural market, but then they gotta like one star review. So should they update their product from EB to hybrid because there are not too many infrastructure out there. Or they should just change the market to urban space with their product. So I named it as an outward pivot and inward pivot meaning in terms of an agent, whether it is trying to change something that is environment out there or changing something that is providing the product. So I'm doing some more in depth analysis for denim. So they over the 30 years they have done several pivots. So I'm mapping each cases with how this comes related with each parameter, but I'm a little heavy, hard time persuading people entrepreneurship about this integrative concept, because every paper concentrate on one row, so I wouldn't need some help and then including persuading my advisor Yeah. Any feedback or
I like the the two by two digit grid.
Oh, because it's
because it's helpful to think about
varying your positioning.
It's just a helpful binary choice. Although maybe it isn't a binary choice.
What's the software you're using?
Oh, this one? Yeah,
like what that would be looking at.
So this is obsidian. The whole thing is obsidian. But I drew the diagram was keynote
but is it too much of a stretch that thinking this as kind of an enterpreneur early stage PhD, like evaluation we always when we write paper, we overestimate the citation we're gonna get right. Yeah, we have some optimism when we're writing a paper but the market of the journals is changing and also we have some uncertainty in what model we can build and also what this problem can solve with our model. Like for instance, SD model, you have a lot of markets for your theory, JSON. So I've been thinking about this a lot, but it's very hard.
I don't fully understand it. digest it. And just understand that thanks.
Okay, let's take 10 minutes break 115
about which paper next are in the I want to see someone suggest sticks sticks don't work
oh
I don't know like carrots I mean they might work in a public goods game
