John: [00:00:00] Welcome to The Flux, where we hear stories from people who have asked what if questions to better understand the world and talk about how data can help tell stories that impact decisions and create an intentional impact on the future. This is your host, John Courtier, CEO at Epistemics. In a world where the flux capacitor from Back to the Future does not yet exist, people have to make difficult decisions without always knowing how the future will play out.
Our guests are people who've taken risks, made decisions when uncertainty was high, and who have assisted decision makers. Transcripts provided by Transcription Outsourcing, LLC.
 Welcome to the next episode of the flux. Today we're on with Aaron Frank. we're doing this podcast, while we're here in Santa Fe, New Mexico, while the complex social system society of America's conference is going on. so Aaron, thanks for joining me on the podcast today. [00:01:00] Thanks for the opportunity.
Yeah. for the people that listen to the podcast, we have some who are very technical, some who are totally new to agent based modeling. Some policy makers, or business execs who are trying to learn about agent based modeling sometimes for the first time. based on your background and getting into agent based modeling,what got you into the field in the first place?
so I, started my career at a very traditional, international relations, national security. Sort of international relations theory, qualitative research. I was, writing reports and thought pieces on national security. I got involved in war gaming, which was really a lot of fun and very exciting to look at sort of the way strategic decision making would interact.
And then I ended up getting sent down to a research center at National Defense University working on a program for, like predicting or forecasting state failure. [00:02:00] And that got me into, the idea of like computational models. And I got exposed to Asian based models and complexity. And I looked at it as well.
These are games that play themselves. And that really made the connection between, the ability to say,the games we do and the ideas that we can do in our head are somewhat limited. But the idea that I can give, a little digital critters behave this way and let them kind of wonder while we interact and see what happens was really exciting.
and so that was my sort of foundation. And so I started out, not coming in with any sort of technical or programming skills, my strengths was always how you set up the problem or what's a good question to ask. And so I learned backwards how I program this, and all those things, but it's been, for me, that's been really a lifelong interest because I was always interested in things that we have a really hard time,formally modeling,what's the value of information to decision makers?
so how do you track information? How do you [00:03:00] track that people? aren't really like rational users of information, they put it into their own sort of preexisting mindsets. So there are already rules that are often unspecified that, that exists, right? Sure. And so I think that for me connecting again, my background in sort of strategic analysis and intelligence analysis with the ability to model information flows and exchange in interactions became a sort of very exciting sort of tool to aligned with ABM.
And this idea that decisions can be the units of our analysis. Sure. So if you think back to some of the earlier wargaming exercises and like strategy sessions that you were in,were people actually building computational simulations or was it let's run through this?
Scenarios sitting on a boardroom without a lot of data involved. Like how's that changed? So there were bits and pieces and I'll say like my experience, I won't say that it's been a long time since I've done like a tabletop game. I've done a few of them at my, my time at RAN for the last [00:04:00] 10 years.
Most of my experience was before that. A lot of the games I personally was involved in were more Kind of seminar style, but there were some games that were computer adjudicated you would, the poor programmers, you'd have a game, morning session, people would, submit some PowerPoint slides with, these are my rules.
I know what I want to do, and the programmers are trying to turn these like slides into how do I put this in the model I have and turn it into an action? Sure. They were out, they were busy all the time. They were up late at night. It was horrible. it's a lot of work to be on that side to make it work.
So there are games that are computer adjudicated, that, that do rely on models. I don't know how many of them at the time were really agent based. certainly more and more things like in combat mod are moving to that.depending on the level of fidelity. I think there's still really big investments and really a big, infrastructure.
For,combat simulation that's used in like force development and other things that are really not what we would regard as like a true agent based model [00:05:00] still. That architecture I think is still expanding. But there are some, I think most games are maybe morestill adjudicated with, Sort of expert opinion and judgment often through discourse where the players and control talk, talk it out.
and then, there are, I'd say, maybe more model assisted. The models might help with certain elements of, adjudication, but ultimately, there's human judgment. There's not a lot of games I think where you just put it into the computer and this is the result. I've done some of those.
I experimented with some of them. but I think that oftentimes the If the purpose of the game is more of an exploration or research game as opposed to a training game where training games are, large and repetitions, very clear objectives. You're trying to teach something to the players as to, there's a right way to do something or something like that.
That's very different. That becomes very, that could be very computer based, right? what we saw was we were testing out like if I give these decision maker this new kind of information. So go back 20 [00:06:00] years ago. Okay. It was, hey, if we gave like a social and influence like map of like how a foreign military organization, thinks and who the leaders are and what would happen if this one left, who would be next kind of thing, right?
would they use it? it was those kind of questions and like it, it didn't factor in, it was too much reading. so you could imagine that's like discovering like, what are the kind of information tools that decision makers will even want to use? so that's where some of the games that I was involved in back then, or what it was coordination games.
everybody got sort of pieces of a problem and what their organizational rules. And information sharing protocols allow them to put all the pieces together to actually see what was going on. So those were the kind of, some of the kind of things that I was involved in. more recent games that I did, still a while ago were things like, Will to fight, like what were the feedback between like combat casualties and,and whether or not like soldiers would persist in fighting or not.
and there you have some stochastic elements in a model, but that was mostly like, [00:07:00] like dice driven or stuff. It was very tabletop. Okay. but a lot of that stuff can be computerized. one of, truthfully, one of the biggest problems is gaming is a very sort of personal, interpersonal conversation and stuff, getting all of the things and that sort of high quality engagement that happens face to face around the table in a room or whatever, into an environment where you have access to sort of computation and models.
Usually the gaming side is most concerned about will the computer get in the way of the event. the computer doesn't work. we had one game where we had a, an email system for transmitting information. The day of the game it just went down. So we're sitting here at the Xerox machine copying messages and handing them out.
So there's just like the logistics of making computers work in games can be really tough. That, that truthfully there's odd barriers. but I think we're getting, we're seeing a generation of, technologies that are, Really overcoming these sort of HCI issues. With that, the ability to bring computation into environments that are more natural.
[00:08:00] Yeah. will increase and that will give new opportunities to, to use games and build games. And there's a lot of ambitious stuff out there. So I think that I see a really exciting future for that. Yeah. Going forward. Definitely. The HCI, Human Computer Interaction side of things, as it improves, just makes all these things more accessible to people.
And creating ways that make interacting with models and interacting with data and going through different scenarios accessible for decision makers. I don't think anyone's done that exceptionally well yet, but we know that, other tools that are coming out and make that easier and easier to do.
when you look at the last 10 years within the agent based modeling space, are there common things that people are starting to ask from a question standpoint about changes in? Maybe that might not have been able to be represented in the past or, being able to like a really granular on a population or, what are some of the bigger changes that you've seen in the field that excite you?
 I was always more interested in the sort of theoretical underpinnings and the kind of [00:09:00] insights we can develop into problems in terms of, How should we be thinking? That's always interested me more than the prediction side. Sure. because I feel like ultimately, because our, agent perspective, I feel like our choices matter.
So I feel like predictions have, they're informative, but they're not actually outside of our ability to make changes. So going back 10 years, the sort of explosion of sort of big data and access, I think there was a sense that we could now get all this data and we can put it in the models and we can make better predictions.
We have, I think, now really started to move more and more towards, so that maybe models became bigger in scale. They became, much more ambitious in their representation, but often only in those areas where we could find data that we could use.I think we're now moving into a next generation where, A lot, I don't think, I don't think I'm really to predict really got a lot better.
I'll say that too. So I think now we're seeing more of an emphasis, especially with, there's my case [00:10:00] like merging large language models and agents to begin to say, there's a lot of knowledge about human behavior and systems and how people think and how they interact. in very dynamic ways that have never been specified as algorithms, have never been specified as systems of equations.
Sure. And so I think we're now going from, we have all this data to parameterize models, do we now have access to all these descriptions of behavior and decision making that we computationally to let agents,change how we represent them and their behavior, which I think gives us a sort of many new opportunities to model problems.
both, I think, in, fine grain detail to, large scale. right now, computational performance is a big deal, But I think there's something coming that's very exciting. And so in many ways, like agent based modeling is Lincoln's axe, right?
It's the same axe, but the handle's been changed 10 times. The head's been changed, 11 times. so much of agent based modeling over my, 20 plus [00:11:00] years remains constant, but under the hood, the representations have changed, the computing architectures have changed, the scale has changed, but the core of it I think still remains entirely recognizable and I think the ambition of how do we really represent, credibly and this is where, I'm.
Sort of the behavioral side that leads to different decisions. you could imagine very small scale decisions. Two people talking where,what's the system of equations for two people talking and predicting what they're where the conversation is going to go? that's actually a hard, complex problem with all sort of the feedback and other things that occur.
And so just that very nature and that gets in the thing that can modeling creativity.so you could imagine that, from there up to how do Mobilize like mass segments of society, the role of disinformation, the role of, using information to partition groups, mobilize them.
How do you demobilize people? I think that we're more and more into [00:12:00] things that are, can be informed by data, but I don't think we have a lot of high quality data about those processes, so we still need to experiment with it. yeah. One of the things that,comes into the argument between, prediction versus simulation and testing a system, it's the data exists, some people might run regressions on it, you can understand what's happened in the past, the world's changing, if you don't have those processes, written up.
or understood from a mechanistic standpoint, it's difficult to see how when information changes or there's new,new leaders within a given 
organization, how that's going to shift behavior and decision making. So where do you feel like agent based model can come in to improve decisions if it's not just about prediction? Because some of our listeners, they're like, I just want to know what's going to happen in the future. It's that's one way to think.
The other way to think is what are the conditions that lead to a better outcome for you versus another set of conditions that lead to worse outcomes? How do you help people think through using agent based modeling from a [00:13:00] Information and understanding perspective. 
Aaron: So one is we've used them for creating signposts or indicators of where qualitatively different futures might arise and what you might be looking for to tell you which path.also we've tried to identify whether or not there are certain sort of behaviors that, an organization or an actor, right?
Like how much sway do they as an agent have in determining which future goes down? so those are areas where you might not necessarily have leverage. true predictive power, but at least, this type of strategy will more likely take things in one down one trajectory versus another. that's always contingent because there's other people making choices.
There's a lot of strategic interaction. but you're looking for what are the kind of indicators, that you should be watching for, types of information that might provide some kind of signal. it's hard I think to [00:14:00] necessarily translate the world of the model, which is always highly constrained, from the real world that's,highly open.
So you're doing a lot of sort of translation of, in this artificial world. We can watch these things. We can see these things, the actions that are available, right? We think that maps onto the real world in these ways. And so there's a little bit of, you can really analyze models, but you can also overanalyze them.
Sure. and so I think that we try and I think work with, our decision makers to show them sort of the possibilities that arise. we can show how.if you believe the world is set up this way, right? You can imagine a country and it's got national leaders, it's got business leaders, right?
it,and it might be very hard to engage with them very overtly in, in politics, but there are ranges of network and they have flows and they have power relations and other things. are there indirect ways that you can demonstrate that at least in a model that says, You may not have direct [00:15:00] access or influence over their leadership, but in the overall sort of internal competition for influence, are there those that your engagement with will provide new resources to and help them become more competitive and effective inside, right?
So other indirect lines of influence. So we've tried to Demonstrate those things. 
You 
Aaron: know, there's always again,the model shows that you can do these kind of things right now. We did not have like course of power, right? but there's sort of ways that you use the model and you use other things around it to get to better judgment.
So I would say that we really try and put stuff. The model is like one piece like I've never produced a report or something where he says, We did the model. This is what the model says. It's, we were trying to answer this problem. We built a model. The model led us to think about these things.
We did some additional work on those things, and together we got to more of a richer and thicker understanding. [00:16:00] Oftentimes we did. I did work in the past where we were modeling,like global transnational movements like,if you think back to,after the invasion of Iraq, there was this whole idea of, a Shia revival,would a transnational Shia political identity become the next major thing in the Middle East and the larger Islamic world?
And, we ended up, doing a lot of modeling and stuff and we did, we didn't see that emerge based on our model. we saw, national identity still remained very important and powerful. we saw sort of other things. and but, and ultimately, our modeling work was included into a larger study that was being done inside, inside the government.
but we became another sort of plank in the overall sort of structure being built. Our work was consistent with other things that were being done independently. And so it became just additional,sort of additional argumentation that strengthens sort of the conclusions that you know, other analysts were making.
John: Okay, so in some of the different use cases that you've seen, there's a lot in the government [00:17:00] space, more on the policy side of things. In the last year or so, there have been a few books coming out on agent based modeling and economics. as you've seen the field evolve over the last 10, 15 years, Are there other areas in social science or in business that you feel like agent based modeling is going to start gaining traction in over the next couple of years?
Aaron: So probably, I think one of the areas that would be excited, two areas that would be exciting to me outside of my kind of home domain of national security is something about public health.and I think moving from,How do you understand the micro level interventions to broader like community health questions?
Sure. And so I think that's probably one area that I think would be,really exciting because again, you have this sort of micro macro relationship, probably the other one, more broadly that really excites me is thinking about, cities as complex systems. [00:18:00] Okay. and I'm really interested inhow do we, understand the way variability and,and I'll say inequality, and I don't mean that strictly in the socioeconomic sense, I mean that broadly in the notion of complex systems 
and 
Aaron: hierarchy, hierarchy and heterogeneity, which includes, which includes inequalities and, But how does that sort of endogenously arise and sustain itself and the implications for the future as the world just becomes more and more urbanized?
Sure. So those are areas where I'm hoping to see more and more sort of agent based 
John: work 
Aaron: coming 
John: out of. Okay. And so let's pull on the agent based modeling connected to like urban design, city planning, those types of things. The use cases would extend into things like housing policy, probably real estate prices.
Developers might be interested in trying to understand, building here. What's that going to [00:19:00] do to the community? How's that going to have different impacts? 
Aaron: transportation, housing, public health again.
even things like, like food.I, and I think that,but even like the arising of subgroups and cultures, right? one of the things that, that I think, all cities have,is they produce just sort of wide variety of likeheterogeneous groups inside of them.
And some of them are, probably incredibly, beneficial, others may be, harmful,but so you could think about public safety and. the emergence of gangs or other things. How does urban design, lead to more or less? Or how do you find these things?
but I think those feedbacks are quite interesting. we did some, data analysis on cities like, Los Angeles. And you can find that, you can actually see if you looked at things like,check ins on social media and stuff. You could decompose that and there were like very distinctive groups that used the city in very different ways.
Some people went all over the city. Some people only stayed in certain areas, right? And there were certain, groups that, rarely [00:20:00] interacted and groups that, only interacted in like public spaces like parks and airports. 
Okay. 
Aaron: and so I think that just getting a better understanding of how the mix is, and I don't know if we yet understand how.
interactions within a city themselves become an indicator of the city's overall ability to be, high quality, growth,produce a vibrant, dynamic, innovation base and tax base that, enables it to create, high quality of life for everybody versus those that are descending into, fragmentation and, and it has to do with things like.
the integration of the economy, right? vertical supply chains versus horizontal and whether or not they enable or they disable those sort of ability to do sort of innovation, and mixing. So I think that those are lots of really like interesting questions. A lot of work on like the health of cities, I think could be really tied to what does that look like?
my fantasy, I imagine like a like a slider bar and you adjust [00:21:00] like the interactive, like structure of the city. And you can go into an agent based model, VR, true, and walk around and see the differences that arise. Okay. that's a longterm kind of fantasy there, but I think it'd be really fascinating to think about agent based models, not as research tools, but almost as ways of experiencing different societies dynamically by putting yourself in it.
John: So a lot of people look at, The game SimCity is what got them into agent based modeling in the first place. just to repeat the fantasy back would be, you go from SimCity into agent based modeling of, actual cities and then the experience of, here's different possible futures that we've created through these scenarios.
Connect that into a VR experience so you can walk around and feel what it does. Is like to be in a future. So I think 
Aaron: That's not something that would be, I think, at this stage, not unimaginable, ambitious, but not impossible. and I think that's really fascinating.
But, if you were to walk down the street like like imagine walking [00:22:00] down, a major street in the city. As you just walk like how much variability do you start to see? As you walk, the architecture changes, the, your sense of safety and community, right? How well maintained is it, the character of the neighborhoods who's occupying it, right?
Like I think that just getting a sense that that, how do you, I think, make the differences of life experiences and ambitions of, The populations come through so that we can make better choices about how to build the society we want to be in.
Sure. One 
John: of the podcasts we did earlier,Tom Bike brought up the idea that, all people have good ideas that could improve a society. And how do those ideas get out and get experienced or,just get into, the rest of the population's mindset? And that could be another way to make those ideas very accessible.
I think one of the challenges in the agent based modeling field is the accessibility of building models. Let's say we, have some ways to solve that, but then it's experiencing what these future scenarios could look like. And that could enable an everyday [00:23:00] person to possibly generate some model or a policy, enable their, elected officials to experience, hey, this is something that could be done.
and it's a totally immersive way of doing. new policy. Yeah. 
Aaron: I think certainly for a lot of people in the policy world, right? the idea of models and interleaved and based models as a policy laboratory has always been very attractive and highly motivating. I think that,the challenge has been that the model, there's both sort of the data level of how do you model and represent.
And I think the hardest part has been how do you actually capture the sort of, The hidden feedbacks of policy. Like the unintended consequences? The unintended consequences, which are not always bad. They're often, reinforcing of something that you may want. But more generally, just the idea that, you make a choice, and, and it just keeps you Echoing through and there's lots of, other choices that get made as a result. And so you don't always really know whether or not You [00:24:00] know, you change a rule you change something Whether or not the feedback around it of people adapting to it. 
undermines it amplifies it Solves it locally, but creates a new problem someplace else.
Sure. And so I think that again, I think we need So Probably most people in the research world will tell you, we really need,to know what it is that we're modeling. We have to build a model with a purpose. And I'm wondering more and more whether or not we need to just create, look, I need a high quality artificial world that has no purpose other than to be its world.
And I can then begin working inside of it to see how The sort of feedback start to arise. I'm beginning to wonder whether or not the way that we're trained about you build a model of the purpose is in fact overly constraining because if we're trying to discover, sure, if we're pre constrained by saying, I know I need these things and I don't want to model those [00:25:00] other things.
If those other things are your low frequency sources of feedback, you're not going to see them. You're not going to find it right. So this is where I think there's somewhere between the way we are trained for modeling versus the ambitions of these sort of artificial worlds that are self sustained ecosystems as laboratories, right?
that's a gap. It's Scientific philosophy and implementation that I don't think we've, fully wrestled with, at least not in my group. 
John: Sure. Yeah, that's a real interesting point. we, if you look at in,Don't Farmer's last book, Making Sense of Chaos, there's this diagram that has, here's the population, here's the economic side of it, here's the transportation, here's the health, here's the social.
here's the information networks and it's like a change in any one of those has a ripple effect across society and being able to have those things in combination eventually gives you that low frequency type feedbackthat's a really important thing and, [00:26:00] yeah, cool. being at the conference, have there been anything, has there been anything that surprised you, and like any of the talks or what you discovered that somebody else is working on, that you're excited about?
I just saw, I thought a really 
Aaron: fascinating talk by,Tim Clancy from the University of Maryland and, work he's done on modeling insurgencies and system dynamics, which, as someone who's very focused on the idea of agents as, discrete actors making choices, system dynamics has been Foreign.
it comes from the same sort of system perspective, but that's been a very different way of modeling. So I really liked what he presented, and it showed some rather, interesting and approaches to things that I've been less exposed to. So I think that was definitely something that was quite interesting to me.
I will say broadly, I'll go back to my earlier thing. I, I remain Concern that for Asian based modeling to really show what it can do and really, I think, achieve the sort [00:27:00] of next level of relevance to decision makers, we need much more ambitious projects and we need to really go from, beginning to end with a research chain that's completely committed to that project and not.
Try and build models out of the data we could find. Not try and,repurpose things for things that they were never intended for. so I, I think that there's a, we push a lot of, narrow sort of toy models, ideas, and stuff that really helps, I think, our thinking.
But as the goals of, the modeling community, as The asks from, I think, the potential decision making community, which may not be invested in, I don't care if it's an agent based model or something else, I want a problem solved, or I want to learn something. I think the resourcing and the ambition needs to really come up to another level to begin to take on much more bigger, higher impact problems.
That to me is [00:28:00] probably my overall takeaway is that, As a community, I'm wondering how do we go to that, that next level of what it means to 
John: show what this work could be. Cool. if you're in a conversation with people at this conference or elsewhere, is there a book, a movie, an infographic that like you could think would help inspire people to think bigger or be bold about the future of agent based modeling?
I don't know. I would 
Aaron: probably go back, actually, and I would need to do more research in the policy analysis world, but there are certain sort of studies that stand out as being highly influential from a policy standpoint that once they were done, they actually changed on the policy side what it meant to have studied a problem to get to making a decision.
Okay. It changed the definition of good work. Sure. And I think that would be where I would look. what comes to mind is at least [00:29:00] the, within RAND, the lore of, Albert Wollstetter and others that were involved in the real sort of systems analysis projects that we're looking at, U.
S. force structure and employment and basing and all these sort of. Early studies of, military, engagement and deterrence in the early Cold War that established a new body of techniques to be applied to policy. But it was a very deep and, I would say probably very expensive commitment of resources to get that work done.
But it changed, I think it changed sort of the broad meaning of what it meant in the policy arena to say. We did a policy analysis that justifies, major policy action. I feel like we need, that's what we need for, so somewhere someone's going to do, a massive sort of like national look at public health and healthcare's relation to the economy, its relationship to education, and there's going to be [00:30:00] at some point some.
very ambitious, large, like national level study, that will have all of these sort of full dimensionality to it. Sure. And that will then change what it means to say, this is the basis for making policy. 
John: I can think of somewhat tangential to public health. this guy, Raj Chetty out of Harvard, who's done a lot on, basically your zip code determines your destiny kind of work, across.
Mainly all things socioeconomic, but it's, now there's this thing called the opportunity atlas and you're able to look at that and it gives you a good representation of what has been and maybe, the big, audacious, bold goal for agent based modeling could be. So now that we understand this stuff, what are we going to do about it?
And like starting to run simulations to test policies, see how that's going to change and shape those outcomes differently. Yeah, that, that would be a massive fantasy that I would have to be able to put together. I think 
Aaron: that would be, yeah, I'm very concerned. there's an old TV show called, Numbers, applied math, to what was predominantly lawful, but there was an [00:31:00] episode, dealing with this socioeconomic money ball.
It's basically that exact thing and, the, you talk about feedback, right? this is where models can become. self fulfilling prophecies, right? Sure. if someone doesn't, if you don't think someone's going to make it because people from their zip code that look like they've never made it before, Sure.
You won't invest in them.and therefore, it's great, right? See? right? Told you Yeah, and we have to really worry that,there's been work on sort of economic markets and Models almost become their own little agents and actors and people do what they say, which makes the models right.
It doesn't mean they have to be. It doesn't mean they're always going to be and, and so we really, I think when models and policy start to connect, you have to be very careful about this sort of performative qualities of doing what the model says. and it's very easy to say, look, massive investment in this,
hundreds of PhDs work on this problem, built this model and gathered all this data. It knows more than I know. So I should do what it says, right? or, Everything it says, it looks good. Sounds good, right? Let's hit the, the easy button and do it. And that [00:32:00] becomes very problematic.
So you have to, I think, understand, analysis and decision are deeply intertwined, but don't confuse, Don't just do what analysis says. and so I feel like there's, again, there's still an art 
John: to 
Aaron: be worked out here. Yeah, 
John: there's the human side of interacting with models and, when the model is going to be used for a decision in the actual world,there's, a level of empathy that data scientists need to have for the policymakers and then vice versa.
ideally, these models are going to end up impacting people in a positive way. Yeah, like we have to make sure the modeling community understands there's a whole lot of empathy that needs to go towards the decision maker, things that they're dealing with. it's not just about this next feature of the model.
Aaron: I was always so so. So I think a lot about, we had this talk earlier in the conference about sort of validation. Validation early is situationally specific. but I've always focused more on [00:33:00] make your work transparent. 
John: Make it so 
Aaron: somebody else can follow it. They know what you did, you know what you did, 
John: right?
Aaron: a lesson, an old mentor of mine told me,the difference between doing what you know and knowing what you do. Okay. And I think we should always be focused on actually understanding and knowing what it is that we're doing and not just repeating what we know how to do. Sure.
Yeah. And I think that really applies very strongly in the modeling world because we develop these, In the modeling 
John: world, in the decision making world. yeah, that, that's a really good piece of advice. Wow. So that, that sticks. So I, yeah, so I feel like, who was that mentor?
Oh, 
Aaron: old, old colleague of mine and mentor. his name was Barry Levin. He was a fantastic friend. He passed away a couple of years ago, but,fantastic person. I, but I think that was really important. and he was an Intel, community, analyst and leader.
And it was always right. Be transparent, be able to explain what you know, what you don't know, what the things you don't know, what might let you know them, what it would [00:34:00] mean. very basic, but important, thinking about your own thinking and the limits of what you're able to do. I think that the policy process is.
If everybody is much more transparent about what their goals are, what their tools can do, the layer at which, you have a model is a mathematical artifact that produces stuff, but there's layers of interpretation involved, and, and all these layers, when they become transparent and they can connect, you can make better use of what's being done when they're hidden, when you hide your flaws, or, you, you don't mention that there's certain limitations or something of the sort.
People begin to misinterpret what you've done. Sure. It makes it,more likely that something is not going to be right. And, and there's critical thinking that's involved in all of this, right? It's great that you found a great data set, but not knowing what's in the data set, not knowing how that data sets built or what's, what's not in that data set can lead to all kinds of things.
We don't worry about that's really what the machine learning world has really discovered over the years is a lot of data, especially [00:35:00] data that's collected out of social processes. Those social processes are embedded in that information in those sort of subtle and often insidious of ways. and so I think that,there's a big difference between seeing the world through data and seeing the world 
John: as it is.
Yeah. one of the things that was really beautiful in that, last riff we were on, transparency leading to a higher level of self awareness for both, modeler, policymaker, yeah, that's a really I think that's a pretty big thought there. the more transparent that,people can be, whether it's in a model, in a dialogue conversation, you know what you're doing a little bit more.
that's really, I think, important for a lot of the listeners. Never write a line of code, but you know, they could apply that in their lives in many different ways. I think we probably have time for one more question. so what I usually like to end on, if you had to give some people advice that are thinking about getting into agent based modeling, where should they go first?
What, like, how should they kind of dip their toe in? what do you recommend to people? [00:36:00] So I,
Aaron: I would say Don't be intimidated by what other people, you know, can do. definitely find a programming language. Put in the work to learn enough of it that you can develop your own, just the simplest of models.
I think that ultimately, this is really a skill that is very learnable, but only learnable by doing it. But you don't have to do a lot, but do enough that you understand, you know, I give an agent this rule, I put it in this environment and it does not do what I expected it to do. What's wrong with my expectations?
What's wrong with my rule? What's wrong with the way the environment is set up? Just those littlest of pieces will begin to hone your skills for thinking about sort of autonomy and interaction and sort of agentic representations of things. And then that will prepare you to work with others that may have [00:37:00] better skills and may prepare you to better understand when people start talking about all the wonderful things models can do, you would say, well, you know, there's these fine grain little detail things like when do agents get to make their choices and what information do they have at the moment they make their choice and what might have changed between their choice and something and someone else that are in the environment or whatever.
I really want my agents to do sort of highly sophisticated, like game theoretic reasoning and right, you know, theories of mind about each other and wow, like the algorithms for doing that are not there yet, you know, things like that, like just getting your toe in the water on a technical level will make you a better, thinker, a better, consumer of, of stuff, a better, asker for things and, and don't, you know, I mean, it's amazing, right?
How much of the work, you know, is still done with sort of agent based mind with net logo, which is, you know, I do a lot of work in is sort of meant to be sort of the low barrier. There's so many more [00:38:00] sophisticated, you know, tools, but even that little layer we can get so much out of. And again, you know, I have ambitions.
I spoke about for much, much bigger, bigger things, but there's still so much that's there that just being able to know takes to make a model work will allow you to engage with others in this community. and really very productive ways. And if you're concerned about not being sophisticated, that sort of, like that level of,Self consciousness will, we'll get in the way from really productive opportunities and, and, and so don't, don't be embarrassed by being simple.
John: Yeah, yeah. We all have a different starting point. Just, just go for it. Yeah. Cool. Thank you so much for being on the podcast. This is great. Thanks for having me.