John: [00:00:00] Welcome to the next episode of the flux, where we talk data
decisions and the stories of people asking what if questions to better understand
the world and create an intentional impact on the future. As always, we hope
you can turn hindsight and lessons from our guests into foresight so you can
uncover how you can make better decisions and create an intentional impact on
the future.

Today, we're on with Rob Axil of George Mason University, where he runs, the
largest graduate program in agent based simulation, on the planet, and gets to
work on some really cool stuff. Rob, glad to have you on today.
Rob: Okay, thanks for the introduction, John, and, happy to be here talking to
you guys about, ABM and other things. yeah, as John said, at George Mason,
we have been running for about 15 years a Ph. D. program. In, computational
social science, which, one of the most important aspects of that is, age based
modeling and its various flavors of related things like social networks, etc.
it is the case that, we, at any instant in time, we have about 50 PhD students.
[00:01:00] And over the last,15 years, we've graduated almost 100. I think
something less than 100, but we're approaching 100 soon, graduates with CSS.

let me just say also by way of background that, at, George Mason we have a
Center for Social Complexity of which,it's is a kind of a research arm of, our,
normal educational, programs.
And we also have for this Center for Social Complexity, more theoretical focus,
does a lot of,big picture things. And when we get, people coming in who say
they want to do more policy things. Then we have a little, kind of computational
policy lab that,can be used for, small scale government projects or some, a littler
investigation that we wanted maybe just to do a one off on, but, overall, a
couple of different research centers and a degree program, and that's, that's the
work that we do at George Mason.
What do most of the students go into after they go through one of those
programs? Is it mainly on the government side, or have you seen more of a mix
of government and commercial lately?
Rob: Yeah, that's a great question, John, and I think it, over time, it's been, ebbs
and flows a little bit.
[00:02:00] of the 50 PhD students,a non trivial fraction of them, let's say, 15 to
20 of them, are going to be part time students who work, In Washington
somewhere, right? That, one of the big agencies or maybe for DOD or
something, and we have several, military officers come through our program,
for example, and go on to teach at the Naval Postgraduate School or at the
Naval Academy or somewhere.
it is the case that,some of the students are part time. So those part time students,
of course, are going to go back to their home base of operations and, that is
typically, giving them time off to do the degree. But the ones who are full time,

I would say it's been a pretty, over time, it's been a pretty stable mix of Maybe 1
3rd, people going into government, whether it be traditional,become a
government employee or could it be something more like Somebody might just
go into the D.
O. D. Contractor base where you're working for the government through
contractors. About 1 3rd of our graduates have gone into some form of research,
pure research, academia, flash think tank world. I love put the Rand Corporation
into that group and [00:03:00] maybe miter, depending on how you count them.
I think you said you've had met caller on before. That's in the motion later and
then 1 3rd are going to be in the private sector, John, including people at
startups. so there could be a variety of,social network startups and, maybe not so
much ABM startups, like which, how I think of epistemics, but, but that's
coming on the horizon too, I think, but more just kind of startups associated
with some maybe machine learning or something like that.
so overall, something like, one third, academic research, one third private
sector, then one third government is a certainly a, approximate way to think
about where our graduates have gone. Cool.
Cool. before we talk about more things from the program and complex social
systems, agent based modeling, and some things you're excited about, is there a
story that you have that got you into agent based simulation or modeling
complex systems?
Like, how did you get started?
Rob: I've been around so long. I'm such an antique, that, I was there a little bit
at the founding of ABM. maybe,the relevant question is something like, what
got me into the study of complex systems?
[00:04:00] And, and there, there's a very clear answer. And that's,in the late 80s.
I was a natural scientist mostly, but with an interest in policy and economics.
And uh,the way it actually worked was that, when the microcomputer
revolution first happened, and all of us were migrating from our mainframe
Fortran compilers over, I was very excited about,the burgeoning nonlinear
dynamic stuff on Chaos, and Mamba Brat Set, and all the cool graphics that
were happening at the time.

And that led me to the Santa Fe Institute. So as a graduate student, even, so I
guess in the, somewhere around 1990 as a PhD student at Carnegie Mellon, I
attended SFI's, it was called the first winter school on complex systems.
And it seemed to be the only
Rob: winter school, I think, because, I didn't really have a building and so that
was held at the University of Arizona and it was held between the fall and
spring and it's held in like January or something.
It was a great session though. And, where Murray Gell Mann spoke about,
scaling laws for cities [00:05:00] and Jeff West was there speaking
about,quantum field theory. But, but then, but also there were good talks by
other founders of complex systems like Mitch Feigenbaum came and spoke
about.
period doubling bifurcations, and, Michael Fisher came and spoke about the
renormalization group anyway. So it's a great exposure to complexity. And from
there, from there after I finished graduate school in 92, I went to Brookings, and
Josh Epstein and I began working together. And we immediately jumped into
what we thought we were doing was, we thought we were doing artificial life,
which at that time was a kind of a, just an up and coming field.
We thought we were doing artificial life for social science. Where, where the
artificial life are mostly applied to biology at that time. So we thought we were
doing a life for social science. In fact, by the way, the original name of the
SugarScape model in all of our first documents was just, we just called it
artificial social life.
As if, as if there would only be one model and you'd be done. Not knowing that
there would be many models you want to write of artificial agents doing social
science things. So that was, That was, the [00:06:00] start of everything, I think,
with computing and ABM, etc. Of course, back then, we didn't have the word
ABM.
As I said, we were calling it artificial life, type modeling,
Okay. But that
Rob: was, for me, that was the start. And I just wanted to say that it was
obviously quite different than, it was clear to me once it got going that it was
quite different from what we had done before, what I had done before in the

sense that, most of my graduate school training had been in mathematical
economics.
And in math economics,you basically, you solve for what is the market clearing
price or you figure out, what sets of quantities are going to make the, make a
firm profitable or whatever. You solve for an answer and you get the answer.
And when, when Josh and I, first drew out the Sugar Skate model on a napkin
and then I got it going, in a simple little, Pascal C dialect, running on a early
Mac computer, the fact that the model just ran and ran and never had any final,
it never spit out like, An answer of 3. 14 or 42, it just kept running. That was
quite novel. I remember it, I still can remember at the time thinking like, this is
quite different than just doing numerical computation, right? This is something
[00:07:00] that is, using computing in a different way altogether.
when you see students come to the program, are there common models that
gave them their a ha moments or, what are some of the, since, since the early
1990s that you've seen get more people into agent based simulation or complex
systems modeling.
Rob: Yeah. So I would say that, yeah, there's definitely a subset of them who
have, at least, come to our program who have,who have read the Sugarscape
book and can see how it can, could naturally be applied to other things that they
care about.
So the Sugarscape model, is a, Point of departure, not because it's going to give
them the answers they want, but because they can see how to use some ideas,
right?
it's also the case that I think that,people understand the Schelling model as being
very foundational. The Schelling segregation model is a way to think about how
people spontaneously,people are not homogeneously distributed across space or
time.
And so people who want to work on the issue of inequality, Would naturally be
drawn to the shelling model.and then Josh and I did some [00:08:00] work on
the spontaneous information of classes, which is along similar lines, but I think
that these days there, there's such a large number of abms and so many different
social science domains that it would you accounting for someone to say I'm
coming to your program because I saw a model of this or that or the other,
which I'm somewhat expert,I'm a tax specialist, or, we now have ABMS that we
use for this or that.

So I wanna learn how to do that. Or I'm a, I'm an environmental scientist and we
have an ABM of a fishery or something. I want to figure out how to do that. So
basically they'll come having seen ABM in some form and they say, obviously
they're, what they're going to do in graduate school is gain the skill to be able to
pursue that.
Okay. are there fields that you've seen more people that people weren't trying to
get into and apply agent based modeling in the last 10 years or so? Good
Rob: question. and I think that, that speaks a little bit, John, to the design of our
program. And let me just say, just one second, a background to that, and that's
that,at time zero, we really tried to span the social sciences.
So we had one person from each social science, from politics, [00:09:00]
sociology, anthropology, geography. We tried to have somebody from every,
you Every different field who is who be able capable of advising the students
and I would handle the ones who want to do economics and finance primarily
over time.
I think it's fair to say that, we've had more from some fields and less from
others. And so it's no longer try to support all fields. For example, being in
Washington here, we do have a fair number of people doing international
relations in politics,
but
Rob: it's less common. I would say that people doing kind of American politics
or doing traditional politics.
We do, we have had some people who work on anthropology, less in sociology.
I'm not exactly sure why. And then we, of course, we have quite a number who
are working in both economics, finance, and also other ones in geography. yeah,
so it is quite,that's some kind of self selection, right?
People come to us wanting to do certain things and,but it's not yet, it's not the
case that we have equal numbers of students in all those different areas these
days.
Sure. with the background on the more finance and economic side of modeling,
is there [00:10:00] like a subfield or a new area that you're really excited about
people exploring or you're exploring yourself?

Rob: Yeah. Good, good point. Good question. I think that,those fields that have
a lot of data available are naturally attractive in the sense that they're going to
help us, those data will help us figure out which models are the best ones. So it
turns out that, as of right now, the areas that are just having a giant change in
the data structure are things like,all manner of kind of consumer, behavior,
consumer sentiment,what do consumers purchase?
There are commercial data sets developed about that. there are, various
government data sets relevant. so I think that trying to build better models of
consumer behavior, that's an important topic growing and growing rapidly. But
then those same data and those models can be used.
To do similar things in other, in, in other areas, things like macroeconomists
always want to know, what people are spending their money on and
what,people buying durable goods or most, or, people using credit to people
buying, buying financial assets, et [00:11:00] cetera.
I think we have ever better models on those things.
Meaning
Rob: means that, for example, there always has been a field of agent based
macroeconomics going way back into the early nineties. the late Professor
UCLA ran a couple workshops, several workshops in the nineties about agent
based macro.
But those macro models were always very small. They were like, a few
thousand agents or something.
But today
Rob: we can do much, much, much larger, more high fidelity models. So agent
based macro has gone from being relatively speculative, just a small scale affair
to being much more empirically grounded, much larger scale now.
So that's, something which is a big change on the rise and very exciting, I think.
Yeah. is just the scale, something that you believe has held agent based
modeling back from other fields, getting more traction. if you look at 10 years
ago, people were talking about machine learning and you see a lot more
machine learning specific graduate programs, even popping up now,so there's
the scale factor, there's the.

Connecting ABMs to actual data factor. what are some other things that you
believe have held edge based modeling back that now that are [00:12:00] being
solved for. created brighter future for the complex social systems. A B M,
Rob: Yes. I think fact that we have more data now should mean the whole thing
should accelerate.
and to be totally honest about it, in some of these areas, in order for an agency
based model to gain traction, has to basically out compete the best mathematical
model or other kinds of model. And because those other models have had a
large investment of time and effort, et cetera, in them.
we have not had capital level of investment only now are we able to compete.
For example, I think it's only now to be on the last year or two that the agent
based macro models have been able to make a run at the state of the art, what
are called DSGE models that the central bankers think about,the U.
S. Federal Reserve Board or the, in the U. K. it'd be the Bank of England. The
way that the central banks use models, they traditionally use a very low
dimensional, numerical models, not agent based, numerical ones, but which
have a lot of investment and statistical estimation and, grounding and data in
[00:13:00] them.
So even though the mo what those models do, basically, is model one agent.
elaborate detail saying what does a representative UK consumer do? how much
does he or she work? How much does she invest in retirement versus current
consumption? How much did invest in financial assets, et cetera.
So measure one very complicated agent is only now are we able to come up
with models that are empirically able to compete with those a highly find how
to highly polished central bank models have been going out for, been used for a
couple decades now. yeah, so I think it's,holding back, ABM historically, so it
was both the computing and the data, but now, it seems like we have enough
computing in it, and so the availability of the data is really letting things take
off.
Okay. you podcast, they're coming at it from a number of different types of
fields. when you look at, the competition of agent based models versus other
types of models. If you wanted to give a 30, 000 foot view on what types of
questions are really good for an agent based model rather [00:14:00] than a
more traditional statistical or mathematical model,it would be great to hear your
take on that.

Rob: Okay, yeah. So I guess there's one kind of zeroth order question, which, or
zeroth order answer to your good question, John, which is that, certainly in a
social science domain, People who are card carrying social scientists typically,
not always, but typically, are going to prefer models where the resolution is
going to be at the level of individual actors, individual people.
you might want to have a model where there are groups of actors, like a firm or
something. but typically, you want to have the individual behaviors in there
somewhere. So the first thing to say is that, neural network or other kind of,
maybe let's call them reduced form models of social phenomena, oftentimes.
even though they may be accurate in some sense and they may be useful for
forecasting, they're oftentimes not very useful for understanding,for gaining an
understanding of how a system is working, just because we have no way to
interpret what the weights are, what the parameters mean, right?
in general, we like having models with large numbers or with, with individual
actors represented. that, [00:15:00] that makes it where it means we're already
naturally in a, in an ABM world, in that sense. Now, having said that, it is also
the case that,we can imagine, say, a small number of agents or a larger number
want a number of them.
And, maybe that, you can get by for certain questions with representative
agents. I can say, I just need to have, ten types of agents,Poor, medium, middle
class, upper class, top 1%, top 10th of 1%, and I can represent, the whole U. S.
inequality distribution or something.
there may be times when you don't need to have every individual in a model,
and you can still do okay. But, having said that, I just, I think I personally
always err on the side of saying, look, if we had enough computing, There's no
reason. If we have computing enough data, why wouldn't we model every
person?
that just seems natural to do.
I'm thinking like over to you mentioned other disciplines over to epidemiology.
obviously, if you think about diseases living on people, it's probably possible to
do good things, right? with, only say, 100 to 1 representation of a population.
But if you really want to get at how do these propagate, you really need to have
a 1 to 1 representation of that population. [00:16:00] So I think that,I think that
the, over the next decade, we're going to basically see is, full scale models in

which every person is represented simply because there's no reason not to do
that.
If you have all the data, you have enough computing right now. Historically, we
never had enough data. We never have had enough computing. So we're on this
cusp right now. We're going through the transition. And I think that,yeah, yes, it
was still, there are still reasons why you want to use models that reduce scale.
But let's just say those reasons are becoming fewer and fewer over time.
So if
Rob: the
agent based modeling communities at somewhat of a tipping point or inflection
point, what are some commonly held beliefs in a world where that level of
highly granular large scale data was unavailable versus now that it is, or soon
will be?
Are there some like older types of thinking in the modeling community that you
think need to get? brought up to speed or,are there any commonly held beliefs
that, you disagree with?
Rob: yeah. [00:17:00] Yeah. maybe I'll just, lemme first part your question,
John, as being a BM versus non a BM modeling communities.
And just to say that, there certainly is, I think from the natural, particularly
people trained, oftentimes people who are ABMers come from some natural
science training or have a good computing background. And it is the case that.
In many areas of natural sciences, also in computer science, you can get by with
using averages and using typical values.
You can often times abstract from maybe the outliers by saying, what is
the,what does the representative person do or something? I think that, at least in,
when it comes to the social world in general, but also economics and finance in
particular, it does seem to me, That,that there's going to be, there are very
important areas where things like, mean values are barely meaningful and,
average values, sorry, and things like second higher moments are simply, not
useful because they're basically noise statistics.
And I always use the following example, which is due to one of my colleagues
at George Mason named Ken Comer. Ken always uses the following example.

He says, imagine [00:18:00] that there's,there's, 80, 000 people at a big football
stadium. And the average height of the people in the stadium is five foot nine.
And in walks, one of the tallest men in the world, Kareem Abdul-Jabbar, and
Jabbar will modify the average height of the people in the stadium by only a
very small amount. Jabbar is, say he's not even two feet tall. He's not, he's.
seven foot nine. He's seven foot two or three or something, right?
So he's going to divide two, two feet over, over 80, 000 people. It's only going
to be a tiny, third, fourth decimal point adjustment. But now imagine you,
you're in the stadium and the average,income or wealth in that stadium is say,
200, 000. average income of 70, 000, average wealth of, 200, 000, just throwing
out numbers, but in walks Bill Gates, right?
Now, what Bill Gates does is he actually raises the average income,up to the
million dollar mark. And he raises the average, wealth up to something very
high also. in the real social world, we have these very skew distributions that are
so [00:19:00] skew, that oftentimes, the high, the variance term is, does not
exist.
there's always, you can always compute it for finite data, but it doesn't really
exist. and so the mean value may or may not be meaningful. But this is, the
distribution of cities, the distribution of firm sizes, the distribution of words and
how commonly they're used, are, the, and, used, a million times more frequently
than, indivisible or something, right?
So the idea is that, that throughout the social world, humans create these
gigantic, ine for lack of a better term, I'll just call them inequalities, Word
frequency is not really, there's no pejorative associated with inequality. We're
just saying some words are used more commonly.
And so I think that one thing that the ABM, ABMs can really do for us is by
representing the entire distribution of a population, we can properly represent
the heterogeneity that's there. And that's something which is basically lost on
anybody who tries to, use, do any kind of model and took our social science
model with a mean and a variance.
and at least, the ABM community can really Help better understand, I think, the
real world when you can say we need to understand, who's out in [00:20:00] the
tail and how do the tails matter, etc. But now, if I also parse your question,
though, is what is it in the ABM world that people don't get right?

and that may be that, I think that,we can build models today that are highly
empirically,grounded. And, there is still this perception where they're both in
the ABM world and without that, ABM is a mostly toy model. the Schelling
model is a toy model of segregation.
It's not a model of how or why Detroit has a bright line across eight mile road
for white people living north of eight mile and black people living south of eight
mile road. so it's I think it's fair to say that as you've said we're on a cusp, John,
right now of transition. And a lot of the a lot of ABMers, still, you're not,fully
equipped to build empirically grounded models, and that's something that, with
synthetic population and other things, which I know you guys know a lot about,
we can certainly do a much better job of that, going forward.
Going back to one of your earlier points where some fields just accept working
with averages, where others might be,getting beyond that. In your professional
career, have you seen [00:21:00] times when people have been basing decisions
on averages and have gotten things totally wrong where looking back, had you
been able to apply an agent based model or a simulation to that, that would
capture more of the variance could have solved some, not so great downstream
consequences.
Rob: Yeah. I just, I think there are many examples. I'm just going to give you a
quick and dirty one. Is that some, so put yourself back in like 2000. 789 time
frame and there are a lot of commentators in Newspapers saying things like, oh,
yes,we know that there are all these subprime mortgages being written Yes, we
know there are a bunch of borrowers who are getting houses with one percent
down or zero percent down But don't worry about it because it's only one or two
or three percent of the mortgages And so even if they blow up, it's not going to
bring down the whole market, right?
That's the logic which is a, that was a, not quite the same thing as the average,
it's an averaging argument in the sense that it says that, look, the average
mortgage does not include anything subprime, so don't sweat it.
But
Rob: what those comments totally lost or totally missed was the idea that, the
way that the [00:22:00] secondary mortgage markets were at that time, bundling
mortgages was to say, look, okay, there's a bunch of these high quality
mortgages.

they're not going to pay very much interest. So let's actually bundle the high
quality ones with some subprime ones just so that people can take a little bit
more risk, get a little bit higher return. And that's fine. As long as the returns are
long as those subprimes are not blowing up and and basically, causing, causing
the other mortgages to the, the mortgage investment products on which the
healthy mortgages are based to blow up, then it's fine.
But of course, what did happen was enough of the subprime stuff went south
that,most of the entire. secondary mortgage market, developed significant
problems. And even though those subprime mortgages, both numerically, in
terms of how many there were, and also in their dollar value, were a very small
slice of the overall mortgage market, the fact that they were bundled with other
ones, really, in essence, caused the financial crisis of 2000,that really
manifested itself in the fall of 2008, right when, the election was happening,
Obama's first election.
But it had started before the year [00:23:00] before that and it took another year
to unwind and Then we had the great recession that took a long time to get out
of so anyway So that's a that was a very concrete example where people who
you can see Economic commentators in the wall street journal new york times
etc in 2007 and 8 saying don't sweat it These you know on average these
subprime mortgages are not going to matter much.
so and that's not to be completely wrong
So the name of our podcast is the flux named after the flux capacitor from back
to the future. And so we would jokingly call that a, what the flux kind of
moment. are there things going on today where you think we might be in one of
those, what the flux type moments where we should be running more
simulations,trying to understand what's going on in the world and how things
might play out into the future.
Rob: that's a good question. Yeah, I think everybody feels that, right? If you
compare, for example, right now, compare 2024 to, say, 2004, things look a lot
different on the international scene and on the, in terms of [00:24:00] stability of
even U. S. democracy. it might be that simulations would help on some of that
stuff.
not positive. it is the case that, the DOD runs lots of simulations of, outcomes of
different wars, et cetera. I don't have any, I don't have a security clearance, so
I'm not privy to say anything, that's probably very,deep about that, but I'll just,

we'll say that,it, it does always seem to me that, for example, we have a, election
coming up, and people would put a lot of stock in, the 538.
com or Nate Silver's predictions, but those are just one simulation that you can
imagine running based on the polling data. You can run, imagine running a lot
of other simulations so called, so I guess maybe in that case, it seems like, More
would be better that we shouldn't have everything coming down to just one or a
few models.
We should have more models of that stuff running. I guess I do think that when
it comes to things like, overall international financial stability, it,it probably is
the case that, We run too few. We have too few predictive models. just as a
concrete example, things like, when you say,what kind of event, today we know
how to run, so called,do shock testing or do stress testing of banks, or,say, even
like mortgage hold mortgages.
if somebody [00:25:00] lost their job would be able to keep paying their
mortgage, that kind of stuff. We know a lot about when idiosyncratic shocks hit.
Financial institutions, what we know less about are potentially large scale
shocks hitting an entire financial system when everything is interlinked and
stuff.
So some work of that type is going on, but my sense is probably we know too
little about that or be useful to know, what are the vulnerabilities. Now, it may
be the case that, once again, I don't have a security clearance. Too many people.
Maybe there's stuff going on in the background, and it's very important that not
be reported,what is the most vulnerable part of the financial system?
It's good that nobody really understands that, because otherwise it might be
people want to attack it or something, I don't know. But I guess the main thing
to say is that,it is, just a general issue, seems to me, John, is that, when it comes
to policy, okay, It has not been the case that policies are made after the conduct
of a large number of computational simulations.
Rather, policies typically made from the point of view of saying, we, this is a
good idea. There's a, there's a deontic that is there's a reason to do it for
[00:26:00] some reason of right or wrong or good or bad or some. Oftentimes
we don't understand what are the secondary effects, right?
What can emerge as an unintended consequence? Now it seems to me that
relatively rudimentary, models that we could build would in fact show up, it
would show us things like, what are the unintended consequences, right? So I

think that across the entire policy spectrum, policies being largely made by
lawyers largely being A numerical that is not very good with numbers.
I think, just as a, just as a contrast, there is a stipulation in the D. O. D. code
somewhere. I don't remember the exact number of something like every project
over, X million dollars. We're actually used to be like 25 million. It's something
that's higher now.
maybe every project over 100 million dollars, there's some threshold, before
you build it before you have the contract on it, you've got to build a simulation
of it to say, is it, is this idea even feasible? It's even going to work, right? you're
going to redesign aircraft carrier to do something or other, is it going to be
possible while you build a simulation of it to see if that's going to, is the whole
thing going to be viable or not?
So for [00:27:00] engineering projects, we have this requirement that
simulations be built in advance to attempt to, to assess their feasibility. We don't
have that for policy. And I think in a world, by the way, in a world where we,
the modeling community could provide low cost simulations to engineers.
all manner of, users,this could become very viable, but we don't have that today.
Absolutely. using the financial crisis of 2007, 8, 9 example, some of these
simulations might require a little bit of a signal to get some momentum built
behind it and for some people to really start, paying attention to it.
If we look at COVID and Neil Ferguson's model, there was some early signal
that then tipped off a whole bunch of other people getting into it. What, right
now, do you see are those early warning signs that could provide signal to then
kick the modeling community off to going into understanding something that
might be, a year, two, three years out on the horizon?
Rob: Yeah,
Or is it just people [00:28:00] getting, or is it just people getting lucky? Yeah,
Rob: Yeah. that's a great question. I think that if, if I knew the answer to that, I
could invest the, invest a bunch of money and make some money out of it. So I
don't know the answer to it, but it's a good, it's a good question.

lemme just say that as a footnote, as is, as another non, another way to non-
answer your question. I'll say that,I, when I was a graduate student, I interacted
a fair amount with Herbert Simon.
And one of
Rob: the things that Herbert Simon, one of Herbert Simon's great dicta in life
was, he said that the hardest thing for a researcher, the hardest thing in a
researcher's career is not necessarily doing the work or not necessarily, learning
some new technique or programming language or whatever.
The hardest thing is knowing what the right problems to work on are. because,
until you do the work, you don't know what's going to come out the other end,
right? It's basic research. And It's a little bit similar to to what your question
was, like, if I knew, over the remainder of the 2020s, what are going to be the
key, pressure points on which the coherence of our society depends?
And if I knew the answer to that, then I would, I could figure out a good, strong
research agenda. But I don't know the answer to that. [00:29:00] So it's a little
bit unclear what the right research agenda is. so I think that you're right. maybe
a different way to say it is that, what's COVID was obviously going to become a
thing.
Ferguson at all, realized that they could morph their flu model over to
something that would be useful for non pharmaceutical interventions. And that
became obviously something that made sense to do. But like even three months
earlier, it was not, that was not obvious. That was the right thing to do.
so these things are always going to be,knowing the right thing to do depends a
lot on the context.
And,
Rob: and as of right now, we have a pretty jumbled context. It's not
obvious,where the big pressure points are coming up, but surely there will be
some, but we don't know them.
we talked with Josh a little bit about, inverse generative modeling. We talked
with a couple of others about applications of, some large language models
informing agents and agents informing some LLMs and even looking at when
you run, a parameter sweep, how do you understand those different range of
outcomes that could come out of the same model?

Are [00:30:00] there any other, I guess technical advances that, you're really
excited about or would want to highlight whether they're at George Mason or
somewhere else that you think others in the modeling community should be
paying more attention to.
Rob: Yeah. So that's a, that's another great question.
Hard question. I think, I fully support,Josh has other people's attempts to, do the
inverse generative social science. I think it's very hard to do it. and I think, it's a
little bit like when you think about, presumably a simpler problem like,Newton
working on gravity and before him, Kepler, before him, Copernicus.
over time, they developed ideas about what gravity was and, and basically they
took innovations of a philosophical kind to actually make that better. And then,
of course, Einstein comes along and says, all that stuff is naive anyway, and
here's how it really works. And now we know today that, black, dark energy and
dark matter means Einstein is wrong, right?
so I guess my point is to say that,we don't have any principled way to go from
data to model. that's the kind of inverse problem, right? going, the forward
problem is given the model,how can I spin it forward to, and then compare it to
the [00:31:00] data?
That's relatively straightforward.
but the inverse problem. That's what science is about, right? Science is about.
I've got this pattern on here. I have this regularity. What is the right underlying
model for it? That's science. And although, I have colleagues, it was the paper
in nature last December about automating chemical synthesis, like having an A.
I actually do the heavy. I not just do all the chemical lab experiments actually
also to you. Figure out what was the most, lucrative, lab experiment to do, or
which was the right, right procedure for doing it.
automating science is something which is on the horizon, not there yet. but I do
think that, but a big bottleneck with age based modeling is trying to figure out
what are the relevant behavioral rules, certainly.
but I also think that,The way that social scientists have approached it in the past,
it's not crazy, we're going to say we're just going to parametrize the rules with
some numbers and then we're going to use the data to figure out what the

numbers are going to be. And no, a crappy model will only explain part of the
variance, a good model will explain most of it.
And I don't think, I don't think that age based modeling is really any different
from [00:32:00] microeconometrics in that way. That,micro, in
microeconometrics there's going to be a model of what, of why people do what
they do and then we use the data to estimate the model. I just think that, it turns
out There is terminology in the, in, in other branches of science where we say
about, what if your model's wrong, right?
What if you have a model, what if there's a structural,break in your model, or
what happens if the, if the structure of the model is inherently wrong, always
you're going to have a crappy, crappy ability to predict stuff, right? I don't know
if, If there's any kind of silver bullet on the horizon for that.
My personal feeling is that LLMs are not the silver bullet. Just because,imagine
we have every agent have its own LLM. my feeling is that, an LLM is perfectly
useful at qualitatively telling me, explaining to me something that I don't know
much about. But it's not very good at explaining, telling me about what I, it
wouldn't be able to tell me what I'm going to have for lunch tomorrow.
it wouldn't be a very good job of forecasting that. it's not going to be doing a
very good job, I don't think, predicting like consumer behavior, which is, which
is, we can get that down to something like, 1%, We have good forecasting
ability now to get that to high verisimilitude.
So I don't think LM, it's not obviously going to help us there, or even if they
did, [00:33:00] we wouldn't know why they're working, so much Anyways, I
don't think that, I'm not sure if that's a dead end, but some people have said, but
I also don't think it's going to be that helpful. And as you guys know, LM's are
all full of things like hallucinations, et cetera, that are hard to know in advance.
And maybe that, in a world of infinite computing, it may be the case that every
agent has its own neural network. I'm fine with that. But I'm not sure that it's
going to look like a transformer, at that point.
we have two questions left. first of those two, if you had to choose one area,
going back to your idea of one of the biggest challenges, knowing what problem
space to be operating in, where in the next, let's say, 10 to 20 years do you think
agent based simulation might be most impactful?

Rob: Okay, so speaking from my own expertise, John, I'm going to have a very
definite answer for you. I'm going to try to justify it as, as something which is,
which is a way to go. It's not the only way to go, but it's a way to go. I do think
in the next decade or two, we are going to see for the first time, in essence,
what, Brian [00:34:00] Arthur and Mitch Waldrop, I'm not sure who actually
said their, used their word, but it's a way to go.
But in the book Complexity from 1992, when Waldrop wrote this about the, the
kind of up and coming SFI, those guys said, what if you could have an economy
under glass? Because once you had an experimental laboratory, in this case
computational, where you would say, I have a 330 million consumer economy,
the size of the U.
S. economy. That economy has got 6 million business firms in it. It's one to one
scale with the real economy. now, it may make no sense to do it for the U. S.
Maybe you want to do, Latvia, Lithuania, or Liechtenstein first. I don't know.
But the point is, the idea of building an ultra high fidelity model at a one to one
scale, And we probably need to make a new term for this.
Because this is not really macroeconomics anymore. It's really something like,
it's like a synthetic economy or, I'm totally happy with the economy under glass,
but it's not very scientifically sounding. But the idea of let's build a model with
all the relevant [00:35:00] pieces in there.
And now, instead of having to conduct like, you're going to. Try a new tax
policy. It's going to take you like, six months to get it passed and then, a year to
get it going and then four years to figure out what the effect on the deficit is,
etc. You're going to be able to figure out what's going to go on with that tax
policy in the next five minutes, right?
That strikes me as being a game changer in a way that we can use, our modeling
approach to a whole new and greater capability for not just, The U. S. and for
civilized people, but also just for humanity in general.
We're going to have
Rob: a much more fine grained, understanding of what's going to happen now.
But now my libertarian friends at George Mason and beyond are going to get
upset at me. They're going to say, if you can do that, if the government can do
that someday, won't they be able to control our lives? And that's not what I'm

saying at all. I'm just saying that we already do, people already try to do what
I'm just describing.
We just do it badly, right? People, we say, let's change the corporate income tax
rate from, 35 percent to 28 percent to 21 percent or whatever, and they just have
no idea that, with all kinds of side effects associated with that, and these people
are going to move their operation to the Cayman [00:36:00] Islands, and this
money actually will not come back from Ireland or whatever, all these things
are going to happen, and they have just no way, and they're just making, it's just
total guesswork as to what will happen, we can at least have some principled,
way in which, we can say, our prediction is this, then you can still try the policy
and see, okay, the prediction is only, Half writer, only three quarters, right?
And what didn't work? And you see the better understanding of how the whole
thing works from looking at it exposed. but I think that we would never, even
though Brian Arthur and Mitch Waldrop discussed this in 1991 92. It comes out
in the book complexity. We still have never realized it. We still don't do not
have this kind of complete autonomous economy running on a machine
somewhere.
And that's gonna be doable here. in the next decade or so.
Final question. So for people that are interested in getting into agent based
simulation, some come from a social science background, other from a
computer science background.if you're talking to a high school student, they
might be getting interested in pursuing an undergrad or a graduate degree.
What might you recommend [00:37:00] them get into? That's a good on ramp
into agent based simulation.
Rob: That's interesting. I do often times encounter the following situation, and
I'll report the pros and cons of it, is that people come to me with very good
NetLogo skills, hoping to be able to do research with NetLogo.
Now, for those of you, those people listening to the podcast who don't know
NetLogo, NetLogo is a very capable, very friendly, very, to be totally honest,
very pleasant piece of software to work with even though it has a non trivial
semantics and syntax. Jack. it is a way, it is that I think the net logo software,
which is free and has a robust models library, is a very nice on ramp to HBS
modeling.

The only, and the only, the main con I'll say about it is that, is that, once again,
if you could be, if you become too facile at net logo, you can look, it can
become a limitation because you then say, I have to go over to write my own
code in Python or, or Java or C or something.
It's a lot more work. And it's true, it's a lot more work. You're going to get a big
performance increase out of it and you can really do research with the other
thing. But certainly I think for high [00:38:00] school students, John, and even
for undergraduates, NetLogo is a great on ramp. You can see applications in
natural sciences and social sciences.
You can see, colorful models for which the visualization is important.
Numerical models for which the,some data analysis is important. I think so
NetLogo has done a great service to our community. And of course, as it's based
on a pedagogy of StarLogo or earlier educational languages at MIT trying to get
kids to learn how to program, et cetera, right?
So it comes out of that tradition. it can today be used for small scale research
projects. It's just not the right vehicle for the kinds of things that I've been
talking about earlier in our conversation. A large scale models. With high
verisimilitude, it's probably not the right approach. But certainly anybody who's
listening to the podcast who wants to get their fingers wet with, with ABM, the
free software called NetLogo is a great place to start.
And, and for example, it's actually how we teach, ABM to undergraduates. At
George Mason, we have an undergraduate course. by the time that we get to
[00:39:00] my graduate course,I recommend people do not use NetLogo just
because they're not going to use it in their own research. But, but it's great for,
It's a non specialist and for people who are just getting in, I think.
Cool. Rob, thank you so much for jumping on. There's a lot of good context in
here for people.
Rob: Enjoy that, John. Thank you so much for having me. I really appreciate it.