Large language models as simulated economic agents (2022) [pdf]

https://john-joseph-horton.com/papers/llm_ask.pdf

by benbreen • 4 years ago

89 42 4 years ago

cs7023• 4 years ago

I love it. Instead of (a) running mathematical experiments that model human beings as utility-maximizing agents in a highly-simplified toy economy (easy and cheap, but unrealistic), or (b) running large-scale social experiments on actual human beings (more realistic, but hard and expensive), the authors propose (c) running large-scale experiments on large language models (LLMs) trained to respond, i.e., behave, like human beings. Recent LLMs seem to model human beings well enough for it!

Abstract:

Newly-developed large language models (LLM)—because of how they are trained and designed—are implicit computational models of humans—a homo silicus. These models can be used the same way economists use homo economicus: they can be given endowments, information, preferences, and so on and then their behavior can be explored in scenarios via simulation. I demonstrate this approach using OpenAI’s GPT3 with experiments derived from Charness and Rabin (2002), Kahneman, Knetsch and Thaler (1986) and Samuelson and Zeckhauser (1988). The findings are qualitatively similar to the original results, but it is also trivially easy to try variations that offer fresh insights. Departing from the traditional laboratory paradigm, I also create a hiring scenario where an employer faces applicants that differ in experience and wage ask and then analyze how a minimum wage affects realized wages and the extent of labor-labor substitution.

bmc7505• 4 years ago

https://link.springer.com/article/10.1007/s11229-020-02950-3

jmeister• 4 years ago

Great reference. Thank you. I think simulation is underused as a thinking aid and pedagogical tool.

In statistics Andrew Gelman has been championing simulation lately.

eternalban3• 4 years ago

Recent LLMs seem to model human beings well enough for it!

Human beings are bundles of emotions and feelings. Quite a lot of economic activity of humans is motivated by irrational impulses that are engendered in society and through interacting in society with other humans. More fundamentally, OP’s assertions regarding homo silicus and “implicit computational models of humans” are precisely the matter under contention. Does language fully capture human existence? Is thought truly simply a side effect of language? I am in the camp that says, no.

cs7023• 4 years ago

No one's saying that recent LLMs "fully capture human existence," whatever that may mean.

But the evidence in this paper suggests they simulate human beings well enough for these kinds of experiments.

rcme• 4 years ago

Are LLMs rational?

astrange2• 4 years ago

That's not what "rational"/"irrational" means in economics.

It means something like, if you're an economist and you see the price of eggs has spiked, you can't just enter the market at the old price and win all the business from existing egg farmers. You don't know more than existing market actors (they're "rational agents") just because you know economics.

andrepd• 4 years ago

Attempting to draw any kind of conclusions about the real world and human behaviour from a chatbot. Can't decide if this is hilarious or disturbing.

john_horton7• 4 years ago

It is so implausible that the training process that creates LLMs might learn features of human behavior that could then be uncovered via experimentation? I showed, empirically, that one can replicate several findings in behavioral economics with AI agents. Perhaps the model "knows" how to behave from these papers, but I think the more plausible interpretation is that it learned about human preferences (against price gouging, status quo bias, & so on) from its training. As such, it seems quite likely that there are other latent behaviors captured by LLMs and yet to be discovered.

c7b2• 4 years ago

Fascinating as a paradigm. I wonder whether there's are ways to scale this to the level of agent-based simulations, ie models with populations of agents that let you study macroeconomic effects as emergent phenomena. You'd need to be able to scale those LLMs computations to many agent and find principled ways of encoding their interactions and decisions.

wcoenen• 4 years ago

You'd also have to fit the past interactions in the context window of the LLM, otherwise it wouldn't remember them.

Fine-tuning individual agents in order to move memories from the context window to the neural network weights, even if possible, would probably get too expensive.

john_horton7• 4 years ago

yeah - so I think this is worth exploring. Given how many tokens you can jam in the prompt even w/ GPT3, I think could do some pretty complex game play, at least compared to what is typical in the lab e.g., I think could easily have it remember how 100 or so other agents behaved in some kind of public goods game.

bilsbie2• 4 years ago

I remember this being an example of human irrationality in economics so I asked chatgpt.

(your utility from the exact same good shouldn’t change depending on where you buy it from.)

Seems like it has the same issue as humans do:

Pretend we’re two friends and we’re at the beach. You give me money to buy you a beer. How much would you want to spend?

Chatgpt: I would want to spend around $5 for a beer at the beach.

Me: What if I could the only place selling beers is a fancy resort? It’s the same beer though.

Chatgpt: In that case, I would be willing to spend around $10 for a beer at the fancy resort. It's still the same beer but the location and atmosphere of the resort may justify the higher price.

lossolo• 4 years ago

That's a bad example, it's fully rational and chatGPT gets that "It's still the same beer but the location and atmosphere of the resort may justify the higher price.", you are not paying only for beer there, you have location and atmosphere included in that price, that's why alcohol price is so high in night clubs. You can go buy the beer and go home or you can go to fancy resort and drink beer there, experience will be different and that experience is included in the price of the beer.

bilsbie2• 4 years ago

Not sure if it was clear but the friend is bringing you the beer. So you wouldn’t experience either location.

The end result would be identical.

astrange2• 4 years ago

(your utility from the exact same good shouldn’t change depending on where you buy it from.)

It's rarely the exact same good in real life.

(If you like a bar you'll accept higher prices because they fund the bar. If you like an artist you'll buy their merch over a different artist even if they're the "same thing".)

Of course this is true for things like grocery stores, but if you see someone doing this in real life you should assume they have a reason for it.

john_horton7• 4 years ago

oh hey - it's my paper! If anyone is interested in exploring these ideas, feel free to get in touch (@johnjhorton, https://www.john-joseph-horton.com/). FWIW - I think it would be really neat to build a Python library w/ some tools for constructing & running experiments of different kinds. I think the paper only scratches surface of what's possible (esp. once GPT4-ish has an API).

anigbrowl• 4 years ago

You might like to include this experiment in future research: https://www.science.org/doi/10.1126/sciadv.1600451

john_horton7• 4 years ago

that's a great suggestion - thanks!

doctoboggan2• 4 years ago

This is really cool. I had a similar thought that GPT3 could be used to simulate political polling. A few weeks ago I tried telling GPT3 that it was part of a specific demographic (age, gender, race, income, political leaning, etc) and then asked it how it would respond to certain political questions (I tried gun control, immigration, abortion and some other issues). GPT3 was able to change its answers in believable ways depending on what demographic I instructed it to be.

My thinking was that this could be used as a quick polling test to see how the real population may respond to certain new ideas.

More work would need to be done to calibrate it, as without specific demographic details the answers tended to be liberal leaning. But its an interesting idea which could be used to create instant focus tests on any number of topics.

capitol_• 4 years ago

GPT3 was able to change its answers in believable ways depending on what demographic I instructed it to be.

Doesn't this just mean that your own preconceptions about those demographics matches the language models preconceptions? How would we know that is matches reality when presented with novel ideas/concepts that we want to get feedback on?

visarga4• 4 years ago

I had a similar thought that GPT3 could be used to simulate political polling.

There's a good paper for that task. See my other post: https://news.ycombinator.com/item?id=34385489

rhelz• 4 years ago

This has got to be a Sokol-style troll. My bullshit detectors are pegged out at max.

theptip• 4 years ago

Interesting. One problem with this approach might be that as we start using RLHF to teach LLMs how to be “nice”, we might dissuade them from making statements that actually reflect objectively real human selfishness.

Shorel• 4 years ago

Now, this is the point where AI advances become interesting. What a year to be alive.

The only limitation I see is, we now need so much more computing power.

visarga4• 4 years ago

This is how I envision the path to AGI - run simulations to learn from outcomes, play games, solve problems and tasks. The model can create its own data. It's data engineering.

visarga4• 4 years ago

Large language models as simulated economic agents (2022) [pdf]

Related Stories