LLM Economist – Mechanism Design for Simulated Agent Societies

https://github.com/sethkarten/LLM-Economist

by milkkarten • 12 months ago

2 9 12 months ago

milkkarten4• 12 months ago

We simulate large-scale agent societies where heterogeneous personas work, adapt, and vote—governed by an in-context planner optimizing social welfare.

The system models decentralized governance, dynamic tax policy, and institutional evolution—entirely via in-context reinforcement learning, no fine-tuning required.

Full paper (arXiv): https://arxiv.org/abs/2507.15815

slwvx2• 12 months ago

I like the idea of simulating a society! I don't pretend to understand everything that you're doing, so please correct me where I'm wrong below.

The right side of Fig 5a shows that your LLM tool has 80% tax for people making between 0 and $11.6k/year, then drops to about 30% for the next tax bracket, with other tax brackets moving around all over the place. This seems to be designed to induce people to NOT pay taxes.For all its faults, I think the US progressive system is fairly rational and does a pretty good job of inducing people to actually pay taxes [1]; specifically the (effectively) negative tax rate in the US for low-income people gets them in the habit of paying taxes. I.e. whatever underlying model of social welfare you are assuming to get the great social welfare on the right side of Fig 5a seems to not model real people. I wonder if some LLM hallucinations are going on under the hood to create the strange behavior in Fig 5a.

Some questions: You don't seem to model the US system of tax credits; is that right? Also, is there a Saez tax below $47.2k in Fig 5a? What about between $244k and $609k? I.e. is the Saez tax ever under the LLM tax?

[1] https://blogs.worldbank.org/en/governance/why-does-progressi...

milkkarten4• 12 months ago

These are the marginal tax rates not the effective tax rate (e.g. 80% of first $10k, 30% of $10k-20k). We do not model tax credits here. We try to keep the system as simple as possible so that we can effectively evaluate changes. As is, the Economic theory is intractable once we introduce bounded rationality from purely rational. We do think in future work we can potentially work out some smoothness in the overall tax rate but try to let the LLM planner try what it thinks is best in order to help test the in-context optimization capabilities.

Also, while there is a complicated tax code in the US, in our simulation there is no way for agents to avoid paying taxes :)

The Saez tax rates are perturbed from the LLM Economist's tax rates to find the theoretically optimal values according to the economic theory.

Thanks for the interest and I hope that this helps clarify some of the details.

MutedEstate453• 12 months ago

Interesting approach, but I'm curious about the practical cost considerations. A 1,000-agent simulation could easily be hundreds of thousands of API calls. The repo recommends gpt-4o-mini over gpt-4 and supports local Llama models, but there's no guidance on the performance trade-offs.

Would love to see cost-per-experiment breakdowns and quality benchmarks across model tiers. Does a local Llama 3.1 8B produce meaningful economic simulations or do you need the reasoning power of frontier models? This could be the difference between $5 and $500 experiments.

milkkarten4• 12 months ago

Using smaller, cheaper agents is one of the goals of the work. There is a Pareto frontier though: by using smaller, faster, cheaper agents, the number of steps required to converge increases. We touch upon this briefly in the paper

MutedEstate453• 12 months ago

Thanks. That Pareto trade-off is exactly what I'm trying to quantify not just qualify. For example, if I've got a $50 budget, what's the sweet spot?

Scenario A: 100 agents × GPT-4o-mini × 500 steps Scenario B: 500 agents × local Llama 3-8B × 1,000+ steps

A quick table like "X agents × Y model × Z steps → tokens, $, convergence score" in the README would let new users budget experiments without having to read the whole paper plus run expensive experiments just to discover basic resource planning.

LLM Economist – Mechanism Design for Simulated Agent Societies

Related Stories