Simulation is a powerful tool particularly for the financial services industry as it provides a controlled environment for developing new trading and investment strategies. Unlike replaying historical market data, which cannot react to an experimental trader’s actions, simulation can account for their market impact. It also overcomes the need for proprietary market data for research purposes. By leveraging simulators, one can use learning techniques such as reinforcement learning (RL) to devise optimal trading strategies for a variety of market participants with diverse objectives. RL in conjunction with multi-agent simulations can offer an attractive framework to model interactions between objective maximizing agents such as those present in financial markets. Simulation also allows for consideration of a range of hypothetical and rare market scenarios, such as flash crashes, for stress testing new strategies before their implementation in real markets [21]. Likewise, one can utilize simulators to inform regulatory policy by answering policy questions, such as the impact of a stock dependent tick size vs a one-size-fits-all tick size in equity markets, by incorporating market participants’ response to policy changes and their systemic effects [4].
For economics researchers and practitioners, simulations provide an appealing approach to conduct counterfactual policy experiments of economies as an Equilibrium Markov Process (EMP) [16]. As with all counterfactual experiments in economics and macro-finance, the challenge is formalizing the appropriate structure for a question of interest - which, in turn, constrains the space of counterfactuals for the EMP. This structure can include a combination of formal decision problems of agents, statistical orthogonality restrictions, and behavioral assumptions. The classic baseline is to formally describe every feature of an agent’s decision making as a Markov Decision Process (MDP) assuming underlying agents form mathematical expectations using the well-specified EMP (i.e., rational expectations in [16]). While this is amenable to estimation, such as with DSGEs [8], and can help economists reason about many important counterfactuals of aggregated variables, it has limitations in its ability to implement rich heterogeneity and plausible sources of bounded rationality (i.e., agents may not have a well specified EMP). This led macroeconomists and financial economists to explore alternatives to the fully parametric model.
One direction is to relax assumptions on rational expectations and investigate the properties of the EMP through simulation as in classic models of learning in macroeconomics while still largely maintaining the MDP structure - such as in models of learning and bounded rationality in [2, 15, 6]. Another approach is to use a simulated EMP for the underlying expectations, as in heterogeneous agent macroeconomic models of [12], which can be interpreted as providing behavioral approximations [14]. Recent innovations in machine learning can enable the solving of EMPs with much larger MDPs [11, 9, 10], alongside investigating richer models of expectation formation with RL.
The other direction is to begin with more computationally-driven models of economic agents and explore EMPs as emergent phenomena from those interactions in what have traditionally been called agent-based models (ABMs) [18, 1]. By removing the requirement for consistency between the agents’ decisions and the resulting EMP, these methods have the advantage of allowing richer simulations of interactions between diverse economic actors like households, firms, banks, financial markets, monetary policy, and fiscal policy. ABMs provide a versatile framework to answer economic policy questions of relevance to governments, by contrasting the effects of alternative policies on the economy [24, 5]. While the lack of structure connecting agents’ decisions with their expectations of the EMP has limited the set of counterfactual experiments that ABMs are appropriate to investigate, recent advances in machine learning and game theory may help better connect the methods. In particular, approaches which bridge ABMs and game theory enable the study of equilibria in games where producing a finite game model is impractical [22]. Here, an empirical game is estimated from simulation data generated over a space of agent strategies [23], which can be constructed based on heuristics [20], or iteratively using RL [13, 17].
Despite their potential, the adoption of simulation tools in real-world decision-making faces challenges related to simulator realism and calibration. In particular, multi-agent simulators are difficult to calibrate as they require the specification of a large number of agent-specific parameters that require (labeled) individual behavioral data. Such data is typically proprietary and/or unavailable in the context of financial markets and economic systems. These challenges led to the development of world agent models, which overcome the need for individual data by using a single world agent to generate the actions of all agents within the simulator [3]. Additionally, simulators can be validated in their ability to reproduce certain stylized facts, which are important statistical properties present in historical data pertaining to real financial and economic systems [7, 19].
Arthur, W. B. Foundations of complexity economics. Nature Reviews Physics 3, 2 (2021), 136–145.
Bray, M., and Kreps, D. M. Rational learning and rational expectations. In Arrow and the ascent of modern economic theory. Springer, 1987, pp. 597–625.
Coletta, A., Moulin, A., Vyetrenko, S., and Balch, T. Learning to simulate realistic limit order book markets from data as a world agent. In Proceedings of the third acm international conference on ai in finance (2022), pp. 428–436.
Dwarakanath, K., Vyetrenko, S., Balch, T., and Oyebode, T. Transparency as delayed observability in multi-agent systems. In 2023 Winter Simulation Conference (WSC) (2023), IEEE, pp. 279–290.
Dwarakanath, K., Vyetrenko, S., Tavallali, P., and Balch, T. ABIDES-Economist: Agent-Based Simulation of Economic Systems with Learning Agents. arXiv preprint arXiv:2402.09563 (2024).
Evans, G. W., and Honkapohja, S. Learning and expectations in macroeconomics. Princeton University Press, 2001.
Fagiolo, G., Guerini, M., Lamperti, F., Moneta, A., and Roventini, A. Validation of agent-based models in economics and finance. Computer simulation validation: fundamental concepts, methodological frameworks, and philosophical perspectives (2019), 763–787.
Fern´andez-Villaverde, J., and Rubio-Ram´ırez, J. F. Estimating macroeconomic models: A likelihood approach. The Review of Economic Studies 74, 4 (2007), 1059–1087.
Gu, Z., Lauriere, M., Merkel, S., and Payne, J. Deep learning solutions to master equations for continuous time heterogeneous agent macroeconomic models. Tech. rep., 2023.
Kahou, M. E., Fern´andez-Villaverde, J., Gomez-Cardona, S., Perla, J., and Rosa, J. Spooky boundaries at a distance: Inductive bias, dynamic models, and behavioral macro. Tech. rep., National Bureau of Economic Research, 2024.
Kahou, M. E., Fern´andez-Villaverde, J., Perla, J., and Sood, A. Exploiting symmetry in high-dimensional dynamic programming. Tech. rep., National Bureau of Economic Research, 2021.
Krusell, P., and Smith, Jr, A. A. Income and wealth heterogeneity in the macroeconomy. Journal of political Economy 106, 5 (1998), 867–896.
Lanctot, M., Zambaldi, V., Gruslys, A., Lazaridou, A., Tuyls, K., P´erolat, J., Silver, D., and Graepel, T. A unified game-theoretic approach to multiagent reinforcement learning. Advances in neural information processing systems 30 (2017).
Moll, B. Heterogeneous-agent macro as a gateway to behavioral macro, 2023.
Sargent, T. J. Bounded rationality in macroeconomics: The Arne Ryde memorial lectures. Oxford University Press, 1993.
Sargent, T. J. Macroeconomics after lucas.
Smith, M., Anthony, T., and Wellman, M. Iterative empirical game solving via single policy best response. In International Conference on Learning Representations (2020).
Tesfatsion, L., and Judd, K. L. Handbook of computational economics: agent-based computational economics. Elsevier, 2006.
Vyetrenko, S., Byrd, D., Petosa, N., Mahfouz, M., Dervovic, D., Veloso, M., and Balch, T. Get real: Realism metrics for robust limit order book market simulations. In Proceedings of the First ACM International Conference on AI in Finance (2020), pp. 1–8.
Walsh, W. E., Das, R., Tesauro, G., and Kephart, J. O. Analyzing complex strategic interactions in multi-agent systems. In AAAI-02 Workshop on Game-Theoretic and Decision-Theoretic Agents (2002), pp. 109–118.
Wang, X., Hoang, C., Vorobeychik, Y., and Wellman, M. P. Spoofing the limit order book: A strategic agent-based analysis. Games 12, 2 (2021), 46.
Wellman, M. P. Methods for empirical game-theoretic analysis. In AAAI (2006), vol. 980, pp. 1552–1556.
Wellman, M. P. Economic reasoning from simulation-based game models. Œconomia. History, Methodology, Philosophy, 10-2 (2020), 257–278.
Zheng, S., Trott, A., Srinivasa, S., Parkes, D. C., and Socher, R. The AI Economist: Taxation policy design via two-level deep multiagent reinforcement learning. Science advances 8, 18 (2022), eabk2607.