TBA

Successful resource allocation mechanisms maximise prosperity by encouraging recipients to make reciprocal contributions to the economy. How, then, can we design allocation mechanisms that foster sustainable exchange? Here, in an iterated multiplayer trust game, we used deep reinforcement learning (RL) to design a resource allocation mechanism that promoted sustainable contributions from human participants to a common resource pool, leading to a large surplus and an inclusive economy. The RL agent maximised human surplus relative to baseline mechanisms based on unrestricted welfare or conditional cooperation, by conditioning its generosity on available resources and temporarily sanctioning defecting players. Examining the agent policy allowed us to develop a heuristic policy that performed at similar levels. Deep reinforcement learning can be used to discover mechanisms that promote sustainable exchange.

Agent-based models (ABMs) are fit to model heterogeneous, interacting systems like financial markets. We present the latest advances in Evology: a heterogeneous, empirically calibrated market ecology agent-based model of the US stock market. Prices emerge endogenously from the interactions of market participants with diverse investment behaviours and their reactions to fundamentals. This approach allows testing trading strategies while accounting for the interactions of this strategy with other market participants and conditions. Those early results encourage a closer association between ABMs and ML algorithms for testing and optimising investment strategies using machine learning algorithms.

Agent-based modelling (ABMing) is a natural and powerful modelling paradigm for complex systems. However, the complexity of these simulators complicates the task of analysing and understanding these models, their full variety of behaviours, and the sensitivities of these behaviours to changes in the parameters. This problem has motivated previous work on approaches to efficiently exploring the parameter space – and therefore the behaviours and parameter sensitivities of the model – based on Fisher information matrix-like objects and through appeal to the notion of “sloppiness”. These works have not however provided a full probabilistic treatment of ABMs, which are often stochastic. In this paper, we propose a framework for constructing Fisher information matrices for ABMs using differentiable, probabilistic surrogate models. We demonstrate a simple implementation of this framework that enables us to consistently identify stiff and sloppy directions in parameter space for a macroeconomic ABM of economic growth.

Forecasting players in sports has grown in popularity due to the potential for a tactical advantage and the applicability of such research to multi-agent interaction systems. Team sports contain a significant social component that influences interactions between teammates and opponents. However, it still needs to be fully exploited. In this work, we hypothesize that each participant has a specific function in each action and that role-based interaction is critical for predicting players' future moves. We create RolFor, a novel end-to-end model for Role-based Forecasting. RolFor uses a new module we developed called Ordering Neural Networks (OrderNN) to permute the order of the players such that each player is assigned to a latent role. The latent role is then modeled with a RoleGCN. Thanks to its graph representation, it provides a fully learnable adjacency matrix that captures the relationships between roles and is subsequently used to forecast the players' future trajectories. Extensive experiments on a challenging NBA basketball dataset back up the importance of roles and justify our goal of modeling them using optimizable models. When an oracle provides roles, the proposed RolFor compares favorably to the current state-of-the-art (it ranks first in terms of ADE and second in terms of FDE errors). However, training the end-to-end RolFor incurs the issues of differentiability of permutation methods, which we experimentally review. Finally, this work restates differentiable ranking as a difficult open problem and its great potential in conjunction with graph-based interaction models.

Agent-based modelling (ABMing) is a powerful and intuitive approach to modelling complex systems; however, the intractability of ABMs’ likelihood functions and the non-differentiability of the mathematical operations comprising these models presents a challenge to their use in the real world. These difficulties have in turn generated research on approximate Bayesian inference methods for ABMs and on constructing differentiable approximations to arbitrary ABMs, but little work has been directed towards designing approximate Bayesian inference techniques for the specific case of differentiable ABMs. In this work, we aim to address this gap and discuss how generalised variational inference procedures may be employed to provide misspecification-robust Bayesian parameter inferences for differentiable ABMs. We demonstrate with experiments on a differentiable ABM of the COVID-19 pandemic that our approach can result in accurate inferences, and discuss avenues for future work.

Stock Trading is one of the popular ways for financial management. However, the market and the environment of economy is unstable and usually not predictable. Furthermore, engaging in stock trading requires time and effort to analyze, create strategies, and make decisions. It would be convenient and effective if an agent could assist or even do the task of analyzing and modeling the past data and then generate a strategy for autonomous trading. Recently, reinforcement learning has been shown to be robust in various tasks that involve achieving a goal with a decision making strategy based on time-series data. In this project, we have developed a pipeline that simulates the stock trading environment and have trained an agent to automate the stock trading process with deep reinforcement learning methods, including deep Q-learning, deep SARSA, and policy gradient method. We evaluate our platform during relatively good (before 2021) and bad (2021 - 2022) situations. The stocks we've evaluated on including Google, Apple, Tesla, Meta, Microsoft, and IBM. These stocks are among the popular ones, and the changes in trends are representative in terms of having good and bad situations. We showed that before 2021, the three reinforcement methods we have tried always provide promising profit returns with total annual rates around 70% to 90%, while maintain a positive profit return after 2021 with total annual rates around 2% to 7%.

The COVID-19 pandemic has highlighted the importance of supply chains and the role of digital management to react to dynamic changes in the environment. In this work, we focus on developing \emph{dynamic} ordering policies for multi-echelon inventory optimization. Traditional inventory optimization methods aim to determine a \emph{static} reordering policy. Thus, these policies are not able to adjust to dynamic changes such as those observed during the COVID-19 crisis. On the other hand, conventional strategies offer the advantage of being interpretable, which is a crucial feature for supply chain managers in order to communicate decisions to their stakeholders. To address this limitation, we propose an interpretable reinforcement learning approach that aims to be as interpretable as the traditional static policies while being as flexible and environment-agnostic as other deep learning-based reinforcement learning solutions. We propose to use Neural Additive Models as an interpretable dynamic policy of a reinforcement learning agent, showing that this approach is competitive with a standard full connected policy. Finally, we use the interpretability property to gain insights into a complex ordering strategy for a simple, linear three-echelon inventory supply chain.

Social dilemmas are situations where groups of individuals can benefit from mutual cooperation but conflicting interests impede them from doing so. This type of situations resembles many of humanity's most critical challenges, and discovering mechanisms that facilitate the emergence of cooperative behaviors is still an open problem. In this paper, we study the behavior of self-interested rational agents that learn world models in a multi-agent reinforcement learning (RL) setting and that coexist in environments where social dilemmas can arise. Our simulation results show that groups of agents endowed with world models outperform all the other tested ones when dealing with scenarios where social dilemmas can arise. We exploit the world model architecture to qualitatively assess the learnt dynamics and confirm that each agent's world model is capable to encode information of the behavior of the changing environment and the other agent's actions. This is the first work that shows that world models are crucial for the emergence of complex coordinated behaviors that enable interacting agents to ``understand'' both environmental and social dynamics.

Understanding belief formation is an increasingly central question in Macroeconomics. I am interested in doing so because I want to understand what conditions allow agents’ beliefs to compound and contribute to market crashes, banking panics, untethered inflation, failure of institutions, and the outbreak of conflict and crime when misjudging threats. The dominant paradigm of full- information rational expectations (FIRE) is insufficient to think about these conditions because we know belief evolution is inconsistent with FIRE (Coibion and Gorodnichenko 2015). I believe I need to understand and model agents’ belief formation and choice behavior in a manner consistent with empirical evidence to have any shot at matching historical episodes of instability/shifts in beliefs and choice behavior. I will tackle this question with a research project, where I utilize new tools from reinforcement learning to build more empirically accurate models of how agents learn and how the economy will evolve.