Complex systems are very hard to predict and control (chaotic dynamics and open-ended outcomes), but the understanding and harnessing of their underlying mechanisms holds great promises to revolutionize many areas of science. Whereas considerable progress has been made at the bench for manipulating/measuring the system activity down to the lowest-level, there is a fundamental gap between knowledge at the low-level and control of the resulting properties at the global scale. Modern tools from machine learning hold great promises to assist humans in mapping and navigating the space of possible outcomes toward novel or hard-to-reach morphological/functional targets. However current methods are largely restricting and biasing the boundary of events that the AI can measure and try to affect. In this talk, I will present how recent computational models of intrinsically motivated learning and exploration can be used to design more open-ended forms of AI "discovery assistant". In particular I will discuss how meta-diversity search, curriculum learning and external guidance (environmental or preference-based) can be key ingredients for shaping the search process. I will show how those ingredients, when implemented in practice, can help solving challenging problems in science. This includes the search of interesting patterns in continuous cellular automata, the study of the origins of sensorimotor agency and the design of novel forms of collective intelligence for AI and biology.

Brains and machine learning systems are selected or designed to exhibit learning and adaptation. But what kinds of systems can learn and adapt spontaneously, without selection or design? We might like the idea that societies, ecological communities or perhaps the biosphere as a whole can learn and exhibit adaptation, but since these are not evolutionary units, by what mechanism can this happen except by intensional design or as fortuitous happenstance? Our work shows general conditions for a dynamical system to adapt spontaneously. ‘Natural induction’ occurs in dynamical systems described by networks of viscoelastic connections (connections that give-way slightly under stress or over time). This is a natural assumption, described by local energy minimisation acting on connection parameters. It does not require the entities, their connections nor the system as a whole to be reproducing units, subject to natural selection or composed of utility-maximising agents (although they might be). When the state variables of this system are subject to episodic stress or occasionally perturbed, such that the system visits a distribution of attractor states, and the relatively slow plasticity of the connections naturally accommodates to these states, the system as a whole exhibits associative learning that produces adaptive organisation or ‘systemic intelligence’. This is not just self-organisation or memory (attractor) formation – the generalisation capabilities of associative learning produce an optimisation ability that can be quantified by an increase in the quality of solutions it finds to a combinatorial optimisation problem. We give some simulation examples of a spring-damper system, a social network and an ecological community solving MAX-2-SAT problems and Sudoku puzzles by natural induction. To the extent that interactions are rarely perfectly elastic, and stresses are rarely constant, natural induction may be ubiquitous in networks of all kinds from autocatalytic chemical networks, to multi-cellular bioelectric networks, to societies and the biosphere. We note (however) that the design principles to enhance systemic intelligence through natural induction are different from those that enhance performance under selection or utility maximisation. In particular, sustained stress (such as sustained profit maximisation, performance improvement or cost reduction) destroys systemic intelligence, causing the system to forget what it has learned (the analogue of overfitting). We discuss the implications for us as individuals, and our relationships with one another and the natural world.

with Mayalen Etcheverry, Alex Mordvintsev & Richard A. Watson

Collective discussion with speakers, authors and audience in GatherTown

In contrast to deep-learning agents, biological neural networks are grown through a self-organized developmental process. In this work, we propose a new hypernetwork approach to grow artificial neural networks based on neural cellular automata (NCA). Inspired by self-organising systems and information-theoretic approaches to developmental biology, we show that NCA can be used to grow neural networks capable of solving standard reinforcement learning tasks. Finally, we explore how the same approach can be used to build developmental networks capable of metamorphosing their weights to solve variations of the initial RL task.

Voxel-based Soft Robots (VSRs) are a form of modular soft robots, composed of several deformable cubes, i.e., voxels. Each VSR is thus an ensemble of simple agents, namely the voxels, which must cooperate to give rise to the overall VSR behavior. Within this paradigm, collective intelligence plays a key role in enabling the emerge of coordination, as each voxel is independently controlled, exploiting only the local sensory information together with some knowledge passed from its direct neighbors (distributed or collective control). In this work, we propose a novel form of collective control, influenced by Neural Cellular Automata (NCA) and based on the bio-inspired Spiking Neural Networks: the embodied Spiking NCA (SNCA). We experiment with different variants of SNCA, and find them to be competitive with the state-of-the-art distributed controllers for the task of locomotion. In addition, our findings show significant improvement with respect to the baseline in terms of adaptability to unforeseen environmental changes, which could be a determining factor for physical practicability of VSRs.

A successful development of a complex multicellular organism took millions of years of evolution. The genome of such a multicellular organism guides the development of its body from a single cell, including its control system. Our goal is to imitate this natural process using a single neural cellular automaton (NCA) as a genome for modular robotic agents. In the introduced approach, called Neural Cellular Robot Substrate (NCRS), a single NCA guides the growth of a robot and the cellular activity which controls the robot during deployment. We also introduce three benchmark environments, which test the ability of the approach to grow different robot morphologies. We evolve the NCRS with covariance matrix adaptation evolution strategy (CMA-ES), and covariance matrix adaptation MAP-Elites (CMA-ME) for quality diversity and observe that CMA-ME generates more diverse robot morphologies with higher fitness scores. While the NCRS is able to solve the easier tasks in the benchmark, the success rate reduces when the difficulty of the task increases. We discuss directions for future work that may facilitate the use of the NCRS approach for more complex domains.

In this paper, we study the problem of networked multi-agent reinforcement learning (MARL), where a number of agents are deployed as a partially connected network and each interacts only with nearby agents. Networked MARL requires all agents make decision in a decentralized manner to optimize a global objective with restricted communication between neighbors over the network. Inspired by the fact that sharing plays a key role in human's learning of cooperation, we propose LToS, a hierarchically decentralized MARL framework that enables agents to learn to dynamically share reward with neighbors so as to encourage agents to cooperate on the global objective through collectives. For each agent, the high-level policy learns how to share reward with neighbors to decompose the global objective, while the low-level policy learns to optimize local objective induced by the high-level policies in the neighborhood. The two policies form a bi-level optimization and learn alternately. We empirically demonstrate that LToS outperforms existing methods in both social dilemma and networked MARL scenario across scales.

The diversity and complexity of living systems on Earth have presumably emerged from a single common ancestor, and before that, from the inorganic components present on the surface of Earth. So far, it is unclear what are the \emph{algorithmic} properties of a process that would display a similar trajectory in its state space. Describing such a process entails characterizing both the state space itself, the possible emergent forms, and the evolutionary process behind the diversification and complexification of forms. Because living systems are hypothesized to correspond to attractors in chemical networks, Artificial Chemistries (AC) are well suited to explore this question because they can simulate the evolution of these networks. Combinatory Chemistry is an AC in which self-reproducing metabolisms emerge from its dynamics. Here, I extend it with a set of mutation reactions, showing that said reactions coupled with the emergent structures in the system enable a more efficient search of complex structures. I conclude that the resulting dynamics constitute an emergent self-organizing search process that could capture the properties of open-ended evolutionary processes.

Complex adaptive systems occur in all domains across all scales, from cells to societies. The question, however, of how the various forms of collective behavior can emerge from individual behavior and feedback to influence those individuals remains open. Complex systems theory focuses on emerging patterns from deliberately simple individuals. Fields such as machine learning and cognitive science emphasize individual capabilities without considering the collective level much. To date, however, little work went into modeling the effects of changing and uncertain environments on emergent collective behavior from individually self-learning agents. To this end, we derive and present deterministic reinforcement learning dynamics where the agents only partially observe the actual state of the environment.This paper aims to obtain an efficient mathematical description of the emergent behavior of biologically plausible and parsimonious learning agents for the typical case of environmental and perceptual uncertainty. We showcase the broad applicability of our dynamics across different classes of agent-environment systems, highlight emergent effects caused by partial observability and show how our method enables the application of dynamical systems theory to partially observable multi-agent learning. The presented dynamics have the potential to become a formal yet practical, lightweight, and robust tool for researchers in biology, social science, and machine learning to systematically investigate the effects of interacting partially observant agents.

Summarizing Societies: Agent Abstraction in Multi-Agent Reinforcement Learning

Efficiently Evolving Swarm Behaviors Using Grammatical Evolution With PPA-style Behavior Trees

Toward Evolutionary Autocurricula: Emergent Sociality from Inclusive Rewards

Continual Learning with Deep Artificial Neurons

Local Learning with Neuron Groups

Know Thy Student: Interactive Learning with Gaussian Processes

Goal-Guided Neural Cellular Automata: Learning to Control Self-Organising Systems

Designing Neural Network Collectives

Open-Ended Evolution as an Emergent Self-Organizing Search Process

HyperNCA: Growing Developmental Networks with Neural Cellular Automata

Non-linear dynamics of collective learning in uncertain environments

Collective control of modular soft robots via embodied Spiking Neural Cellular Automata

A Unified Substrate for Body-Brain Co-evolution

Learning to Share in Multi-Agent Reinforcement Learning

This is a look under the hood of "The Lifecycle of Software Objects," a work of fiction about artificial lifeforms called "digients." I'll offer my rationales for the digients having the characteristics they have, and talk about the questions I was interested in exploring in writing the novella.

In this talk I will discuss how knowledge and skills can accumulate over generations within populations, a phenomenon described as cumulative cultural evolution. Cumulative cultural evolution is proposed to underpin many of the behavioural characteristics of modern humans. I will also discuss how this phenomenon can be studied under laboratory conditions, and how those experiments can provide insights into the mechanisms by which beneficial discoveries can be retained and even built upon, in spite of continual population turnover.

People are lifelong learners who operate in natural environments with complex multiscale temporal correlations. Psychologists have argued that mechanisms of human learning and memory are optimized to the structure of the environments in which they operate. If the goal of continual learning is to design agents that interact with natural environments, we should be designing and evaluating continual learning methods for such environments.

with Ted Chiang, Christine Caldwell, Michael Mozer & Jessica Flack

Collective Discussion with speakers, authors and audience in GatherTown

Embryos and regenerating organs produce very complex, robust anatomical structures and stop growth and remodeling when those structures are complete. One of the most remarkable things about morphogenesis is that it is not simply a feed-forward emergent process, but one that has massive plasticity: even when disrupted by manipulations such as damage or changing the sizes of cells, the system often manages to achieve its morphogenetic goal. How do cell collectives know what to build and when to stop? Constructing and repairing anatomies in novel circumstances is a remarkable example of the collective intelligence of a biological swarm. In this talk, I will review examples of problem-solving in morphospace and other challenging, high-dimensional spaces by living things at diverse scales of organization, and propose that a multi-scale competency architecture is how evolution exploits physics to achieve robust machines that solve novel problems. I will describe what is known about developmental bioelectricity - a precursor to neurobiology which is used for cognitive binding in biological collectives, that scales their intelligence and the size of the goals they can pursue. I will conclude with examples of synthetic living machines - a new biorobotics platform that uses some of these ideas to build novel primitive intelligences, and speculate about a future that integrates deeply across biological and synthetic agents.

The unexpected capacities of Large Language Models to understand concepts and engage in convincing dialogue, including the use of theory of mind, is remarkable, given that these models are at bottom nothing but sequence learners. In this talk, I’ll relate recent developments in large unsupervised foundation models with both “society of mind” models of cognition, and with theories of intelligence that place our evolutionary development and that of other large-brained animals in a social framework. Some implications of this emerging multiscale social picture of cognition will be sketched.

with Michael Levin, David Wolpert & Blaise Agüera y Arcas

Collective Discussion with speakers, authors and audience in GatherTown

In many agent societies with complex and vast interactions, agents cannot make sense of the world through direct consideration of small-scale low-level agent identities, rather it is imperative to recognize emergent collective identities among populations of agents. In this paper, we take a first step towards developing a framework for recognizing this structure in low-level agents so that they can be modeled as a much smaller number of high-level agents – a process that we call agent abstraction. Specifically, we build on the literature of bisimulation metrics for state abstraction in reinforcement learning and take steps to broaden the scope of this theory to the setting of multi-agent reinforcement learning, in which an agent is necessarily faced with a non-stationary environment resulting from the presence of other learning agents. We formulate a new set of bisimulation metrics on the joint action space of other agents and analyze a straightforward, if crude, abstraction based on a metric that distinguishes experienced joint actions. We show that this joint action space abstraction improves the minimax regret of a reinforcement learning agent by a transparent factor that inspires a measure for the utility of abstracting the joint action space of a subset of agents. We then test this measure on a large dataset of human play of the popular social dilemma game Diplomacy, we find that it correlates strongly with the degree of ground- truth abstraction of low-level units into the human players that control them and reveals key moments of stronger top-down control during the game.

Evolving swarm behaviors with artificial agents is computationally expensive and challenging. Because reward structures are often sparse in swarm problems, only a few simulations among hundreds evolve successful swarm behaviors. Additionally, swarm evolutionary algorithms typically rely on ad hoc fitness structures, and novel fitness functions need to be designed for each swarm task. This paper evolves swarm behaviors by systematically combining Postcondition-Precondition-Action (PPA) canonical Behavior Trees (BT) with a Grammatical Evolution. The PPA structure replaces ad hoc reward structures with systematic postcondition checks, which allows a common grammar to learn solutions to different tasks using only environmental cues and BT feedback. The static performance of learned behaviors is poor because no agent learns all necessary subtasks, but performance while evolving is excellent because agents can quickly change behaviors in new contexts. The evolving algorithm succeeded in 75\% of learning trials for both foraging and nest maintenance tasks, an eight-fold improvement over prior work.

The competitive and cooperative forces of natural selection have driven the evolution of intelligence for many millions of years, eventually culminating in nature's vast biodiversity and the complexity of our human minds. In this paper, we present a novel multi-agent reinforcement learning framework, inspired by the process of evolution. We assign a genotype to each agent, and propose an inclusive reward that optimizes for the fitness of an agent's genes. Since an agent's genetic material can be present in other agents as well, our inclusive reward also takes genetically related individuals into account. We study the effect of inclusion on the resulting social dynamics in two network games with prisoner's dilemmas, and find that our results follow well-established principles from biology. Furthermore, we lay the foundation for future work in a more open-ended 3D environment, where agents have to ensure the survival of their genes in a natural world with limited resources. We hypothesize the emergence of an arms race of strategies, where each new strategy will be a gradual improvement in response to an earlier adaptation from other agents, effectively creating a multi-agent autocurriculum similar to biological evolution. Our evolutionary rewards provide a novel social dimension that features a non-stationary spectrum of cooperation due to the finite environmental resources and changing population distribution. It has the potential to create increasingly advanced strategies, where agents learn to balance cooperative and competitive incentives in a more complex and dynamic setup than previous works, where agents were often confined to predefined team setups that did not entail the social intricacies that biological evolution has. We argue this could be an important contribution towards creating advanced, general and socially intelligent agents.

Neurons in real brains are complex computational units, capable of input-specific damping, inter-trial memory, and context-dependent signal processing. Artificial neurons, on the other hand, are usually implemented as simple weighted sums. Here we explore if increasing the computational power of individual neurons can yield more powerful neural networks. Specifically, we introduce Deep Artificial Neurons (DANs)—small neural networks with shared, learnable parameters embedded within a larger network. DANs act as filters between nodes in the net-work; namely, they receive vectorized inputs from multiple neurons in the previous layer, condense these signals into a single output, then send this processed signal to the neurons in the subsequent layer. We demonstrate that it is possible to meta-learn shared parameters for the various DANS in the network in order to facilitate continual and transfer learning during deployment. Specifically, we present experimental results on (1) incremental non-linear regression tasks and (2)unsupervised class-incremental image reconstruction that show that DANs allow a single network to update its synapses (i.e., regular weights) over time with minimal forgetting. Notably, our approach uses standard backpropagation, does not require experience replay, and does need separate wake/sleep phases.

Traditional deep network training methods optimize a monolithic objective function jointly for all the components. This can lead to various inefficiencies in terms of potential parallelization. Local learning is an approach to model-parallelism that removes the standard end-to-end learning setup and utilizes local objective functions to permit parallel learning amongst model components in a deep network. Recent works have demonstrated that variants of local learning can lead to efficient training of modern deep networks. However, in terms of how much computation can be distributed, these approaches are typically limited by the number of layers in a network. In this work we propose to study how local learning can be applied by splitting layers or modules into sub-components, introducing a notion of width-wise modularity to the existing depth-wise modularity associated with local learning. We investigate local-learning penalties that permit such models to be trained efficiently. Our experiments on the CIFAR-10 dataset demonstrate that introducing width-level modularity can lead to computational advantages over existing methods based on local learning and opens potential opportunities for improved model-parallel training. This type of approach increases the potential of distribution and could be used as a backbone when conceiving collaborative learning frameworks.

Learning often involves interaction between multiple agents. Human teacher-student settings best illustrate how interactions result in efficient knowledge passing where the teacher constructs a curriculum based on their students' abilities. Prior work in machine teaching studies how the teacher should construct optimal teaching datasets assuming the teacher knows everything about the student.However, in the real world, the teacher doesn't have complete information and must probe before teaching.Our work proposes a simple probing algorithm which uses Gaussian processes for inferring student-related information, before constructing a teaching dataset.We study this in two knowledge settings where the student is a tabula rasa or has partial knowledge of the domain.We study this in the ridge regression, support vector machines and offline reinforcement learning domains.Our experiments highlight the importance of probing before teaching, demonstrate how students can learn much more efficiently with the help of an interactive teacher, and outline where probing combined with machine teaching would be more desirable than passive learning.

Inspired by cellular growth and self-organization, Neural Cellular Automata (NCAs) have been capable of "growing" artificial cells into images, 3D structures, and even functional machines. NCAs are flexible and robust computational systems, but are inherently uncontrollable during and after their growth process. In this work, we attempt to control these systems using Goal-Guided Neural Cellular Automata (GoalNCA), which leverages goal encodings to control cell behavior dynamically at every step of cellular growth. This enables the NCA to continually change behavior, and in some cases, generalize its behavior to unseen scenarios. We also demonstrate the robustness of the NCA with its ability to preserve task performance, even when only a portion of cells receive goal information.

Artificial neural networks have demonstrated exemplary learning capabilities in a wide range of tasks, including computer vision, natural language processing and, most recently, graph-based learning. Many of the advances in deep learning have been made possible by the large design-space for neural network architectures. We believe that this diversity in architectures may lead to novel and emergent learning capabilities, especially when architectures are connected into a collective system. In this work, we outline a form of neural network collectives (NNC), motivated by recent work in the field of collective intelligence, and give details about the specific sub-components that an NNC may have.

The diversity and complexity of living systems on Earth have presumably emerged from a single common ancestor, and before that, from the inorganic components present on the surface of Earth. So far, it is unclear what are the algorithmic properties of a process that would display a similar trajectory in its state space. Describing such a process entails characterizing both the state space itself, the possible emergent forms, and the evolutionary process behind the diversification and complexification of forms. Because living systems are hypothesized to correspond to attractors in chemical networks, Artificial Chemistries (AC) are well suited to explore this question because they can simulate the evolution of these networks. Combinatory Chemistry is an AC in which self-reproducing metabolisms emerge from its dynamics. Here, I extend it with a set of mutation reactions, showing that said reactions coupled with the emergent structures in the system enable a more efficient search of complex structures. I conclude that the resulting dynamics constitute an emergent self-organizing search process that could capture the properties of open-ended evolutionary processes.

In contrast to deep reinforcement learning agents, biological neural networks are grown through a self-organized developmental process. Here we propose a new hypernetwork approach to grow artificial neural networks based on neural cellular automata (NCA). Inspired by self-organising systems and information-theoretic approaches to developmental biology, we show that our HyperNCA method can grow neural networks capable of solving common reinforcement learning tasks. Finally, we explore how the same approach can be used to build developmental metamorphosis networks capable of transforming their weights to solve variations of the initial RL task.

Complex adaptive systems occur in all domains across all scales, from cells to societies. The question, however, of how the various forms of collective behavior can emerge from individual behavior and feedback to influence those individuals remains open. Complex systems theory focuses on emerging patterns from deliberately simple individuals. Fields such as machine learning and cognitive science emphasize individual capabilities without considering the collective level much. To date, however, little work went into modeling the effects of changing and uncertain environments on emergent collective behavior from individually self-learning agents. To this end, we derive and present deterministic memory mean-field temporal-difference reinforcement learning dynamics where the agents only partially observe the actual state of the environment. This paper aims to obtain an efficient mathematical description of the emergent behavior of biologically plausible and parsimonious learning agents for the typical case of environmental and perceptual uncertainty. We showcase the broad applicability of our dynamics across different classes of agent-environment systems, highlight emergent effects caused by partial observability and show how our method enables the application of dynamical systems theory to partially observable multi-agent learning. The presented dynamics have the potential to become a formal yet practical, lightweight, and robust tool for researchers in biology, social science, and machine learning to systematically investigate the effects of interacting partially observant agents.

Voxel-based Soft Robots (VSRs) are a form of modular soft robots, composed of several deformable cubes, i.e., voxels. Each VSR is thus an ensemble of simple agents, namely the voxels, which must cooperate to give rise to the overall VSR behavior. Within this paradigm, collective intelligence plays a key role in enabling the emerge of coordination, as each voxel is independently controlled, exploiting only the local sensory information together with some knowledge passed from its direct neighbors (distributed or collective control). In this work, we propose a novel form of collective control, influenced by Neural Cellular Automata (NCA) and based on the bio-inspired Spiking Neural Networks: the embodied Spiking NCA (SNCA). We experiment with different variants of SNCA, and find them to be competitive with the state-of-the-art distributed controllers for the task of locomotion. In addition, our findings show significant improvement with respect to the baseline in terms of adaptability to unforeseen environmental changes, which could be a determining factor for physical practicability of VSRs.

The discovery of complex multicellular organism development took millions of years of evolution. The genome of such a multicellular organism guides the development of its body from a single cell, including its control system. Our goal is to imitate this natural process using a single neural cellular automaton (NCA) as a genome for modular robotic agents. In the introduced approach, called Neural Cellular Robot Substrate (NCRS), a single NCA guides the growth of a robot and the cellular activity which controls the robot during deployment. In this paper, NCRSs are trained with covariance matrix adaptation evolution strategy (CMA-ES), and covariance matrix adaptation MAP-Elites (CMA-ME) for quality diversity, which we show leads to more diverse robot morphologies with higher fitness scores. While the NCRS can solve the easier tasks from our benchmark environments, the success rate reduces when the difficulty of the task increases. We discuss directions for future work that may facilitate the use of the NCRS approach for more complex domains.

In this paper, we study the problem of networked multi-agent reinforcement learning (MARL), where a number of agents are deployed as a partially connected network and each interacts only with nearby agents. Networked MARL requires all agents make decisions in a decentralized manner to optimize a global objective with restricted communication between neighbors over the network. Inspired by the fact that sharing plays a key role in human's learning of cooperation, we propose LToS, a hierarchically decentralized MARL framework that enables agents to learn to dynamically share reward with neighbors so as to encourage agents to cooperate on the global objective through collectives. For each agent, the high-level policy learns how to share reward with neighbors to decompose the global objective, while the low-level policy learns to optimize local objective induced by the high-level policies in the neighborhood. The two policies form a bi-level optimization and learn alternately. We empirically demonstrate that LToS outperforms existing methods in both social dilemma and networked MARL scenario across scales.