Geometry, Dynamics and Particles in Deep Learning

Recent years have seen the emergence of a new point of view on deep learning: neural networks are dynamical systems, and their basic constituents, neurons, features or tokens, behave as interacting particles. The idea appeared first for wide two-layer networks whose training becomes, as the number of neurons grows, a gradient flow in Wasserstein space, and most recently for Transformers, the architecture behind large language models. Indeed, the attention mechanism couples the tokens of a sequence to one another in a mean-field manner, so that a prompt passes through the layers as a system of interacting particles moving on a sphere, and the forward pass, which propagates and transforms the information carried by the input, is the evolution of this system. Information processing in deep neural networks thus falls squarely within the territory of interacting particle systems and kinetic theory, from the Boltzmann and Vlasov equations to mean-field limits and propagation of chaos, and of continuum mechanics, geometric flows and optimal transport.

Much has been established yet many questions remain, ranging from the rigorous derivation of the limiting equations and the study of their long-time behavior, to the role of the learned interaction and of the geometry of the underlying space, to the connection between the dynamics of the tokens and the computation that emerges from it. These questions require sophisticated mathematics in an area where theory lags far behind practice, and the researchers able to answer them currently have limited interaction. The purpose of the workshop is to bring them together, to expose each community to the questions and tools of the others, and to identify the key open problems.