Speaker
Deoliveira, McGill University
Abstract
In this work, under suitable assumptions, we prove the occurrence of the Weak-to-Strong generalization phenomenon, where a strong model learns from labels generated by a weak model and still outperforms it. We study this phenomenon in a two-layer random features model, where the model strength is determined by the number of neurons. Using tools from random matrix theory, in particular deterministic equivalents obtained through a self-consistent Dyson equation, we characterize the population errors of the weak teacher and the strong student. This allows us to find the exact exponent characterizing how the strong model’s population error improves relative to the weak model’s error for linear and ReLU activation functions and polynomial target functions. This is a joint work with Elliot Paquette.