Physics Colloquium: Exploring the properties of neural networks with Statistical Mechanics

Zoom: https://tau-ac-il.zoom.us/j/86547186243

Abstract: 

The training of neural networks is a complex, high-dimensional, non-convex and noisy optimization problem whose theoretical understanding is interesting both from an applicative perspective and for fundamental reasons. A core challenge is to understand the geometry and topography of the landscape that guides the optimization. I will present two projects in which we employ Statistical Mechanics methods to study this landscape. First, we show that using Langevin dynamics we can explore the neural landscape of a network performing classification tasks. Analyzing the fluctuation statistics, in analogy to thermal dynamics at a constant temperature, we infer a clear geometric description of the low-loss region. We find that it is a low-dimensional manifold whose dimension can be readily obtained from the fluctuations. Furthermore, this dimension is controlled by the number of data points that reside near the classification decision boundary, and its structure fundamentally defies a quadratic approximation, due to the exponential nature of the decision boundary and the flatness of the low-loss region. Second, I will show how the framework of critical collective behavior can explain a recently discovered phenomenon termed “grokking”, in which a model learns to generalize long after it has overfit the training data. Using general Stat Mech reasoning and tools from Random Matrix Theory, we demonstrate that delayed generalization may be simply explained in terms of "critical slowing down". That is, a dilation of time scales which is generically expected at the vicinity of critical points, which in this context are non analytical points in the long time limit of training dynamics.

Event Organizer: Dr. Yohai Bar Sinai

Recordings of Past Colloquium >