Physics Colloquium: Exploring the properties of neural networks with Statistical Mechanics

Dr. Yohai Bar-Sinai, TAU

04 August 2024, 14:00 
Shenkar Building, Melamed Hall 006 
Physics Colloquium

 

Zoom: https://tau-ac-il.zoom.us/j/86547186243

 

Abstract: 

The training of neural networks is a complex, high-dimensional, non-convex and noisy optimization problem whose theoretical understanding is interesting both from an applicative perspective and for fundamental reasons. A core challenge is to understand the geometry and topography of the landscape that guides the optimization. I will present two projects in which we employ Statistical Mechanics methods to study this landscape. First, we show that using Langevin dynamics we can explore the neural landscape of a network performing classification tasks. Analyzing the fluctuation statistics, in analogy to thermal dynamics at a constant temperature, we infer a clear geometric description of the low-loss region. We find that it is a low-dimensional manifold whose dimension can be readily obtained from the fluctuations. Furthermore, this dimension is controlled by the number of data points that reside near the classification decision boundary, and its structure fundamentally defies a quadratic approximation, due to the exponential nature of the decision boundary and the flatness of the low-loss region. Second, I will show how the framework of critical collective behavior can explain a recently discovered phenomenon termed “grokking”, in which a model learns to generalize long after it has overfit the training data. Using general Stat Mech reasoning and tools from Random Matrix Theory, we demonstrate that delayed generalization may be simply explained in terms of "critical slowing down". That is, a dilation of time scales which is generically expected at the vicinity of critical points, which in this context are non analytical points in the long time limit of training dynamics.

 

 

 

 

 

Event Organizer: Dr. Yohai Bar Sinai

 

Recordings of Past Colloquium >

 

Tel Aviv University makes every effort to respect copyright. If you own copyright to the content contained
here and / or the use of such content is in your opinion infringing, Contact us as soon as possible >>