Physics Colloquium: The role of a layer in deep learning

Abstract:

Deep artificial neural networks (DNNs) have been driving many of the recent advancements in machine learning. An important question on the theory side of DNNs concerns the role played by each layer in the network. Recently two bold conjectures were made: The first is that DNNs learn to perform a series of Renormalization-Group (RG) transformations on the data they are given. The second claims that each subsequent layer in a DNN increases more and more a certain conditional-entropy. In this talk, I’ll discuss some tests and refinements of these two conjectures. In particular, I’ll present an information-theory based formulation of real-space RG and compare it with more conventional training algorithms for DNNs. Time permitting I’ll also discuss the training of DNNs using the above conditional-entropy based goal.

Event Organizer: Dr. Liron Barak