绿帽社

鈥�Scaling Effects and Efficiency in Deep Learning Algorithms鈥�

Abstract: We are interested in the design of provably-efficient (both statistically and computationally) deep learning algorithms. The cornerstone of our approach is to view the neural network output as a stochastic process, i.e., a random object evolving over time. In this talk, I will present our recent work on studying the effect of scaling in the statistical behavior (such as bias and variance) of deep neural networks as well as on the test accuracy and generalization properties of the given architecture. We find that in terms of variance reduction of the neural network鈥檚 output and increased test accuracy the best choice is to choose the scaling of the hidden layers to correspond to the so-called mean-field scaling. We also find that this is particularly true for the outer layer, in that the neural network鈥檚 behavior is more sensitive to the scaling of the outer layer as opposed to the scaling of the inner layers. The mechanism for the mathematical analysis is an asymptotic expansion for the neural network鈥檚 output yielding a clean bias-variance decomposition. An important practical consequence of the analysis is that it provides a systematic and mathematically informed way to choose hyperparameters (such as learning rates) and to compare different architectures leading to provably-efficient and stable algorithms. Such choices guarantee that the resulting neural network architectures behave in statistically robust ways during training. I will discuss applications of these ideas to the design of deep learning algorithms for scientific problems including solving high dimensional partial differential equations (PDEs), closure of dynamical systems (like PDEs, moment equations etc.) and reinforcement learning with applications to financial engineering, turbulence, computational biology. I will conclude by presenting an overview of our research program, aimed at advancing foundational understanding to develop efficient and reliable artificial intelligence.

听