(12/06/17) Understanding the Dynamics of Online Learning Algorithms via Scaling and Mean-Field Limits

December 6, 2017
In our recent paper, we present a tractable and asymptotically exact framework for analyzing the dynamics of online learning algorithms in the high-dimensional scaling limit. Our results are applied to two concrete examples: online regularized linear regression and principal component analysis. As the ambient dimension tends to infinity, and with proper time scaling, we show that the time-varying joint empirical measures of the target feature vector and its estimates provided by the algorithms will converge weakly to a deterministic measured-valued process that can be characterized as the unique solution of a nonlinear PDE.