Generalization Bounds for Noisy Iterative Algorithms Using Properties of Additive Noise Channels
Hao Wang, Rui Gao, Flavio P. Calmon; 24(26):1−43, 2023.
Abstract
This paper examines the generalization of models trained by noisy iterative algorithms and analyzes their distribution-dependent generalization bounds. By establishing a connection between noisy iterative algorithms and additive noise channels in communication and information theory, the authors derive these generalization bounds. The bounds have implications for various applications, including differentially private stochastic gradient descent (DP-SGD), federated learning, and stochastic gradient Langevin dynamics (SGLD). The authors also provide numerical experiments to demonstrate the practicality of these bounds in understanding the generalization phenomena of neural networks.
[abs]