Memory-Based Optimization Methods for Model-Agnostic Meta-Learning and Personalized Federated Learning
Bokun Wang, Zhuoning Yuan, Yiming Ying, Tianbao Yang; 24(145):1−46, 2023.
Abstract
The field of model-agnostic meta-learning (MAML) has gained popularity in recent years. However, the stochastic optimization of MAML is still not fully developed. Existing MAML algorithms rely on the concept of “episodes,” where a few tasks and data points are sampled to update the meta-model in each iteration. However, these algorithms either lack convergence guarantee with a constant mini-batch size or require processing a large number of tasks in every iteration, which is not suitable for continual learning or cross-device federated learning scenarios where only a small number of tasks are available per iteration or per round. To address these issues, this paper proposes memory-based stochastic algorithms for MAML that converge with vanishing error. These algorithms only require sampling a constant number of tasks and data samples per iteration, making them suitable for continual learning scenarios. Additionally, the paper introduces a communication-efficient memory-based MAML algorithm for personalized federated learning in cross-device (with client sampling) and cross-silo (without client sampling) settings. The proposed algorithms are supported by our theoretical analysis, which improves the optimization theory for MAML, and our empirical results confirm the validity of our theoretical findings.
[abs]