Buffered Asynchronous SGD for Byzantine Learning
Yi-Rui Yang, Wu-Jun Li; 24(204):1−62, 2023.
Abstract
Distributed learning has become a popular area of research due to its wide range of applications in cluster-based large-scale learning, federated learning, edge computing, and more. Traditional distributed learning methods often assume no failure or attack, but in real-world scenarios, unexpected cases like communication failure or malicious attacks can occur. This has led to the emergence of Byzantine learning (BL), which focuses on distributed learning with failure or attack. While most existing BL methods are synchronous, they are impractical in cases with heterogeneous or offline workers. Asynchronous BL (ABL) is typically preferred in such scenarios. This paper introduces a novel method called buffered asynchronous stochastic gradient descent (BASGD) for ABL. BASGD is the first ABL method capable of resisting non-omniscient attacks without storing any instances on the server. Additionally, an improved variant of BASGD called BASGD with momentum (BASGDm) is proposed, which incorporates local momentum into BASGD. Compared to methods that require instance storage on the server, both BASGD and BASGDm have a broader range of applications. They are compatible with various aggregation rules and have been proven to be convergent and resistant to failure or attack. Empirical results demonstrate that our methods outperform existing ABL baselines significantly when workers experience failure or attack.
[abs]