The focus of this paper is on continual learning (CL), which involves learning new tasks without forgetting previous tasks. However, existing CL methods require a large amount of raw data, which is often unavailable due to copyright and privacy concerns. Instead, stakeholders typically provide pre-trained machine learning models as a service (MLaaS) through APIs. This paper introduces two new practical CL settings: data-efficient CL (DECL-APIs) and data-free CL (DFCL-APIs), which enable CL using APIs with limited or no raw data. However, performing CL under these settings presents challenges such as unavailable raw data, unknown model parameters, diverse models of varying architectures and scales, and the issue of catastrophic forgetting of previous APIs.
To address these challenges, the paper proposes a novel data-free cooperative continual distillation learning framework. This framework distills knowledge from a stream of APIs into a CL model by generating pseudo data through API queries. The framework consists of two cooperative generators and one CL model, trained as an adversarial game. Firstly, the CL model and the current API serve as fixed discriminators to train the generators using a derivative-free method. The generators aim to generate hard and diverse synthetic data that maximizes the difference in response between the CL model and the API. Secondly, the CL model is trained by minimizing the response gap between itself and the black-box API on the synthetic data, allowing the transfer of knowledge from the API to the CL model. Additionally, a regularization term based on network similarity is proposed to prevent catastrophic forgetting of previous APIs.
Experimental results show that our method performs comparably to classic CL with full raw data on the MNIST and SVHN datasets in the DFCL-APIs setting. In the DECL-APIs setting, our method achieves performance levels of 0.97x, 0.75x, and 0.69x compared to classic CL on the CIFAR10, CIFAR100, and MiniImageNet datasets, respectively.