Learning a Predictive Function for Hyper-parameters Conditioned on Tasks
Authors: Jun Shu, Deyu Meng, Zongben Xu; Published in 2023, Volume 24, Issue 186, Pages 1-74.
Abstract
Recently, meta learning has gained significant attention in the machine learning community. While conventional machine learning focuses on learning prediction rules to predict labels for new query data, meta learning aims to learn the learning methodology for machine learning from observed tasks in order to generalize to new query tasks by leveraging the meta-learned learning methodology. In this study, we propose a learning methodology called Simulating Learning Methodology (SLeM) by learning an explicit hyper-parameter prediction function that is shared by all training tasks. This function, referred to as the meta-learner, is a parameterized function that maps a training/test task to its suitable hyper-parameter setting, extracted from a pre-specified function set known as the meta learning machine. The use of this setting ensures that the meta-learned learning methodology can adapt to diverse query tasks, rather than relying on fixed hyper-parameters as seen in many current meta learning methods, which have limited adaptability to variations in query tasks. The understanding of meta learning presented in this study also allows for the application of traditional learning theory to analyze its generalization bounds with different losses, tasks, and models. The theory naturally leads to feasible strategies for controlling and improving the quality of the extracted meta-learner. These strategies have been verified to enhance the generalization capability of the meta-learner in various meta learning applications, including few-shot regression, few-shot classification, and domain generalization. The source code of our method is available at https://github.com/xjtushujun/SLeM-Theory.
[abs]