Leaky Hockey Stick Loss: The First Negatively Divergent Margin-based Loss Function for Classification
Authors: Oh-Ran Kwon, Hui Zou; Journal of Machine Learning Research, 24(239):1−40, 2023.
Abstract
Many modern classification algorithms are formulated using the regularized empirical risk minimization (ERM) framework, where the risk is defined based on a loss function. However, the non-negativity of the loss function in ERM is not necessary for classification calibration and producing a Bayes consistent classifier. In this paper, we introduce the leaky hockey stick loss (LHS loss), which is the first negatively divergent margin-based loss function. We prove that the LHS loss is classification-calibrated. By replacing the hinge loss with the LHS loss in the ERM approach for deriving the kernel support vector machine (SVM), we obtain a well-defined solution called the kernel leaky hockey stick classifier (LHS classifier). Under mild regularity conditions, we prove that the kernel LHS classifier is Bayes risk consistent. Our theoretical analysis overcomes challenges caused by the negative divergence of the LHS loss, which is not present in the analysis of usual kernel machines. We also provide a computationally efficient algorithm to solve the kernel LHS classifier and compare it to the kernel SVM on simulated data and fifteen benchmark datasets. Additionally, we present a class of negatively divergent margin-based loss functions that have similar theoretical properties to the LHS loss. Interestingly, the LHS loss can be seen as a limiting case of this family of negatively divergent margin-based loss functions.
[abs]