On Tilted Losses in Machine Learning: Theory and Applications
Tian Li, Ahmad Beirami, Maziar Sanjabi, Virginia Smith; 24(142):1−79, 2023.
Abstract
Exponential tilting is a technique commonly employed in statistics, probability, information theory, and optimization to introduce parametric distribution shifts. Despite its wide use in related fields, tilting has not been widely adopted in machine learning. This study aims to bridge this gap by exploring the application of tilting in risk minimization. We examine an extension to empirical risk minimization (ERM) called tilted empirical risk minimization (TERM), which utilizes exponential tilting to flexibly adjust the impact of individual losses. The resulting framework offers several valuable properties: TERM can increase or decrease the influence of outliers, enabling fairness or robustness respectively; it possesses variance-reduction properties that contribute to generalization; and it can be seen as a smooth approximation to the tail probability of losses. This work establishes connections between TERM and related objectives such as Value-at-Risk, Conditional Value-at-Risk, and distributionally robust optimization (DRO). We develop batch and stochastic first-order optimization methods for solving TERM, provide convergence guarantees for the solvers, and demonstrate that the framework can be efficiently solved compared to common alternatives. Additionally, we showcase the versatility of TERM in various machine learning applications, including subgroup fairness enforcement, outlier mitigation, and handling class imbalance. Despite the straightforward modification that TERM introduces to traditional ERM objectives, we find that the framework consistently outperforms ERM and achieves competitive performance with state-of-the-art, problem-specific approaches.
[abs]
[code]