From Classification Accuracy to Proper Scoring Rules: Elicitability of Probabilistic Top List Predictions
Johannes Resin; 24(173):1−21, 2023.
Abstract
In the field of forecasting, the importance of probabilistic assessments has been widely acknowledged to account for uncertainty. However, when it comes to classification, the evaluation of classifiers often focuses on single-class predictions using simple accuracy measures, disregarding any probabilistic uncertainty quantification. This article introduces probabilistic top lists as a new type of prediction in classification, bridging the gap between single-class predictions and predictive distributions. The elicitation of the probabilistic top list functional is achieved through the use of strictly consistent evaluation metrics. These evaluation metrics are based on symmetric proper scoring rules and enable comparisons among different types of predictions, ranging from single-class point predictions to fully specified predictive distributions. The Brier score is particularly suitable for this type of comparison.
[abs]