Evaluation of Algorithm Portfolios using Item Response Theory
Authors: Sevvandi Kandanaarachchi, Kate Smith-Miles; Published in Journal of Machine Learning Research, Volume 24, Issue 177, 2023.
Abstract
Item Response Theory (IRT) is a method used in Educational Psychometrics to evaluate student ability, test question difficulty, and discrimination power. Recently, IRT has been applied to assess the performance of machine learning algorithms on classification datasets. In this paper, we propose a modified IRT-based framework that evaluates a portfolio of algorithms across multiple datasets while capturing additional characteristics such as algorithm consistency and anomalousness. These characteristics are derived from a novel reinterpretation of the traditional IRT model without the need for additional dataset feature computations. We demonstrate the applicability of this framework on algorithm portfolios for various applications, providing valuable insights into algorithm evaluation. The IRT parameters also offer an explainable nature, enhancing our understanding of algorithm portfolios.
[Abstract]