[Submitted on 31 Aug 2023]
Download a PDF of the paper titled “Everything, Everywhere All in One Evaluation: Using Multiverse Analysis to Evaluate the Influence of Model Design Decisions on Algorithmic Fairness” by Jan Simson, Florian Pfisterer, and Christoph Kern here.
Abstract: Algorithmic decision-making (ADM) systems are widely used across various industries to automate decisions that were previously made by humans. While well-designed ADM systems promise more objective decisions and resource savings, poorly designed ones can lead to unfair decisions that discriminate against certain societal groups. To understand the fairness implications of design decisions in ADM systems, we propose a method called multiverse analysis. This method makes explicit the implicit design decisions and evaluates their impact on fairness. By combining different decision combinations, we create a grid of all possible “universes” and compute metrics of fairness and performance for each. Through a case study on predicting public health coverage, we demonstrate how multiverse analysis can provide insights into the variability and robustness of algorithmic fairness. Our findings highlight the surprising effects of design decisions on fairness and how multiverse analysis can help detect and understand these effects.