Variational Gibbs Inference for Statistical Model Estimation from Incomplete Data
Authors: Vaidotas Simkus, Benjamin Rhodes, Michael U. Gutmann; Journal of Machine Learning Research, 24(196):1−72, 2023.
Abstract
Statistical models play a crucial role in machine learning, finding applications in various downstream tasks. These models rely on free parameters that are typically estimated from data using maximum-likelihood estimation or its approximations. However, when working with real-world datasets, a common challenge arises: the models are formulated based on fully-observed data, while the available datasets often have missing data. The problem of estimating statistical models from incomplete data is conceptually similar to that of estimating latent-variable models, where techniques like variational inference (VI) are commonly used. However, unlike standard latent-variable models, parameter estimation with incomplete data often involves estimating exponentially-many conditional distributions of the missing variables, making conventional VI methods infeasible. To bridge this gap, we introduce variational Gibbs inference (VGI), a novel general-purpose method for estimating the parameters of statistical models from incomplete data. We validate the effectiveness of VGI on synthetic and real-world estimation tasks, demonstrating its capability to estimate important machine learning models such as variational autoencoders and normalizing flows from incomplete data. Despite being a general-purpose method, VGI achieves competitive or superior performance compared to existing model-specific estimation methods.
[abs]