Prior Specification for Bayesian Matrix Factorization via Prior Predictive Matching
Eliezer de Souza da Silva, Tomasz Kuśmierczyk, Marcelo Hartmann, Arto Klami; 24(67):1−51, 2023.
Abstract
The choice of prior distributions in Bayesian models used in machine learning is crucial and often relies on hyperparameters selected through Bayesian optimization or cross-validation. However, this requires costly posterior inference. In this paper, we propose an alternative approach for selecting good priors without the need for posterior inference. We leverage the prior predictive distribution to estimate virtual statistics for data generated by the prior predictive distribution. Then, we optimize over the hyperparameters to find the ones that match the target values provided by the user or estimated from a subset of the observed data. We demonstrate the effectiveness of this approach in the context of probabilistic matrix factorization, a problem for which prior selection solutions have been missing. We analytically determine the hyperparameters, including the number of factors, that best replicate the target statistics for Poisson factorization models. We also investigate the sensitivity of this approach to model mismatch and present a model-independent procedure for determining hyperparameters in general models using stochastic optimization. We showcase this extension in the context of hierarchical matrix factorization models.
[abs]