Variational flows have been successful in learning complex continuous distributions, but there is still a challenge in approximating discrete distributions. Current methods often involve embedding the discrete target in a continuous space, but this approach has limitations. It can result in a surrogate target that does not accurately represent the original discrete target, and it may have biased or unstable gradients. Additionally, optimizing these continuous-embedding flows can be difficult.
To address these issues, we propose a new approach for handling discrete distributions without any continuous embedding. We introduce a measure-preserving and discrete (MAD) invertible map that preserves the discrete target, ensuring that the characteristics of the original distribution are maintained. Based on this map, we develop a mixed variational flow called MAD Mix. This flow allows for more reliable approximations of discrete distributions compared to continuous-embedding flows, and it can be trained significantly faster.
Furthermore, we extend MAD Mix to handle joint models that incorporate both discrete and continuous variables. This enables us to model more complex distributions that involve a combination of discrete and continuous features.
Our experimental results demonstrate the effectiveness of MAD Mix in producing reliable approximations of discrete distributions. Compared to continuous-embedding flows, MAD Mix achieves superior performance while also offering faster training times.