DIME: Deterministic Information Maximizing Autoencoder

Variational autoencoders (VAEs) offer a theoretically sound and popular framework for deep generative models. However, learning a VAE from data presents unresolved theoretical questions and significant practical challenges. (i) It has been observed that the learned decoder distribution tends to be the same for all points in the latent space, implying that the latent space is not dependent on data space. This results in a poor latent representation of data. (ii) Additionally, due to the stochastic nature of VAE's decoder, it tends to produce blurry images that do not align well with the real data distribution, resulting in high FID scores. In this work, we propose a deterministic approach that addresses the limitations of traditional VAEs by learning a more informative latent space. Our method leverages a von-Mises Fisher (vMF) family-based kernel to regularize hyperspherical latent spaces in simple deterministic autoencoders. This regularization can be interpreted as maximizing the mutual information between the data and the latent space, leading to a more informative representation. We investigate how this regularization can create a better and more meaningful latent space than traditional VAE. In a rigorous empirical study, we show that our proposed model can generate samples that are comparable to, or better than, those of VAEs and other state-of-the-art autoencoders when applied to images as well as other challenging data such as equations.