Skip to yearly menu bar Skip to main content

In-Person Poster presentation / poster accept

A VAE for Transformers with Nonparametric Variational Information Bottleneck

James Henderson · Fabio Fehr

MH1-2-3-4 #44

Keywords: [ Bayesian nonparametrics ] [ VIB ] [ natural language ] [ transformers ] [ vae ] [ Deep Learning and representational learning ]


We propose a Variational AutoEncoder (VAE) for Transformers by developing a Variational Information Bottleneck (VIB) regulariser for Transformer embeddings. We formalise such attention-based representations as mixture distributions, and use Bayesian nonparametrics to develop a Nonparametric VIB (NVIB) for them. The variable number of mixture components supported by nonparametrics captures the variable number of vectors supported by attention, and exchangeable distributions from nonparametrics capture the permutation invariance of attention. Our Transformer VAE (NVAE) uses NVIB to regularise the information passing from the Transformer encoder to the Transformer decoder. Evaluations of a NVAE, trained on natural language text, demonstrate that NVIB can regularise the number of mixture components in the induced embedding whilst maintaining generation quality and reconstruction capacity.

Chat is not available.