Variational Autoencoder (VAE)

What It Is

A generative model that learns to encode data into a continuous latent distribution $q(z|x)$ and decode samples from that distribution back to data — training both encoder and decoder jointly with a reconstruction loss plus a KL divergence regularizer that keeps the latent space smooth and contiguous.

Why It Matters

VAEs are the compression backbone of Latent Diffusion Models. The VQ-VAE (vector-quantized variant) in Stable Diffusion compresses images to 4-8x smaller latent representations. Diffusion then operates in this compressed space, cutting training and inference cost dramatically while preserving perceptual quality.

How It Works

The encoder outputs a mean $\mu$ and variance ${\sigma }^2$ rather than a single point: $q_{\varphi }(z|x)=\mathcal{N}({\mu }_{\varphi }(x),{\sigma }_{\varphi }^2(x))$. The training objective is the Evidence Lower BOund (ELBO):

$\mathcal{L}={\mathbb{E}}_{q_{\varphi }(z|x)}\left[\log p_{\theta }(x|z)\right]-D_{KL}(q_{\varphi }(z|x)\Vert p(z))$

The reconstruction term pushes the decoder to faithfully reproduce inputs. The KL term regularizes the latent space toward a standard normal $\mathcal{N}(0,I)$, ensuring smooth interpolation and valid samples from random $z\sim \mathcal{N}(0,I)$.

Key Sources

• latent-diffusion-models-high-resolution-image-synthesis

Related Concepts

• latent-space
• diffusion-models