GANs What is the difference between generative model and discriminative models KL Divergence - Kullback-Leibler Divergence F-Divergence Convex conjugate (also known as the Fenchel conjugate or the Legendre transform) GANs Naive GANs Wasserstein metric Wasserstein GANs Frechet Inception Distance (FID) Conditional GANs(cGANs) Domain adversarial networks VAEs Variational encoding(VAEs) Aggregated posterior mismatch VAE as regularized autoencoder and Beta VAEs Info VAEs DDPM Denoising Diffusion Probabilistic Models (DDPM) - part 1 Denoising Diffusion Probabilistic Models (DDPM) - part 2 Denoising Diffusion Probabilistic Models (DDPM) - part 3 Score matching Score matching part 1 Score matching part 2 Score matching part 3 Difference between SMLD and DDPM Conditional score matching SDEs Precursor to Stochastic Differential Equations (SDEs) Stochastic Differential Equations (SDEs) DDPMs as SDE SMLD as SDE DDIM Sequence2sequence models Transformers State space models & Mamba Self supervised learning Noise contrastive estimation JEPA - Joint Embedding Prediction and Autoencoding Misc Cheatsheet Cheatsheet

DDPM as special case of variational autoencoders (VAE) Impose following structure on the VAE: Multiple latent variables, where each latent variable is associated with a timestep. Dim of all the latent variables is the same as the dim of the data. Assume a non learnable encoder unlike VAE where the encoder was learnable. Notations $X_1, …, X_T$: Denote latent variables at different timesteps $p(X_0)$: Denote the model data distribution $X_{1:T} = (X_1, X_2, …, X_T)$: Denote the sequence of all latent variables $q(X_{1:T}|X_0)$: Denote the latent posterior Latent posterior Because $q(X_{1:T}|X_0)$ is not learnable, we need to define it - we make it first order markovian chain with gaussian transitions: ...

We were using latent variables $X_t$ to model the data as: $$\log p_\theta(x_0) = \log \int p_\theta(x_{0:T}) dx_{1:T}$$ This formulation expresses the log-likelihood of the observed data $x_0$ as an integral over all possible latent variable sequences $x_{1:T}$. The model’s goal is to maximize this log-likelihood, which involves learning the parameters $\theta$ that define the generative process from $x_T$ back to $x_0$. Also: $p_\theta(x_T) = \mathcal{N}(0, I)$ - The prior distribution of the final latent variable is a standard normal distribution $p_\theta(x_{0:T}) = p_\theta(x_T) \prod_{t=1}^T p_\theta(x_{t-1}|x_t)$ - The joint distribution of all latent variables is factorized using the chain rule $q(x_t|x_{t-1}) = \mathcal{N}(x_t; \sqrt{\alpha_t}x_{t-1}, (1-\alpha_t)I)$ - The forward process transition probability is a Gaussian distribution $q(x_{1:T}|x_0) = \prod_{t=1}^T q(x_t|x_{t-1})$ - The posterior of all latent variables given the data is factorized using the chain rule Because the log likelihood is intractable, we constructed the ELBO of the model: ...

Recall that: $$T_3 = - \sum_{t=2}^T \mathbb{E}_{q(x_t|x_0)} [D_{KL}(q(x_{t-1}|x_t,x_0) \| p_\theta(x_{t-1}|x_t))]$$ Here: $q(x_{t-1}|x_t,x_0)$ is the noising distribution after applying the forward process for $t-1$ steps $p_\theta(x_{t-1}|x_t)$ is the model’s predicted denoising distribution after applying the reverse process We also know that: $$q(x_{t-1}|x_t,x_0) = \mathcal{N}(x_{t-1}; \mu_q(x_t,x_0), \Sigma_q(t))$$ where $\mu_q(x_t,x_0)$ and $\Sigma_q(t)$ are the mean and covariance of the posterior distribution and: $$ \mu_q(x_t, x_0) = \frac{\sqrt{\alpha_t}(1-\bar{\alpha}_{t-1})x_t + \sqrt{\bar{\alpha}_{t-1}}(1-\alpha_t)x_0}{1-\bar{\alpha}_t} $$ The covariance $\Sigma_q(t)$ is given by: ...

Note that: NCSN: Noise Conditional Score Network is same as SMLD: Score matching Langevin Dynamics Inference The inference processes also differ between SMLD and DDPM: SMLD (Noise Conditional Score Networks): We start with a sequence of noise scales arranged from largest to smallest: $${\sigma_i}_{i=1}^L, \quad \sigma_1 > \sigma_2 > … > \sigma_L \quad (L \approx 10)$$ Start with random Gaussian noise: $$x_0 \sim \mathcal{N}(0, \sigma_1^2I)$$ For each noise scale $\sigma_i$ from $i=1$ to $L$: Perform $K$ Langevin steps ($K \approx 100$) at fixed $\sigma_i$: $$x_{k+1} = x_k + \alpha s_\theta(x_k, \sigma_i) + \sqrt{2\alpha}\epsilon_k, \quad \epsilon_k \sim \mathcal{N}(0,I)$$ for $k = 0,1,…,K-1$ Use final sample $x_K$ as starting point for next scale $\sigma_{i+1}$ This gradually denoises the image by: Getting a good sample at high noise levels (large $\sigma$) Using that sample as initialization for next lower noise level Repeating until reaching smallest noise scale $\sigma_L$ DDPM (Denoising Diffusion Probabilistic Models): Uses a deterministic reverse process Starting from random Gaussian noise, applies: $$x_{t-1} = \frac{1}{\sqrt{\alpha_t}}\left(x_t - \frac{\beta_t}{\sqrt{1-\bar{\alpha}_t}}\epsilon_\theta(x_t,t)\right) + \sigma_t z$$ Where: $\alpha_t = 1 - \beta_t$ $\bar{\alpha}_t = \prod_{s=1}^t \alpha_s$ $\sigma_t^2 = \beta_t$ (simplified version) $z \sim \mathcal{N}(0,I)$ One step per noise level Follows the reverse Markov chain exactly ...

Also known as Unsupervised domain adaptation (UDA) Introduction Domain adversarial networks (DANs) are a type of neural network architecture designed to address the problem of domain shift, where the distribution of data differs between the training and testing phases. DANs use adversarial training to learn domain-invariant features that are robust across different domains. Aim The aim is to learn a classifier that performs well on both the source and target domains, even though the target domain may have different characteristics than the source domain. For example, we train a model to classify medical images at company A and then we want to use this model to classify medical images at some other hospital. ...

Let the data be $D = {t_i}_{i=1}^N$ where $t_i \in \mathbb{R}^d$ are iid $p_t(t)$ data points. Latent variable models Let each data point $x_i$ be associated with a latent variable $z_i$ that is not observed. $z_i$ is a random variable that takes values in some finite set ${1, \ldots, K}$ and represents the membership of $x_i$ to one of $K$ clusters. Hence the data can be represented as $$D = \{(t_i, z_i)\}_{i=1}^N$$ where $t_i$ is the observed data and $z_i$ is the latent variable and $(t_i,z_i)$ are iid $p_{tz}$. The marginal distribution of the observed data is given by: ...

F-divergence is a generalized measure of difference between two probability distributions. For probability distributions $P$ and $Q$ over a event space $X$, the F-divergence is defined as: $$D_f(P || Q) = ∫_X q(x) f(\frac{p(x)}{q(x)}) dx$$ Where: Generator function $f: R^+ → R$ is a lower semi-continuous convex function with $f(1) = 0$. The F in F-divergence comes from generator “f"unction. $p(x)$ and $q(x)$ are the probability density functions of $P$ and $Q$ respectively Some examples of F-divergences include: ...

Why generative models cannot be evaluated like discriminative models Usually, we train a deep learning model on training data and then evaluate it on test data. However, for generative models, we cannot directly compare the generated samples to the test data in the same way we evaluate discriminative models. This is because: Generative models produce new, unique samples rather than predictions on existing data. There’s no one-to-one correspondence between generated samples and real data points. We need to assess the quality, diversity, and realism of the generated samples, which requires different metrics. Traditional evaluation metrics like accuracy or mean squared error are not applicable to generative tasks. Therefore, we need specialized methods to evaluate generative models, such as the Frechet Inception Distance (FID), which measures the similarity between the distribution of generated samples and the distribution of real data. ...

Recall that generative models try to learn the data distribution $P_{data}$. GANs try to learn $P_{data}$ by approximating it by $P_{generator}$ by minimizing F-divergence $D_f(P_{data} || P_{generator})$. Derivation of GANs Let $\theta$ be the parameters of the generator such that it maps a sample $z \sim N(0, I)$ to a sample $x \sim p_{generator}$. Then, the optimal parameters for the generator can be expressed as: $$\theta^* = \underset{\theta}{\text{argmin}} D_f(P_{data} || P_{generator})$$ ...