Posts

DDPM as special case of variational autoencoders (VAE) Impose following structure on the VAE: Multiple latent variables, where each latent variable is associated with a timestep. Dim of all the latent variables is the same as the dim of the data. Assume a non learnable encoder unlike VAE where the encoder was learnable. Notations $X_1, …, X_T$: Denote latent variables at different timesteps $p(X_0)$: Denote the model data distribution $X_{1:T} = (X_1, X_2, …, X_T)$: Denote the sequence of all latent variables $q(X_{1:T}|X_0)$: Denote the latent posterior Latent posterior Because $q(X_{1:T}|X_0)$ is not learnable, we need to define it - we make it first order markovian chain with gaussian transitions: ...

We were using latent variables $X_t$ to model the data as: $$\log p_\theta(x_0) = \log \int p_\theta(x_{0:T}) dx_{1:T}$$ This formulation expresses the log-likelihood of the observed data $x_0$ as an integral over all possible latent variable sequences $x_{1:T}$. The model’s goal is to maximize this log-likelihood, which involves learning the parameters $\theta$ that define the generative process from $x_T$ back to $x_0$. Also: $p_\theta(x_T) = \mathcal{N}(0, I)$ - The prior distribution of the final latent variable is a standard normal distribution $p_\theta(x_{0:T}) = p_\theta(x_T) \prod_{t=1}^T p_\theta(x_{t-1}|x_t)$ - The joint distribution of all latent variables is factorized using the chain rule $q(x_t|x_{t-1}) = \mathcal{N}(x_t; \sqrt{\alpha_t}x_{t-1}, (1-\alpha_t)I)$ - The forward process transition probability is a Gaussian distribution $q(x_{1:T}|x_0) = \prod_{t=1}^T q(x_t|x_{t-1})$ - The posterior of all latent variables given the data is factorized using the chain rule Because the log likelihood is intractable, we constructed the ELBO of the model: ...

Recall that: $$T_3 = - \sum_{t=2}^T \mathbb{E}_{q(x_t|x_0)} [D_{KL}(q(x_{t-1}|x_t,x_0) \| p_\theta(x_{t-1}|x_t))]$$ Here: $q(x_{t-1}|x_t,x_0)$ is the noising distribution after applying the forward process for $t-1$ steps $p_\theta(x_{t-1}|x_t)$ is the model’s predicted denoising distribution after applying the reverse process We also know that: $$q(x_{t-1}|x_t,x_0) = \mathcal{N}(x_{t-1}; \mu_q(x_t,x_0), \Sigma_q(t))$$ where $\mu_q(x_t,x_0)$ and $\Sigma_q(t)$ are the mean and covariance of the posterior distribution and: $$ \mu_q(x_t, x_0) = \frac{\sqrt{\alpha_t}(1-\bar{\alpha}_{t-1})x_t + \sqrt{\bar{\alpha}_{t-1}}(1-\alpha_t)x_0}{1-\bar{\alpha}_t} $$ The covariance $\Sigma_q(t)$ is given by: ...

Note that: NCSN: Noise Conditional Score Network is same as SMLD: Score matching Langevin Dynamics Inference The inference processes also differ between SMLD and DDPM: SMLD (Noise Conditional Score Networks): We start with a sequence of noise scales arranged from largest to smallest: $${\sigma_i}_{i=1}^L, \quad \sigma_1 > \sigma_2 > … > \sigma_L \quad (L \approx 10)$$ Start with random Gaussian noise: $$x_0 \sim \mathcal{N}(0, \sigma_1^2I)$$ For each noise scale $\sigma_i$ from $i=1$ to $L$: ...

Teacher model is a large model that was trained on a large dataset as usual. Student model is a smaller model that is trained to mimic the behavior of the teacher model. The student model has 2 losses: $$ \mathcal{L} = \mathcal{L}_\text{task} + \mathcal{L}_\text{distill} $$ - $\mathcal{L}_\text{task}$ is the task-specific loss. - $\mathcal{L}_\text{distill}$ is the distillation loss - This loss is used to make the student model's output close to the teacher model's softened outputs.

Also known as Unsupervised domain adaptation (UDA) Introduction Domain adversarial networks (DANs) are a type of neural network architecture designed to address the problem of domain shift, where the distribution of data differs between the training and testing phases. DANs use adversarial training to learn domain-invariant features that are robust across different domains. Aim The aim is to learn a classifier that performs well on both the source and target domains, even though the target domain may have different characteristics than the source domain. For example, we train a model to classify medical images at company A and then we want to use this model to classify medical images at some other hospital. ...

Ensemble learning is a machine learning technique that combines multiple models to improve the overall performance and robustness of the prediction. It is based on the idea that by aggregating the predictions of several models, we can achieve better results than any single model alone.

Let the data be $D = {t_i}_{i=1}^N$ where $t_i \in \mathbb{R}^d$ are iid $p_t(t)$ data points. Latent variable models Let each data point $x_i$ be associated with a latent variable $z_i$ that is not observed. $z_i$ is a random variable that takes values in some finite set ${1, \ldots, K}$ and represents the membership of $x_i$ to one of $K$ clusters. Hence the data can be represented as $$D = \{(t_i, z_i)\}_{i=1}^N$$ where $t_i$ is the observed data and $z_i$ is the latent variable and $(t_i,z_i)$ are iid $p_{tz}$. The marginal distribution of the observed data is given by: ...

F-divergence is a generalized measure of difference between two probability distributions. For probability distributions $P$ and $Q$ over a event space $X$, the F-divergence is defined as: $$D_f(P || Q) = ∫_X q(x) f(\frac{p(x)}{q(x)}) dx$$ Where: Generator function $f: R^+ → R$ is a lower semi-continuous convex function with $f(1) = 0$. The F in F-divergence comes from generator “f"unction. $p(x)$ and $q(x)$ are the probability density functions of $P$ and $Q$ respectively Some examples of F-divergences include: ...

Why generative models cannot be evaluated like discriminative models Usually, we train a deep learning model on training data and then evaluate it on test data. However, for generative models, we cannot directly compare the generated samples to the test data in the same way we evaluate discriminative models. This is because: Generative models produce new, unique samples rather than predictions on existing data. There’s no one-to-one correspondence between generated samples and real data points. We need to assess the quality, diversity, and realism of the generated samples, which requires different metrics. Traditional evaluation metrics like accuracy or mean squared error are not applicable to generative tasks. Therefore, we need specialized methods to evaluate generative models, such as the Frechet Inception Distance (FID), which measures the similarity between the distribution of generated samples and the distribution of real data. ...