Motivation We explored different sampling methods for generative models: Langevin dynamics (LE): $$x_{t+1} = x_t + \alpha \nabla_x \log p(x_t) + \sqrt{2\alpha}\epsilon, \quad \epsilon \sim \mathcal{N}(0,I)$$ DDPM (Denoising Diffusion Probabilistic Models): $$x_{t+1} = \sqrt{1-\beta_t}x_t + \sqrt{\beta_t}\epsilon, \quad \epsilon \sim \mathcal{N}(0,I)$$ SMLD (Score Matching with Langevin Dynamics): $$x_{t+1} = x_t + \sqrt{\sigma_{t+1}^2 - \sigma_t^2}\epsilon, \quad \epsilon \sim \mathcal{N}(0,I)$$ A natural question arises: What is the unifying mathematical framework that connects all these sampling methods? ...

Q-Learning is a powerful model-free reinforcement learning algorithm that learns the optimal action-value function directly without requiring a model of the environment. Understanding Model-Based vs. Model-Free Approaches Model-Based Reinforcement Learning A model of the environment in reinforcement learning refers to explicit knowledge of the system dynamics, specifically: State Transition Probabilities: $p_{ij}(u)$ - The probability of transitioning from state $i$ to state $j$ when taking action $u$ Reward/Cost Functions: $g(i,u,j)$ - The immediate reward or cost received when transitioning from state $i$ to state $j$ via action $u$ With a complete model, we can use dynamic programming methods like value iteration or policy iteration to compute optimal policies without direct environment interaction. ...

Sampling as optimization In probabilistic modeling, sampling can be viewed as an optimization problem where we aim to find the most probable sample from a probability distribution. This can be formulated mathematically as: $$x^* = \mathop{\arg \max}_{x} p(x)$$ Taking the log of the probability (which preserves the optimum due to monotonicity of log): $$x^* = \mathop{\arg \max}_{x} \log p(x)$$ And converting to a minimization problem: $$x^* = \mathop{\arg \min}_{x} - \log p(x)$$ ...

Recap $J_{ESM}(\theta) = \mathbb{E}_{p(x)} \left[ \frac{1}{2} | \hat{s}(x; \theta) - \nabla_x \log p(x) |^2 \right]$ $J_{ISM}(\theta) = \mathbb{E}_{p(x)} \left[ \frac{1}{2} | \hat{s}(x; \theta) |^2 + \text{tr}(\nabla_x \hat{s}(x; \theta)) \right] + C$ Theorem: $J_{ISM}(\theta) = J_{ESM}(\theta) + C$ $\theta^* = \mathop{\arg \min}_{\theta} J_{ISM}(\theta)$ $x_{t+1} = x_t + \alpha \cdot \hat{s}(x_t; \theta^*) + \sqrt{2\alpha} \epsilon, \quad \epsilon \sim \mathcal{N}(0, I)$ $\hat{s}(x; \theta)$ modelled using a neural network There is a problem with calculating the trace term The implicit score matching loss function, as we discussed earlier, is given by: ...

Recap Langevin equation: $x_{t+1} = x_t + \alpha \cdot \nabla_x \log p(x_t) + \sqrt{2\alpha} \epsilon, \quad \epsilon \sim \mathcal{N}(0, I)$ score function: $\nabla_x \log p(x) = s(x)$ $J_{ESM}(\theta) = \mathbb{E}_{p(x)} \left[ \frac{1}{2} | \hat{s}(x; \theta) - \nabla_x \log p(x) |^2 \right]$ $J_{ISM}(\theta) = \mathbb{E}_{p(x)} \left[ \frac{1}{2} | \hat{s}(x; \theta) |^2 + \text{tr}(\nabla_x \hat{s}(x; \theta)) \right] + C$ Theorem: $J_{ISM}(\theta) = J_{ESM}(\theta) + C$ $J_{PSM}(\theta) = \frac{1}{2} \mathbb{E}_v \mathbb{E}_{p(x)} \left[ \| v^\top \hat{s}_\theta(x) - v^\top s(x) \|^2 \right]$ $J_{SSM}(\theta) = \mathbb{E}_v \mathbb{E}_{p(x)} \left[ \frac{1}{2}(v^\top \hat{s}_\theta(x))^2 + v^\top (\nabla_x \hat{s}_\theta(x)) v \right]$ Theorem: $J_{PSM}(\theta) = J_{SSM}(\theta) + C$ Big picture plan Denoising score matching Denoising score matching (DSM) is an alternative approach to score matching. It introduces an auxiliary variable and works with conditional scores, which offers several advantages. ...

Let the data be $$D = {(x_i, y_i)}_{i=1}^N \text{ iid } \sim p(x, y).$$ here $x_i = { x_i^1, x_i^2, \ldots, x_i^k }$, where $x_j^i \in \mathbb{R}^d$ represents a sequence of $k$ vectors of dimension $d$.They are called tokens in usual NLP models. here $y_i = { y_i^1, y_i^2, \ldots, y_i^m }$, where $y_j^i \in \mathbb{R}^{d’}$ represents a sequence of $m$ vectors of dimension $d’$. Note that $d’ \neq d$. It represents a softmax distribution over a vocabulary of size $d’$ in NLP models. The models that map $x_i$ to $y_i$ are called seq2seq models. ...

Step 1 of big picture (Stochastic forward process to continuous SDE) We know SMLD (Score Matching with Langevin Dynamics) forward process is: $$x_i = x_{i-1} + \sqrt{\sigma_{i}^2 - \sigma_{i-1}^2}z_{i-1}, \quad z_{i-1} \sim \mathcal{N}(0,I)$$ where $\sigma_i$ is the noise level at step $i$, increasing with $i$. Let’s convert the discrete SMLD process to a continuous SDE: First, let’s write the discrete process in terms of continuous time: Let $\Delta t = \frac{1}{N}$ be the time step Map discrete index $i$ to continuous time $t$: $i = \frac{t}{N}$ Write $\sigma_i^2 = \sigma(t)^2$ for continuous noise schedule The discrete process becomes: $$x(t + \Delta t) = x(t) + \sqrt{\sigma(t + \Delta t)^2 - \sigma(t)^2}z_t$$ where $z_t \sim \mathcal{N}(0,I)$ Rearranging to get the change in x: $$x(t + \Delta t) - x(t) = \sqrt{\sigma(t + \Delta t)^2 - \sigma(t)^2}z_t$$ We use a first-order Taylor expansion to approximate the change in $\sigma(t)^2$: $$\sigma(t + \Delta t)^2 \approx \sigma(t)^2 + \frac{d(\sigma(t)^2)}{dt}\Delta t$$ Subtracting $\sigma(t)^2$ from both sides gives: $$\sigma(t + \Delta t)^2 - \sigma(t)^2 \approx \frac{d(\sigma(t)^2)}{dt}\Delta t$$ This linearizes the change in $\sigma(t)^2$ over the small interval $\Delta t$. Also, $z_t\sqrt{\Delta t} = dB_t$ for Brownian motion increments Substituting these we get: $$dx = \sqrt{\frac{d(\sigma(t)^2)}{dt}\Delta t}\frac{dB_t}{\sqrt{\Delta t}}$$ Simplifying: $$dx = \sqrt{\frac{d(\sigma(t)^2)}{dt}}dB_t$$ ...

Motivation for state space models RNNs can handle variable length inputs however they are not parallelizable. CNNs are parallelizable however they cannot handle variable length inputs. Can we get the best of both worlds? Linear State Space Models A linear state space model consists of two equations: $$h’(t) = Ah(t) + Bx(t)$$ $$y(t) = Ch(t) + Dx(t)$$ where: $h(t)$ is the hidden state at time $t$ $x(t)$ is the input at time $t$ $y(t)$ is the output at time $t$ $h’(t)$ is the derivative of the hidden state with respect to time $A, B, C, D$ are learnable parameter matrices that define the dynamics The first equation describes how the hidden state evolves over time based on: ...

Big picture of what we are going to do: Big picture of SDEs This topic comes under stochastic calculus, which deals with functions whose derivatives yield random vectors rather than deterministic points. In traditional calculus, when we take the derivative of a function at a point, we get a fixed value ie a scalar that represents the instantaneous rate of change. However, in stochastic calculus, the derivative at any point is a random variable. ...

Dataset is linearly separable Dataset $D = {(x_i, y_i)}_{i=1}^n$ where $x_i \in \mathbb{R}^d$ and $y_i \in {-1, 1}$ is said to be linearly separable if there exists a hyperplane that can separate the two classes of data points with zero training error. Mathematically, a dataset is linearly separable if there exist weights $w$ and bias $b$ such that: $$ w^T x_i + b > 1 \text{ for } y_i = 1 $$ $$ w^T x_i + b < -1 \text{ for } y_i = -1 $$ ...