Posts

Regularization is a technique to increase model bias to reduce variance by constraining empirical risk minimization. regularized empirical risk minimization: $$\text{Reg } ERM = \min_{h \in H} \hat{R}(h_\theta) \quad \text{s.t. } \Omega(h_\theta) < k$$ here $\Omega(h_\theta) < k$ is the regularization function which is a design choice. This can be solved using the method of Lagrange multipliers. $h(x) = \arg\min_{h_\theta \in H} \left( \hat{R}(h_\theta) + \lambda \Omega(\theta) \right)$ here $\lambda$ is the regularization parameter which is a design choice. ...

Recall that the loss function $L(y_i, \hat{y_i})$ measures the error between the predicted label and the true label for a single data point. The risk function $R(h)$ is the expected loss over all data points in the dataset. It is defined as: $$R(h) = \mathbb{E}_{(x,y) \sim P}[L(y, h(x))]$$ where $P$ is the true data distribution. Conditional risk The conditional risk $R(h|x)$ is the expected loss for a given input $x$. It is defined as: $$R(h(x)|x) = \mathbb{E}_{y \sim P(y|x)}[L(y, h(x))]$$ ...

Sampling as optimization In probabilistic modeling, sampling can be viewed as an optimization problem where we aim to find the most probable sample from a probability distribution. This can be formulated mathematically as: $$x^* = \mathop{\arg \max}_{x} p(x)$$ Taking the log of the probability (which preserves the optimum due to monotonicity of log): $$x^* = \mathop{\arg \max}_{x} \log p(x)$$ And converting to a minimization problem: $$x^* = \mathop{\arg \min}_{x} - \log p(x)$$ ...

Recap $J_{ESM}(\theta) = \mathbb{E}_{p(x)} \left[ \frac{1}{2} | \hat{s}(x; \theta) - \nabla_x \log p(x) |^2 \right]$ $J_{ISM}(\theta) = \mathbb{E}_{p(x)} \left[ \frac{1}{2} | \hat{s}(x; \theta) |^2 + \text{tr}(\nabla_x \hat{s}(x; \theta)) \right] + C$ Theorem: $J_{ISM}(\theta) = J_{ESM}(\theta) + C$ $\theta^* = \mathop{\arg \min}_{\theta} J_{ISM}(\theta)$ $x_{t+1} = x_t + \alpha \cdot \hat{s}(x_t; \theta^*) + \sqrt{2\alpha} \epsilon, \quad \epsilon \sim \mathcal{N}(0, I)$ $\hat{s}(x; \theta)$ modelled using a neural network There is a problem with calculating the trace term The implicit score matching loss function, as we discussed earlier, is given by: ...

Recap Langevin equation: $x_{t+1} = x_t + \alpha \cdot \nabla_x \log p(x_t) + \sqrt{2\alpha} \epsilon, \quad \epsilon \sim \mathcal{N}(0, I)$ score function: $\nabla_x \log p(x) = s(x)$ $J_{ESM}(\theta) = \mathbb{E}_{p(x)} \left[ \frac{1}{2} | \hat{s}(x; \theta) - \nabla_x \log p(x) |^2 \right]$ $J_{ISM}(\theta) = \mathbb{E}_{p(x)} \left[ \frac{1}{2} | \hat{s}(x; \theta) |^2 + \text{tr}(\nabla_x \hat{s}(x; \theta)) \right] + C$ Theorem: $J_{ISM}(\theta) = J_{ESM}(\theta) + C$ $J_{PSM}(\theta) = \frac{1}{2} \mathbb{E}_v \mathbb{E}_{p(x)} \left[ \| v^\top \hat{s}_\theta(x) - v^\top s(x) \|^2 \right]$ $J_{SSM}(\theta) = \mathbb{E}_v \mathbb{E}_{p(x)} \left[ \frac{1}{2}(v^\top \hat{s}_\theta(x))^2 + v^\top (\nabla_x \hat{s}_\theta(x)) v \right]$ Theorem: $J_{PSM}(\theta) = J_{SSM}(\theta) + C$ Big picture plan Denoising score matching Denoising score matching (DSM) is an alternative approach to score matching. It introduces an auxiliary variable and works with conditional scores, which offers several advantages. ...

Let the data be $$D = {(x_i, y_i)}_{i=1}^N \text{ iid } \sim p(x, y).$$ here $x_i = { x_i^1, x_i^2, \ldots, x_i^k }$, where $x_j^i \in \mathbb{R}^d$ represents a sequence of $k$ vectors of dimension $d$.They are called tokens in usual NLP models. here $y_i = { y_i^1, y_i^2, \ldots, y_i^m }$, where $y_j^i \in \mathbb{R}^{d’}$ represents a sequence of $m$ vectors of dimension $d’$. Note that $d’ \neq d$. It represents a softmax distribution over a vocabulary of size $d’$ in NLP models. The models that map $x_i$ to $y_i$ are called seq2seq models. ...

Step 1 of big picture (Stochastic forward process to continuous SDE) We know SMLD (Score Matching with Langevin Dynamics) forward process is: $$x_i = x_{i-1} + \sqrt{\sigma_{i}^2 - \sigma_{i-1}^2}z_{i-1}, \quad z_{i-1} \sim \mathcal{N}(0,I)$$ where $\sigma_i$ is the noise level at step $i$, increasing with $i$. Let’s convert the discrete SMLD process to a continuous SDE: First, let’s write the discrete process in terms of continuous time: Let $\Delta t = \frac{1}{N}$ be the time step Map discrete index $i$ to continuous time $t$: $i = \frac{t}{N}$ Write $\sigma_i^2 = \sigma(t)^2$ for continuous noise schedule The discrete process becomes: $$x(t + \Delta t) = x(t) + \sqrt{\sigma(t + \Delta t)^2 - \sigma(t)^2}z_t$$ where $z_t \sim \mathcal{N}(0,I)$ ...

Let the data be D = {($x_1$, $y_1$), ($x_2$, $y_2$), …, ($x_n$, $y_n$)} where $y_i \in {1, 2, …, K}$ is the multiclass label of the feature vector $x_i$, and $K$ is the number of classes. Then the softmax regression model is defined as: $$P(y_i = k | x_i) = \frac{e^{w_k^T x_i + b_k}}{\sum_{j=1}^K e^{w_j^T x_i + b_j}}$$ where $w_k$ is the parameter vector for class $k$, and $b_k$ is the bias term for class $k$. ...

Example 1 - convert continuous SDE to discrete soln without drift term The continuous stochastic differential equation (SDE) is given by: $$dx = \epsilon \sqrt{dt} \quad // \text{cont eqn}$$ where $\epsilon \sim \mathcal{N}(0,1)$. To convert this continuous SDE to a discrete solution, we consider the increments: $$X_{t_{i+1}} - X_{t_i} = \sqrt{t_{i+1} - t_i} \cdot \epsilon$$ Assuming $t$ were not discrete, we can express it as: $$X_{t_{i+1}} = X_{t_i} + \sqrt{t_{i+1} - t_i} \cdot \epsilon$$ ...

Motivation for state space models RNNs can handle variable length inputs however they are not parallelizable. CNNs are parallelizable however they cannot handle variable length inputs. Can we get the best of both worlds? Linear State Space Models A linear state space model consists of two equations: $$h’(t) = Ah(t) + Bx(t)$$ $$y(t) = Ch(t) + Dx(t)$$ where: $h(t)$ is the hidden state at time $t$ $x(t)$ is the input at time $t$ $y(t)$ is the output at time $t$ $h’(t)$ is the derivative of the hidden state with respect to time $A, B, C, D$ are learnable parameter matrices that define the dynamics The first equation describes how the hidden state evolves over time based on: ...