Big picture of what we are going to do: Big picture of SDEs This topic comes under stochastic calculus, which deals with functions whose derivatives yield random vectors rather than deterministic points. In traditional calculus, when we take the derivative of a function at a point, we get a fixed value ie a scalar that represents the instantaneous rate of change. However, in stochastic calculus, the derivative at any point is a random variable. ...
VAE: Hierarchical VAE: DDPM(Hierarchical VAE with deterministic forward process): Notice that $\phi$ has been removed from $q_\phi()$. Also see that: The ELBO for a Variational Autoencoder (VAE) is given by: $$ \text{ELBO (VAE)} = \mathbb{E}_{q_\phi(z|x)} \left[ \log \frac{p(x|z)}{q_\phi(z|x)} \right] $$ - For Denoising Diffusion Models (DDPM), the ELBO is expressed as: $$ \text{ELBO (DM)} = \mathbb{E}_{q(x_{1:T}|x_0)} \left[ \log \frac{p(x_{0:T})}{q(x_{1:T}|x_0)} \right] $$
Recall that in support measure machine (SCM), we were trying to maximize the margin between the two classes of data points. $$ \min_{w,b,\xi} \frac{1}{2} |w|^2 + C \sum_{i=1}^n \xi_i $$ $$ \text{s.t. } y_i (w^T x_i + b) \geq 1 - \xi_i \text{ for all } i $$ $$ \xi_i \geq 0 \text{ for all } i $$ SCM as ERM This is an optimization problem that can be formulated as a empirical risk minimization (ERM) problem. ...
Dataset is linearly separable Dataset $D = {(x_i, y_i)}_{i=1}^n$ where $x_i \in \mathbb{R}^d$ and $y_i \in {-1, 1}$ is said to be linearly separable if there exists a hyperplane that can separate the two classes of data points with zero training error. Mathematically, a dataset is linearly separable if there exist weights $w$ and bias $b$ such that: $$ w^T x_i + b > 1 \text{ for } y_i = 1 $$ $$ w^T x_i + b < -1 \text{ for } y_i = -1 $$ ...
Dataset is not linearly separable Unlike the case when the dataset is linearly separable, we cannot find a hyperplane that separates the two classes of data points with zero training error. Soft margin To handle this case, we introduce a slack variable $\xi_i$ for each data point $x_i$ to allow some points to be on the wrong side of the margin or even in the wrong class. Optimization problem The optimization problem becomes: ...
Prerequisites - Kernel trick Kernel function $k(x, y) = \phi(x)^T \phi(y)$ where $\phi(x)$ is a feature mapping that maps the data points to a higher dimensional space without actually computing the feature mapping, i.e., $\phi(x)$. In other words, we can compute the kernel function $k(x, y)$ without actually computing the feature mapping $\phi(x)$. Examples of kernel functions: Polynomial kernel: $k(x_1, x_2) = \phi(x_1)^T \phi(x_2) = (1 + x_1^T x_2)^p$ Sigmoid kernel: $k_s(x_1, x_2) = \frac{1}{1 + \exp(a x_1^T x_2)}$ RBF/Gaussian kernel: $k(x_1, x_2) = \exp\left(-\frac{| x_1 - x_2 |^2}{\sigma^2}\right)$ Motivation If the dataset $D = {(x_i, y_i)}_{i=1}^n$ where $x_i \in \mathbb{R}^d$ and $y_i \in {-1, 1}$ is not linearly separable in the original $d$-dimensional space, we can use a feature mapping $\phi(x)$ to map the data points to a higher dimensional space where they become linearly separable and then use the SVM optimization problem to find the optimal hyperplane. ...
Temporal Difference (TD) learning, introduced by Rich Sutton in 1984, is a fundamental reinforcement learning method that combines ideas from Monte Carlo methods and dynamic programming. Recap of Monte Carlo Methods Recall that Monte Carlo methods estimate the value function by: $$ V_\pi(s) = E[G_n | S_0 = s] $$ where: $G_n = r_{n+1} + r_{n+2} + … + r_N$ is the return (sum of rewards until terminal state) $N$ is the terminal instant of the episode Monte Carlo methods work with sample average data collected over complete trajectories, requiring episodes to terminate before learning can occur. ...
Temporal Difference (TD) learning is a fundamental reinforcement learning technique that combines ideas from Monte Carlo methods and dynamic programming. It allows agents to learn from experience without requiring a model of the environment, making it particularly valuable for real-world applications. Understanding Policy Evaluation Mathematical Foundation Policy evaluation refers to the process of determining the value function $J_π$ for a given policy $π$. The value function represents the expected cumulative reward when following policy $π$ from a given state. ...
We can model distributions using neural networks in two ways: Deterministic way In this way the output of neural network is a sample from the distribution. Probabilistic way In this way the output of neural network are parameters of the distribution.
Definitions Expected Single-Stage Cost The expected single-stage cost under policy $\mu$ in state $x$, denoted as $\bar{g}(x,\mu(x))$, is: $$ \underbrace{\bar{g}(x,\mu(x))}_{\substack{\text{Expected} \\ \text{single-stage cost}}} = \sum_{j=0}^n \underbrace{p_{xj}(\mu(x))}_{\substack{\text{Transition} \\ \text{probability}}} \underbrace{g(x,\mu(x),j)}_{\substack{\text{Stage cost for} \\ \text{this transition}}} $$ where: $p_{xj}(\mu(x))$ is the probability of transitioning from state $x$ to state $j$ under action $\mu(x)$ $g(x,\mu(x),j)$ is the stage cost for this transition The sum is over all possible next states $j$ This represents the immediate expected cost when taking the action prescribed by policy $\mu$ in state $x$. ...