Posts

A Naive GAN (Generative Adversarial Network) refers to the basic or “vanilla” version of GANs, introduced by Ian Goodfellow in 2014. Derivation Recall the genereral expression for the GANs that we got by minimizing F-divergence: $$\theta^* = \underset{\theta}{\text{argmin}} \underset{\phi}{\sup} \left[\mathbb{E}_{x \sim p_{data}}[T_\phi(x)] - \mathbb{E}_{x \sim p_{generator}}[f^*(T_\phi(x))]\right]$$ where: $\theta$: Parameters of the generator network $\phi$: Parameters of the discriminator (critic) network $\theta^*$: Optimal parameters for the generator $\text{argmin}_\theta$: Argument that minimizes the expression with respect to $\theta$ $\sup_\phi$: Supremum (least upper bound) with respect to $\phi$ $T_\phi(x)$: The discriminator function, parameterized by $\phi$ $f^*$: The convex conjugate of the function $f$ used in the F-divergence $p_{data}$: The true data distribution $p_{generator}$: The distribution of the generated data Write $T_\phi(x)$ as composite function $T_\phi(x) = \sigma(V_\phi(x))$, where $\sigma$ is the sigmoid function Substitute this in the expression for GANs: ...

A neural network is a computational model inspired by the structure and function of biological neural networks. Mathematically, it can be defined as a series of function compositions: $$ f(x) = f_L(f_{L-1}(…f_2(f_1(x)))) $$ where $L$ is the number of layers in the network, and each function $f_i$ represents a layer operation. For a single layer, the operation can be expressed as: $$ f_i(x) = \sigma(W_i x + b_i) $$ where: ...

Idea The core idea behind Noise Contrastive Estimation (NCE) is: Given samples from a distribution, if you know how to distinguish them from noise samples, then you implicitly know the underlying distribution. If you know how to tell what is noise from what is real data, you must have learned something about the true data distribution. This provides an alternative way to learn probability distributions by transforming the density estimation problem into a binary classification problem between real and noise samples. ...

What is parametric density estimation? In parametric density estimation, we assume that the data is generated from a known distribution, such as the normal distribution, and we estimate the parameters of the distribution using various methods like maximum likelihood estimation or risk minimization. What is non-parametric density estimation? However, in non-parametric density estimation, we make no assumptions about the form of the distribution and estimate the density directly from the data. ...

Parzen window estimate is also known as the kernel density estimate Basic idea Recall that we had derived the following equation for non-parametric density estimation: $$p(x) = \frac{k}{nV}$$ where: $k$ is the number of data points in region $R$ $n$ is the total number of data points in the dataset $D$ $V$ is the volume of the region $R$ In Parzen window estimate, we fix the volume $V$ and count $k$ by using a window function. ...

In value iteration, we were focused on finding the optimal value function $J^*$. In policy iteration, we start with a random policy and then improve it iteratively. Algorithm Description Policy iteration alternates between two main steps: Policy Evaluation: Given a policy $\mu_k$, compute its cost-to-go vector $J^{\mu_k}$ by solving the system of linear equations: $$ J^{\mu_k}(i) = \sum_{j=1}^n p_{ij}(\mu_k(i))(g(i,\mu_k(i),j) + J^{\mu_k}(j)) \quad i=1,\ldots,n $$ where $J^{\mu_k}(1), J^{\mu_k}(2), \ldots, J^{\mu_k}(n)$ are the unknowns to be solved for. ...

Proposition: Optimality Conditions in Policy Iteration Let’s examine a fundamental proposition that establishes optimality conditions for policy iteration in infinite horizon discounted cost problems. This proposition provides crucial insights into why policy iteration converges to optimal policies. Statement of the Proposition Consider a stationary policy $\bar{\mu}$ that satisfies the Bellman optimality condition: $$ \underbrace{T_{\bar{\mu}}J^{\bar{\mu}} = TJ^{\bar{\mu}}}_{\substack{\text{Policy } \bar{\mu} \text{ achieves the minimum} \\ \text{in the Bellman equation for all states}}} $$ This can be expressed more explicitly for all states $i = 1,2,\ldots,n$ as: ...

Motivation We explored different sampling methods for generative models: Langevin dynamics (LE): $$x_{t+1} = x_t + \alpha \nabla_x \log p(x_t) + \sqrt{2\alpha}\epsilon, \quad \epsilon \sim \mathcal{N}(0,I)$$ DDPM (Denoising Diffusion Probabilistic Models): $$x_{t+1} = \sqrt{1-\beta_t}x_t + \sqrt{\beta_t}\epsilon, \quad \epsilon \sim \mathcal{N}(0,I)$$ SMLD (Score Matching with Langevin Dynamics): $$x_{t+1} = x_t + \sqrt{\sigma_{t+1}^2 - \sigma_t^2}\epsilon, \quad \epsilon \sim \mathcal{N}(0,I)$$ A natural question arises: What is the unifying mathematical framework that connects all these sampling methods? ...

Problem Setup Data: Let the dataset be $X = {x_i}_{i=1}^N$, where each data point $x_i \in \mathbb{R}^d$. Goal: We want to reduce the dimensionality of the data from $d$ to $k$ while preserving as much of the data’s variance as possible. Intuition PCA aims to find a new coordinate system for the data that maximizes the variance along the first coordinate (principal component), while minimizing the variance along the other coordinates. This is useful for data visualization, noise reduction, and feature extraction. ...

Q-Learning is a powerful model-free reinforcement learning algorithm that learns the optimal action-value function directly without requiring a model of the environment. Understanding Model-Based vs. Model-Free Approaches Model-Based Reinforcement Learning A model of the environment in reinforcement learning refers to explicit knowledge of the system dynamics, specifically: State Transition Probabilities: $p_{ij}(u)$ - The probability of transitioning from state $i$ to state $j$ when taking action $u$ Reward/Cost Functions: $g(i,u,j)$ - The immediate reward or cost received when transitioning from state $i$ to state $j$ via action $u$ With a complete model, we can use dynamic programming methods like value iteration or policy iteration to compute optimal policies without direct environment interaction. ...