Generative Model Let the training dataset be $$data = \{(x_i)\}_{i=1}^N$$ where $x_i \in \mathbb{R}^d$ are called data points. These data points are iid samples from the true data distribution $P(x)$. Aim of generative model is to extimate $P(x)$ using the training data and sample new data points from it. GANs, GMMs etc are a example of generative model. In GANs we use training data to train a neural network to learn the distribution of the data. Once we have learnt this distribution we can sample new data points from it. ...

In infinite horizon discounted cost problems, we consider scenarios without a termination state and an infinite sequence of decisions. Let’s understand the key components and mathematical formulation. Problem Setting State Space: Set of states $S = {1,2,…,n}$. Note that there is no terminal state unlike in finite horizon problems. Action Space: For each state $i$, set of feasible actions $A(i)$ Total action space: $A = \bigcup_{i \in S} A(i)$ Cardinality: Both $|S|$ and $|A|$ are finite Optimal Cost Function The optimal cost function $J^*(i)$ represents the minimum total discounted cost starting from state $i$: ...

InfoVAE addresses the aggregated posterior mismatch problem in standard VAEs by modifying the objective function to explicitly match the aggregated posterior with the prior distribution. Key Ideas If the aggregated posterior $q_\phi(z)$ becomes a Dirac delta, all inputs $x$ map to the same latent code $z$ In such a scenario, the latent code $z$ would contain no information about the input $x$ The standard VAE objective does not directly encourage the matching between $q_\phi(z)$ and the prior $p(z)$ InfoVAE Objective Function The standard VAE objective is given by: ...

Controlled Markov Chain (CMC) A Controlled Markov Chain (CMC) is a stochastic process ${X_n}_{n \geq 0}$ that evolves over discrete time steps $n = 0, 1, 2, \ldots$. At each time step $n$, the system is in a state $X_n$ belonging to a state space $S$. To influence the evolution of this process, we apply a control or action $Z_n$ from a set of feasible actions $A(X_n)$ that depends on the current state $X_n$. Thus, $Z_n \in A(X_n)$. The defining characteristic of a Controlled Markov Chain is its transition probability, which satisfies the Markov property conditioned on the chosen action. Specifically, the probability of transitioning to a next state $X_{n+1} = j$ depends only on the current state $X_n = i$ and the action $Z_n = a$ taken at the current time step, and not on the history of states and actions. Mathematically, this is expressed as: ...

Core Idea of Reinforcement Learning Reinforcement Learning (RL) is a computational approach to learning whereby an agent seeks to maximize some notion of cumulative reward by taking actions in an environment. This process is akin to training a dog to perform a trick: you cannot dictate every move, but you can provide rewards when it performs correctly. In RL, the agent learns through trial and error, receiving feedback in the form of rewards or penalties based on its actions. The primary objective is to develop a policy, a strategy that guides the agent to take actions that maximize the total accumulated reward over time. ...

Idea Joint Embedding Predictive Architecture (JEPA) combines predictive learning with representation learning in a unified framework. Formulation Let us formalize the JEPA framework: Given: Input space $\mathcal{X}$ Context patch $x_c \in \mathcal{X}$ Target patch $x_t \in \mathcal{X}$ Context encoder $f_\theta: \mathcal{X} \rightarrow \mathcal{Z}$ Target encoder $g_\phi: \mathcal{X} \rightarrow \mathcal{Z}$ Predictor network $h_\psi: \mathcal{Z} \rightarrow \mathcal{Z}$ The process works as follows: Encode context: $z_c = f_\theta(x_c)$ Encode target: $z_t = g_\phi(x_t)$ Predict target embedding: $\hat{z}_t = h_\psi(z_c)$ The objective is to minimize prediction error in embedding space: ...

K-means clustering also known as K-means algorithm or hard clustering Recall that in mixture model clustering Data was: $$D = \{x_i\}_{i=1}^N$$ We assumed latent variable $z_i$ is associated with each data point $x_i$. Hence data became $D = \{(x_i, z_i)\}_{i=1}^N$. We assumed $$p_\theta(x) = \sum_{j=1}^m p_\theta(x|z=j) p_\theta(z=j) = \sum_{j=1}^m N(x; \mu_j, \Sigma_j) \pi_j$$ Where: $N(x; \mu_j, \Sigma_j)$ is the Gaussian probability density function for cluster $j$, with mean $\mu_j$ and covariance matrix $\Sigma_j$. $\pi_j$ is the probability of a data point belonging to cluster $j$. K-means clustering Now let’s make the following assumptions to simplify our model: ...

KL divergence is a measure of how one probability distribution differs from another probability distribution. Mathematically, for two probability distributions $P(x)$ and $Q(x)$, the KL divergence from $Q$ to $P$ is defined as: $$D_{KL}(P || Q) = \sum_{x} p(x) \log\left(\frac{p(x)}{q(x)}\right) = E_{x \sim p(x)}\left[\log\left(\frac{p(x)}{q(x)}\right)\right]$$ for discrete distributions $$D_{KL}(P || Q) = \int p(x) \log\left(\frac{p(x)}{q(x)}\right) dx = E_{x \sim p(x)}\left[\log\left(\frac{p(x)}{q(x)}\right)\right]$$ for continuous distributions where the sum/integral is over all possible events $x$. And $p(x)$ and $q(x)$ are the probability density functions of distributions $P(x)$ and $Q(x)$ respectively. ...

Recap We have covered several key topics in reinforcement learning: Fundamentals and Problem Formulation Basic concepts and terminology of reinforcement learning and problem formulation Multi-armed bandit problems(Single stage multiple action problems) Solved using greedy, ε-greedy, UCB, and gradient-based methods Markov Decision Processes ie MDPs(Known system model (state transition probabilities and reward/cost functions)) Infinite horizon problems($n<\infty$ but deterministic) - Dynamic programming solutions to the Bellman equation for finding the optimal policy Stochastic shortest path problems($n<\infty$ but random) - Bellman equation formulations, value iteration, policy iteration Discounted cost problems ($n \to \infty$) - Bellman equation formulations, policy iteration, value iteration Learning the system model from data From now on we shall assume we do not know the system model (state transition probabilities and reward/cost functions) and we need to learn it from data. The Data is in the form of trajectories generated by the system: ...

Let the data be $D = {(x_i, y_i)}_{i=1}^N$ where $x_i \in \mathbb{R}^d$ and $y_i \in {1, 2}$ Let’s assume the parametric form of the conditional density $p(x|y)$ is: $$p(x|y=1) \sim N(x; \mu_1, \Sigma)$$ $$p(x|y=0) \sim N(x; \mu_2, \Sigma)$$ where $N(x; \mu, \Sigma)$ denotes a multivariate Gaussian distribution with mean $\mu$ and covariance matrix $\Sigma$. Note that we assume the covariance matrix $\Sigma$ is the same for both classes, which is a key assumption in LDA. ...