Posts

Core Idea of Reinforcement Learning Reinforcement Learning (RL) is a computational approach to learning whereby an agent seeks to maximize some notion of cumulative reward by taking actions in an environment. This process is akin to training a dog to perform a trick: you cannot dictate every move, but you can provide rewards when it performs correctly. In RL, the agent learns through trial and error, receiving feedback in the form of rewards or penalties based on its actions. The primary objective is to develop a policy, a strategy that guides the agent to take actions that maximize the total accumulated reward over time. ...

Idea Joint Embedding Predictive Architecture (JEPA) combines predictive learning with representation learning in a unified framework. Formulation Let us formalize the JEPA framework: Given: Input space $\mathcal{X}$ Context patch $x_c \in \mathcal{X}$ Target patch $x_t \in \mathcal{X}$ Context encoder $f_\theta: \mathcal{X} \rightarrow \mathcal{Z}$ Target encoder $g_\phi: \mathcal{X} \rightarrow \mathcal{Z}$ Predictor network $h_\psi: \mathcal{Z} \rightarrow \mathcal{Z}$ The process works as follows: Encode context: $z_c = f_\theta(x_c)$ Encode target: $z_t = g_\phi(x_t)$ Predict target embedding: $\hat{z}_t = h_\psi(z_c)$ The objective is to minimize prediction error in embedding space: ...

Basic idea Recall that we had derived the following equation for non-parametric density estimation: $$p(x) = \frac{k}{nV}$$ where: $k$ is the number of data points in region $R$ $n$ is the total number of data points in the dataset $D$ $V$ is the volume of the region $R$ In K-Nearest Neighbour (KNN) method, we fix the volume $V$ and count $k$ Problem setting Given a set of data points $D = {x_1, y_1}, {x_2, y_2}, …, {x_n, y_n}$, where $x_i \in \mathbb{R}^d$ is the feature vector and $y_i \in {1, 2, …, C}$ is the class label. ...

K-means clustering also known as K-means algorithm or hard clustering Recall that in mixture model clustering Data was: $$D = \{x_i\}_{i=1}^N$$ We assumed latent variable $z_i$ is associated with each data point $x_i$. Hence data became $D = \{(x_i, z_i)\}_{i=1}^N$. We assumed $$p_\theta(x) = \sum_{j=1}^m p_\theta(x|z=j) p_\theta(z=j) = \sum_{j=1}^m N(x; \mu_j, \Sigma_j) \pi_j$$ Where: $N(x; \mu_j, \Sigma_j)$ is the Gaussian probability density function for cluster $j$, with mean $\mu_j$ and covariance matrix $\Sigma_j$. $\pi_j$ is the probability of a data point belonging to cluster $j$. K-means clustering Now let’s make the following assumptions to simplify our model: ...

KL divergence is a measure of how one probability distribution differs from another probability distribution. Mathematically, for two probability distributions $P(x)$ and $Q(x)$, the KL divergence from $Q$ to $P$ is defined as: $$D_{KL}(P || Q) = \sum_{x} p(x) \log\left(\frac{p(x)}{q(x)}\right) = E_{x \sim p(x)}\left[\log\left(\frac{p(x)}{q(x)}\right)\right]$$ for discrete distributions $$D_{KL}(P || Q) = \int p(x) \log\left(\frac{p(x)}{q(x)}\right) dx = E_{x \sim p(x)}\left[\log\left(\frac{p(x)}{q(x)}\right)\right]$$ for continuous distributions where the sum/integral is over all possible events $x$. And $p(x)$ and $q(x)$ are the probability density functions of distributions $P(x)$ and $Q(x)$ respectively. ...

KL divergence is a measure of how one probability distribution diverges from second, probability distribution. Mathematically, for two probability distributions $P(x)$ and $Q(x)$, the KL divergence from $Q$ to $P$ is defined as: $D_{KL}(p || q) = Σ p(x) * log\left(\frac{p(x)}{q(x)}\right)$ for discrete distributions $D_{KL}(p || q) = ∫ p(x) * log\left(\frac{p(x)}{q(x)}\right) dx$ for continuous distributions where the sum/integral is over all possible events $x$. And $p(x)$ and $q(x)$ are the probability density functions of distributions $P(x)$ and $Q(x)$ respectively. ...

We are given a dataset $D = {(x_i, y_i)}_{i=1}^n$ sampled from an unknown distribution $P$. We want to find a hypothesis function $h$ out of the complete hypothesis space $H$ that minimizes the risk function $R(h)$.

Recap We have covered several key topics in reinforcement learning: Fundamentals and Problem Formulation Basic concepts and terminology of reinforcement learning and problem formulation Multi-armed bandit problems(Single stage multiple action problems) Solved using greedy, ε-greedy, UCB, and gradient-based methods Markov Decision Processes ie MDPs(Known system model (state transition probabilities and reward/cost functions)) Infinite horizon problems($n<\infty$ but deterministic) - Dynamic programming solutions to the Bellman equation for finding the optimal policy Stochastic shortest path problems($n<\infty$ but random) - Bellman equation formulations, value iteration, policy iteration Discounted cost problems ($n \to \infty$) - Bellman equation formulations, policy iteration, value iteration Learning the system model from data From now on we shall assume we do not know the system model (state transition probabilities and reward/cost functions) and we need to learn it from data. The Data is in the form of trajectories generated by the system: ...

Let the data be $D = {(x_i, y_i)}_{i=1}^N$ where $x_i \in \mathbb{R}^d$ and $y_i \in {1, 2}$ Let’s assume the parametric form of the conditional density $p(x|y)$ is: $$p(x|y=1) \sim N(x; \mu_1, \Sigma)$$ $$p(x|y=0) \sim N(x; \mu_2, \Sigma)$$ where $N(x; \mu, \Sigma)$ denotes a multivariate Gaussian distribution with mean $\mu$ and covariance matrix $\Sigma$. Note that we assume the covariance matrix $\Sigma$ is the same for both classes, which is a key assumption in LDA. ...

Let the data be $(x_i, y_i)$ for $i = 1, 2, …, n$ where $x_i \in \mathbb{R}^d$ and $y_i \in \mathbb{R}$. then we can model $y_i$ as: $$Y = \beta_0 + \beta_1X + \varepsilon$$ Where: $Y \in \mathbb{R}$ is the dependent variable $X \in \mathbb{R}^d$ is the independent variable (or feature vector in higher dimensions) $\beta_0$ is the y-intercept (bias term) $\beta_1$ is the slope (or coefficient vector in higher dimensions) $\varepsilon$ is the error term Ideal regressor For mean squared error loss, the ideal regressor is defined as: ...