Posts on Victor

Andrew Tate on Big Bedrooms

Tue, 22 Jul 2025 00:00:00 +0000

These are some things that I have learned from watching Andrew Tate’s videos:

Politicians

Politicians are not stupid: Do not think that politicians are stupid; that is a very optimistic view. They are not stupid; they know what they are doing. It is you who is confused because you think they work for you. You believe the politicians are your representatives, and when they don’t represent your interests, you think they are dumb. No! They work for someone else. They use this democracy thing to only allow you to accept it. They are doing exactly what the people who pay them want them to do; in fact, they are doing it fantastically. Your confusion stems from thinking that they care about you. They never have.

Freedom

How to maintain illusion of freedom: By putting freaks to the front. This person chopped his dick off, see we are free!(You are free to hurt yourself). But do not be fooled you are not free to say anything against the power, anything important.

Anxiety

You have anxiety because you should be anxious, Why is a mouse anxious ? Same for you! The large bear does not get anxious. A mouse think a noise (anxious) it might kill me! Bear thinks I might want to kill it! Its a different vibe.
Anxiety is not a disorder - it is a natural conclusion of your ineptitude, its your soul giving you feedback. Anxiety is cured by strength.
Dying ain’t so bad untill unless you die for a good reason. Dying is not the worst thing that can heppen, living without a purpose is the worst thing that can happen.

Decision making framework using probability theory

Sun, 29 Jun 2025 00:00:00 +0000

This is how you can use Probability Theory to make decisions.

Fundamental Terminologies

Before diving into the mathematical framework, let’s establish the key concepts that form the foundation of decision-making under uncertainty.

Action

An action is a choice or decision that a decision-maker can take from a set of available alternatives. It represents what we can control or decide upon in a given situation.

Mathematical Notation: $$ a \in A $$

I Asked Robert Herjavec How He Made $500 Million

Sat, 28 Jun 2025 00:00:00 +0000

2:29 - Money stays in shape: Rick people are fit
3:29 - No is halfway to yes
4:32 - More important than money is WHAT YOU ARE MADE OF?
16:19 - On Networking - If you do not know anybody, you got to become the person other people want to know
16:40 - Motivation is easy, passion is easy, all that stuff fades, OBSESSION LASTS!
17:10 - The biggest flex of wealth for the rich is being in shape: If you are not using your money to stay in shape, then what is the point?
18:44 - Beware of the man in the T Shirt: If you walk in the room, you see 10 guys in a suit and 1 guy in the back in a t shirt, BEWARE OF THE GUY IN THE T SHIRT!
19:57 - Sell Joy: People should enjoy talking to you

Connect Supabase To n8n

Tue, 27 May 2025 00:00:00 +0000

Follow the instructions in this video to connect Supabase to n8n

How to Connect Google Drive To n8n

Tue, 27 May 2025 00:00:00 +0000

Follow the instructions in this video to connect Google Drive to n8n

How to Generate a Google Gemini API Key

Tue, 27 May 2025 00:00:00 +0000

Follow the instructions in this video to generate a Google Gemini API key

How to Generate Mistral API Keys

Tue, 27 May 2025 00:00:00 +0000

Follow the instructions in this video to generate Mistral API Keys

How to Generate OCR Space API Keys

Tue, 27 May 2025 00:00:00 +0000

The OCR API provides a simple way of parsing images and multi-page PDF documents (PDF OCR) and getting the extracted text results returned in a JSON format.

Here is the link to up to date instructions on how to generate OCR Space API Keys - https://ocr.space/OCRAPI

How to Generate Open AI API Keys

Tue, 27 May 2025 00:00:00 +0000

Follow the instructions in this video to generate Open AI API Keys

How to open a workflow in n8n

Tue, 27 May 2025 00:00:00 +0000

A step-by-step guide to opening a workflow in n8n

How to setup n8n cloud

Tue, 27 May 2025 00:00:00 +0000

A step-by-step guide to setting up n8n cloud platform for workflow automation

How to Setup Supabase for Postgres memory node in n8n

Tue, 27 May 2025 00:00:00 +0000

Follow the instructions in this video to setup Supabase for Postgres memory node in n8n

Applying RL to derive Optimal Policy for human behavior

Sun, 25 May 2025 00:00:00 +0000

RL setting

In the reinforcement learning framework, we can model human behavior as an agent interacting with the world. Let’s formalize this mathematically.

Markov Decision Process (MDP) Formulation

The human experience can be modeled as a Markov Decision Process (MDP) defined by the tuple:

$$\mathcal{M} = (\mathcal{S}, \mathcal{A}, \mathcal{P}, \mathcal{R}, \gamma)$$

where:

$\mathcal{S}$ is the state space (all possible states of the world and internal states)
$\mathcal{A}$ is the action space (all possible actions the human can take)
$\mathcal{P}: \mathcal{S} \times \mathcal{A} \times \mathcal{S} \rightarrow [0,1]$ is the transition probability function
$\mathcal{R}: \mathcal{S} \times \mathcal{A} \times \mathcal{S} \rightarrow \mathbb{R}$ is the reward function
$\gamma \in [0,1]$ is the discount factor

Environment is the world

The environment encompasses everything external to the agent (brain). Mathematically, the environment dynamics are governed by:

I Was Broke.. Found God.. Now I Make $15M/Year

Sun, 25 May 2025 00:00:00 +0000

9:11 - It’s the things that we don’t know that we don’t know that are the problem

Andrew Tate on Big Bedrooms

Sat, 24 May 2025 00:00:00 +0000

Link to the video: https://rumble.com/v6vkhsl-andrew-tate-on-big-bedrooms.html?e9s=src_v1_ucp_v

5:04 - Do not show the pictures of me or you to a girl, just show the pictures of the bedroom. And ask them where it will be cooler to have sex.

Andrew Tate on Big Bedrooms

Sat, 24 May 2025 00:00:00 +0000

Link to the video: https://rumble.com/v6vo3x9-real-training-tate-confidential-ep-308.html?e9s=src_v1_ucp_v

4:55 - You cannot have loyalty from someone who is weak, even if they have your best interest at heart. Do not like weak people, do not have weak people around you.
7:54 - Learn to switch your mind off - Zombify yourself.
9:15 - You need to have something about you. You need to have a chi. Its like a power that people can feel. There is something that is evolved in us that can feel the predators.

Asking Beverly Hills Billionaires How They Got So RICH

Sat, 24 May 2025 00:00:00 +0000

2:15 - You got to keep trying mf
2:43 - Be a good salesman, sell yourself first: Wear nicer clothes
5:36 - Be frugal, pul all your money back in the business
7:38 - Do not be afraid to spend money, scared money never made money, even spending money on enjoyment is ok
10:17 - Do what you do best and delegate the rest
12:19 - You are what you hang around with: Do not hang around with looseers
14:57 - Do not expect anyone to help you You will have to do it yourself or hire someone do not seek help from anyone in your family
17:49 - If the home front is not good, it will take away your attention

Asking Beverly Hills Billionaires How They Got So RICH

Sat, 24 May 2025 00:00:00 +0000

3:16 - You need send and delete kind of persons - I will only hire send and delete kind of persons
4:34 - Everyone is not born for entrepreneurship - If you have to force someone to be an entrepreneur, then they are not meant to be an entrepreneur
5:13 - Master negotiation tactics - Whoever speaks first loses - Listen, Learn then negotiate
6:27 - Most improtant advice - Keep your word - Keep your word

Asking Politicians How They Got Wealthy

Sat, 24 May 2025 00:00:00 +0000

20:41 - Whatever you do, get to where you want to get to by 40 - because by 40 that is where you are going to make most of your money - that is when you takeoff

How To Deal With Stress In Jobs ? | Baramati PSU Bank Manager Case

Sat, 24 May 2025 00:00:00 +0000

6:27 - Have you ever seen a politician working hard? Have you ever seen a politician stressing about whether every area is clean, or worrying that things are not working properly? Yet, here you are, working in a petty job, and you are the one taking all the tension. Isn’t that surprising?
7:10 - Stop being sensitive and stop feeling disrespected.
10:50 - Asking top management that people run away by taking thousands of crores of loan and you do not do anything, yet you harass junior employees if they make minor mistakes.
13:49 - You are not getting sex, you are not getting food, You do not live with your family? Why are you working?
17:09 - Owners and promoters are selling the shares of their company. They are not interested. Then why are you interested?

I Asked Adin Ross How He Made $100 Million

Sat, 24 May 2025 00:00:00 +0000

2:15 - When you make a sale, wrap it up - If a customer is not ready to buy, do not show them other options
2:23 - God gave you 2 ears and 1 mouth for a reason - Twice the amount of listening than talking
3:29 - Sales is statistical
[8:08 - Be a crystal clear communicator]

I Asked Adin Ross How He Made $100 Million

Sat, 24 May 2025 00:00:00 +0000

4:18 - Money does not come from chasing money, money comes from chasing problems and monetizing the solution to those problems

The Complete Guide to the X 'For You' Algorithm

Sat, 24 May 2025 00:00:00 +0000

A technical breakdown of how X's recommendation algorithm actually works, based on the open-source code. Not speculation—just what the code says.

What is Retrieval Augmented Generation? (RAG)

Sat, 24 May 2025 00:00:00 +0000

Limitations of Traditional LLMs

Large Language Models (LLMs), while powerful, face several fundamental constraints that limit their effectiveness in real-world applications:

Knowledge Cutoff and Staleness

Static Training Data: LLMs are trained on data up to a specific cutoff date, making them unable to access or reason about recent information, events, or developments
Knowledge Decay: Information becomes increasingly outdated over time, reducing the model’s relevance for current queries

Context Window Constraints

Limited Input Size: Most LLMs have finite context windows, restricting the amount of information that can be processed in a single query
Information Loss: When dealing with large documents or datasets, critical information may be truncated or lost due to context limitations

Computational and Economic Costs

Expensive Long Contexts: While some models support extended context windows (100K+ tokens), processing large amounts of text significantly increases computational costs and latency
Inefficient Information Processing: Passing entire documents through the model when only specific sections are relevant leads to unnecessary resource consumption

Hallucination and Accuracy Issues

Knowledge Gaps: When LLMs lack specific information, they may generate plausible-sounding but incorrect responses (hallucinations)
Inability to Cite Sources: Traditional LLMs cannot provide verifiable sources or references for their generated content, making fact-checking difficult
Lack of Traceability: There’s no way to trace back which specific information or training data influenced a particular response, making it impossible to verify the source of generated content

Domain-Specific Knowledge Limitations

Specialized Information: LLMs may lack deep expertise in highly specialized domains or proprietary knowledge that wasn’t part of their training data
Real-time Data: Cannot access live data, current market conditions, or dynamic information that changes frequently

When do you need RAG vs Fine-tuning?

Understanding when to use Retrieval Augmented Generation (RAG) versus Fine-tuning is crucial for building effective AI systems. Each approach has distinct advantages and is suited for different scenarios.

Overview of Reinforcement Learning

Fri, 09 May 2025 00:00:00 +0000

Basic Terminologies in Reinforcement Learning(RL)

Reinforcement Learning is defined by the following components:

Agent: The entity that interacts with the environment and makes decisions.
Environment: The world in which the agent operates. Mathematically defined as a state transition system: $\mathcal{E}: \mathcal{S} \times \mathcal{A} \rightarrow \mathcal{S}$
State ($s \in \mathcal{S}$): The current configuration of the environment.
- The state space $\mathcal{S}$ is the set of all possible states.
- At time step $t$, the state is denoted as $s_t$.
Action ($a \in \mathcal{A}$): The decision made/ action taken by the agent after observing the state $s_t$ of the environment based on its policy $\pi$.

How reinforcement learning works in Unity

Sat, 01 Mar 2025 00:00:00 +0000

Unity Packages for Reinforcement Learning

Unity’s modular nature is facilitated by its Package Manager, which allows developers to add specific functionalities to their projects. For implementing Reinforcement Learning (RL) within Unity, the primary and most widely used package is:

ML-Agents Toolkit:
- Purpose: This is Unity’s official open-source package specifically designed to enable games and simulations to serve as environments for training intelligent agents using RL and imitation learning.
- Functionality: It provides the necessary tools and SDKs to:
  - Define Agents: The actors within the environment that learn to make decisions.
  - Define Observations: The information the agent receives from the environment (e.g., position, velocity, sensor readings).
  - Define Actions: The decisions the agent can make (e.g., move forward, jump, turn).
  - Define Rewards: The signals given to the agent to indicate whether its actions are good or bad towards achieving a goal.
  - Connect to Training Algorithms: It integrates with established Python-based RL libraries (like TensorFlow via mlagents-learn) to perform the actual training process, often using algorithms like PPO (Proximal Policy Optimization) or SAC (Soft Actor-Critic).
- Integration: ML-Agents allows seamless communication between the Unity environment (where the simulation runs) and the Python training process.

While ML-Agents is the cornerstone for RL in Unity, developers could potentially integrate other third-party libraries or custom solutions, but these would typically require significantly more manual setup and integration effort compared to the streamlined workflow offered by the official ML-Agents package. For most users looking to apply RL in Unity, ML-Agents is the recommended and standard choice.

AI Roadmap

Sat, 01 Feb 2025 00:00:00 +0000

AI Learning Roadmap

I have made this roadmap that I will give to my younger brother who is starting his learning journey in AI. The key points of this roadmap are:

The roadmap assumes you have basic knowledge of maths and programming of a 12th class pass out.
All the resources are free.
The roadmap is linear, follow it step by step. However, if you have a good grasp of a topic, you can skip it and move to the immediate next topic.
The roadmap is not exhaustive, I have designed it so that you learn 90% of the most important topics in AI.
I have seen most AI learning resources to either focus on maths or programming. I have designed this roadmap to cover both.
This is not the only possible roadmap, there are many other roadmaps that are possible. This is just a roadmap that I think is best for a beginner learning on his own.
There are thousands of students from all walks of life studying this roadmap in this Discord server
I have made the Discord server for students who are following this roadmap to help them learn and grow together. Here is the link to the server: Join our Discord Server. Do join thousands of students who would be learning along with you.
I also encourage you to learn about the topics from other sources as well in addition to the ones mentioned in the roadmap if you have more time.
In case you are not able to understand a particular topic from the resources mentioned in the roadmap, do explore other resources and share your experience.
As and when I find a better new resource, I will update the roadmap. If you have any suggestions, please let me know so that I can update the roadmap.
This roadmap is up to date. As and when an important breakthrough comes in the field of AI, that is important, I will update the roadmap.

After completing the roadmap:

You should pick one or more area of expertise like Computer Vision, Natural Language Processing, etc. and start applying your knowledge and learning advance topics in that domain.
There is no use of learning AI if you are not applying your knowledge. You should start working on real world projects. Either start working on your own projects or contribute to the projects at your company/open source community.
AI is one of the fastest growing field, so do keep yourself updated with the latest research, news and products.

My tips for the Discord server:

The Discord server is free and open to all with a learning mindset. Do invite your friends and classmates by sharing the server link on your social media handles and bring them along on this learning journey so that you can learn and grow together. Moreover the more members in the server, the more easier it will be to get help from each other and more resources will be shared for everyone’s benefit.
Do not hesitate to ask questions. Moreover, if you have time do help others by answering their questions as this will help you solidify your understanding of the topic.
Do find friends in this Discord server who are learning along with you and do form a study group(if possible find friends that you can even meet in person).

Here is the roadmap:

1. Maths prerequisites

1.1 Linear Algebra

1.2 Calculus and Optimization

Multivariate Calculus by Imperial College London

1.3 Probability

Probability Theory from The Bright Side of Mathematics

2. Coding

Python for Beginners by Mosh (Skip if you already know basics of Python)
Learn Git from Mosh (Skip if you already know Git)
Learn to make virtual environment in Python
Pytorch beginner series by PyTorch (Do not try to understand everything about PyTorch at once, just get some idea then learn more about it as you code AI models from Machine Learning and Deep Learning section)

3. Machine Learning

Machine Learning by Josh Starmer from StatQuest
Do implement all the basic machine learning models like Linear Regression, Logistic Regression, K-Nearest Neighbors, Decision Trees, Random Forest, Gradient Boosting, etc. that are taught in the playlist in PyTorch. Youtube has a lot of tutorials on how to implement these models in PyTorch.

4. Deep Learning

Deep Learning by Josh Starmer from StatQuest
Neural networks from 3Blue1Brown
Implement all the models learned in the playlists in PyTorch. You can follow this playlist
Become proficient in coding Deep Learning models by following Andrej Karpathy’s Zero to Hero series

Finally I will again emphasize that this is not the end of the learning journey, rather it is the beginning. I have designed this roadmap so that you learn 90% of the most important topics in AI which will act as a foundation for your further learning a particular subdomain of AI.

Asking Bugatti Owners How They Got RICH

Sat, 01 Feb 2025 00:00:00 +0000

2:30 - Fuck the Bussiness: It does not matter how large the business is, the only metric that matters is the money in your pocket. Even after the taxes.
3:13 - Create friendships with your clients: And then be like brotheers to them.
3:37 - Most proffessors are too scared to enter the real world: That is why they keep professing! They are pros at nothing. They do not know how the real world works.
3:53 - Never quit: That is what separates winners from losers.
5:48 - Watches YouTube
6:41 - The world will teach you if you listen: Fail fast, fail cheap, fail often. If you never swing you never hit a homerun(thats the game right there).
7:12 - You have to love what you do, otherwise you do not work so hard: Do not make descisons that have potential to take joy away from the project.
7:14 - Always over deliver on the promise: Always keep your word. Even though there are contracts but if i shake your hand its a done deal..
10:17 - Go through the day looking at the problems that can be solved: What’s a better way? Look for entreprenurial opportunities everywhere. And think out of the box.
14:51 - If you pay peanuts, you will attract monkeys: If you pay low, you will get guaranteed low quality.
17:14 - Stop trying to convince people about your vision: Do not discuss your vision with all tom dick and harry.
18:47 - People better than you will give permission to become better yourself: There can even be people in your family and friends who are not doing that, they are telling you to be humble, quit, grateful.

Free Resources

Sat, 01 Feb 2025 00:00:00 +0000

📖 Read my free books

🎯 Learn AI for Free

Complete AI Roadmap & Learning Community

🔄 Keep up to date with AI progress

Directory of AI Resources

📚 Read my handwritten notes for free

🎓 IISC Bangalore Resources

💝 Support Me

Buy me a coffee

How This 76 Year Old Turned thousands into millions

Sat, 01 Feb 2025 00:00:00 +0000

2:58 - Bullshit flies if you are selling: And its OK. People will believe it.
3:37 - People will buy it: You just have to say it and people will buy it.
4:03 - Urgency is a great motivator
4:53 - Learn to listen, people are always talking: The harder you listen, the better you will be at making deal.
7:41 - Want it so bad that you can almost taste it
10:11 - Who is a great entrepreneur?: Someone who is going to get back up. This is the “only” difference. You are mostly born with it, but can be exagerrated.
11:01 - I alwayas loose money, who cares, its a long long game: Going up and down have fun.
12:05 - You will always have doubts
15:15 - Be authentic: Everyone like the authentic ones.
16:20 - Nothing wrong with over preparing

Motivational links

Sat, 01 Feb 2025 00:00:00 +0000

These are some things that I stumbled on that motivate me:

https://youtube.com/shorts/q1txLXeFyLg?si=0cSsSBFv93QzMN6I

Resources to keep up with AI

Sat, 01 Feb 2025 00:00:00 +0000

I have not listed the resources in any particular order(say these are the resources for keeping update with research, these are the resources for latest news, etc). The resources are listed paltform wise. These resources are the best resources to keep updated with research, controversies, latest news, etc all rolled into one place(That is how we keep the things intresting)

Youtube

What should i do if AI is improving

Sat, 01 Feb 2025 00:00:00 +0000

Given that

Seems that AI is improving at a rapid pace

Following jobs will be lost

Text to video models etc will replace Actors, producers, writers etc.
Text to speech models will replace Singers
AI pimps will replace Only fans, Ticktok creators etc
Tools like Claude computer use will replace all remote workers
Indie hackers relying on microsaas will be replaced by AI. Infact most SaaS will be replaced by AI.
Bloggers and most sites will be replaced by SearchGPTs.
All consultants like lawyers who do not go to court, accountants etc.
I cannot find a job where you do 9 to 5 on a computer and you not getting replaced by AI.
99% of all book authors except for people like Marcus Aurelius will be replaced by AI.
Online courses, teachers even nice people like Khan Academy will be replaced by AI. Heck i do not think that even great colleges like IISC and IITs teaching computer science will be spared.
Copywriters
Affiliate marketers because of tools like perplexity shopping

If we are able to build robots

If we are able to get robots which also seems likely then even jobs that require physical actions will also be replaced by AI.

How to prepare for this brave new world

Invest all your money to buy AI tools and tech upgrades.
Be a super learner.
Obviously it will become easier to make a software/videos/agents etc in the future but you still have to push the current tools to absolute limits - so that you are prepared for the future when these tools mature.

These effors will not be in vain

Building an audience on X.
Keeping up with AI progress. Learning about AI.
Evergreen phylosophy etc
Spending time with family and friends.
I building AI agents will be the only thing that matters if there was one thing that mattered.

What i think will still be valuable

Bitcoin, Ethereum maybe
Land
I will not bet that thousand year old trend of gold being valueable will be replaced.

I feel that somehow these jobs will still be there

Sports
I would have written artists, actors etc but most of them are crooks in real life. We do not watch them because we are inspired by them in real life.

Contarary to popular belief

I think AI will be able to empatize even better than humans.

These are the things that excite me

AR glasses with a personal coach/secretary.
GPT 5/6 level AI agents.

Why we need a virtual environment

Sat, 01 Feb 2025 00:00:00 +0000

What is a Programming Language?

A programming language is a formal system that allows humans to give instructions to computers. It acts as a bridge between human-readable commands and machine code. Python, which we’ll focus on in this article, is a popular high-level programming language known for its readable syntax and extensive library ecosystem.

What is a Module?

A module in Python is a file containing Python code that can be imported and used in other Python programs. It’s a fundamental way to organize and reuse code by grouping related functions, classes, and variables together.

Adaboost

Wed, 01 Jan 2025 00:00:00 +0000

Recall

Recall from our discussion on boosting we defined the ensemble model $H_T(x)$ as:

$$ H_T(x) = \sum_{t=1}^T \alpha_t h_t(x) $$

the problem of finding the next weak learner reduces to:

$$ h_T = \arg\min_{h \in H} \sum_{i=1}^n r_i \cdot h(x_i) $$

where $$r_i = -\frac{\partial R(H_{T-1}(x_i))}{\partial H_{T-1}(x_i)}$$

Problem setting

Let the dataset be $D = {(x_i, y_i)}_{i=1}^n$ where $y_i \in {-1, 1}$.

The loss function is given by:

$$L(H(x_i), y_i) = \exp(-y_i H(x_i))$$

Aggregated posterior mismatch

Wed, 01 Jan 2025 00:00:00 +0000

Why did we not have such a problem in EM algorithm?

The Variational Autoencoder (VAE) architecture can be visualized as follows:

Before expanding the KL divergence term, recall that:

latent variable $q_{\phi}(z|x)$ is the approximate posterior (i.e., how likely the latent variable $z$ is given the input $x$). $p_{\theta}(z|x)$ is the true posterior (i.e., the actual distribution of $z$ given $x$ under the model). The KL divergence measures how much $q_{\phi}(z|x)$ deviates from $p_{\theta}(z|x)$:

Anderson's result

Wed, 01 Jan 2025 00:00:00 +0000

Anderson’s result states that for a forward SDE of the form:

$$dx = f(x,t)dt + g(t)dB_t$$

The corresponding reverse SDE takes the form:

$$dx = \left(f(x,t) - g^2(t)\nabla_x(\log p_t(x))\right)dt + g(t)dB_t$$

where:

$f(x,t)$ is the drift term of the forward process
$g(t)$ is the diffusion coefficient
$\nabla_x(\log p_t(x))$ is the score function
The reverse process has the same diffusion term $g(t)dB_t$ as the forward process

This result shows that to reverse a diffusion process, we need to:

Backpropagation

Wed, 01 Jan 2025 00:00:00 +0000

Notations

$L$: number of layers in the network.
$w_{jk}^l$: weight connecting $k^{th}$ neuron of $(l-1)^{th}$ layer to $j^{th}$ neuron of $l^{th}$ layer
$b_j^l$: bias of $j^{th}$ neuron in $l^{th}$ layer
$a_j^l$: output of $j^{th}$ neuron in $l^{th}$ layer
$z_j^l$: preactivation output of $j^{th}$ neuron in $l^{th}$ layer

Here: $$z_j^l = \sum_k w_{jk}^l a_k^{l-1} + b_j^l$$

Also: $$a_j^l = \sigma(z_j^l)$$ $$a_j^l = \sigma(\sum_k w_{jk}^l a_k^{l-1} + b_j^l)$$

Derivation

Let’s assume we use squared loss function: $$L = \frac{1}{2} |a^L - y|^2$$

Bagging and Random Forest

Wed, 01 Jan 2025 00:00:00 +0000

Bagging

Let the original data be D.
Create n new datasets by sampling with replacement from D.
Each new dataset will have the same number of samples as the original dataset, but some samples will be repeated, and some will be excluded.
Train a model on each of the new datasets.
The final prediction is made by aggregating the predictions of all models:
- For classification, the final prediction is made by taking the majority vote of the predictions of all models.
- For regression, the final prediction is made by taking the average of the predictions of all models.

Random Forest - Bagged descion trees

Bagging of decision trees.
Each decision tree is trained on a random subset of the features.
The final prediction is made by taking the majority vote of the predictions of all decision trees.

Bayes classifier

Wed, 01 Jan 2025 00:00:00 +0000

Let the data be D = {($x_1$, $y_1$), ($x_2$, $y_2$), …, ($x_n$, $y_n$)} where $y_i$ is the label of the feature vector $x_i$.

Let the hypothesis space be $H = {h | h: X \to Y}$.

The Bayes classifier is defined as: $$h_B(x) = \begin{cases} 1 & \text{if } P(y=1 | x=x_i) > P(y=0 | x=x_i) \ 0 & \text{if } P(y=1 | x=x_i) \leq P(y=0 | x=x_i) \end{cases}$$

Notations

$P(y)$ is also called the prior probability of class $y$ or class prior probability.
$P(x|y)$ is also called the likelihood of $x$ given class $y$.
$P(y|x)$ is also called the posterior probability of class $y$ given $x$.
$P(x)$ is also called the evidence.

Bayes classifier is the best classifier for 0-1 loss function

To prove that the Bayes classifier is the best classifier for the 0-1 loss function, we need to show that it minimizes the expected risk (error) for any given input $x$.

Bias variance tradeoff

Wed, 01 Jan 2025 00:00:00 +0000

It can be shown that average risk for square error loss can be decomposed into three components:

R_{\text{avg}}(h) = \mathbb{E}_{P_D} \mathbb{E}_{P_{x,y}} \left[ \left( h_D(x) - \hat{h}(x) \right)^2 \right] \quad \text{// variance (sensitivity to dataset)}

$$ + \mathbb{E}_{P_{x,y}} \left[ \left( \hat{h}(x) - h^*(x) \right)^2 \right] \quad \text{// bias (how different is avg classifier from optimum classifier)} $$

+ \mathbb{E}_{P_{x,y}} \left[ \left( h^*(x) - y \right)^2 \right] \quad \text{// irreducible noise (nothing can be done about this)}

where:

Boosting

Wed, 01 Jan 2025 00:00:00 +0000

In bagging, we train each model independently on a random subset of the data. In boosting, we train each model sequentially on the same data, with each subsequent model focusing on correcting the errors of combined previous model by increasing weights of misclassified datapoints

Unlike bagging, each model depends on the previous ones and its contribution to the final prediction is weighted differently.

Mathematical Formulation of Boosting

Let’s define the ensemble model $H_T(x)$ as:

Cheat sheet

Wed, 01 Jan 2025 00:00:00 +0000

Formulas:

K L divergence: $D_{KL}(p; q) = \int p(x) \log \frac{p(x)}{q(x)} dx$
F-divergence: $D_{F}(p \parallel q) = \int q(x) f\left(\frac{p(x)}{q(x)}\right) dx$
- $f: \mathbb{R}^+ \rightarrow \mathbb{R}$ is lower semi continous convex function such that $f(1) = 0$
- If $f(x) = x \log x$, then $D_{F}(p \parallel q) = D_{KL}(p \parallel q)$
- If $f(x) = x\log(x) - (x+1)\log(x+1)$, then $f^*=-\log(1-\exp^x)$, then $D_{F}(p \parallel q)$ = Jensen-Shannon divergence = $D_{JS}(p \parallel q) = \frac{1}{2}D_{KL}(p \parallel m) + \frac{1}{2}D_{KL}(q \parallel m)$, where $m = \frac{1}{2}(p + q)$
Jensen’s inequality: For a convex function $f$ and a random variable $X$: $f(\mathbb{E}[X]) \leq \mathbb{E}[f(X)]$

Conditional GANs(cGANs)

Wed, 01 Jan 2025 00:00:00 +0000

A Conditional Generative Adversarial Network (cGAN) is a type of Generative Adversarial Network (GAN) where the generation process is conditioned on some additional information, such as class labels or data from other modalities.

Architecture modification

Generator

Concatenate the conditional information Y with the noise input Z and feed it to the generator.

Here Y can be:

One hot encoded class label
Text embedding etc

Discriminator

The discriminator in a cGAN is modified similarly to the generator. Concatenate the conditional information Y with the input image X and feed this combined input (X, Y) to the discriminator.

Conditional score matching

Wed, 01 Jan 2025 00:00:00 +0000

When we had unlabeled data, we estimated the score function:

$$s(x) = \nabla_x \log p(x)$$

Now, we have labeled data ie $D = {(x_i, y_i)}_{i=1}^N$ we instead estimate the conditional score function:

$$s(x|y) = \nabla_x \log p(x|y)$$

Where:

$s(x)$ is the unconditional score function
$s(x|y)$ is the conditional score function
$p(x)$ is the data distribution
$p(x|y)$ is the conditional data distribution given some condition $y$

This can be done in 2 ways:

Convex conjugate

Wed, 01 Jan 2025 00:00:00 +0000

Convex conjugate is also known as the Fenchel conjugate or the Legendre transform

Definition

Let $f: R^n → R$ be a convex function. The convex conjugate of $f$, denoted as $f^*$, is defined as:

$$f^*(y) = \underset{x \in \text{dom} f}{\sup} (y^T x - f(x))$$

Where:

$y \in R^n$ is the variable
$\text{dom} f$ is the domain of $f$
$\sup$ denotes the supremum (least upper bound)

Properties

1. Convexity: $f^{**}(x)$ is also convex

Proof: Let $y_1, y_2 \in \mathbb{R}^n$ and $\lambda \in [0,1]$. We need to show that:

DDIM (Denoising Diffusion Implicit Models)

Wed, 01 Jan 2025 00:00:00 +0000

Motivation

During inference, we often had to sequentially traverse through 1000 time steps because of the Markov property assumption, which can be computationally intensive.

Is it possible to give up the Markov property assumption so that we can train the same way but sample faster - not have to traverse through 1000 time steps?

Recall the DDPM

Forward process: $$q(x_t | x_0) = \int q(x_{t:1} | x_0) dx_{1:(t-1)}$$ $$q(x_t | x_0) = \mathcal{N}(x_t; \sqrt{\alpha_t} x_0, (1 - \alpha_t) \mathbb{I})$$ $$\Rightarrow x_t = \sqrt{\alpha_t} x_0 + \sqrt{1 - \alpha_t} \epsilon, \quad \epsilon \sim \mathcal{N}(.)$$ Note that here $\alpha_t$ is $\bar{\alpha}$ that we studied in DDPM.

DDIM inversion

Wed, 01 Jan 2025 00:00:00 +0000

$X_T$ to $X_0$

For $\sigma = 0$, the DDIM process becomes deterministic:

$$x_{t-1} = \sqrt{\alpha_{t-1}}\left(\frac{x_t - \sqrt{1-\alpha_t}\epsilon_\theta^{(t)}(x_t)}{\sqrt{\alpha_t}}\right) + \sqrt{1-\alpha_{t-1}}\epsilon_\theta^{(t)}(x_t)$$

where:

$\epsilon_\theta^{(t)}(x_t)$ is the predicted noise from our trained model
$\alpha_t$ is the cumulative product of $(1-\beta_t)$

This deterministic nature means that:

For a given $x_T$, we can find a unique $x_0$ by following the forward process
Given this $x_T$, we can recover the exact same $x_0$ through the reverse process

This property makes DDIM particularly useful for:

DDPMs as SDEs

Wed, 01 Jan 2025 00:00:00 +0000

Step 1 of big picture (Stochastic forward process to continuous SDE)

Now let’s see how DDPMs can be viewed as discretizations of SDEs.

The forward process in DDPMs is given by: $$x_{i+1} = \sqrt{1-\beta_{i+1}}x_i + \sqrt{\beta_{i+1}}\epsilon_i \quad \text{where} \quad \epsilon_i \sim \mathcal{N}(0,I)$$

Let’s derive the corresponding continuous-time SDE:

First, rewrite the discrete equation: $$x_{i+1} - x_i = (\sqrt{1-\beta_{i+1}} - 1)x_i + \sqrt{\beta_{i+1}}\epsilon_i$$
As the time step $\Delta t \to 0$, we can approximate: $$\sqrt{1-\beta_{i+1}} \approx 1 - \frac{\beta_{i+1}}{2}$$ This approximation comes from the Taylor series expansion of $\sqrt{1-x}$ around $x=0$: $$\sqrt{1-x} = 1 - \frac{1}{2}x - \frac{1}{8}x^2 - \frac{1}{16}x^3 + \cdots$$ We keep only the first two terms, assuming $\beta_{i+1}$ is small.

Decision Trees

Wed, 01 Jan 2025 00:00:00 +0000

How to descion tree works

At each node, a question is asked about the data that splits the data into two or more non-overlapping subsets. $$ \text{Question: } x_j \leq \theta $$ then the data is split into two subsets, one where $x_j \leq \theta$ and one where $x_j > \theta$.
The process is repeated till we reach leaf node, that classifies the datapoint to a region of the feature space.

Denoising Diffusion Probabilistic Models (DDPM) - part 1

Wed, 01 Jan 2025 00:00:00 +0000

DDPM as special case of variational autoencoders (VAE)

Impose following structure on the VAE:

Multiple latent variables, where each latent variable is associated with a timestep.
Dim of all the latent variables is the same as the dim of the data.
Assume a non learnable encoder unlike VAE where the encoder was learnable.

Notations

$X_1, …, X_T$: Denote latent variables at different timesteps
$p(X_0)$: Denote the model data distribution
$X_{1:T} = (X_1, X_2, …, X_T)$: Denote the sequence of all latent variables
$q(X_{1:T}|X_0)$: Denote the latent posterior

Latent posterior

Because $q(X_{1:T}|X_0)$ is not learnable, we need to define it - we make it first order markovian chain with gaussian transitions:

Denoising Diffusion Probabilistic Models (DDPM) - part 2

Wed, 01 Jan 2025 00:00:00 +0000

We were using latent variables $X_t$ to model the data as:

$$\log p_\theta(x_0) = \log \int p_\theta(x_{0:T}) dx_{1:T}$$

This formulation expresses the log-likelihood of the observed data $x_0$ as an integral over all possible latent variable sequences $x_{1:T}$. The model’s goal is to maximize this log-likelihood, which involves learning the parameters $\theta$ that define the generative process from $x_T$ back to $x_0$.

Also:

$p_\theta(x_T) = \mathcal{N}(0, I)$ - The prior distribution of the final latent variable is a standard normal distribution
$p_\theta(x_{0:T}) = p_\theta(x_T) \prod_{t=1}^T p_\theta(x_{t-1}|x_t)$ - The joint distribution of all latent variables is factorized using the chain rule
$q(x_t|x_{t-1}) = \mathcal{N}(x_t; \sqrt{\alpha_t}x_{t-1}, (1-\alpha_t)I)$ - The forward process transition probability is a Gaussian distribution
$q(x_{1:T}|x_0) = \prod_{t=1}^T q(x_t|x_{t-1})$ - The posterior of all latent variables given the data is factorized using the chain rule

Because the log likelihood is intractable, we constructed the ELBO of the model:

Denoising Diffusion Probabilistic Models (DDPM) - part 3

Wed, 01 Jan 2025 00:00:00 +0000

Recall that:

T_3 = - \sum_{t=2}^T \mathbb{E}_{q(x_t|x_0)} [D_{KL}(q(x_{t-1}|x_t,x_0) \| p_\theta(x_{t-1}|x_t))]

Here:

$q(x_{t-1}|x_t,x_0)$ is the noising distribution after applying the forward process for $t-1$ steps
$p_\theta(x_{t-1}|x_t)$ is the model’s predicted denoising distribution after applying the reverse process

We also know that:

$$q(x_{t-1}|x_t,x_0) = \mathcal{N}(x_{t-1}; \mu_q(x_t,x_0), \Sigma_q(t))$$

where $\mu_q(x_t,x_0)$ and $\Sigma_q(t)$ are the mean and covariance of the posterior distribution and:

\mu_q(x_t, x_0) = \frac{\sqrt{\alpha_t}(1-\bar{\alpha}_{t-1})x_t + \sqrt{\bar{\alpha}_{t-1}}(1-\alpha_t)x_0}{1-\bar{\alpha}_t}

The covariance $\Sigma_q(t)$ is given by:

Difference between SMLD and DDPM

Wed, 01 Jan 2025 00:00:00 +0000

Note that: NCSN: Noise Conditional Score Network is same as SMLD: Score matching Langevin Dynamics

Inference

The inference processes also differ between SMLD and DDPM:

SMLD (Noise Conditional Score Networks):
- We start with a sequence of noise scales arranged from largest to smallest: $${\sigma_i}_{i=1}^L, \quad \sigma_1 > \sigma_2 > … > \sigma_L \quad (L \approx 10)$$
- Start with random Gaussian noise: $$x_0 \sim \mathcal{N}(0, \sigma_1^2I)$$
- For each noise scale $\sigma_i$ from $i=1$ to $L$:

Distillation

Wed, 01 Jan 2025 00:00:00 +0000

Teacher model is a large model that was trained on a large dataset as usual.
Student model is a smaller model that is trained to mimic the behavior of the teacher model.
The student model has 2 losses:

\mathcal{L} = \mathcal{L}_\text{task} + \mathcal{L}_\text{distill}

- $\mathcal{L}_\text{task}$ is the task-specific loss. - $\mathcal{L}_\text{distill}$ is the distillation loss - This loss is used to make the student model's output close to the teacher model's softened outputs.

Domain adversarial networks (DANs)

Wed, 01 Jan 2025 00:00:00 +0000

Also known as Unsupervised domain adaptation (UDA)

Introduction

Domain adversarial networks (DANs) are a type of neural network architecture designed to address the problem of domain shift, where the distribution of data differs between the training and testing phases. DANs use adversarial training to learn domain-invariant features that are robust across different domains.

Aim

The aim is to learn a classifier that performs well on both the source and target domains, even though the target domain may have different characteristics than the source domain. For example, we train a model to classify medical images at company A and then we want to use this model to classify medical images at some other hospital.

Ensemble learning

Wed, 01 Jan 2025 00:00:00 +0000

Ensemble learning is a machine learning technique that combines multiple models to improve the overall performance and robustness of the prediction. It is based on the idea that by aggregating the predictions of several models, we can achieve better results than any single model alone.

Expectation Maximization (EM) algorithm

Wed, 01 Jan 2025 00:00:00 +0000

Let the data be $D = {t_i}_{i=1}^N$ where $t_i \in \mathbb{R}^d$ are iid $p_t(t)$ data points.

Latent variable models

Let each data point $x_i$ be associated with a latent variable $z_i$ that is not observed. $z_i$ is a random variable that takes values in some finite set ${1, \ldots, K}$ and represents the membership of $x_i$ to one of $K$ clusters.

Hence the data can be represented as

D = \{(t_i, z_i)\}_{i=1}^N

where $t_i$ is the observed data and $z_i$ is the latent variable and $(t_i,z_i)$ are iid $p_{tz}$.

The marginal distribution of the observed data is given by:

F-Divergence

Wed, 01 Jan 2025 00:00:00 +0000

F-divergence is a generalized measure of difference between two probability distributions. For probability distributions $P$ and $Q$ over a event space $X$, the F-divergence is defined as:

$$D_f(P || Q) = ∫_X q(x) f(\frac{p(x)}{q(x)}) dx$$ Where:

Generator function $f: R^+ → R$ is a lower semi-continuous convex function with $f(1) = 0$.
The F in F-divergence comes from generator “f"unction.
$p(x)$ and $q(x)$ are the probability density functions of $P$ and $Q$ respectively

Some examples of F-divergences include:

Frechet Inception Distance (FID)

Wed, 01 Jan 2025 00:00:00 +0000

Why generative models cannot be evaluated like discriminative models

Usually, we train a deep learning model on training data and then evaluate it on test data. However, for generative models, we cannot directly compare the generated samples to the test data in the same way we evaluate discriminative models. This is because:

Generative models produce new, unique samples rather than predictions on existing data.
There’s no one-to-one correspondence between generated samples and real data points.
We need to assess the quality, diversity, and realism of the generated samples, which requires different metrics.
Traditional evaluation metrics like accuracy or mean squared error are not applicable to generative tasks.

Therefore, we need specialized methods to evaluate generative models, such as the Frechet Inception Distance (FID), which measures the similarity between the distribution of generated samples and the distribution of real data.

Generalized Linear Regression

Wed, 01 Jan 2025 00:00:00 +0000

Generalized Linear Regression extends the concept of linear regression by projecting the data into a higher-dimensional space before performing linear regression. This allows us to capture more complex relationships in the data.

Let the data be $(x_i, y_i)$ for $i = 1, 2, …, n$ where $x_i \in \mathbb{R}^d$ and $y_i \in \mathbb{R}$.

We define a feature map $\phi: \mathbb{R}^d \rightarrow \mathbb{R}^m$, where $m > d$. This map projects our original features into a higher-dimensional space.

Generative Adversarial Networks (GANs)

Wed, 01 Jan 2025 00:00:00 +0000

Recall that generative models try to learn the data distribution $P_{data}$.

GANs try to learn $P_{data}$ by approximating it by $P_{generator}$ by minimizing F-divergence $D_f(P_{data} || P_{generator})$.

Derivation of GANs

Let $\theta$ be the parameters of the generator such that it maps a sample $z \sim N(0, I)$ to a sample $x \sim p_{generator}$.

Then, the optimal parameters for the generator can be expressed as:

$$\theta^* = \underset{\theta}{\text{argmin}} D_f(P_{data} || P_{generator})$$

Generative vs Discriminative Models

Wed, 01 Jan 2025 00:00:00 +0000

Generative Model

Let the training dataset be

data = \{(x_i)\}_{i=1}^N

where $x_i \in \mathbb{R}^d$ are called data points. These data points are iid samples from the true data distribution $P(x)$.

Aim of generative model is to extimate $P(x)$ using the training data and sample new data points from it.
GANs, GMMs etc are a example of generative model. In GANs we use training data to train a neural network to learn the distribution of the data. Once we have learnt this distribution we can sample new data points from it.

Hypothesis space (H)

Wed, 01 Jan 2025 00:00:00 +0000

Let the data be D = {($x_1$, $y_1$), ($x_2$, $y_2$), …, ($x_n$, $y_n$)} where $y_i$ is the label of the feature vector $x_i$.

A hypothesis function $h$ is a function that maps the feature vector $x$ to the label $y$, ie $$h: X \to Y$$

The hypothesis space is the set of all possible hypothesis functions, denoted as $$H = {h | h: X \to Y}$$

During the learning process, we try to find the best hypothesis function $h$ from the hypothesis space $H$ that minimizes the error between the predicted label and the true label.

Index for Artificial Intelligence Topics

Wed, 01 Jan 2025 00:00:00 +0000

Index for Deep Learning Topics

Wed, 01 Jan 2025 00:00:00 +0000

GANs

VAEs

DDPM

Score matching

SDEs

Sequence2sequence models

Self supervised learning

Misc

Distillation

Cheatsheet

Cheatsheet

Infinite horizon discounted cost problems

Wed, 01 Jan 2025 00:00:00 +0000

In infinite horizon discounted cost problems, we consider scenarios without a termination state and an infinite sequence of decisions. Let’s understand the key components and mathematical formulation.

Problem Setting

State Space: Set of states $S = {1,2,…,n}$. Note that there is no terminal state unlike in finite horizon problems.
Action Space: For each state $i$, set of feasible actions $A(i)$
- Total action space: $A = \bigcup_{i \in S} A(i)$
Cardinality: Both $|S|$ and $|A|$ are finite

Optimal Cost Function

The optimal cost function $J^*(i)$ represents the minimum total discounted cost starting from state $i$:

Info VAEs: Information Maximizing Variational Autoencoders

Wed, 01 Jan 2025 00:00:00 +0000

InfoVAE addresses the aggregated posterior mismatch problem in standard VAEs by modifying the objective function to explicitly match the aggregated posterior with the prior distribution.

Key Ideas

If the aggregated posterior $q_\phi(z)$ becomes a Dirac delta, all inputs $x$ map to the same latent code $z$
In such a scenario, the latent code $z$ would contain no information about the input $x$
The standard VAE objective does not directly encourage the matching between $q_\phi(z)$ and the prior $p(z)$

InfoVAE Objective Function

The standard VAE objective is given by:

Info-NCE

Wed, 01 Jan 2025 00:00:00 +0000

Idea

Instead of using 1 negative sample, why not use more?

Formulation

Let us formalize the Info-NCE framework:

Given:
- A data point $x \in \mathcal{X}$ from input space $\mathcal{X}$
- A context $c \in \mathcal{C}$ from context space $\mathcal{C}$
- A positive sampling distribution $p_{pos}(x,c)$
- A negative sampling distribution $p_{neg}(x,c)$
We sample:
- One positive sample $x^+ \sim p_{pos}(x,c)$
- $N-1$ negative samples
$\{x^-_i\}_{i=1}^{N-1} \sim p_{neg}(x,c)$
The InfoNCE loss is defined as:

\mathcal{L}_{\text{InfoNCE}} = -\mathbb{E}\left[\log \frac{\exp(f_\theta(x)^\top f_\theta(x^+))}{\exp(f_\theta(x)^\top f_\theta(x^+)) + \sum_{i=1}^{N-1} \exp(f_\theta(x)^\top f_\theta(x^-_i))}\right]

where:

Introduction to Controlled Markov Chains and Markov Decision Processes

Wed, 01 Jan 2025 00:00:00 +0000

Controlled Markov Chain (CMC)

A Controlled Markov Chain (CMC) is a stochastic process ${X_n}_{n \geq 0}$ that evolves over discrete time steps $n = 0, 1, 2, \ldots$.
At each time step $n$, the system is in a state $X_n$ belonging to a state space $S$. To influence the evolution of this process, we apply a control or action $Z_n$ from a set of feasible actions $A(X_n)$ that depends on the current state $X_n$. Thus, $Z_n \in A(X_n)$.

Introduction to Reinforcement Learning

Wed, 01 Jan 2025 00:00:00 +0000

Core Idea of Reinforcement Learning

Reinforcement Learning (RL) is a computational approach to learning whereby an agent seeks to maximize some notion of cumulative reward by taking actions in an environment. This process is akin to training a dog to perform a trick: you cannot dictate every move, but you can provide rewards when it performs correctly. In RL, the agent learns through trial and error, receiving feedback in the form of rewards or penalties based on its actions. The primary objective is to develop a policy, a strategy that guides the agent to take actions that maximize the total accumulated reward over time.

JEPA - Joint Embedding Prediction and Autoencoding

Wed, 01 Jan 2025 00:00:00 +0000

Idea

Joint Embedding Predictive Architecture (JEPA) combines predictive learning with representation learning in a unified framework.

Formulation

Let us formalize the JEPA framework:

Given:
- Input space $\mathcal{X}$
- Context patch $x_c \in \mathcal{X}$
- Target patch $x_t \in \mathcal{X}$
- Context encoder $f_\theta: \mathcal{X} \rightarrow \mathcal{Z}$
- Target encoder $g_\phi: \mathcal{X} \rightarrow \mathcal{Z}$
- Predictor network $h_\psi: \mathcal{Z} \rightarrow \mathcal{Z}$
The process works as follows:
- Encode context: $z_c = f_\theta(x_c)$
- Encode target: $z_t = g_\phi(x_t)$
- Predict target embedding:
$\hat{z}_t = h_\psi(z_c)$
The objective is to minimize prediction error in embedding space:

K Nearest neighbour (KNN)

Wed, 01 Jan 2025 00:00:00 +0000

Basic idea

Recall that we had derived the following equation for non-parametric density estimation:

$$p(x) = \frac{k}{nV}$$

where:

$k$ is the number of data points in region $R$
$n$ is the total number of data points in the dataset $D$
$V$ is the volume of the region $R$

In K-Nearest Neighbour (KNN) method, we fix the volume $V$ and count $k$

Problem setting

Given a set of data points $D = {x_1, y_1}, {x_2, y_2}, …, {x_n, y_n}$, where $x_i \in \mathbb{R}^d$ is the feature vector and $y_i \in {1, 2, …, C}$ is the class label.

K-means clustering

Wed, 01 Jan 2025 00:00:00 +0000

K-means clustering also known as K-means algorithm or hard clustering

Recall that in mixture model clustering

Data was:

D = \{x_i\}_{i=1}^N

We assumed latent variable $z_i$ is associated with each data point $x_i$. Hence data became $D = \{(x_i, z_i)\}_{i=1}^N$.

We assumed

$$p_\theta(x) = \sum_{j=1}^m p_\theta(x|z=j) p_\theta(z=j) = \sum_{j=1}^m N(x; \mu_j, \Sigma_j) \pi_j$$

Where:

$N(x; \mu_j, \Sigma_j)$ is the Gaussian probability density function for cluster $j$, with mean $\mu_j$ and covariance matrix $\Sigma_j$.
$\pi_j$ is the probability of a data point belonging to cluster $j$.

K-means clustering

Now let’s make the following assumptions to simplify our model:

KL Divergence - Kullback-Leibler Divergence

Wed, 01 Jan 2025 00:00:00 +0000

KL divergence is a measure of how one probability distribution differs from another probability distribution.
Mathematically, for two probability distributions $P(x)$ and $Q(x)$, the KL divergence from $Q$ to $P$ is defined as:

$$D_{KL}(P || Q) = \sum_{x} p(x) \log\left(\frac{p(x)}{q(x)}\right) = E_{x \sim p(x)}\left[\log\left(\frac{p(x)}{q(x)}\right)\right]$$ for discrete distributions

$$D_{KL}(P || Q) = \int p(x) \log\left(\frac{p(x)}{q(x)}\right) dx = E_{x \sim p(x)}\left[\log\left(\frac{p(x)}{q(x)}\right)\right]$$ for continuous distributions

where the sum/integral is over all possible events $x$. And $p(x)$ and $q(x)$ are the probability density functions of distributions $P(x)$ and $Q(x)$ respectively.

KL Divergence - Kullback-Leibler Divergence

Wed, 01 Jan 2025 00:00:00 +0000

KL divergence is a measure of how one probability distribution diverges from second, probability distribution.

Mathematically, for two probability distributions $P(x)$ and $Q(x)$, the KL divergence from $Q$ to $P$ is defined as:

$D_{KL}(p || q) = Σ p(x) * log\left(\frac{p(x)}{q(x)}\right)$ for discrete distributions

$D_{KL}(p || q) = ∫ p(x) * log\left(\frac{p(x)}{q(x)}\right) dx$ for continuous distributions

where the sum/integral is over all possible events $x$. And $p(x)$ and $q(x)$ are the probability density functions of distributions $P(x)$ and $Q(x)$ respectively.

Learning problem

Wed, 01 Jan 2025 00:00:00 +0000

We are given a dataset $D = {(x_i, y_i)}_{i=1}^n$ sampled from an unknown distribution $P$.

We want to find a hypothesis function $h$ out of the complete hypothesis space $H$ that minimizes the risk function $R(h)$.

Learning the system model from data and Monte Carlo methods

Wed, 01 Jan 2025 00:00:00 +0000

Recap

We have covered several key topics in reinforcement learning:

Fundamentals and Problem Formulation
- Basic concepts and terminology of reinforcement learning and problem formulation
Multi-armed bandit problems(Single stage multiple action problems)
- Solved using greedy, ε-greedy, UCB, and gradient-based methods
Markov Decision Processes ie MDPs(Known system model (state transition probabilities and reward/cost functions))
1. Infinite horizon problems($n<\infty$ but deterministic) - Dynamic programming solutions to the Bellman equation for finding the optimal policy
2. Stochastic shortest path problems($n<\infty$ but random) - Bellman equation formulations, value iteration, policy iteration
3. Discounted cost problems ($n \to \infty$) - Bellman equation formulations, policy iteration, value iteration

Learning the system model from data

From now on we shall assume we do not know the system model (state transition probabilities and reward/cost functions) and we need to learn it from data. The Data is in the form of trajectories generated by the system:

Linear Discriminant Analysis(LDA)

Wed, 01 Jan 2025 00:00:00 +0000

Let the data be $D = {(x_i, y_i)}_{i=1}^N$ where $x_i \in \mathbb{R}^d$ and $y_i \in {1, 2}$

Let’s assume the parametric form of the conditional density $p(x|y)$ is:

$$p(x|y=1) \sim N(x; \mu_1, \Sigma)$$ $$p(x|y=0) \sim N(x; \mu_2, \Sigma)$$

where $N(x; \mu, \Sigma)$ denotes a multivariate Gaussian distribution with mean $\mu$ and covariance matrix $\Sigma$. Note that we assume the covariance matrix $\Sigma$ is the same for both classes, which is a key assumption in LDA.

Linear regression

Wed, 01 Jan 2025 00:00:00 +0000

Let the data be $(x_i, y_i)$ for $i = 1, 2, …, n$ where $x_i \in \mathbb{R}^d$ and $y_i \in \mathbb{R}$.

then we can model $y_i$ as:

$$Y = \beta_0 + \beta_1X + \varepsilon$$

Where:

$Y \in \mathbb{R}$ is the dependent variable
$X \in \mathbb{R}^d$ is the independent variable (or feature vector in higher dimensions)
$\beta_0$ is the y-intercept (bias term)
$\beta_1$ is the slope (or coefficient vector in higher dimensions)
$\varepsilon$ is the error term

Ideal regressor

For mean squared error loss, the ideal regressor is defined as:

Logistic regression also known as logit regression or binary classification

Wed, 01 Jan 2025 00:00:00 +0000

Let the data be D = {($x_1$, $y_1$), ($x_2$, $y_2$), …, ($x_n$, $y_n$)} where $y_i \in {0, 1}$ is the binary label of the feature vector $x_i$.

Then the logistic regression model is defined as: $$P(y_i = 1 | x_i) = \frac{1}{1 + e^{-w^T x_i+b}}$$ $$P(y_i = 0 | x_i) = \frac{1}{1 + e^{w^T x_i+b}}$$

where $w$ is the parameter vector.

Note that $$P(y_i = 1 | x_i) + P(y_i = 0 | x_i) = 1$$

Loss function (L)

Wed, 01 Jan 2025 00:00:00 +0000

Let the data be D = {($x_1$, $y_1$), ($x_2$, $y_2$), …, ($x_n$, $y_n$)} where $y_i$ is the label of the feature vector $x_i$.

Let the hypothesis function be $h: X \to Y$.

Let $\hat{y_i} = h(x_i)$ be the prediction of this hypothesis function for the feature vector $x_i$, whereas the true label is $y_i$.

Then the loss function $L(y_i, \hat{y_i})$ is a function that measures the error between the predicted label and the true label.

Main takeaway

Wed, 01 Jan 2025 00:00:00 +0000

The main takeaway from this exploration is that whenever you see a random process converting one distribution to another, SDEs should come to mind as a powerful mathematical framework. The three sampling methods we studied (Langevin dynamics, DDPM, SMLD) are all discrete approximations of continuous SDEs, differing mainly in their drift and diffusion terms. Understanding them through the SDE lens helps unify these approaches and provides a foundation for developing new sampling techniques.

Make Book by Peter Levels

Wed, 01 Jan 2025 00:00:00 +0000

My takeaways from the book

Record your ideas

If you do not have idea then you need to live a better life - Be creative. Find problems in your life and solve them - then sell them as a product. Your idea list is very precious. Carry it everywhere. Note down ideas immediately. Ideas can come at any time. Do not miss them.

Templates

You should have templates for everything - Boiler plate code, launch procedures etc

Automations

Use automations exhaustively.
Only hire humans under dire circumstances otherwise just use automations.
Do not take external capital. Do not have co-founders - avoid groupthink drama. Most important thing is to not take any descions that may reduce your freedom or having fun.
Avoid physical products.

Focus on core functionality

Build only the core functionality first.

Ask for money

Ask people to pay you for your product, do not have ads. Do not sell users data.

Masked reconstruction

Wed, 01 Jan 2025 00:00:00 +0000

Idea

Contrastive learning is one way to learn representations from unlabeled data. Masked reconstruction is another way.

Formulation

Let us formalize the masked reconstruction framework:

Given:
- Input space $\mathcal{X}$
- A data point $x \in \mathcal{X}$
- A masking operator $\hat{x}$ that corrupts the input
- An encoder-decoder model $T_\theta(x)$ parameterized by $\theta$
The process works as follows:
- Apply masking operator: $\hat{x} = \text{mask}(x)$
- Pass masked input through model: $\tilde{x} = T_\theta(\hat{x})$
- Reconstruct original input: $\hat{x} \rightarrow \tilde{x} \approx x$
The objective is to minimize reconstruction loss:

Maximum aposteriori estimate(MAP)

Wed, 01 Jan 2025 00:00:00 +0000

The Maximum A Posteriori (MAP) estimate is defined as:

\theta^*_{MAP} = \arg\max_{\theta} p(\theta|V) = \arg\max_{\theta} p(V|\theta) p(\theta)

where - $\theta^*_{MAP}$ is the MAP estimate of the parameter $\theta$ - $p(\theta|V)$ is the posterior probability of the parameter $\theta$ given the observed data $V$ - $p(V|\theta)$ is the likelihood of the observed data $V$ given the parameter $\theta$ - $p(\theta)$ is the prior probability of the parameter $\theta$

Note that:

Unlike MLE, MAP estimation incorporates prior knowledge about the parameter $\theta$ and observed data $V$ where as MLE only depends on the observed data $V$.
MAP provides a balance between the likelihood of the data and the prior beliefs, leading to more robust estimates, especially when the data is limited.

Conjugate prior

A conjugate prior is a type of prior distribution $p(\theta)$ such that when multiplied with the likelihood function $p(V|\theta)$ the posterior distribution $p(\theta|V)$ that we get is in the same form as the prior $p(\theta)$.

Minimizing KL Divergence is Equivalent to Maxmimizing Likelihood

Wed, 01 Jan 2025 00:00:00 +0000

For a typical ML problem, all we have are samples from the true distribution $P(x)$ ie $data = {(x_i)}_{i=1}^N$ where $x_i \in \mathbb{R}^d$ are data points. We do not know the distribution $P(x)$ explicitly.

We try our best to estimate the true distribution $P(x)$ by $Q(x; \theta)$ where $\theta$ are the parameters of the model.

We want to know how well our model $Q(x; \theta)$ is performing. We can do this by calculating the KL divergence between the true distribution $P(x)$ and the estimated distribution $Q(x; \theta)$.

Miscellaneous Machine Learning Terms

Wed, 01 Jan 2025 00:00:00 +0000

Epoch

One complete pass of the entire training dataset for training the model.

Batch Size

Risk is defined as

$$L = \frac{1}{N} \sum_{i=1}^N l(y_i, \hat{y}_i)$$

Where $l(y_i, \hat{y}_i)$ is a general loss function that measures the discrepancy between the true value $y_i$ and the predicted value $\hat{y}_i$ for each sample.

Instead of calculating the risk over the entire dataset, however calculating the risk over complete dataset is computationally expensive. Hence we calculate the risk over a small subset of the dataset, called a batch to perform back propogation.

Mixture model clustering also known as soft clustering

Wed, 01 Jan 2025 00:00:00 +0000

Problem Setup

Data: Let the dataset be $X = {x_1, x_2, …, x_N}$, where each data point $x_i \in \mathbb{R}^d$. The data points are unlabeled.

Latent Variable: Let $z \in {1, …, m}$ be a latent variable representing the clusters, where $m$ is the total number of clusters.

We can extend the dataset as $D = {(x_i, z_i)}_{i=1}^N$, where each data point $x_i$ is associated with its (unknown) cluster $z_i$.

Multi-Armed Bandit Problem

Wed, 01 Jan 2025 00:00:00 +0000

Multi-Armed Bandit Problem

The Multi-Armed Bandit problem is a fundamental concept in reinforcement learning, where an agent must repeatedly select from multiple options, known as “arms” (akin to slot machines), to maximize its cumulative reward. This scenario is simplified by the fact that the environment is state-less, meaning the outcome of an action is solely dependent on the action itself, without any influence from a changing state.

Key characteristics of the Multi-Armed Bandit problem include:

Naive GANs

Wed, 01 Jan 2025 00:00:00 +0000

A Naive GAN (Generative Adversarial Network) refers to the basic or “vanilla” version of GANs, introduced by Ian Goodfellow in 2014.

Derivation

Recall the genereral expression for the GANs that we got by minimizing F-divergence:

\theta^* = \underset{\theta}{\text{argmin}} \underset{\phi}{\sup} \left[\mathbb{E}_{x \sim p_{data}}[T_\phi(x)] - \mathbb{E}_{x \sim p_{generator}}[f^*(T_\phi(x))]\right]

where:

$\theta$: Parameters of the generator network
$\phi$: Parameters of the discriminator (critic) network
$\theta^*$: Optimal parameters for the generator
$\text{argmin}_\theta$: Argument that minimizes the expression with respect to $\theta$
$\sup_\phi$: Supremum (least upper bound) with respect to $\phi$
$T_\phi(x)$: The discriminator function, parameterized by $\phi$
$f^*$: The convex conjugate of the function $f$ used in the F-divergence
$p_{data}$: The true data distribution
$p_{generator}$: The distribution of the generated data

Write $T_\phi(x)$ as composite function $T_\phi(x) = \sigma(V_\phi(x))$, where $\sigma$ is the sigmoid function Substitute this in the expression for GANs:

Neural network

Wed, 01 Jan 2025 00:00:00 +0000

A neural network is a computational model inspired by the structure and function of biological neural networks. Mathematically, it can be defined as a series of function compositions:

$$ f(x) = f_L(f_{L-1}(…f_2(f_1(x)))) $$

where $L$ is the number of layers in the network, and each function $f_i$ represents a layer operation.

For a single layer, the operation can be expressed as:

$$ f_i(x) = \sigma(W_i x + b_i) $$

where:

Noise Contrastive Estimation

Wed, 01 Jan 2025 00:00:00 +0000

Idea

The core idea behind Noise Contrastive Estimation (NCE) is:

Given samples from a distribution, if you know how to distinguish them from noise samples, then you implicitly know the underlying distribution.
If you know how to tell what is noise from what is real data, you must have learned something about the true data distribution.

This provides an alternative way to learn probability distributions by transforming the density estimation problem into a binary classification problem between real and noise samples.

Non parametric density estimation

Wed, 01 Jan 2025 00:00:00 +0000

What is parametric density estimation?

In parametric density estimation, we assume that the data is generated from a known distribution, such as the normal distribution, and we estimate the parameters of the distribution using various methods like maximum likelihood estimation or risk minimization.

What is non-parametric density estimation?

However, in non-parametric density estimation, we make no assumptions about the form of the distribution and estimate the density directly from the data.

Parzen window estimate

Wed, 01 Jan 2025 00:00:00 +0000

Parzen window estimate is also known as the kernel density estimate

Basic idea

Recall that we had derived the following equation for non-parametric density estimation:

$$p(x) = \frac{k}{nV}$$

where:

$k$ is the number of data points in region $R$
$n$ is the total number of data points in the dataset $D$
$V$ is the volume of the region $R$

In Parzen window estimate, we fix the volume $V$ and count $k$ by using a window function.

Policy Iteration

Wed, 01 Jan 2025 00:00:00 +0000

In value iteration, we were focused on finding the optimal value function $J^*$. In policy iteration, we start with a random policy and then improve it iteratively.

Algorithm Description

Policy iteration alternates between two main steps:

Policy Evaluation: Given a policy $\mu_k$, compute its cost-to-go vector $J^{\mu_k}$ by solving the system of linear equations:

$$ J^{\mu_k}(i) = \sum_{j=1}^n p_{ij}(\mu_k(i))(g(i,\mu_k(i),j) + J^{\mu_k}(j)) \quad i=1,\ldots,n $$

where $J^{\mu_k}(1), J^{\mu_k}(2), \ldots, J^{\mu_k}(n)$ are the unknowns to be solved for.

Policy iteration for infinite horizon discounted cost problems

Wed, 01 Jan 2025 00:00:00 +0000

Proposition: Optimality Conditions in Policy Iteration

Let’s examine a fundamental proposition that establishes optimality conditions for policy iteration in infinite horizon discounted cost problems. This proposition provides crucial insights into why policy iteration converges to optimal policies.

Statement of the Proposition

Consider a stationary policy $\bar{\mu}$ that satisfies the Bellman optimality condition:

$$ \underbrace{T_{\bar{\mu}}J^{\bar{\mu}} = TJ^{\bar{\mu}}}_{\substack{\text{Policy } \bar{\mu} \text{ achieves the minimum} \\ \text{in the Bellman equation for all states}}} $$

This can be expressed more explicitly for all states $i = 1,2,\ldots,n$ as:

Precursor: Stochastic Differential Equations (SDEs)

Wed, 01 Jan 2025 00:00:00 +0000

Motivation

We explored different sampling methods for generative models:

Langevin dynamics (LE): $$x_{t+1} = x_t + \alpha \nabla_x \log p(x_t) + \sqrt{2\alpha}\epsilon, \quad \epsilon \sim \mathcal{N}(0,I)$$
DDPM (Denoising Diffusion Probabilistic Models): $$x_{t+1} = \sqrt{1-\beta_t}x_t + \sqrt{\beta_t}\epsilon, \quad \epsilon \sim \mathcal{N}(0,I)$$
SMLD (Score Matching with Langevin Dynamics): $$x_{t+1} = x_t + \sqrt{\sigma_{t+1}^2 - \sigma_t^2}\epsilon, \quad \epsilon \sim \mathcal{N}(0,I)$$

A natural question arises: What is the unifying mathematical framework that connects all these sampling methods?

Principal Component Analysis (PCA)

Wed, 01 Jan 2025 00:00:00 +0000

Problem Setup

Data: Let the dataset be $X = {x_i}_{i=1}^N$, where each data point $x_i \in \mathbb{R}^d$.

Goal: We want to reduce the dimensionality of the data from $d$ to $k$ while preserving as much of the data’s variance as possible.

Intuition

PCA aims to find a new coordinate system for the data that maximizes the variance along the first coordinate (principal component), while minimizing the variance along the other coordinates. This is useful for data visualization, noise reduction, and feature extraction.

Q Learning

Wed, 01 Jan 2025 00:00:00 +0000

Q-Learning is a powerful model-free reinforcement learning algorithm that learns the optimal action-value function directly without requiring a model of the environment.

Understanding Model-Based vs. Model-Free Approaches

Model-Based Reinforcement Learning

A model of the environment in reinforcement learning refers to explicit knowledge of the system dynamics, specifically:

State Transition Probabilities: $p_{ij}(u)$ - The probability of transitioning from state $i$ to state $j$ when taking action $u$
Reward/Cost Functions: $g(i,u,j)$ - The immediate reward or cost received when transitioning from state $i$ to state $j$ via action $u$

With a complete model, we can use dynamic programming methods like value iteration or policy iteration to compute optimal policies without direct environment interaction.

Regularization

Wed, 01 Jan 2025 00:00:00 +0000

Regularization is a technique to increase model bias to reduce variance by constraining empirical risk minimization.

regularized empirical risk minimization:

$$\text{Reg } ERM = \min_{h \in H} \hat{R}(h_\theta) \quad \text{s.t. } \Omega(h_\theta) < k$$

here $\Omega(h_\theta) < k$ is the regularization function which is a design choice.

This can be solved using the method of Lagrange multipliers.

$h(x) = \arg\min_{h_\theta \in H} \left( \hat{R}(h_\theta) + \lambda \Omega(\theta) \right)$

here $\lambda$ is the regularization parameter which is a design choice.

Risk function (R)

Wed, 01 Jan 2025 00:00:00 +0000

Recall that the loss function $L(y_i, \hat{y_i})$ measures the error between the predicted label and the true label for a single data point.

The risk function $R(h)$ is the expected loss over all data points in the dataset. It is defined as: $$R(h) = \mathbb{E}_{(x,y) \sim P}[L(y, h(x))]$$

where $P$ is the true data distribution.

Conditional risk

The conditional risk $R(h|x)$ is the expected loss for a given input $x$. It is defined as: $$R(h(x)|x) = \mathbb{E}_{y \sim P(y|x)}[L(y, h(x))]$$

Score matching part 1

Wed, 01 Jan 2025 00:00:00 +0000

Sampling as optimization

In probabilistic modeling, sampling can be viewed as an optimization problem where we aim to find the most probable sample from a probability distribution. This can be formulated mathematically as:

$$x^* = \mathop{\arg \max}_{x} p(x)$$

Taking the log of the probability (which preserves the optimum due to monotonicity of log):

$$x^* = \mathop{\arg \max}_{x} \log p(x)$$

And converting to a minimization problem:

$$x^* = \mathop{\arg \min}_{x} - \log p(x)$$

Score matching part 2

Wed, 01 Jan 2025 00:00:00 +0000

Recap

$J_{ESM}(\theta) = \mathbb{E}_{p(x)} \left[ \frac{1}{2} | \hat{s}(x; \theta) - \nabla_x \log p(x) |^2 \right]$
$J_{ISM}(\theta) = \mathbb{E}_{p(x)} \left[ \frac{1}{2} | \hat{s}(x; \theta) |^2 + \text{tr}(\nabla_x \hat{s}(x; \theta)) \right] + C$
Theorem: $J_{ISM}(\theta) = J_{ESM}(\theta) + C$
$\theta^* = \mathop{\arg \min}_{\theta} J_{ISM}(\theta)$
$x_{t+1} = x_t + \alpha \cdot \hat{s}(x_t; \theta^*) + \sqrt{2\alpha} \epsilon, \quad \epsilon \sim \mathcal{N}(0, I)$
$\hat{s}(x; \theta)$ modelled using a neural network

There is a problem with calculating the trace term

The implicit score matching loss function, as we discussed earlier, is given by:

Score matching part 3

Wed, 01 Jan 2025 00:00:00 +0000

Recap

Langevin equation: $x_{t+1} = x_t + \alpha \cdot \nabla_x \log p(x_t) + \sqrt{2\alpha} \epsilon, \quad \epsilon \sim \mathcal{N}(0, I)$
score function: $\nabla_x \log p(x) = s(x)$
$J_{ESM}(\theta) = \mathbb{E}_{p(x)} \left[ \frac{1}{2} | \hat{s}(x; \theta) - \nabla_x \log p(x) |^2 \right]$
$J_{ISM}(\theta) = \mathbb{E}_{p(x)} \left[ \frac{1}{2} | \hat{s}(x; \theta) |^2 + \text{tr}(\nabla_x \hat{s}(x; \theta)) \right] + C$
Theorem: $J_{ISM}(\theta) = J_{ESM}(\theta) + C$
$J_{PSM}(\theta) = \frac{1}{2} \mathbb{E}_v \mathbb{E}_{p(x)} \left[ \| v^\top \hat{s}_\theta(x) - v^\top s(x) \|^2 \right]$
$J_{SSM}(\theta) = \mathbb{E}_v \mathbb{E}_{p(x)} \left[ \frac{1}{2}(v^\top \hat{s}_\theta(x))^2 + v^\top (\nabla_x \hat{s}_\theta(x)) v \right]$
Theorem: $J_{PSM}(\theta) = J_{SSM}(\theta) + C$

Big picture plan

Denoising score matching

Denoising score matching (DSM) is an alternative approach to score matching. It introduces an auxiliary variable and works with conditional scores, which offers several advantages.

Sequence to sequence models (Transformers)

Wed, 01 Jan 2025 00:00:00 +0000

Let the data be $$D = {(x_i, y_i)}_{i=1}^N \text{ iid } \sim p(x, y).$$

here $x_i = { x_i^1, x_i^2, \ldots, x_i^k }$, where $x_j^i \in \mathbb{R}^d$ represents a sequence of $k$ vectors of dimension $d$.They are called tokens in usual NLP models.
here $y_i = { y_i^1, y_i^2, \ldots, y_i^m }$, where $y_j^i \in \mathbb{R}^{d’}$ represents a sequence of $m$ vectors of dimension $d’$. Note that $d’ \neq d$. It represents a softmax distribution over a vocabulary of size $d’$ in NLP models.

The models that map $x_i$ to $y_i$ are called seq2seq models.

SMLD as SDE

Wed, 01 Jan 2025 00:00:00 +0000

Step 1 of big picture (Stochastic forward process to continuous SDE)

We know SMLD (Score Matching with Langevin Dynamics) forward process is: $$x_i = x_{i-1} + \sqrt{\sigma_{i}^2 - \sigma_{i-1}^2}z_{i-1}, \quad z_{i-1} \sim \mathcal{N}(0,I)$$

where $\sigma_i$ is the noise level at step $i$, increasing with $i$.

Let’s convert the discrete SMLD process to a continuous SDE:

First, let’s write the discrete process in terms of continuous time:
- Let $\Delta t = \frac{1}{N}$ be the time step
- Map discrete index $i$ to continuous time $t$: $i = \frac{t}{N}$
- Write $\sigma_i^2 = \sigma(t)^2$ for continuous noise schedule
The discrete process becomes: $$x(t + \Delta t) = x(t) + \sqrt{\sigma(t + \Delta t)^2 - \sigma(t)^2}z_t$$ where $z_t \sim \mathcal{N}(0,I)$

Softmax regression also known multiclass regression

Wed, 01 Jan 2025 00:00:00 +0000

Let the data be D = {($x_1$, $y_1$), ($x_2$, $y_2$), …, ($x_n$, $y_n$)} where $y_i \in {1, 2, …, K}$ is the multiclass label of the feature vector $x_i$, and $K$ is the number of classes.

Then the softmax regression model is defined as: $$P(y_i = k | x_i) = \frac{e^{w_k^T x_i + b_k}}{\sum_{j=1}^K e^{w_j^T x_i + b_j}}$$

where $w_k$ is the parameter vector for class $k$, and $b_k$ is the bias term for class $k$.

Some example problems

Wed, 01 Jan 2025 00:00:00 +0000

Example 1 - convert continuous SDE to discrete soln without drift term

The continuous stochastic differential equation (SDE) is given by:

$$dx = \epsilon \sqrt{dt} \quad // \text{cont eqn}$$

where $\epsilon \sim \mathcal{N}(0,1)$.

To convert this continuous SDE to a discrete solution, we consider the increments:

$$X_{t_{i+1}} - X_{t_i} = \sqrt{t_{i+1} - t_i} \cdot \epsilon$$

Assuming $t$ were not discrete, we can express it as:

$$X_{t_{i+1}} = X_{t_i} + \sqrt{t_{i+1} - t_i} \cdot \epsilon$$

State space models & Mamba

Wed, 01 Jan 2025 00:00:00 +0000

Motivation for state space models

RNNs can handle variable length inputs however they are not parallelizable.
CNNs are parallelizable however they cannot handle variable length inputs.

Can we get the best of both worlds?

Linear State Space Models

A linear state space model consists of two equations:

$$h’(t) = Ah(t) + Bx(t)$$ $$y(t) = Ch(t) + Dx(t)$$

where:

$h(t)$ is the hidden state at time $t$
$x(t)$ is the input at time $t$
$y(t)$ is the output at time $t$
$h’(t)$ is the derivative of the hidden state with respect to time
$A, B, C, D$ are learnable parameter matrices that define the dynamics

The first equation describes how the hidden state evolves over time based on:

Stochastic Differential Equations (SDEs)

Wed, 01 Jan 2025 00:00:00 +0000

Big picture of what we are going to do:

Big picture of SDEs

This topic comes under stochastic calculus, which deals with functions whose derivatives yield random vectors rather than deterministic points.

In traditional calculus, when we take the derivative of a function at a point, we get a fixed value ie a scalar that represents the instantaneous rate of change. However, in stochastic calculus, the derivative at any point is a random variable.

Summarizing DDPM

Wed, 01 Jan 2025 00:00:00 +0000

VAE:
Hierarchical VAE:
DDPM(Hierarchical VAE with deterministic forward process):

Notice that $\phi$ has been removed from $q_\phi()$.

Also see that:

The ELBO for a Variational Autoencoder (VAE) is given by:

\text{ELBO (VAE)} = \mathbb{E}_{q_\phi(z|x)} \left[ \log \frac{p(x|z)}{q_\phi(z|x)} \right]

- For Denoising Diffusion Models (DDPM), the ELBO is expressed as:

\text{ELBO (DM)} = \mathbb{E}_{q(x_{1:T}|x_0)} \left[ \log \frac{p(x_{0:T})}{q(x_{1:T}|x_0)} \right]

Support measure machine (SCM) as Empirical Risk Minimization (ERM)

Wed, 01 Jan 2025 00:00:00 +0000

Recall that in support measure machine (SCM), we were trying to maximize the margin between the two classes of data points.

$$ \min_{w,b,\xi} \frac{1}{2} |w|^2 + C \sum_{i=1}^n \xi_i $$ $$ \text{s.t. } y_i (w^T x_i + b) \geq 1 - \xi_i \text{ for all } i $$ $$ \xi_i \geq 0 \text{ for all } i $$

SCM as ERM

This is an optimization problem that can be formulated as a empirical risk minimization (ERM) problem.

Support vector machine(SVM) when data is linearly separable

Wed, 01 Jan 2025 00:00:00 +0000

Dataset is linearly separable

Dataset $D = {(x_i, y_i)}_{i=1}^n$ where $x_i \in \mathbb{R}^d$ and $y_i \in {-1, 1}$ is said to be linearly separable if there exists a hyperplane that can separate the two classes of data points with zero training error.

Mathematically, a dataset is linearly separable if there exist weights $w$ and bias $b$ such that: $$ w^T x_i + b > 1 \text{ for } y_i = 1 $$ $$ w^T x_i + b < -1 \text{ for } y_i = -1 $$

Support vector machine(SVM) when data is not linearly separable

Wed, 01 Jan 2025 00:00:00 +0000

Dataset is not linearly separable

Unlike the case when the dataset is linearly separable, we cannot find a hyperplane that separates the two classes of data points with zero training error.

Soft margin

To handle this case, we introduce a slack variable $\xi_i$ for each data point $x_i$ to allow some points to be on the wrong side of the margin or even in the wrong class.

Optimization problem

The optimization problem becomes:

SVM with kernel

Wed, 01 Jan 2025 00:00:00 +0000

Prerequisites - Kernel trick

Kernel function $k(x, y) = \phi(x)^T \phi(y)$ where $\phi(x)$ is a feature mapping that maps the data points to a higher dimensional space without actually computing the feature mapping, i.e., $\phi(x)$.

In other words, we can compute the kernel function $k(x, y)$ without actually computing the feature mapping $\phi(x)$.

Examples of kernel functions:

Polynomial kernel: $k(x_1, x_2) = \phi(x_1)^T \phi(x_2) = (1 + x_1^T x_2)^p$
Sigmoid kernel: $k_s(x_1, x_2) = \frac{1}{1 + \exp(a x_1^T x_2)}$
RBF/Gaussian kernel: $k(x_1, x_2) = \exp\left(-\frac{| x_1 - x_2 |^2}{\sigma^2}\right)$

Motivation

If the dataset $D = {(x_i, y_i)}_{i=1}^n$ where $x_i \in \mathbb{R}^d$ and $y_i \in {-1, 1}$ is not linearly separable in the original $d$-dimensional space, we can use a feature mapping $\phi(x)$ to map the data points to a higher dimensional space where they become linearly separable and then use the SVM optimization problem to find the optimal hyperplane.

TD Learning

Wed, 01 Jan 2025 00:00:00 +0000

Temporal Difference (TD) learning, introduced by Rich Sutton in 1984, is a fundamental reinforcement learning method that combines ideas from Monte Carlo methods and dynamic programming.

Recap of Monte Carlo Methods

Recall that Monte Carlo methods estimate the value function by:

$$ V_\pi(s) = E[G_n | S_0 = s] $$

where:

$G_n = r_{n+1} + r_{n+2} + … + r_N$ is the return (sum of rewards until terminal state)
$N$ is the terminal instant of the episode

Monte Carlo methods work with sample average data collected over complete trajectories, requiring episodes to terminate before learning can occur.

Temporal difference learning

Wed, 01 Jan 2025 00:00:00 +0000

Temporal Difference (TD) learning is a fundamental reinforcement learning technique that combines ideas from Monte Carlo methods and dynamic programming. It allows agents to learn from experience without requiring a model of the environment, making it particularly valuable for real-world applications.

Understanding Policy Evaluation

Mathematical Foundation

Policy evaluation refers to the process of determining the value function $J_π$ for a given policy $π$. The value function represents the expected cumulative reward when following policy $π$ from a given state.

Two types of neural networks

Wed, 01 Jan 2025 00:00:00 +0000

We can model distributions using neural networks in two ways:

Deterministic way

In this way the output of neural network is a sample from the distribution.

Probabilistic way

In this way the output of neural network are parameters of the distribution.

Understanding Stochastic Shortest Path Problems: A Deep Dive into Mathematical Foundations

Wed, 01 Jan 2025 00:00:00 +0000

Definitions

Expected Single-Stage Cost

The expected single-stage cost under policy $\mu$ in state $x$, denoted as $\bar{g}(x,\mu(x))$, is:

\underbrace{\bar{g}(x,\mu(x))}_{\substack{\text{Expected} \\ \text{single-stage cost}}} = \sum_{j=0}^n \underbrace{p_{xj}(\mu(x))}_{\substack{\text{Transition} \\ \text{probability}}} \underbrace{g(x,\mu(x),j)}_{\substack{\text{Stage cost for} \\ \text{this transition}}}

where:

$p_{xj}(\mu(x))$ is the probability of transitioning from state $x$ to state $j$ under action $\mu(x)$
$g(x,\mu(x),j)$ is the stage cost for this transition
The sum is over all possible next states $j$

This represents the immediate expected cost when taking the action prescribed by policy $\mu$ in state $x$.

Unsupervised learning

Wed, 01 Jan 2025 00:00:00 +0000

In unsupervised learning, we don’t have labeled data. We want to find structure in the data. Hence the data is of the form $X = {x_1, …, x_N}$ where $x_i \in \mathbb{R}^d$.

VAE as regularized autoencoder

Wed, 01 Jan 2025 00:00:00 +0000

Understanding VAE as a Regularized Autoencoder

Recall that the Variational Autoencoder (VAE) objective is given by:

F_{\theta}(q) = \mathbb{E}_{q_{\phi}(z|x)} \left[ \log p_{\theta}(x|z) \right] - D_{KL} \left( q_{\phi}(z|x) \| p(z) \right)

We can rewrite this as:

F_{\theta}(q) = \mathbb{E}_{q_{\phi}(z|x)} \left[ \|x - \hat{x}\|^2 \right] - D_{KL} \left( q_{\phi}(z|x) \| p(z) \right)

Where:

The first term is the reconstruction error
The second term acts as a regularizer

This formulation shows that the VAE objective can be interpreted as a regularized autoencoder, where the regularization term encourages the latent distribution to be close to the prior $p(z)$.

Value Iteration

Wed, 01 Jan 2025 00:00:00 +0000

Value iteration is a numerical method for solving Stochastic Shortest Path (SSP) problems that relies on the dynamic programming operator $T_\mu$. Let’s examine how this operator works and its key properties:

The Dynamic Programming Operator $T_\mu$

For a given policy $\mu$, the dynamic programming operator $T_\mu$ transforms one value function into another. Mathematically, for any state $x$, we have:

\underbrace{(T_\mu J)(x)}_{\substack{\text{New value function} \\ \text{at state } x}} = \underbrace{E_{x_{k+1}}}_{\substack{\text{Expected value over} \\ \text{next states}}} \left[ \underbrace{g(x,\mu(x),x_{k+1})}_{\substack{\text{Stage cost under} \\ \text{policy } \mu}} + \underbrace{J(x_{k+1})}_{\substack{\text{Cost-to-go from} \\ \text{next state}}} \biggm\vert x_k=x \right]

where:

Variational encoding(VAEs)

Wed, 01 Jan 2025 00:00:00 +0000

Problem setting

Given a set of data points $D = ({x_i})_{i=1}^N$
Where $x_i \in \mathbb{R}^d$ are iid samples from some unknown distribution $p_{\text{data}}$.
We want to model $p_{\text{data}}$ as $p_{\theta}$ using a neural network.
We model $p_{\theta}$ as a latent variable model

Modelling

Log likelihood is given by:

$$ \ell(\theta) = \log p_{\theta}(x) $$

We assume that each data point $x_i$ is associated with a latent variable $z_i$. Hence, we will introduce the latent variable $z$ and marginalize over it:

Vector Quantized Variational Autoencoders (VQ-VAEs)

Wed, 01 Jan 2025 00:00:00 +0000

Vector Quantized Variational Autoencoders (VQ-VAEs) were introduced to address several limitations of regular VAEs:

The “posterior collapse” problem where the latent code is ignored
The continuous latent space can make it difficult to model discrete structures
The aggregated posterior mismatch between $q_\phi(z)$ and the prior $p(z)$

VQ-VAEs solve these issues by using discrete latent variables through vector quantization. The architecture consists of:

An encoder network $E_{\phi}$ that maps input $x \in \mathbb{R}^N$ to continuous latent vectors $z_e = E_{\phi}(x) \in \mathbb{R}^{D}$

Wasserstein GANs (WGANs)

Wed, 01 Jan 2025 00:00:00 +0000

Perfect Discriminator theorem

The Perfect Discriminator theorem states that if two distributions $p_x$ and $q_x$ have support on two disjoint subsets $M$ and $P$ respectively, there always exists a discriminator $D^*: x \rightarrow [0,1]$ that has accuracy 1:

$$\forall x \in M \cup P, D^*(x) = \begin{cases} 1 & \text{if } x \in M \ 0 & \text{if } x \in P \end{cases}$$

This discriminator achieves perfect classification, as it correctly identifies the origin of every sample with 100% accuracy.

Wasserstein metric

Wed, 01 Jan 2025 00:00:00 +0000

Definition

We can use Wasserstein metric in place of F-divergence to define a distance between two distributions $P$ and $Q$ both defined over the same space $X$.

The Wasserstein distance $W_p(P,Q)$ of order $p$ is defined as:

$$W_p(P,Q) = \left( \inf_{\gamma \in \Gamma(P,Q)} \mathbb{E}_{(x,y) \sim \gamma} [d(x,y)^p] \right)^{1/p}$$

where:

$\Gamma(P,Q)$ is the set of all possible joint distributions of $(x,y)$ where $x,y \in X$ with marginals $x \sim P$ and $y \sim Q$ (also called couplings)
$\gamma(x,y)$ is a particular joint distribution chosen out of all possible joint distributions in $\Gamma(P,Q)$ that minimizes $\mathbb{E}_{(x,y) \sim \gamma} [d(x,y)^p]$
$d(x,y)$ is the metric (distance) between points $x$ and $y$ in the space $X$
$p \geq 1$ is the order of the Wasserstein distance. Common choices are $p=1$ (first-order Wasserstein distance) and $p=2$ (second-order Wasserstein distance).

Properties of Wasserstein metric

When $p=1$, the Wasserstein distance is equivalent to the Earth Mover’s Distance (EMD), because it can be interpreted as the minimum amount of work required to transform one distribution into another.
When $p=2$, the Wasserstein distance is equivalent to the squared Euclidean distance.
The Wasserstein distance is always finite, unlike the F-divergence, which can be infinite.
The Wasserstein distance is a true metric, meaning it satisfies the triangle inequality.
When KL divergence is $0$ then Wasserstein distance is $0$, but not vice versa.

Proof for 5th property - KL divergence is 0 then Wasserstein distance is 0

To prove that when the KL divergence is 0, the Wasserstein distance is also 0, we start with the definitions of both metrics.

XGBoost - Gradient boosted regression tree

Wed, 01 Jan 2025 00:00:00 +0000

In XGBoost, the weak learners are decision trees.

The algorithm builds these trees sequentially, with each new tree aiming to correct the errors of the combined previous trees.

Recall that $r_i = -\frac{\partial R(H_{T-1}(x_i))}{\partial H_{T-1}(x_i)}$ are the pseudo-residuals.
XGBoost formulates the optimization problem for finding the next weak learner as:

$$ h_T = \arg\min_{h \in H} \sum_{i=1}^n (r_i \cdot h(x_i) + \frac{1}{2} h(x_i)^2) $$
We can rewrite this by defining $\hat{y}_i = -r_i$: