Victor

This is how you can use Probability Theory to make decisions. Fundamental Terminologies Before diving into the mathematical framework, let’s establish the key concepts that form the foundation of decision-making under uncertainty. Action An action is a choice or decision that a decision-maker can take from a set of available alternatives. It represents what we can control or decide upon in a given situation. Mathematical Notation: $$ a \in A $$ ...

RL setting In the reinforcement learning framework, we can model human behavior as an agent interacting with the world. Let’s formalize this mathematically. Markov Decision Process (MDP) Formulation The human experience can be modeled as a Markov Decision Process (MDP) defined by the tuple: $$\mathcal{M} = (\mathcal{S}, \mathcal{A}, \mathcal{P}, \mathcal{R}, \gamma)$$ where: $\mathcal{S}$ is the state space (all possible states of the world and internal states) $\mathcal{A}$ is the action space (all possible actions the human can take) $\mathcal{P}: \mathcal{S} \times \mathcal{A} \times \mathcal{S} \rightarrow [0,1]$ is the transition probability function $\mathcal{R}: \mathcal{S} \times \mathcal{A} \times \mathcal{S} \rightarrow \mathbb{R}$ is the reward function $\gamma \in [0,1]$ is the discount factor Environment is the world The environment encompasses everything external to the agent (brain). Mathematically, the environment dynamics are governed by: ...

Everything in this guide comes from the actual source code—not speculation, not “what worked for me,” not guru theories. Chapter 1: Introduction Forget Everything You’ve Heard You’ve heard the advice. “Post at 9am for maximum reach.” “Use trending hashtags.” “The algorithm suppresses certain topics.” “Blue checkmarks get boosted.” “Follow-to-follower ratio matters.” “Post more to grow faster.” You’ve heard the gurus—the ones selling courses, the ones with “X Growth Expert” in their bio, the ones who claim they’ve “cracked the algorithm.” ...

Limitations of Traditional LLMs Large Language Models (LLMs), while powerful, face several fundamental constraints that limit their effectiveness in real-world applications: Knowledge Cutoff and Staleness Static Training Data: LLMs are trained on data up to a specific cutoff date, making them unable to access or reason about recent information, events, or developments Knowledge Decay: Information becomes increasingly outdated over time, reducing the model’s relevance for current queries Context Window Constraints Limited Input Size: Most LLMs have finite context windows, restricting the amount of information that can be processed in a single query Information Loss: When dealing with large documents or datasets, critical information may be truncated or lost due to context limitations Computational and Economic Costs Expensive Long Contexts: While some models support extended context windows (100K+ tokens), processing large amounts of text significantly increases computational costs and latency Inefficient Information Processing: Passing entire documents through the model when only specific sections are relevant leads to unnecessary resource consumption Hallucination and Accuracy Issues Knowledge Gaps: When LLMs lack specific information, they may generate plausible-sounding but incorrect responses (hallucinations) Inability to Cite Sources: Traditional LLMs cannot provide verifiable sources or references for their generated content, making fact-checking difficult Lack of Traceability: There’s no way to trace back which specific information or training data influenced a particular response, making it impossible to verify the source of generated content Domain-Specific Knowledge Limitations Specialized Information: LLMs may lack deep expertise in highly specialized domains or proprietary knowledge that wasn’t part of their training data Real-time Data: Cannot access live data, current market conditions, or dynamic information that changes frequently When do you need RAG vs Fine-tuning? Understanding when to use Retrieval Augmented Generation (RAG) versus Fine-tuning is crucial for building effective AI systems. Each approach has distinct advantages and is suited for different scenarios. ...

Basic Terminologies in Reinforcement Learning(RL) Reinforcement Learning is defined by the following components: Agent: The entity that interacts with the environment and makes decisions. Environment: The world in which the agent operates. Mathematically defined as a state transition system: $\mathcal{E}: \mathcal{S} \times \mathcal{A} \rightarrow \mathcal{S}$ State ($s \in \mathcal{S}$): The current configuration of the environment. The state space $\mathcal{S}$ is the set of all possible states. At time step $t$, the state is denoted as $s_t$. Action ($a \in \mathcal{A}$): The decision made/ action taken by the agent after observing the state $s_t$ of the environment based on its policy $\pi$. The action space $\mathcal{A}$ is the set of all possible actions. At time step $t$, the action is denoted as $a_t$. Reward ($r \in \mathbb{R}$): The feedback signal received by the agent from the environment after taking an action. Mathematically defined as: $r: \mathcal{S} \times \mathcal{A} \times \mathcal{S} \rightarrow \mathbb{R}$ is a real-valued function that maps the current state, action, and next state to a scalar reward. At time step $t$, the reward is denoted as $r_t = r(s_t, a_t, s_{t+1})$, which represents the reward received at time step $t$. Note that in some books the reward function is defined as $r_{t+1} = r(s_t, a_t, s_{t+1})$, which represents the reward received at time step $t+1$. However I like to use the earlier definition to avoid confusion. Policy ($\pi$): The strategy that the agent follows to select actions. Deterministic policy: $\pi: \mathcal{S} \rightarrow \mathcal{A}$ Stochastic policy: $\pi: \mathcal{S} \times \mathcal{A} \rightarrow [0,1]$, where $\pi(a|s)$ is the probability of taking action $a$ in state $s$. State Transition Probability ($p$): The probability of transitioning to state $s’$ after taking action $a$ in state $s$. Mathematically: $p(s’|s,a) = \mathbb{P}(s_{t+1}=s’|s_t=s, a_t=a)$ This defines the dynamics of the environment and captures how actions influence state changes. For deterministic environments: $p(s’|s,a) = 1$ for exactly one next state $s’$ and 0 for all others. For stochastic environments: $\sum_{s’ \in \mathcal{S}} p(s’|s,a) = 1$ for all state-action pairs $(s,a)$. Episode: A sequence of state-action-reward transitions from an initial state to a terminal state. Mathematically: $\tau = (s_0, a_0, r_0, s_1, a_1, r_1, …, s_T, a_T, r_T)$ $T$ is the time step when the terminal state $s_T$ is reached. Trajectory: A sequence of states, actions, and rewards generated by following a policy $\pi$ from an initial state $s_0$. Mathematically: $\tau = (s_0, a_0, r_0, s_1, a_1, r_1, …, s_T, a_T, r_T)$ A trajectory represents the complete history of an agent’s interaction with the environment during an episode. Return: The cumulative reward received by the agent over a trajectory $\tau$. Mathematically: $$G(\tau) = \sum_{t=0}^{T} \gamma^t r_t$$ Where: $G(\tau)$ is the return for the trajectory when agent follows policy $\pi$ $\gamma \in [0,1]$ is the discount factor that determines the present value of future rewards $r_t$ is the reward received at time step $t$ $T$ is the final time step in the trajectory ...

Unity Packages for Reinforcement Learning Unity’s modular nature is facilitated by its Package Manager, which allows developers to add specific functionalities to their projects. For implementing Reinforcement Learning (RL) within Unity, the primary and most widely used package is: ML-Agents Toolkit: Purpose: This is Unity’s official open-source package specifically designed to enable games and simulations to serve as environments for training intelligent agents using RL and imitation learning. Functionality: It provides the necessary tools and SDKs to: Define Agents: The actors within the environment that learn to make decisions. Define Observations: The information the agent receives from the environment (e.g., position, velocity, sensor readings). Define Actions: The decisions the agent can make (e.g., move forward, jump, turn). Define Rewards: The signals given to the agent to indicate whether its actions are good or bad towards achieving a goal. Connect to Training Algorithms: It integrates with established Python-based RL libraries (like TensorFlow via mlagents-learn) to perform the actual training process, often using algorithms like PPO (Proximal Policy Optimization) or SAC (Soft Actor-Critic). Integration: ML-Agents allows seamless communication between the Unity environment (where the simulation runs) and the Python training process. While ML-Agents is the cornerstone for RL in Unity, developers could potentially integrate other third-party libraries or custom solutions, but these would typically require significantly more manual setup and integration effort compared to the streamlined workflow offered by the official ML-Agents package. For most users looking to apply RL in Unity, ML-Agents is the recommended and standard choice. ...

AI Learning Roadmap I have made this roadmap that I will give to my younger brother who is starting his learning journey in AI. The key points of this roadmap are: The roadmap assumes you have basic knowledge of maths and programming of a 12th class pass out. All the resources are free. The roadmap is linear, follow it step by step. However, if you have a good grasp of a topic, you can skip it and move to the immediate next topic. The roadmap is not exhaustive, I have designed it so that you learn 90% of the most important topics in AI. I have seen most AI learning resources to either focus on maths or programming. I have designed this roadmap to cover both. This is not the only possible roadmap, there are many other roadmaps that are possible. This is just a roadmap that I think is best for a beginner learning on his own. There are thousands of students from all walks of life studying this roadmap in this Discord server I have made the Discord server for students who are following this roadmap to help them learn and grow together. Here is the link to the server: Join our Discord Server. Do join thousands of students who would be learning along with you. I also encourage you to learn about the topics from other sources as well in addition to the ones mentioned in the roadmap if you have more time. In case you are not able to understand a particular topic from the resources mentioned in the roadmap, do explore other resources and share your experience. As and when I find a better new resource, I will update the roadmap. If you have any suggestions, please let me know so that I can update the roadmap. This roadmap is up to date. As and when an important breakthrough comes in the field of AI, that is important, I will update the roadmap. After completing the roadmap: You should pick one or more area of expertise like Computer Vision, Natural Language Processing, etc. and start applying your knowledge and learning advance topics in that domain. There is no use of learning AI if you are not applying your knowledge. You should start working on real world projects. Either start working on your own projects or contribute to the projects at your company/open source community. AI is one of the fastest growing field, so do keep yourself updated with the latest research, news and products. My tips for the Discord server: The Discord server is free and open to all with a learning mindset. Do invite your friends and classmates by sharing the server link on your social media handles and bring them along on this learning journey so that you can learn and grow together. Moreover the more members in the server, the more easier it will be to get help from each other and more resources will be shared for everyone’s benefit. Do not hesitate to ask questions. Moreover, if you have time do help others by answering their questions as this will help you solidify your understanding of the topic. Do find friends in this Discord server who are learning along with you and do form a study group(if possible find friends that you can even meet in person). Here is the roadmap: 1. Maths prerequisites 1.1 Linear Algebra Linear Algebra by Imperial College London 3Blue1Brown’s “Essence of Linear Algebra” playlist 1.2 Calculus and Optimization Multivariate Calculus by Imperial College London 1.3 Probability Probability Theory from The Bright Side of Mathematics 2. Coding Python for Beginners by Mosh (Skip if you already know basics of Python) Learn Git from Mosh (Skip if you already know Git) Learn to make virtual environment in Python Pytorch beginner series by PyTorch (Do not try to understand everything about PyTorch at once, just get some idea then learn more about it as you code AI models from Machine Learning and Deep Learning section) 3. Machine Learning Machine Learning by Josh Starmer from StatQuest Do implement all the basic machine learning models like Linear Regression, Logistic Regression, K-Nearest Neighbors, Decision Trees, Random Forest, Gradient Boosting, etc. that are taught in the playlist in PyTorch. Youtube has a lot of tutorials on how to implement these models in PyTorch. 4. Deep Learning Deep Learning by Josh Starmer from StatQuest Neural networks from 3Blue1Brown Implement all the models learned in the playlists in PyTorch. You can follow this playlist Become proficient in coding Deep Learning models by following Andrej Karpathy’s Zero to Hero series Finally I will again emphasize that this is not the end of the learning journey, rather it is the beginning. I have designed this roadmap so that you learn 90% of the most important topics in AI which will act as a foundation for your further learning a particular subdomain of AI. ...

What is a Programming Language? A programming language is a formal system that allows humans to give instructions to computers. It acts as a bridge between human-readable commands and machine code. Python, which we’ll focus on in this article, is a popular high-level programming language known for its readable syntax and extensive library ecosystem. What is a Module? A module in Python is a file containing Python code that can be imported and used in other Python programs. It’s a fundamental way to organize and reuse code by grouping related functions, classes, and variables together. ...

Recall Recall from our discussion on boosting we defined the ensemble model $H_T(x)$ as: $$ H_T(x) = \sum_{t=1}^T \alpha_t h_t(x) $$ the problem of finding the next weak learner reduces to: $$ h_T = \arg\min_{h \in H} \sum_{i=1}^n r_i \cdot h(x_i) $$ where $$r_i = -\frac{\partial R(H_{T-1}(x_i))}{\partial H_{T-1}(x_i)}$$ Problem setting Let the dataset be $D = {(x_i, y_i)}_{i=1}^n$ where $y_i \in {-1, 1}$. The loss function is given by: $$L(H(x_i), y_i) = \exp(-y_i H(x_i))$$ ...

Why did we not have such a problem in EM algorithm? The Variational Autoencoder (VAE) architecture can be visualized as follows: Before expanding the KL divergence term, recall that: latent variable $q_{\phi}(z|x)$ is the approximate posterior (i.e., how likely the latent variable $z$ is given the input $x$). $p_{\theta}(z|x)$ is the true posterior (i.e., the actual distribution of $z$ given $x$ under the model). The KL divergence measures how much $q_{\phi}(z|x)$ deviates from $p_{\theta}(z|x)$: ...