What is a Programming Language?

A programming language is a formal system that allows humans to give instructions to computers. It acts as a bridge between human-readable commands and machine code. Python, which we’ll focus on in this article, is a popular high-level programming language known for its readable syntax and extensive library ecosystem.

What is a Module?

A module in Python is a file containing Python code that can be imported and used in other Python programs. It’s a fundamental way to organize and reuse code by grouping related functions, classes, and variables together.

Key Aspects of Modules:

  1. Code Organization: Modules help break down large programs into manageable files. Each module typically focuses on a specific functionality.

  2. Code Reusability: Once created, modules can be imported and used in multiple programs:

    import math  # importing the math module
result = math.sqrt(16)  # using a function from the module

  3. Namespace Management: Modules create their own namespace (a container that maps names to objects, ensuring that identifiers like function and variable names remain unique within that module’s scope), preventing naming conflicts:

    import module1
import module2

module1.function_name()  # using function from module1
module2.function_name()  # using different function with same name from module2

Types of Modules:

  1. Built-in Modules: Come with Python installation

    • math: Mathematical functions
    • random: Random number generation
    • datetime: Date and time handling

  2. Third-party Modules: Created by others, installed via pip (pip is a package manager for Python - just like how you download apps from Google Play Store on your Android phone, pip allows you to download and install Python packages from the Python Package Index (PyPI). It handles downloading, installing and managing these packages automatically)

    • Can be part of larger packages

  3. User-defined Modules: Created by the programmer

    # mymodule.py
def greet(name):
    return f"Hello, {name}!"

# main.py
import mymodule
print(mymodule.greet("Alice"))

Module Import Methods:

  1. Import Entire Module:

    import math
math.sqrt(16)

  2. Import Specific Items:

    from math import sqrt
sqrt(16)

  3. Import with Alias:

    import numpy as np
np.array([1, 2, 3])

What is a package?

A package in Python is a way to organize related modules into a directory hierarchy. It’s essentially a directory that contains multiple Python modules and a special __init__.py file that marks it as a package. Think of it as a folder containing multiple Python files (modules) that work together.

Key Components:

  1. Directory Structure: A package must have:

    • A directory containing Python modules (.py files)
    • A special __init__.py file (can be empty)

  2. __init__.py: This file:

    • Marks the directory as a Python package
    • Can initialize package-level variables
    • Can import specific items to make them available when the package is imported

Example of a basic package structure:

Key Characteristics of Packages:

  1. Organization: Packages help organize related code into manageable units. For example:

    mypackage/
├── __init__.py
├── module1.py
├── module2.py
└── subpackage/
    ├── __init__.py
    └── module3.py

  2. Namespace Management: Packages create hierarchical namespaces that help avoid naming conflicts. For instance:

    from mypackage.module1 import function1
from mypackage.subpackage.module3 import function2

  3. Distribution: Packages can be easily distributed and installed using package managers like pip. For example:

    pip install numpy  # numpy is a package

Types of Packages:

  1. Built-in Packages: Come pre-installed with Python

    • os: Operating system interface
    • sys: System-specific parameters
    • math: Mathematical functions

  2. Third-party Packages: Created by the Python community

    • numpy: Numerical computing
    • pandas: Data analysis
    • requests: HTTP library

  3. Local Packages: Created by developers for their specific projects

Package vs Module:

  • A module is a single Python file containing code
  • A package is a collection of modules
  • Every package is a module, but not every module is a package

What is a library?

A library is a collection of pre-written code (modules and packages) that can be easily reused. Libraries provide ready-to-use implementations of various functionalities, saving developers from having to write common code from scratch.

Key Characteristics of Libraries:

  1. Reusability: Libraries contain code that can be used across different projects

    import numpy as np  # Reusing numpy's array functionality
array = np.array([1, 2, 3])

  2. Abstraction: Libraries hide complex implementations behind simple interfaces

    import requests  # Complex HTTP logic abstracted into simple methods
response = requests.get('https://api.example.com/data')

  3. Maintenance: Libraries are typically well-maintained and optimized

    import pandas as pd  # Regularly updated with bug fixes and improvements
df = pd.read_csv('data.csv')

Types of Libraries:

  1. Standard Library: Comes with Python installation

    • datetime: Date and time handling
    • json: JSON encoding/decoding
    • random: Random number generation

  2. Third-party Libraries: External libraries that extend Python’s capabilities

    • matplotlib: Data visualization
    • scipy: Scientific computing
    • django: Web development

  3. Domain-specific Libraries: Focused on particular areas

    • tensorflow: Machine learning
    • beautifulsoup4: Web scraping
    • pygame: Game development

Library vs Package:

  • A library is a broader term that may contain multiple packages
  • A package is a way of organizing modules, while a library provides functionality
  • For example, scipy is a library that contains multiple packages for scientific computing

Different Package Versions May Be Incompatible

There are several key reasons why different versions of packages may be incompatible with each other:

1. Breaking Changes in APIs

  • API Changes: Newer versions of packages often introduce changes to their Application Programming Interface (API)
    # Old version (1.0)
model.train(data)

# New version (2.0) - Breaking change
model.fit(data)  # Method name changed from train to fit

2. Dependency Chain Conflicts

  • Nested Dependencies: Packages often depend on other packages, creating a chain of dependencies
    PackageA (v2.0) → requires PackageB (v1.5+)
PackageC (v1.0) → requires PackageB (v1.0-1.4)
    Here, you cannot use PackageA v2.0 and PackageC v1.0 together due to conflicting PackageB requirements

3. Python Version Requirements

  • Different package versions may require different Python versions
    # Package requirements example
tensorflow 1.x → Python 2.7 or 3.3-3.6
tensorflow 2.x → Python 3.7+

4. Operating System Dependencies

  • System Libraries: Packages may rely on different versions of system-level libraries
    # Example: Package requiring specific system libraries
opencv-python 3.x → requires libGL.so.1
opencv-python 4.x → requires newer version of libGL.so.1

5. Internal Data Structures

  • Data Format Changes: Different versions may use incompatible internal data structures
    # Old version serialized data
model_v1.save('model.pkl')  # Uses format A

# New version cannot read old format
model_v2.load('model.pkl')  # Expects format B

6. Feature Deprecation and Removal

  • Removed Features: Newer versions may remove features that older dependent packages rely on
    # Old version
pandas.DataFrame.as_matrix()  # Works

# New version
pandas.DataFrame.as_matrix()  # Raises AttributeError - Method removed

7. Security Updates

  • Security Patches: Newer versions may implement security fixes that change how features work
    # Old version - vulnerable
requests.get(url, verify=False)  # Allows insecure connections

# New version - stricter security
requests.get(url)  # Requires SSL verification by default

These are just some of the reasons why we cannot just use any package version for any project.

This is a bad solution

A common but problematic approach that many beginners take is installing all packages globally in their base Python environment. Here’s what this typically looks like:

The “Install Everything in Base” Approach

  1. Initial Project Setup

    # Project A needs tensorflow 1.15
pip install tensorflow==1.15
pip install numpy==1.16
pip install other-packages==old-versions

  2. Starting a New Project

    # Project B needs tensorflow 2.0
pip uninstall tensorflow  # Remove old version
pip install tensorflow==2.0  # Install new version
pip uninstall numpy  # Remove incompatible version
pip install numpy==1.19  # Install compatible version

Problems with this Approach

  1. Time-Consuming Package Management

    • Need to constantly install and uninstall packages
    • Have to remember exact versions needed for each project
    • Must reinstall all dependencies when switching projects

  2. Dependency Conflicts

    # Project A's code breaks because tensorflow 2.0 is not backward compatible
import tensorflow as tf
tf.placeholder()  # This function exists in 1.x but not in 2.x

  3. Version Tracking Nightmare

    • No clear record of which versions work for which project
    • Easy to forget dependencies when sharing code
    • Hard to reproduce environment on another machine

  4. System Instability

    • Frequent package changes can corrupt Python environment
    • System-wide Python installation becomes unstable
    • Risk of breaking system tools that depend on Python

  5. Project Isolation Issues

    # Project A and B running simultaneously
import tensorflow  # Which version will this use?
# Both projects might try to use different versions
# Leading to unpredictable behavior

  6. Debugging Complexity

    • Hard to determine if issues are from:
      • Current package versions
      • Leftover conflicts from previous installations
      • Incomplete package removals
      • System-level Python problems

This approach is like having one toolbox for all your different jobs - when you need a different set of tools, you have to take everything out and put new tools in. It’s inefficient, prone to errors, and makes it difficult to switch between projects quickly.

A better solution - Virtual Environment

How Virtual Environments Help with Segregation

Virtual environments solve our dependency problems by providing complete isolation between projects. Let’s see how:

  1. Project-Specific Package Management

    # Project A's virtual environment
project_a_env/
   python==3.7
   tensorflow==1.15
   numpy==1.16

# Project B's virtual environment  
project_b_env/
   python==3.8
   tensorflow==2.0
   numpy==1.19

    Each project gets its own “container” with exactly the packages it needs.

  2. Independent Python Versions

    • Different projects can use different Python versions
    • Example:
      # Web project using Python 3.9
web_env/
   python==3.9
   django==4.2

# ML project using Python 3.7
ml_env/
   python==3.7
   tensorflow==1.15

  3. Clean Dependency Management

    # requirements.txt for Project A
tensorflow==1.15
numpy==1.16
pandas==1.0.3

# requirements.txt for Project B
tensorflow==2.0
numpy==1.19
pandas==1.3.0

    Each project maintains its own list of exact package versions.

  4. Easy Environment Reproduction

    # Create and activate new environment for Project A
python -m venv project_a_env
source project_a_env/bin/activate  # On Unix/macOS

# Install Project A's dependencies
pip install -r requirements.txt

    Anyone can recreate the exact environment needed.

  5. Safe Experimentation

    # Create temporary environment for testing
python -m venv test_env

# Try new packages without affecting other projects
pip install experimental_package

# If things go wrong, just delete the environment
deactivate
rm -rf test_env

    You can experiment without risking your stable environments.

  6. Multiple Projects Running Simultaneously

    # Terminal 1 (Project A)
source project_a_env/bin/activate
python script.py  # Uses tensorflow 1.15

# Terminal 2 (Project B)
source project_b_env/bin/activate
python script.py  # Uses tensorflow 2.0

    Run different projects with different dependencies at the same time.

This segregation provides several benefits:

  • Stability: Each project has its own stable environment
  • Reproducibility: Environments can be exactly recreated
  • Flexibility: Easy to switch between projects
  • Safety: Experiments won’t affect other projects
  • Clarity: Clear documentation of dependencies
  • Portability: Projects can be easily shared and deployed

Think of virtual environments like separate workshops - each project gets its own space with its own set of tools, and there’s no risk of tools from one project interfering with another.

How to create a virtual environment

Using Python’s built-in venv module

The built-in venv module provides a lightweight way to create isolated Python environments. Here’s how to use it:

  1. Creating a Virtual Environment

    # Basic syntax
python -m venv environment_name

# Example
python -m venv myproject_env

  2. Activating the Environment

    # On Windows
myproject_env\Scripts\activate

# On Unix/macOS
source myproject_env/bin/activate

    After activation, your prompt will change to show the active environment.

  3. Installing Packages

    # Install packages using pip
pip install package_name

# Example
pip install numpy pandas

# Install from requirements file
pip install -r requirements.txt

  4. Deactivating the Environment

    deactivate

Limitations of venv

Think of venv like a basic toolbox - it works fine for simple projects but has some key drawbacks. It can’t automatically figure out when tools might conflict with each other, only works with Python tools (not other types of software), needs different commands on Windows vs Mac/Linux, and isn’t great for data science projects that need special high-performance tools. This is why many people use Conda instead, which is like a more advanced toolbox that can handle all these complex situations better - it manages tool conflicts automatically, works the same way on all computers, can handle non-Python tools, and comes with optimized tools for data science work.

Using Conda

Conda is a powerful package and environment management system that’s particularly popular in data science. Here’s a comprehensive guide to using Conda effectively:

Basic Environment Management

  1. Creating Environments
    # Create with specific Python version (recommended)
conda create --name myenv python=3.10

# Activate environment (works on all platforms)
conda activate myenv

# List environments
conda env list

# Remove environment
conda remove --name myenv --all

Package Management

  1. Installing and Managing Packages
    # Install packages
conda install numpy pandas scikit-learn

# View installed packages
conda list

# Search for package
conda list | grep numpy

# Check dependencies
conda depends package_name
conda depends --reverse-depends package_name

# Test installation compatibility
conda install --dry-run new_package

Environment Documentation

  1. Export/Import
    # Export environment
conda env export > environment.yml

# Export core packages only
conda env export --from-history > environment.yml

# Create from file
conda env create -f environment.yml

Key Reminder: Always document configurations, test thoroughly, and maintain version control for reproducibility.

Some tips when learning from online resources

I love learning from YouTube tutorials, but since Python packages evolve so rapidly, I often find myself watching videos that are a few years old. This can lead to version mismatches between what’s shown in the tutorial and what’s currently available. Here’s how I handle these situations:

Case 1: If following a tutorial series/playlist/course

When working through a structured tutorial series:

  1. Initial Setup: Environment Matching Tutorial Versions

    To ensure a smooth learning experience when following tutorials, especially older ones, it is crucial to replicate the environment used in the tutorial as closely as possible. This primarily involves using the same versions of Python and the relevant packages.

    # Step 1: Create a dedicated conda environment with the Python version from the tutorial
conda create --name tutorial_env python=3.x  # Replace 3.x with the Python version mentioned in the tutorial (e.g., 3.6, 3.7)

# Step 2: Activate the newly created environment
conda activate tutorial_env

# Step 3: Install packages with the EXACT versions specified in the tutorial
conda install package1==version1 package2==version2 package3==version3 # ... and so on

    For example, if a tutorial uses TensorFlow 1.14 and Keras 2.2.4, the installation command would be:

    conda install tensorflow==1.14 keras==2.2.4

    Rationale:

    • Ensures Code Compatibility: By using the same package versions, you minimize the risk of encountering errors or unexpected behavior due to version mismatches. The code in the tutorial is designed to work with specific versions, and replicating this environment ensures that the code runs as intended.
    • Focus on Core Concepts: Learning becomes more efficient when you are not constantly debugging version-related issues. Matching the tutorial environment allows you to concentrate on understanding the fundamental concepts and algorithms being taught, rather than spending time troubleshooting compatibility problems.
    • Directly Follow Tutorial Instructions: You can follow the tutorial step-by-step without deviations caused by different software versions. This direct alignment simplifies the learning process, especially for beginners.

Case 2: Integrating tutorial concepts into existing projects

When you’re learning specific concepts from tutorials for your current project, you’re in a situation where you already have a defined project environment. In this case, directly using the tutorial’s packages might not be feasible or desirable. Here’s how to effectively learn from tutorials and apply the knowledge to your existing project:

  1. Focus on Conceptual Understanding

    • Extract Core Principles: Focus on understanding the fundamental concepts, algorithms, and design patterns being taught, independent of specific implementations
    • Identify Version-Independent Logic: Distinguish between core logic that remains constant across versions versus syntax/API details that may change
    • Document Version Context: Make note of which features are version-specific or deprecated to build awareness of library evolution and best practices
    • Build Mental Models: Create clear mental models of how different components interact, making it easier to adapt concepts across versions

  2. Strategic Code Adaptation

    • Pattern-Based Translation: Rather than direct code copying, identify the underlying patterns and implement them using your environment’s idioms
    • Maintain Environment Integrity: Carefully evaluate additions to prevent dependency conflicts and maintain project stability
    • Progressive Enhancement: Add functionality incrementally while testing for compatibility at each step
    • Example of Modern Adaptation:
      # Tutorial (older version):
# from tensorflow.keras.layers import Dense

# Modern adaptation options:
from keras.layers import Dense  # Standalone Keras
# or
from tensorflow.keras.layers import Dense  # TF 2.x
# or
import torch.nn as nn  # PyTorch equivalent
dense_layer = nn.Linear(in_features, out_features)

  3. Systematic Package Integration

In case a new package is required, then we cannot blindly install it as mentioned in the tutorial as it might not be compatible with the current environment. Here is how we can do it:

  • Environment Analysis:

    # Review current state
conda list  # Full package inventory
conda list | grep package_name  # Search specific package

  • Compatibility Verification:

    # Pre-installation compatibility check
conda install --dry-run new_package

# Version-specific check
conda install --dry-run "new_package>=2.0,<3.0"

  • Version Management:

    # List all available versions
conda search new_package

# Analyze package dependencies
conda depends new_package
conda depends --reverse-depends new_package  # What depends on it?

  • Smart Installation Process:

    1. Begin with latest stable version compatible with your environment
    2. If conflicts arise, systematically test earlier versions
    3. Leverage additional channels when needed:
      # Try conda-forge for additional versions
conda install -c conda-forge new_package

# Install specific version with constraints
conda install "new_package=2.1.*" --channel conda-forge

Conclusion

In today’s development landscape, we constantly switch between multiple projects and learning resources, each with unique package requirements and dependencies.

Without proper environment management, this juggling act can quickly become overwhelming, leading to conflicts and compatibility issues. This is where Conda proves invaluable - while it requires an initial time investment to learn, its robust environment management capabilities make it an essential tool for modern development.

The ability to create isolated, reproducible environments for each project or tutorial ensures smooth transitions between different tasks and eliminates dependency headaches. The time spent mastering Conda is minimal compared to the hours saved in debugging environment issues and managing project dependencies.

Also note that i have not made this post as a how to guide on how to use or install Conda, as that keeps on changing as new versions are released hence refer to the Conda documentation for the latest information. This post is just to motivate you to use a virtual environment for your projects.