Generalized Linear Regression extends the concept of linear regression by projecting the data into a higher-dimensional space before performing linear regression. This allows us to capture more complex relationships in the data.

Let the data be $(x_i, y_i)$ for $i = 1, 2, …, n$ where $x_i \in \mathbb{R}^d$ and $y_i \in \mathbb{R}$.

We define a feature map $\phi: \mathbb{R}^d \rightarrow \mathbb{R}^m$, where $m > d$. This map projects our original features into a higher-dimensional space.

Then we model $y_i$ as:

$$Y = \beta_0 + \beta_1\phi(X) + \varepsilon$$

Where:

  • $Y \in \mathbb{R}$ is the dependent variable
  • $X \in \mathbb{R}^d$ is the original independent variable
  • $\phi(X) \in \mathbb{R}^m$ is the projected feature vector
  • $\beta_0$ is the y-intercept (bias term)
  • $\beta_1$ is the coefficient vector in the higher-dimensional space
  • $\varepsilon$ is the error term

Feature Maps

Common choices for the feature map $\phi$ include:

  1. Polynomial features: $\phi(x) = [x, x^2, x^3, …, x^p]$
  2. Radial Basis Functions (RBF): $\phi_i(x) = \exp(-\gamma ||x - c_i||^2)$
  3. Sigmoid: $\phi_i(x) = \tanh(ax_i + b)$

The choice of feature map allows us to capture different types of non-linear relationships in the data while still using the linear regression framework.

Next steps

Now the data has been transformed into a higher-dimensional space, we can perform linear regression in this space as usual.