Mixture model clustering also known as soft clustering

Problem Setup

Data: Let the dataset be $X = {x_1, x_2, …, x_N}$, where each data point $x_i \in \mathbb{R}^d$. The data points are unlabeled.

Latent Variable: Let $z \in {1, …, m}$ be a latent variable representing the clusters, where $m$ is the total number of clusters.

We can extend the dataset as $D = {(x_i, z_i)}_{i=1}^N$, where each data point $x_i$ is associated with its (unknown) cluster $z_i$.

The GMM defines the probability of observing a data point $x$ as:

$p_\theta(x) = \sum_{j=1}^m p_\theta(x|z=j) p_\theta(z=j) = \sum_{j=1}^m N(x; \mu_j, \Sigma_j) \pi_j$

Where:

$N(x; \mu_j, \Sigma_j)$ is the Gaussian probability density function for cluster $j$, with mean $\mu_j$ and covariance matrix $\Sigma_j$.
$\pi_j$ is the probability of a data point belonging to cluster $j$.

The set of parameters $\theta$ that we need to estimate consists of: $$\theta = {\mu_1, …, \mu_m, \Sigma_1, …, \Sigma_m, \pi_1, …, \pi_m}$$

We use the EM algorithm to get ideal parameters $\theta^* = {\mu^, \Sigma^, \pi^*}$.

Use the parameters $\theta^* = {\mu^, \Sigma^, \pi^*}$ to get the soft clustering:

p_{\theta^*}(z=z_i|x=x_i) = \frac{p_{\theta^*}(x_i|z_i) p_{\theta^*}(z_i)}{\sum_z p_{\theta^*}(x_i|z) p_{\theta^*}(z)}

p_{\theta^*}(z=z_i|x=x_i) = \frac{N(x_i; \mu^*_{z_i}, \Sigma^*_{z_i}) \pi^*_i}{\sum_j N(x_i; \mu^*_j, \Sigma^*_j) \pi^*_j}

$$p_{\theta^*}(z=z_i|x=x_i) = \frac{N(x_i; \mu^*_{z_i}, \Sigma^*_{z_i}) \pi^*_i}{\sum_j N(x_i; \mu^*_j, \Sigma^*_j) \pi^*_j}$$

This is a discrete distribution over the $m$ clusters.

This soft clustering can also be used for embedding or dimensionality reduction:

Compute the probability distribution for each data point over the $m$ clusters.
Use these probabilities as an $m$-dimensional embedding for each data point.
This new representation captures the data’s structure in a lower-dimensional space.