The same two steps as k-means — but soft.
k-means asks “which centroid is nearest?” and every point must pick one. A mixture model
asks “how likely is it that this point came from component j?” and the answer is
a set of fractions that add to 1, called the responsibilities.
E-step recomputes those fractions for every point; M-step refits each component
— its weight, its mean, and its covariance — using every point, weighted by how
much of it that component owns. That is the entire algorithm.
The two steps: Expectation and Maximization.
How to read the picture. Each point is painted by blending the component
colors in proportion to its responsibilities, so a point that is 50/50 comes out muddy. Each
component is drawn as two ellipses, at one and two standard deviations. The background is the same
blend, so the boundaries between clusters are gradients, not the hard straight edges you see
on the k-means page.
Three things worth trying.
1. Overlapping, k = 2. Run it, then look at the band between the two blobs: those points
stay muddy for ever, because they genuinely are ambiguous and the model says so. Only 64% of
the points are ever more than 90% committed to a side — on Three blobs that figure is
100%. k-means has no way to express the difference: it must hand every one of them to a side.
This is the whole reason to use a mixture model: the uncertainty is part of the answer.
2. Elongated, k = 2. With Full the two ellipses tilt and lie along the cigars, and
every point lands in the right one. Now press Circular and re-run: forced to be round, the
components stop separating the two cigars and start cutting across them — only about
65% of points end up in the right group, and the log-likelihood falls from −1962 to
−2102. That is the punchline: k-means is a mixture model with circular, equal-sized
components and hard assignment. Everything GMM adds over k-means is in that one word,
covariance.
3. Uniform, and step k from 1 upward. The log-likelihood climbs every single time you add a
component — −2312, −2298, −2265, −2250, −2243, −2222 —
on data with no structure at all. It never peaks, so you cannot choose k by picking the best
likelihood: that is what AIC and BIC are for, and it is the same trap as picking the smallest J in
k-means.
| Step | The formula | In one sentence |
|---|---|---|
| E-step | r(i,j) = w₁⋅N(xᵢ | μ₁, Σ₁) ÷ Σₗ wₗ⋅N(xᵢ | μₗ, Σₗ) |
How much of point i does component j own? The fractions add to 1. |
| M-step | μ₁ = Σᵢ r(i,j)⋅xᵢ ÷ Σᵢ r(i,j) |
Refit each component from every point, weighted by what it owns. |
| Score | log L = Σᵢ log Σₗ wₗ⋅N(xᵢ | μₗ, Σₗ) |
The log-likelihood. EM can never make it go down. |
n_init restarts it for exactly that reason.