Description

1. [16 points] Gaussian Mixture Models & EM
Consider a Gaussian mixture model with K components (k ∈ {1,…,K}), each having mean µk, variance σk2, and mixture weight πk. All these are parameters to be learned, and we subsume them in the set θ. Further, we are given a dataset X = {xi}, where xi ∈ R. We also use Z = {zi} to denote the latent variables, such that zi = k implies that xi is generated from the kth Gaussian.
(a) What is the log-likelihood of the data logp(X;θ) according to the Gaussian Mixture Model? (use µk, σk, πk, K, xi, and X). Don’t use any abbreviations.

(b) For learning θ using the EM algorithm, we need the conditional distribution of the latent variables Z given the current estimate of the parameters θ(t) (we will use the superscript (t) for parameter estimates at step t). What is the posterior probability p(zi = k|xi;θ(t))? To simplify, wherever possible, use N(xi|µk,σk) to denote a Gaussian distribution over xi ∈ R having mean µk and variance σk2.

(c) Find Ezi|xi;θ(t)[logp(xi,zi;θ)]. Denote p(zi = k|xi;θ(t)) as zik, and use all previous notation simplifications.
Your answer:
K
X
Ezi|xi;θ(t)[logp(xi,zi;θ)] = P(zik = 1|xi)logp(xi,zik = 1)
k=1
K
= Xzik log(N(xi|µ(kt),σk(t))) k=1
(d) θ(t+1) is obtained as the maximizer of )]. Find , and , by using your answer to the previous question.

(e) How are kMeans and Gaussian Mixture Model related? (There are three conditions)
Your answer: k-means is obtained from E-M on GMMs via:
• uniform mixture weights
• diagonal covariances cI
• c ↓ 0
2