Topic Models: LDA, Mixed Membership and Interpretable Word Groups

A document can contain several subjects

A support ticket can discuss delivery and battery life in the same paragraph. Assigning it to one cluster forces a single membership; a topic mixture allows both subjects to contribute. The topics themselves are distributions over vocabulary entries, not human-authored labels.

For \(K\) topics, let \(\theta_d\) be document \(d\)’s topic mixture and \(\phi_k\) topic \(k\)’s word distribution. LDA places Dirichlet priors on those distributions, then draws a topic \(z_{dn}\) and word \(w_{dn}\) for each token:

The topic assigned to one token need not be the topic assigned to the next. That token-level mixed membership is what allows a document to combine subjects.

Marginalise the hidden topic

Conditional on the mixtures and topic distributions,

Suppose a ticket is 0.75 hardware and 0.25 shipping. If battery has probability 0.20 under hardware and 0.01 under shipping, its probability in the document is \(0.75(0.20)+0.25(0.01)=0.1525\). This is a mixture of word distributions, not a claim that 75% of the characters describe hardware.

A word can have substantial probability in several topics. Very common words can dominate unless vocabulary choices and priors are handled carefully. Removing words indiscriminately can also discard useful distinctions, so inspect the resulting topics rather than relying on a generic stop list.

Priors control concentration

For a symmetric document prior with parameter \(\alpha\) per topic, values below one tend to favour mixtures near the simplex boundaries, while larger values favour more even mixtures. The total concentration is \(K\alpha\). Keeping \(\alpha\) fixed while changing \(K\) therefore also changes the prior’s total concentration.

The topic–word prior similarly controls how probability is distributed across the vocabulary. These are preferences over distributions, not hard limits on how many topics a document may use or how many words a topic may contain.

One collapsed Gibbs update

A common inference approach integrates out \(\theta\) and \(\phi\) and samples token assignments. Excluding the token currently being updated,

The first factor prefers topics already used in the document. The second prefers topics that already use the observed word. Normalise these weights across topics before sampling.

For two topics, suppose document counts are 3 and 1, word counts are 4 and 0, topic totals are both ten, \(\alpha=1\) and \(\beta=1\) over a five-word vocabulary. The weights are \(4(5/15)=4/3\) and \(2(1/15)=2/15\). Their normalised probabilities are \(10/11\) and \(1/11\). The current token must be removed before computing those counts; otherwise its previous assignment reinforces itself incorrectly.

Interpretability is not likelihood

A topic’s top words may look coherent while the document assignments are poor. Conversely, a model can improve held-out likelihood by capturing frequent statistical patterns that humans find unhelpful. Evaluate predictive fit, stability across runs and usefulness for the intended exploration task separately.

Topic labels are permutation-invariant: “topic 2” in one run need not correspond to “topic 2” in another. Align topics by their distributions before comparing them. Also inspect representative documents, not only the ten largest word probabilities.

Bag-of-words topic models discard order. “The robot damaged the parcel” and “the parcel damaged the robot” have the same counts and therefore the same document evidence under that representation. For event roles, move to semantic role labelling rather than asking topics to recover information they never received.

References

1. Blei, Ng and Jordan. Latent Dirichlet Allocation. JMLR, 2003.
2. Chang et al. Reading Tea Leaves: How Humans Interpret Topic Models. NeurIPS, 2009.