GloVe and fastText: Global Counts and Subword Sharing

Two different questions about word vectors

After word2vec, two natural questions remain. Can we fit the global co-occurrence matrix directly? Can a rare or unseen spelling borrow parameters from familiar parts? GloVe addresses the first; fastText’s subword representation addresses the second. These are different modelling dimensions, so “which is more modern?” is not the useful comparison.

Let \(X_{ij}\) count how often context word \(j\) appears around word \(i\). The count depends on window construction and any distance weighting. Preserve that definition when comparing algorithms.

GloVe as weighted regression

A standard GloVe objective is

The vectors model interactions, while biases absorb some word and context frequency effects. The weighting function limits the influence of unreliable small counts and prevents very large counts from dominating purely through their magnitude. A common family uses \(f(x)=(x/x_{\max})^\alpha\) below \(x_{\max}\) and one above it.

Why logarithms? A multiplicative count difference becomes additive. If one pair occurs 100 times and another ten times, their log-count difference is \(\log 10\). The model can express such differences through vector dot products and biases without requiring a score proportional to the raw count.

Inspect one residual

Suppose \(X_{ij}=4\), the weight is 0.5, and the current predicted log count is 1.0. The residual is \(1-\log4\approx-0.3863\). This pair contributes about \(0.5(0.3863)^2\approx0.0746\) to the objective. Its derivative with respect to the predicted score is \(2(0.5)(1-\log4)\approx-0.3863\), so gradient descent increases that score.

This arithmetic makes the objective concrete: it is fitting a numeric target for each observed pair. Zero counts cannot enter through \(\log0\) and are excluded in this formulation. Absence from the observed matrix therefore has a different role than sampled negative evidence in skip-gram training.

fastText shares pieces across words

Represent a word through a collection of character n-grams, commonly with boundary markers and a whole-word component. Abstractly,

where \(z_g\) is a learned vector for a subword feature. The precise inclusion and aggregation conventions must be recorded. The important property is shared parameters: robot and robots reuse several character fragments even if the latter is rare.

For a toy three-character inventory, <cat> supplies <ca, cat and at>. <cats> supplies <ca, cat, ats and ts>. The first two fragments are shared. If the plural word was absent during training, those shared fragments still carry learned values, while unfamiliar fragments may contribute little useful evidence.

What sharing buys, and what it confuses

Morphologically related forms often benefit from shared fragments. Misspellings may also remain close enough for useful predictions. But shared spelling does not guarantee shared meaning, and unrelated words can overlap in fragments. Conversely, irregular forms such as go and went share little surface material despite a close morphological relationship.

Hashing n-grams into a fixed bucket table bounds memory but permits collisions: unrelated fragments can update the same parameter. Evaluate this as a resource–accuracy tradeoff rather than assuming every bucket names one linguistic unit.

An unseen-word vector is computable, but its quality is an empirical question. Compare frequent, rare, misspelled and genuinely novel words separately. For sequence labelling, also test whether casing and inflectional distinctions survive the preprocessing used to build those vectors.

References

1. Pennington, Socher and Manning. GloVe: Global Vectors for Word Representation. EMNLP, 2014.
2. Bojanowski et al. Enriching Word Vectors with Subword Information. TACL, 2017.