Alessio Borgi

Alessio Borgi

PhD in Graph Neural Networks and Generative AI 👨🏻‍💻🤖

Word2vec: Skip-Gram, Negative Sampling and What the Vectors Learn

4 minute read

Published:

TL;DR: Skip-gram predicts context words from a centre word. Negative sampling trains observed pairs against sampled noise pairs, avoiding a full vocabulary normalisation at every update. The resulting vectors encode distributional regularities under the chosen corpus, window and noise distribution.

Turn a sentence into prediction pairs

From “the robot carries a parcel”, a window of radius one around carries gives the positive pairs (carries, robot) and (carries, a). A larger window adds more distant evidence. Dynamic windows and frequent-word subsampling change the distribution of training pairs, so they are part of the model specification.

Skip-gram uses a centre embedding \(v_w\) and a context embedding \(u_c\). These are different parameter tables even when the vocabulary is shared. With a full softmax,

\[ P(c\mid w)=\frac{\exp(u_c^\top v_w)}{\sum_{c'\in V}\exp(u_{c'}^\top v_w)}. \]

The denominator couples every context word. A direct update requires work involving the whole vocabulary, which motivates alternative objectives or normalisation strategies.

Negative sampling changes the objective

For one observed pair and \(K\) sampled noise contexts \(n_1,\ldots,n_K\), maximise

\[ \ell=\log\sigma(u_c^\top v_w)+\sum_{i=1}^{K}\log\sigma(-u_{n_i}^\top v_w), \qquad \sigma(s)=\frac{1}{1+e^{-s}}. \]

This trains binary discrimination between data pairs and noise pairs. It is not merely an exact softmax computed cheaply, and its pair scores are not automatically a normalised next-word distribution. A noise sample means “drawn from the noise process”, not “a word that can never be a valid context”.

The noise distribution matters. Sampling frequent contexts more often puts more pressure on the model to distinguish true associations from generic frequency. Changing \(K\) changes the balance of positive and negative evidence as well as computational cost.

One gradient update

Let \(s=u_c^\top v_w\). For a positive pair,

\[ \frac{\partial\log\sigma(s)}{\partial v_w}=(1-\sigma(s))u_c. \]

For a noise pair the contribution is \(-\sigma(u_n^\top v_w)u_n\). The positive gradient moves the centre vector toward the observed context; the negative term pushes against noise contexts. Both effects become smaller when the corresponding classification is already confident.

Take \(v_w=(1,0)\) and \(u_c=(0,1)\). Their dot product is zero, so \(\sigma(s)=0.5\). Ignoring noise for this illustration, a gradient-ascent step with learning rate 0.1 changes the centre vector to \((1,0.05)\) while holding the context vector fixed. The new dot product is 0.05: the observed pair receives a higher score.

In actual training, update all participating vectors from a consistent gradient computation. Updating one in place before computing the other’s gradient silently changes the step.

CBOW reverses the prediction direction

Continuous bag of words combines surrounding embeddings to predict the centre word. If the combination is a mean, the order within that context bag is lost. Skip-gram creates individual centre–context prediction events; CBOW pools context before prediction. This changes the learning signal rather than only renaming an architecture.

After training, choose which vector table to export and whether to combine the tables. Record this choice before comparing similarities. Normalising vectors for cosine retrieval is another transformation with its own effect.

What an analogy can and cannot establish

Vector offsets can expose regularities in a trained space, but a few successful analogies do not establish compositional reasoning. Results depend on the candidate vocabulary, similarity metric and exclusion rules. A nearest-neighbour test can reward frequency or topical associations rather than the relation you intended to test.

Use held-out task data alongside intrinsic tests. Include rare words, polysemous words and antonyms. A single static vector for bank blends its observed uses; lexical disambiguation explicitly asks which sense is active in a particular sentence.

Key Insight: The model learns to distinguish observed contexts from a specified noise process. Its geometry reflects both distributions, so the training sampler is part of what the vectors mean.
Common pitfall: Do not report negative-sampling sigmoid scores as a probability distribution over the vocabulary. They need not sum to one and were trained for a different discrimination problem.

Recap

  • Skip-gram predicts contexts; CBOW predicts a centre from combined context.
  • Centre and context embeddings are distinct parameters.
  • Negative sampling replaces full-vocabulary normalisation with a discrimination objective.
  • Static similarities require task-based validation and careful handling of polysemy.

References

  1. Mikolov et al. Distributed Representations of Words and Phrases and their Compositionality. NeurIPS, 2013.
  2. Levy and Goldberg. Neural Word Embedding as Implicit Matrix Factorization. NeurIPS, 2014.

You May Also Enjoy

The Standard Library Modules Worth Knowing Before You pip install

8 minute read

Published:

Python ships with a counter, a queue, a memoiser, a CLI parser, a test framework’s smaller cousin, and a JSON-safe date library already installed, most of what people reach for a package to solve is one import away.

Files and Context Managers: Why with Is Not Optional

9 minute read

Published:

A file object that falls out of scope without with does get closed eventually, but ‘eventually’ means whenever the garbage collector gets around to it, which is not a promise any program handling more than a handful of files can live with.

Errors and Exceptions: EAFP, the Hierarchy, and Reading a Traceback

9 minute read

Published:

Exceptions in Python are not exceptional. They are a normal control-flow mechanism that the language leans on so heavily that the idiomatic style is to try the operation and handle the failure, rather than check first, which is faster, shorter, and free of race conditions.