Neural Language Models: Shared Embeddings, Recurrence and Training

Share evidence across similar histories

A count model treats every stored history as a separate case. A neural model maps tokens to vectors and uses common parameters to turn the context into a prediction. Histories containing related words can then share statistical strength, although whether the learned similarity is useful depends on training.

A fixed-window feed-forward model uses a chosen number of previous embeddings. A recurrent model updates a state after each token. Both still implement the same conditional factorisation introduced in n-gram language models; the architecture changes how the conditional probabilities are estimated.

A recurrent parameterisation

Let \(e_t=E[w_t]\) be the embedding of the current token. A simple recurrent model uses

If embeddings have dimension \(d\), states dimension \(H\) and the vocabulary size \(V\), then \(W_h\) is \(H\times d\), \(U_h\) is \(H\times H\) and \(W_o\) is \(V\times H\). Writing the dimensions catches a common confusion: the embedding lives in a different space from the hidden state unless the design makes them equal.

The state can depend on the whole prefix in principle, but it has fixed dimension and must learn what to retain. Gated recurrent units and LSTMs change the state dynamics to make some forms of information retention easier; they do not guarantee perfect memory.

Train with shifted targets

For a sentence containing robot moves, the inputs might be [START, robot, moves] and the targets [robot, moves, END]. Each target follows its input prefix. Feeding the target token into the context at the same prediction position leaks the answer.

The average negative log likelihood over valid target positions is

where \(m_t\) masks padding. If three targets receive probabilities 0.5, 0.25 and 0.8, the mean loss is \(-(\log0.5+\log0.25+\log0.8)/3\approx0.7675\). Perplexity is its exponential, approximately 2.154. Averaging losses separately per sentence gives a different weighting from averaging over all tokens.

During teacher-forced training, the context contains observed previous tokens. During generation, it contains the model’s earlier outputs. Errors can therefore move generation into contexts encountered less often in training. The decoding chapter examines how the inference procedure uses these distributions.

Gradients through time

Backpropagation follows the recurrent state through earlier steps. Repeated Jacobian products can shrink or grow strongly, making long-range credit assignment difficult. Gradient clipping controls the size of an update caused by large gradients; it does not recover information from gradients that have vanished.

Truncated backpropagation detaches the computation graph at chosen boundaries. The numeric hidden state may continue across a chunk boundary while the gradient stops there. Resetting the state is a different operation. Decide which is appropriate for sentence boundaries, independent documents and streams.

Evaluate what the model was allowed to see

Use the same tokenisation, vocabulary and boundary policy for comparisons. Ensure padded positions do not contribute to loss. If state is carried across documents, report the ordering and whether that matches deployment; accidental state carryover can make one document influence another.

Inspect performance by context length and rare-word frequency. A lower aggregate perplexity does not prove the model resolves reference, respects negation or answers questions faithfully. Those claims require language tasks with corresponding outputs and checks.

References

1. Bengio et al. A Neural Probabilistic Language Model. JMLR, 2003.
2. Mikolov et al. Recurrent Neural Network Based Language Model. Interspeech, 2010.
3. Hochreiter and Schmidhuber. Long Short-Term Memory. Neural Computation, 1997.