Multilingual NLP: Transfer, Tokenisation and Uneven Language Coverage

A shared model does not imply a shared task

An English named-entity scheme may rely heavily on capitalisation. A language without the same casing convention removes that cue. A classifier trained on words separated by spaces encounters a different segmentation problem in another writing system. Rich inflection can also increase the number of surface forms for the same underlying lexeme.

Define the output consistently enough to compare it, while acknowledging where linguistic categories or annotation conventions differ. A translated label name does not guarantee that annotators applied the same operational rule.

Three routes to transfer

Translate-train translates labelled training text into the target language. Translate-test translates target-language input into a language supported by the model. Shared multilingual representations train common parameters so a task learned in one language may transfer to another.

Each route moves a bottleneck. Translation can change entity boundaries, sentiment intensity or cultural references. Shared representations avoid explicit translation at inference but may still encode unequal coverage from the pretraining corpus. Zero-shot transfer means no target-language task supervision under a specified protocol, not that the model has never encountered that language.

Sampling balances frequency and exposure

Suppose language \(\ell\) has \(n_\ell\) training examples. A temperature-like sampling policy can use

At \(\alpha=1\), sampling follows dataset size. At \(\alpha=0\), each nonempty language receives equal probability. Intermediate values flatten the distribution.

For languages with 100 and one units of data, \(\alpha=1\) allocates approximately 99% and 1% of samples. With \(\alpha=0.5\), the weights are ten and one, yielding about 91% and 9%. The smaller language receives more exposure, but repeatedly sampling the same examples does not create new linguistic evidence. Watch for overfitting and negative transfer.

Measure token fragmentation

Let a held-out corpus contain \(W\) whitespace-delimited words under a declared convention and \(T\) model tokens. The ratio \(T/W\) is one fragmentation diagnostic for languages where that word convention is meaningful. For cross-script comparisons, tokens per character or per byte can answer different questions; none is universally language-neutral.

If one corpus averages two model tokens per annotated word and another four, a 512-token window contains roughly half as many annotated words in the second. Vocabulary sharing therefore changes the effective context budget, not only the embedding table size.

Inspect rare scripts, borrowed words, code-switching and transliterated text. A language-identification front end that rejects mixed-language input can fail before the task model receives any text.

Keep script and language separate

A script can serve several languages, and a language can be written in more than one script or in informal transliteration. Script detection alone is not language identification. Unicode normalisation can align equivalent encodings but does not translate spelling conventions or repair all mixed-script confusables.

Preserve original text and offsets when annotations refer to spans. If a translation or transliteration changes length, propagate annotations through an explicit alignment rather than reusing old offsets.

Evaluate the distribution you intend to support

Report per-language scores and both macro and deployment-weighted aggregates. A macro mean treats languages equally; a traffic-weighted mean reflects usage frequency. Neither replaces the individual results. Include uncertainty when a test set is small.

Check whether evaluation data is naturally authored or translated. Translationese can differ systematically from native text and may make a benchmark easier or less representative. Also control duplicated or parallel content across splits: translating a training sentence does not necessarily create an independent test example.

Diagnose transfer rather than assuming it

Compare a shared model with a relevant monolingual or language-specific baseline where resources permit. Examine whether gains concentrate on shared names and cognates while grammar-sensitive cases remain weak. Validate annotation quality with speakers competent in the language and domain.

Language coverage is an empirical claim about data and behaviour. A model that accepts a script without crashing has demonstrated encoding coverage, not reliable understanding of the language written in it.

References

1. Joshi et al. The State and Fate of Linguistic Diversity and Inclusion in the NLP World. ACL, 2020.
2. Conneau et al. Unsupervised Cross-lingual Representation Learning at Scale. ACL, 2020.