How Do Multilingual Language Models Really Scale? DeepMind’s ATLAS Has an Answer
As large language models expand beyond English to support hundreds of languages, a fundamental question has remained unanswered: how do multilingual models actually scale? Google DeepMind’s new research framework, called ATLAS, takes a major step toward resolving that uncertainty by introducing scaling laws designed specifically for multilingual AI systems.
Until now, most scaling laws—the formulas used to predict how performance improves as models grow—have been derived from English-only or single-language training. These laws have guided the development of modern AI, but they fail to account for the complex interactions that emerge when many languages are trained together. ATLAS challenges the assumption that multilingual scaling is simply monolingual scaling multiplied by more data.
Why Multilingual Scaling Is Different
Training a multilingual language model is not just about adding more text. Languages differ in scripts, grammar, vocabulary, and available data. When multiple languages share a single model, they compete for capacity, but they can also help one another through shared linguistic structure.
This creates a tension between positive transfer, where one language improves performance in another, and interference, where adding languages reduces overall efficiency. Existing scaling laws do not capture this trade-off, leaving researchers to rely largely on intuition and trial-and-error.
ATLAS addresses this gap by explicitly modeling how languages interact as models grow in size and as training data becomes more diverse.
A Large and Controlled Experimental Setup
To build ATLAS, DeepMind researchers conducted 774 controlled training experiments, using models ranging from 10 million to 8 billion parameters. The training data spanned over 400 languages, with performance evaluated across 48 carefully selected target languages.
This systematic approach allowed the researchers to observe how changes in model size, data volume, and language mixture affected performance. Instead of treating multilingual training as a black box, ATLAS breaks it down into measurable components.
The result is a set of scaling laws that better predict how multilingual models behave—especially as the number of supported languages increases.
The Cross-Lingual Transfer Matrix
At the heart of ATLAS is a cross-lingual transfer matrix, a tool that quantifies how training on one language affects performance in another. This matrix reveals that language interactions are highly structured rather than random.
Languages with shared scripts or common linguistic roots tend to exhibit strong positive transfer. Scandinavian languages, for example, benefit significantly from being trained together. Similarly, Malay and Indonesian form a high-transfer pair due to their close linguistic relationship.
In contrast, languages that are structurally distant or poorly represented in training data show weaker transfer and are more susceptible to interference.
Not All Transfer Is Equal
One of ATLAS’s most important findings is that cross-lingual transfer is asymmetric. Training on a high-resource language may significantly help a low-resource one, but the reverse is often not true.
Languages like English, French, and Spanish emerge as particularly strong contributors. Their large and diverse datasets allow models to learn general representations that transfer broadly across languages. However, adding smaller languages rarely improves performance in these dominant ones.
This insight challenges the assumption that multilingual training benefits all languages equally and highlights the importance of carefully selecting language mixtures.
Efficiency and Capacity Trade-Offs
ATLAS also formalizes the efficiency costs of multilingual training. As more languages are added, a model’s capacity is spread thinner unless parameter count and data scale increase accordingly. Without careful balancing, performance gains can plateau or even decline.
By quantifying these trade-offs, ATLAS gives researchers a way to predict when adding a new language will be beneficial and when it may require scaling the model further to avoid negative effects.
This has practical implications for building multilingual systems that aim to support underrepresented languages without sacrificing overall quality.
Q&A: Understanding ATLAS and Multilingual Scaling
Q: Why don’t existing scaling laws work well for multilingual models?
A: Most existing scaling laws assume a single language and do not account for cross-lingual interactions. Multilingual models introduce transfer and interference effects that fundamentally change how performance scales.
Q: What is cross-lingual transfer in simple terms?
A: Cross-lingual transfer occurs when learning one language helps a model perform better in another. This often happens when languages share scripts, grammar, or vocabulary.
Q: Does adding more languages always improve a model?
A: No. ATLAS shows that adding languages can help or hurt performance depending on linguistic similarity, data balance, and model capacity.
Q: Why are languages like English so influential in multilingual models?
A: High-resource languages provide large, diverse datasets that help models learn general language patterns, which can transfer to many other languages.
Q: How does ATLAS help AI developers?
A: ATLAS provides predictive tools for deciding how large a model should be, how much data is needed, and which language combinations are most efficient.
A New Lens on Global AI
ATLAS reframes multilingual AI as a distinct scaling problem rather than a simple extension of English-centric models. By making language interactions measurable, it offers a roadmap for building more efficient, inclusive, and globally capable AI systems.
As language models continue to expand their reach, frameworks like ATLAS may prove essential in ensuring that scaling up also means scaling wisely.
