Scaling laws and the drivers of LLM improvement (2017–2026)

On 3 September 2026, OpenAI introduced GPT-6 Astra. The announcement describes it as the result of “years of research and big bets across pre-training, reinforcement learning, and alignment.” Aidan Clark, the company’s vice president of research, added a striking detail in statements to Fortune: it was the first time they had pretrained a model on more than 100,000 GPUs, at their Stargate site in Texas. It was “by far” the largest training run in the company’s history.

The detail is surprising if you remember what was being said less than two years earlier. In December 2024, Ilya Sutskever, co-founder of OpenAI, stated at NeurIPS, the main machine-learning conference, that “pre-training as we know it will unquestionably end.” His argument was that compute kept growing, but data did not: “we have but one internet”. At the end of February 2025, OpenAI launched GPT-4.5, the largest model it had released until then. It improved at conversation and knowledge, but it fell behind smaller models that reasoned before answering, and the company retired it from its API in July 2025.

Had scale run out or not? Both things fit together if we stop talking about “scale” in the singular. Language models have not improved thanks to a single rule, but to several levers, each with its own way of turning more resources into better results. When one became more expensive, the effort shifted to another; and the earlier ones were not abandoned, they added up.

Some of those levers follow a scaling law. The name is a little misleading: it is not a law of physics, but a regularity that has been observed again and again. If more is invested in something specific, such as parameters, data or compute, the result improves so predictably that it can be calculated before spending the money. Other levers do not have a law of their own: what they do is change what is scaled or how much it costs. This article goes through the seven that, in our view, best explain the path from GPT-3 to today, and an eighth that is still in the labs.

Our own compilation. Each bar starts with the work or the release that made that lever widespread; the dashed line marks an earlier precedent, such as the 2017 mixture-of-experts proposal or the 2022 chain of thought. The dates are explained in each section.

2020: the first law turns progress into a budget

To understand the first law, we need three words. A language model learns by reading texts and trying to guess what comes next, fragment by fragment. Those fragments are called tokens: a short word is usually one token and a long one, several.

What the model adjusts while it learns are its parameters, billions of internal numbers that decide what it answers. That is why the size of a model is measured in parameters.

And to know how much it has learned, its loss is measured: put simply, how wrong it is when guessing the next token in texts it has not seen. The lower, the better.

Two pieces from 2017 and 2018 allowed this recipe to grow. The first was the Transformer, the architecture Google presented in 2017. It processes all the words of a text at once, instead of one after another, and that makes it possible to spread training across thousands of chips. The second is that the task needs no labels: any text can be used to learn, because the correct answer is always the fragment that comes next. OpenAI brought the two together in GPT, in 2018, and in GPT-2, in 2019.

In January 2020, an OpenAI team led by Jared Kaplan trained models of very different sizes and compared their loss in the paper Scaling Laws for Neural Language Models. It found an unusual regularity. When the model was given more parameters, more data or more compute, the loss always fell in the same way, and some of those trends held over ranges of more than ten million times. Design details, on the other hand, mattered much less than had been thought.

The practical consequence is easy to state: each time compute is multiplied by ten, the loss falls by the same proportion, around 11 % according to their measurements. This is what is called a power law. Improving is possible, but each step costs ten times more than the previous one. The advantage is that it can be predicted.

Diagram built with the exponent that Kaplan and his team measured for compute (loss proportional to compute raised to the power of −0.05). Each point is ten times more compute than the previous one. Real curves also include a part of the loss that cannot be eliminated.

A few months later, in May 2020, OpenAI published GPT-3, with 175 billion parameters, more than a hundred times as many as GPT-2. As well as writing better, it did something small models barely managed: solving a new task from a few examples written in the request itself, without retraining it. Since then, the compute used to train frontier models has grown steadily. Epoch AI, which compiles this data, estimates that it grew four to five times per year between 2010 and 2024.

Training compute of notable AI systems, in petaFLOP (10¹⁵ operations), on a logarithmic scale. Epoch AI data processed by Our World in Data.

Before 2020, improving a language model was above all a design problem: trying architectures and training tricks in the hope that one of them would work. Afterwards it also became a budget problem. It was possible to calculate, before spending the money, how much the loss of a model ten times larger would fall. That predictability has a limit worth keeping in mind: the law predicts the loss, not which specific tasks the model will be able to do. A smooth improvement in loss can translate into sudden jumps on a task, as we explain in “Why AI seems to improve suddenly at some tasks”.

2022: a large model also needs to read more

Kaplan’s work left a recommendation: with a fixed budget, it is best to spend it mainly on making the model larger, even if it reads relatively little. The large models of 2020 and 2021 were built that way. In March 2022, DeepMind published a study with more than 400 models that reached a different conclusion, known by the name of the model that demonstrated it: Chinchilla. To make good use of the budget, size and data must grow together: if the model doubles in size, it should read twice as much. By that criterion, the large models of the time had read too little.

To demonstrate it, they trained Chinchilla, a 70-billion-parameter model, with the same compute as Gopher, their previous 280-billion-parameter model. The difference is that Chinchilla read 1.4 trillion tokens, almost five times as many. With a model four times smaller, it outperformed Gopher on almost every test; on MMLU, an exam of questions across 57 subjects, it scored 67.5 %, more than seven points higher. The rule of thumb that became popular was about 20 tokens per parameter.

Model	Year	Parameters	Training tokens	Tokens per parameter
GPT-3	2020	175 billion	300 billion	≈ 2
Gopher	2021	280 billion	300 billion	≈ 1
Chinchilla	2022	70 billion	1.4 trillion	≈ 20
Llama 3 8B	2024	8 billion	more than 15 trillion	≈ 1,900

A trillion is a million million. Data from the GPT-3 and Chinchilla papers and from the Llama 3 announcement.

The last row shows that the Chinchilla rule did not become dogma either. In April 2024, Meta explained that the optimal amount of data for its 8-billion-parameter model, according to that criterion, was around 200 billion tokens. It trained it on more than 15 trillion, about 75 times more, and the model kept improving. The explanation is economic. Chinchilla optimises the cost of training, which is paid once. But a model is used millions of times, and a small model that has read a lot is cheaper to run than a large one of the same quality.

Reading more has an obvious limit: the text available. Epoch AI estimates that public text written by people amounts to around 300 trillion tokens and that, at the current pace, models would exhaust it at some point between 2026 and 2032. That is the concern Sutskever expressed when he said that “data is the fossil fuel of AI.” The responses have been to filter better, reuse data, generate synthetic data with other models and, above all, look for improvements that do not depend only on reading more.

2022: knowing is not the same as helping

A pretrained model continues texts. If you write it a request, it does not necessarily interpret it as an instruction: it may continue it as if it were the beginning of a list. The InstructGPT paper, published by OpenAI in 2022, includes an example of this problem.

GPT-3 vs InstructGPT · 2022 The same request, two behaviours

"Write a short story about a frog who travels back in time to ancient Greece in French."

GPT-3: responds with more requests of the same kind. "Write a story about a child who wants to know everything about the games of the gods…", "Write a story about a young man who has an adventure in a distant era…".

InstructGPT: writes the story. "The frog was lost and exhausted. She had only one goal in mind: to travel in time and reach ancient Greece…".

What changes: the first model treats the request as the start of a list of assignments; the second, as an assignment it must carry out. Both models have 175 billion parameters.

Source: figure 8 of Ouyang et al. (2022). Example translated from French and trimmed.

To achieve this, OpenAI used the technique of reinforcement learning from human preferences, known as RLHF. First, several people wrote example answers to real requests. Then they compared different answers from the model and ranked them from best to worst. Those comparisons were used to train a second model, the reward model, which learns to predict which answer a person would prefer. Finally, the main model was tuned with reinforcement learning: it generates answers, the reward model scores them and training makes the ones that get good marks more likely.

The result challenged the idea that everything depended on size. The people who evaluated the answers preferred those of the 1.3-billion-parameter InstructGPT to those of the 175-billion-parameter GPT-3, a model more than a hundred times larger. And that tuning cost little: according to OpenAI, it used less than 2 % of the compute and data of pretraining. The company itself described it as a way to “unlock” capabilities GPT-3 already had. ChatGPT, launched in November 2022, applied this recipe to a general-purpose product.

This lever does not add knowledge; it changes the goal: from continuing texts to helping whoever asks. The 2 % figure is worth remembering, because three years later the share of compute devoted to this post-pretraining stage would be very different.

2023: many parameters, few at a time

In a conventional model, usually called dense, all the parameters work on every token. If the model is ten times larger, each word costs about ten times more to compute.

Mixture of experts, known by its English acronym MoE, breaks that relationship. It divides parts of the model into several blocks, called experts, and adds a router that chooses, for each token, which few experts process it. The others do nothing with that token.

Diagram of an MoE layer with the configuration of Mixtral 8x7B: the router chooses two of eight experts for each token and combines their outputs. The other six compute nothing for that token, although the next one may go to other experts.

The idea dates from 2017, when Noam Shazeer and other Google researchers applied it to networks of up to 137 billion parameters. In language models it became popular with Mixtral 8x7B, which Mistral released in December 2023: it had 46.7 billion parameters, but each token used only 12.9 billion.

The name “expert” is a little confusing. There is no medicine expert and another for history: the router decides its own division of labour during training, and in Mixtral that division seemed to follow the syntax of the text more than its topic.

A year later, DeepSeek-V3 showed how far the savings could go. It has 671 billion parameters, of which each token uses 37 billion, and according to its technical report the final training run cost about $5.6 million in GPU rental. The figure does not include the earlier research, but it was very low for a model that competed with the best closed models, and the MoE design was one of the reasons.

Are all models MoE? No, but in 2026 almost all large open-weight models, the ones anyone can download, are. The table brings together the most recent ones alongside three earlier ones for comparison.

Model	Date	Architecture	Total parameters	Active per token
Mixtral 8x7B	Dec 2023	MoE	46.7 billion	12.9 billion
DeepSeek-V3	Dec 2024	MoE	671 billion	37 billion
Llama 4 Maverick	Apr 2025	MoE	400 billion	17 billion
GLM-5	Feb 2026	MoE	744 billion	40 billion
Qwen3.5-397B-A17B	Feb 2026	MoE	397 billion	17 billion
Gemma 4 31B	Apr 2026	dense	30.7 billion	all
Gemma 4 26B A4B	Apr 2026	MoE	25.2 billion	3.8 billion
DeepSeek-V4-Pro	Apr 2026	MoE	1.6 trillion	49 billion
Kimi K3	Jul 2026	MoE	2.8 trillion	104 billion
Qwen3.8-27B	Aug 2026	dense	27 billion	all

Data from each model’s technical card, linked in the sources. It only includes open-weight models, whose architecture is public.

The table reveals a pattern. From a few hundred billion parameters upwards, in 2026 practically everything is MoE: Meta adopted the design with Llama 4, its first MoE family, and Kimi K3 reaches 2.8 trillion parameters while using less than 4 % of them for each token. Dense models are still present at medium sizes, where the savings make up less for the complexity; Google even releases both variants in the same family, Gemma 4. For the closed models of OpenAI, Anthropic or Google, nobody knows for certain, because they do not publish their architecture. That GPT-4 was an MoE is often repeated, but OpenAI has never confirmed it.

There is an important nuance: an MoE saves compute, but not memory. All the experts have to be loaded, because any token may need any of them. DeepSeek-V4-Pro computes each word like a model of about 50 billion parameters, but it takes up as much memory as its 1.6 trillion. We explain it in more detail in our article on local inference.

Mixture of experts is not a new scaling law, but a change in price. Before, having ten times more parameters meant paying about ten times more for each word generated. With MoE, a model’s capacity to store knowledge and the amount of compute per word no longer always go together.

2024: spending compute when answering

So far, all the levers act on training. Once trained, the model devotes more or less the same effort to each token, whether it is asked for the capital of France or an olympiad problem. The next lever consists of letting it spend more compute on the questions that need it.

The first step was almost a wording trick. In January 2022, a Google team published the work on chain of thought: if the request includes examples solved step by step, the model also writes the intermediate steps before giving the answer, and gets more right.

Chain of thought · 2022 Writing the steps before answering

The request includes a problem solved with its steps: Roger has 5 tennis balls and buys 2 cans of 3 balls each. "Roger started with 5 balls. 2 cans of 3 tennis balls each is 6 tennis balls. 5 + 6 = 11. The answer is 11."

"The cafeteria had 23 apples. If they used 20 to make lunch and bought 6 more, how many apples do they have?"

If the example only gives the final answer: the model answers 27.

If the example gives the steps: "The cafeteria had 23 apples originally. They used 20 to make lunch. So they had 23 - 20 = 3. They bought 6 more apples, so they have 3 + 6 = 9. The answer is 9."

Correct answer: 9. Writing the intermediate result gives the model something to lean on; without it, it tries to reach the answer in one leap and fails.

Source: figure 1 of Wei et al. (2022). The model is PaLM, from Google, with 540 billion parameters.

On GSM8K, a collection of primary-school maths problems, PaLM went from 17.9 % to 56.9 % correct answers with this change. But the effect only appeared in large models, of around 100 billion parameters; in small ones, writing steps did not help and even made the result worse. Chain of thought was, in itself, an effect of scale.

If writing some steps helps, does writing more help? Or attempting the problem several times and keeping the best answer? In August 2024, Snell and other researchers from Google and Berkeley compared two ways of spending the same compute: using a larger model or letting a small one think more. On the problems the small model already solved sometimes, thinking more could beat a model fourteen times larger. On the hardest ones, it could not: there, a better model was still needed.

A month later, on 12 September 2024, OpenAI introduced o1, the first commercial model designed to think for a while before answering. Its announcement included two widely cited charts. In both, accuracy on a mathematics exam rose steadily: in one, when more compute was given to training; in the other, when it was given more time to think about each answer. It was the same shape as the 2020 law, applied to a new axis. The figures for that jump are in the article on sigmoids.

In 2026, this axis is one more control in products. GPT-6 Astra is offered with several effort levels, and an open model such as the large Qwen3.8 includes a “reasoning effort” setting to choose between thinking a little, the normal amount or a lot, depending on the depth and cost desired. In the Astra announcement, Lovable’s CTO explains that more effort allows more iterations and more checks with tests in the browser.

Before 2022, a model’s quality was fixed when its training ended. Now part of it is decided with each question, depending on how much compute it is allowed to spend. The cost per answer stops being fixed and becomes a decision: a simple question does not deserve the same effort as a code audit.

2025: training on what can be checked

A model writing more steps does not guarantee that it thinks better: it can go round in circles without making progress. o1 made good use of that time because it had been trained for it with reinforcement learning. And that is where a problem appeared. In RLHF, the one giving the marks is a reward model that imitates people’s tastes, and it is an imperfect judge. If you train too hard against it, the model learns to please it instead of doing the task well, like a student who studies for the examiner and not for the subject. That is why that stage was kept short.

The way out was to train on problems that can be marked automatically, like an exam with an answer key: mathematics with a known answer or code that has to pass some tests. There the mark is reliable and training can go on for much longer.

In January 2025, DeepSeek published how it had done this with DeepSeek-R1. Its first version, R1-Zero, learned only from that kind of mark, without examples of reasoning written by people. The striking part is what happened during training: its answers grew longer on their own, from hundreds to thousands of reasoning tokens. Nobody asked it to think more; it learned to because thinking more got it more right answers. The authors even record the moment when an intermediate version interrupts itself with “Wait, wait. Wait. That’s an aha moment I can flag here” and revises its approach.

This reinforcement, moreover, scaled. When introducing o3, in April 2025, OpenAI said that the same rule held for it as for pretraining: more compute, better performance. In July 2025, xAI said it had trained Grok 4 with reinforcement “at pretraining scale” and with verifiable problems from many more fields than mathematics and programming. The comparison with 2022 sums up the change: InstructGPT devoted less than 2 % of pretraining compute to this stage; three years later, some labs devote a comparable amount to it.

It still had to be measured rigorously. In October 2025, a team from Meta and several universities published the first systematic study of how reinforcement scales, with more than 400,000 GPU hours. The improvement followed a regular curve, regular enough to predict a large training run from smaller ones. But there was a difference from the 2020 law: this curve ends up flattening. Each training recipe has its ceiling, and many adjustments only change how quickly it is reached, not its height.

With RLHF, the model learned to give answers people liked. With verifiable rewards, it learns to solve problems whose solution can be checked. The scarce resource shifts too: it is no longer only texts to read, but problems with a reliable way of knowing whether they have been solved correctly.

2025: from being able to call a tool to knowing when to use it

A language model, on its own, is bad at long calculations, does not know what has happened since its training and cannot act outside the conversation. Tools solve all three. In February 2023, Meta presented Toolformer, a model that learned to insert calls to a calculator, a search engine or a translator into its text. In June 2023, OpenAI added function calling to its API: the developer describes the available tools and the model responds with a structured request to use them.

That was an interface. The model could ask for a tool, but it had not learned to decide when it was worth doing so within a long task, or to change plan depending on what the tool returned. The change came from combining tools with reinforcement. In the o3 and o4-mini announcement, OpenAI explained that it had trained them to use tools through reinforcement learning, teaching them “not just how to use tools, but to reason about when to use them.” The announcement itself included an example of what that means.

o3 · April 2025 A question that requires chaining tools

"How will summer energy usage in California compare to last year?"

To answer, the model searches the web for public consumption data, writes Python code to build a forecast, generates a chart and explains the factors behind it. If a search does not return what it needs, it can run another.

What changes: nobody tells it which tool to use or in what order. Choosing them and chaining them is part of what the model has learned.

Source: example described in the o3 and o4-mini announcement, 16 April 2025.

Tools fit with the previous lever for a practical reason: they can also be used to check. A model that can run the code it writes knows whether it works before handing it over. In the same announcement, o4-mini solved 99.5 % of the problems on the 2025 AIME exam with access to a Python interpreter. OpenAI warned that this figure should not be compared with that of models without tools, but it illustrates how the model makes use of the resources it has.

In 2026, the tests that releases highlight no longer ask whether the model can use a calculator, but whether it can operate a computer. GPT-6 Astra scores 72.6 % on OSWorld 2.0, a test of real tasks on an operating system, spending about 40 minutes on average on each one. The environment, whether a computer, a browser or a code repository, has become part of what the model knows how to make use of. We tell how releases came to measure this kind of work in our history of benchmarks.

In the lab: thinking without writing

All the ways of thinking we have seen have something in common: the model reasons by writing. Each intermediate step becomes tokens, and each token has to be computed and paid for. There is also a less visible drawback. Internally, the model represents each step with thousands of numbers that can contain several possibilities at once. When it chooses a word, it keeps only one and discards the rest.

From this comes a line of research known as reasoning in latent space. Latent space is that internal language of numbers the model works with before choosing each word. The idea is to let it think there, without going through text.

Illustrative diagram with the apples problem. At the top, the model turns each internal state into text before continuing. At the bottom, it passes the full state to the next step and only writes the answer.

The best-known proposals follow two paths:

Feeding the thought back in. In Coconut, presented by Meta in December 2024, the model does not write the next step: it passes its internal state to itself as if it were the next input. This way it can keep several paths open at once, like someone exploring a maze through several corridors, and it outperformed chain of thought on logic problems that require planning.
Going round more times internally. Instead of writing more, the model repeats the same block of layers as many times as it needs: it thinks in depth, not in length. The prototype by Geiping and other researchers, from February 2025, had 3.5 billion parameters and kept improving on reasoning tests up to spending the compute of a 50-billion-parameter model. In October 2025, the Ouro looped models took the idea into pretraining: with 1.4 and 2.6 billion parameters they performed like models of 4 and 8 billion. According to their authors, not because they knew more, but because they handled what they knew better.

In October 2026 these are still research models, small compared with frontier ones. No lab has confirmed that its leading models reason this way, although they do not publish their architecture either. That is why it does not appear in the chart at the beginning: it is a candidate lever, not an established one.

Its main obstacle is not technical. The written chain of thought is today one of the best windows for monitoring what a model does, because it makes it possible to detect when it considers breaking the rules. In July 2025, researchers from OpenAI, Google DeepMind, Anthropic and other institutions published a joint paper that described it as a “new and fragile opportunity” for AI safety and asked labs to take into account how their design decisions affect that window. In December, OpenAI published evaluations according to which monitoring the chain of thought is much more effective than monitoring only actions and outputs.

That window is already narrowing a little without any need for new architectures. A study from July 2026 found that thirteen advanced models improved by up to 13 points on synthetic reasoning tasks when writing filler tokens with no meaning at all: part of the computation happens without leaving a readable trace. And in the GPT-6 Astra announcement, OpenAI acknowledges that, in tests where it was asked to evade oversight, its written reasoning was harder to monitor than its predecessor’s, partly because it solves problems with fewer written steps.

Thinking without writing would be another way of spending compute when answering, probably cheaper per step. The price would be closing that window. Whether it becomes a lever will depend on how well it performs, but also on whether labs, and those who regulate them, accept that trade-off.

Has scale run out?

With this overview, we can go back to the question at the beginning. Those who argue that scale is running out have good arguments:

Each multiplication by ten costs more money and energy, and data centres take years to build.
Text written by people is finite.
GPT-4.5 showed that enlarging pretraining no longer gave, on its own, a noticeable jump on the tasks that mattered most.
The reinforcement law has a ceiling, and reinforcement works above all where there is an automatic way of checking the result.

All of them are true, but each one describes the limit of a specific lever, not of the whole. When pretraining became more expensive, compute at answer time and reinforcement took over.

Moreover, the levers rest on one another. The authors of DeepSeek-R1 tried applying reinforcement directly to a 32-billion-parameter model, and it worked worse than teaching it to imitate the reasoning of the large model. Their conclusion was that, to go further, more powerful base models and more reinforcement will be needed: reinforcement makes the most of what pretraining has built. GPT-6 Astra fits that reading. It is a return to the first lever, but on top of all the others: OpenAI’s largest pretraining run, followed by reinforcement, tools and effort levels.

One caveat remains that applies to all the laws in this article. They are observations, not guarantees. They measure loss or a score on a test, not usefulness in real work, and they can stop holding without warning, just as Kaplan’s recommendation on data stopped holding.

What to look at instead of size

For years, the number of parameters was the shortcut for comparing models. Today it says little on its own. A model can have 2.8 trillion parameters and use 104 billion per word; it may have read 75 times what a 2022 rule recommended; it may be worth much more at high effort than at low effort. The useful question is no longer how much a model has grown, but what has been scaled and how it has been checked.

That question has consequences for anyone using these models in a real process. Quality has partly become a decision for each task: how much effort, how much time and how much cost per answer each step deserves. And models improve first where the result can be verified. In an organisation, the processes that already have automatic checks, such as software tests, reconciliations or data validations, are the first in which an agent can come to work reliably. Building those checks is, to a large extent, the same work that labs do to train their models.

Each lever in this story had its moment of seeming to be the definitive one. None was, and none disappeared.

Not just bigger: what has made language models improve

2020: the first law turns progress into a budget

2022: a large model also needs to read more

2022: knowing is not the same as helping

2023: many parameters, few at a time

2024: spending compute when answering

2025: training on what can be checked

2025: from being able to call a tool to knowing when to use it

In the lab: thinking without writing

Has scale run out?

What to look at instead of size

Sources

2020: the first law turns progress into a budget

2022: a large model also needs to read more

2022: knowing is not the same as helping

2023: many parameters, few at a time

2024: spending compute when answering

2025: training on what can be checked

2025: from being able to call a tool to knowing when to use it

In the lab: thinking without writing

Has scale run out?

What to look at instead of size

Sources

Keep reading

Why AI seems to improve suddenly at some tasks

From completing a sentence to taking on a job: what we expect from AI

From the pelican to Blender: a history of AI told through its absurd tests

You choose what we can measure