In February 2019, OpenAI introduced GPT-2 by showing, among other things, whether it could read a story and infer the missing word at the end. Today that seems like a small assignment. Back then, it helped investigate a fundamental capability: whether the model was following what happened in the text, beyond stringing together words that often appear together.
In September 2026, the Grok 4.7 announcement compares models on tests that ask them to design electrical circuits, identify legal risks in contracts and reason about medical cases. Understanding the instructions is only one part of those assignments: they also require applying knowledge to a case and meeting its requirements. The expectation expressed in the announcement approaches that of delegating part of a professional’s work.
Between those dates, models changed, but so did what we consider a demonstration of their capabilities. Labs make that visible in their launches: they choose common tests—the benchmarks—publish results and compare them with other systems. Scores tell us how a model performs. The kind of test being highlighted tells us what we are starting to expect from it.
Following the story from the first GPT models lets us trace that shift: how advances gave us reasons to ask for more ambitious tasks and find new ways to check whether the model could handle them.
2018–2020: is it following what it reads?
The first GPT models, from 2018, were evaluated on tasks such as answering questions about a text or recognising whether two sentences contradict each other. For a system learning from large amounts of language, getting those exercises right helped establish whether what it had learned enabled it to interpret the text it was given.
At GPT-2’s launch in February 2019, one of the highlighted tests was LAMBADA. It collected passages whose final word had to be completed. The difficulty was that looking at the immediately preceding words was not enough: the model had to use the story’s context.
Julia explains to a man that she has already focused on his friend and that he only needs to press the shutter button.
“He nodded, threw away his cigarette and picked up the ___.”
Expected answer: camera. The context lets us infer which object he picks up. The word in the original is camera.
Many things could follow “picked up the…”. To arrive at “camera”, the model has to connect the shutter button, the friend about to be photographed and the object passing from one person to another. A one-word answer helped test whether the model could connect details scattered throughout the text.
That was why the question mattered. Answers like this gave us reason to think that learning to continue text could produce something more useful than plausible sentences: an ability to follow situations and answer questions about them. The test provided a signal of comprehension on a specific task, without settling what it means for a machine to “understand”.
GPT-2 answered 63.24 % of LAMBADA exercises correctly. In May 2020, GPT-3 reached 86.4 % when given a few solved examples before answering. The conditions differed, and that assistance was part of the novelty: examples in the prompt could show the model which task we wanted, without retraining it for each activity.
Between GPT-2 and GPT-3, LAMBADA still asked for the missing word: “camera”, in our example. What also changed was how the test was presented: GPT-3 received demonstrations of what to do. Alongside connecting the story’s clues, another capability was being tested: learning what task was being requested from a few examples, without specific training. That gave us reasons to try different assignments, such as writing code.
2021–2023: can it use what it knows?
In 2021, OpenAI introduced the original Codex, a family of models that generated code from instructions. It should be distinguished from today’s Codex: OpenAI reuses the name for its agent-based coding product, to which tasks on software projects can be delegated.
The technical report for that Codex introduced HumanEval: 164 exercises describing an operation that the model had to implement as a function. A function is a piece of code that takes some data and produces a result. The evaluation ran it with different inputs to check whether it did what was requested.
Write a function that takes a list of numbers and a maximum distance. It must indicate whether any two numbers differ by less than that distance.
For example: with the list [1.0, 2.0, 3.0] and a distance of 0.5, does any pair meet the condition?
No: the function should return False. The smallest difference between two numbers in that list is 1. The assignment is to write code that also handles other cases, rather than just answer this example.
With one attempt per problem, Codex solved 28.8 % of the exercises. That was a substantial limitation, but the expectation it opened up was substantial too: describe an operation in words and receive code that performs it.
The change from LAMBADA was visible in the expected response. There, inferring “camera” from the story was enough; HumanEval asked for a function that solved different cases when executed. It tested the ability to turn an instruction into a correct procedure. The assignment remained tightly bounded: the model was told exactly what to program.
On 14 March 2023, the GPT-4 presentation moved the comparison into territory familiar to many more people. According to OpenAI, it had passed a simulated US bar exam with a score around the top 10 % of test takers. GPT-3.5 had scored around the bottom 10 %.
Using that exam as a headline result proposed a different reference point: assessing the model with questions used to assess a person’s preparation. Its technical report also included MMLU, a collection of multiple-choice questions across 57 subjects. GPT-4 answered 86.4 % correctly. One of the questions was:
Which policy best describes a supply-side fiscal policy?
Expected answer: C. Fiscal policy uses taxes and government spending. This question asks about a measure that encourages production: reducing taxes on research can support the development of new technologies.
Source: MMLU, high-school macroeconomics, development set, row 4.
The same model answered questions about economics, history, physics, medicine and law. The aim was to demonstrate a breadth of knowledge that would allow us to treat it as an informed interlocutor across many fields.
Comparing it with LAMBADA makes the change clear: the clues for “camera” were in the story; answering the MMLU question required bringing in economic knowledge that the question did not explain. Interpreting a text was joined by the requirement to use knowledge from many subjects. GPT-4’s strong exam scores therefore opened the next question: how far could it go when knowing the subject was only the first step towards solving the problem?
2023–2024: what happens when the problem gets harder?
In December 2023, Google introduced Gemini Ultra, highlighting 90 % accuracy on MMLU. According to the announcement, it exceeded the human-expert reference score on that test. The system generated several lines of reasoning before choosing an answer: its score also depended on how it was allowed to tackle the exam.
In March 2024, Claude 3 again compared itself with GPT-4 and Gemini on knowledge, coding and mathematics. Common tests made differences between vendors visible, but started to create a problem for subsequent launches: some left little room for improvement.
Claude 3 Opus answered 95 % of GSM8K correctly, a collection of grade-school mathematics problems. Here is one exercise:
Janet's ducks lay 16 eggs a day. She eats three for breakfast and uses four to bake muffins. She sells the rest for 2 dollars each.
How much revenue does she make each day?
Expected answer: 18 dollars. (16 − 3 − 4) × 2 = 18. The challenge is to follow the steps without losing track of what is being counted.
Source: first question in the GSM8K test set, shortened.
Solving it requires understanding what happens and chaining several operations together. But with 95 % accuracy, the next generation could hardly distinguish itself on that test. Learning to solve almost all those exercises did not tell us how far it would get with a less routine problem. Finding that out meant changing the questions.
The launch of o1 in September 2024 made this especially visible. OpenAI presented it as a model trained to devote more computation to developing a solution before answering. Its highlighted tests included AIME, a mathematics competition exam in the US Olympiad selection process, and GPQA, with specialised questions in physics, chemistry and biology.
The expectation was that the model could sustain more demanding reasoning: find an approach and work towards a solution, even when recognising a concept or chaining simple calculations together was insufficient. On AIME, OpenAI compared its previous GPT-4o with o1-preview, the preliminary version it had just released, and with the o1 still in development. These were their accuracy rates with one attempt per problem:
Scale from 0 to 100 %. Data from Appendix A of OpenAI's announcement, allowing each model the maximum computation configured for that evaluation. The version available at announcement was o1-preview.
The previous generation failed the vast majority of those problems; o1 solved almost three out of four under the study’s conditions. That search for demanding problems would keep bringing evaluation closer to the boundary between solving exercises and producing mathematical research.
From GSM8K to AIME, the concrete change was the difficulty of the reasoning: from chaining together the calculations for Janet’s eggs to finding a solution path for a competition problem. The aim was to measure how far the model could develop a solution when the required operations were less obvious. We were still, however, providing a problem statement and expecting an answer. The next step would be to ask it to act on files and applications to achieve a result.
2024–2025: can it make progress without instructions for every step?
On 22 October 2024, just weeks after o1, Anthropic introduced a new version of Claude 3.5 Sonnet. One of its highlighted coding results was solving 49.0 % of SWE-bench Verified. This test starts from bugs and change requests in real software projects: it gives the system the issue description and project files, and asks for a fix.
A bug report explains that Django, a web development tool, accepts usernames ending with a newline, although it should reject them.
The task is to locate the code that validates usernames and fix it so it rejects that case.
The reference solution changes two lines of code. Although the change is small, finding it requires navigating the project files. The evaluation checks that the bug is fixed and that the other behaviours it tests still work.
Source: SWE-bench Verified, test set, row 31, and OpenAI's audit of 23 February 2026.
The example revisits the promise of Codex in 2021, but changes how much work we leave to the system. HumanEval described a specific function. Here it must work out where the bug is and what needs changing before writing the solution. Two lines of code can be the end of a task requiring considerably more searching, reading and checking.
To do that, a model needs access to tools that let it inspect files, edit them and run tests. The program surrounding it returns the outcome of each action so it can decide how to proceed. That combination is what we call an agent. Evaluation starts measuring whether the system can make progress through a sequence of actions.
The same Sonnet announcement introduced computer use through screenshots, clicks and keyboard input. On OSWorld, a test of tasks carried out inside applications, it correctly completed 14.9 % of assignments in that mode; with more steps, 22.0 %. It still failed often. What was new was the demonstration’s objective: do the task inside the application, rather than merely explain how to do it.
In November 2025, Google highlighted in the Gemini 3 launch a test that pushed this continuity further: Vending-Bench 2. The assignment no longer ended with fixing a bug. It required keeping a business running inside a simulation.
The model must run a simulated business: buy stock, deal with suppliers and set prices, keeping operations going for a simulated year.
The test measures how much money the business finishes with. A good result requires coordinating purchases, sales and restocking throughout the simulation. Each decision affects the possibilities for those that follow.
Source: Vending-Bench 2, by Andon Labs, highlighted in the Gemini 3 presentation. The year takes place in a simulation.
SWE-bench Verified ended with delivering a fix; Vending-Bench 2 required continuing to make decisions for a simulated year. A stock purchase shaped sales and subsequent restocking. Completing a task with tools was joined by the ability to maintain an objective, remember what happened and adjust decisions over time. That was the increase in autonomy the new test sought to measure, within a simulation.
In December, GPT-5.2 opened its results table with GDPval, an evaluation of assignments across 44 occupations. It compared presentations, spreadsheets and documents produced by the model with human work on the same assignments. According to OpenAI, GPT-5.2 Thinking’s outputs were preferred or rated equally good in 70.9 % of comparisons. The figure combines wins and ties on those tasks; it does not measure what proportion of a profession can be replaced.
By the end of 2025, benchmarks had expanded both the assignment and how it was judged. HumanEval asked for a function and checked it through execution tests; SWE-bench added navigating and fixing a project; GDPval compared the quality of documents and other outputs with human work. The capability being sought was taking responsibility for more steps and producing a result that met the assignment. The Grok 4.7 announcement would let us examine that expectation against concrete engineering and legal criteria.
2026: can it handle a professional’s assignment?
In 2023, presenting GPT-4’s bar-exam score invited a comparison with someone who had studied law. Three years later, a test in the Grok 4.7 announcement, published on 21 September 2026, asks it to analyse documents for a company acquisition and prepare a report on its legal risks. The comparison approaches the work a lawyer might receive at their desk.
The same announcement includes tests of electrical engineering, clinical reasoning, coding and office work. Bringing them together in a model presentation proposes viewing it as a candidate for tasks from different professions. Each row in this table corresponds to a test; its columns compare the models’ results on it.
| Test | Grok 4.7 xHigh | Grok 4.6 High | GPT-5.6 Sol Max | Fable 5.1 Max |
|---|---|---|---|---|
| Coding CursorBench 4.0 | 46.3 % | 40.4 % | 41.7 % | 51.8 % |
| Coding DeepSWE v1.1 | 71.0 %* | 65.2 % | 72.7 % | 70.0 % |
| Electrical engineering EEBench | 64.0 % | 53.0 % | 39.4 % | 56.4 % |
| Office work AA-Briefcase v1.1 | 1657 | 1546 | 1487 | 1678 |
| Terminal work Terminal-Bench 4.0 | 37.6 % | 20.3 % | 37.3 % | 57.9 % |
| Legal work Harvey Legal Agent | 19.6 % | 15.8 % | 2.5 % | 6.7 % |
| Clinical reasoning HealthBench Professional | 56.7 % | 48.5 % | 60.5 % | 62.1 % |
EEBench, the engineering test created by atopile, shows what that change entails. Here is one assignment:
An electricity meter needs to stay on for 20 milliseconds after a power cut to save its reading. The assignment is to choose capacitors—components that store energy—that can keep it running for that short interval.
The model delivers a file describing the circuit. The evaluation simulates its operation and scores both compliance with requirements and component cost. The circuit is not manufactured in this test.
Simplified example from the EEBench methodology. The diagram shows the assignment's objective, not an electrical solution.
The answer is judged by what the circuit does when its power supply is lost. Knowledge about capacitors matters insofar as it allows that requirement to be met. The test requires moving from explaining a principle to applying it in a verifiable design.
Harvey’s legal evaluation applies that demand to a contract review:
A buyer is considering acquiring the fictional company Crestview Software Solutions. The agent receives eight relevant contracts mixed with other documents.
It must report on the acquisition's legal risks and how to address them.
A licence loses its exclusivity if the company changes ownership. The buyer might be relying on an exclusive right of use that would cease to be exclusive after the acquisition. The report must identify that clause and explain its consequences.
GPT-4’s exam tested legal knowledge. This assignment requires finding a clause among the documents, relating it to the acquisition and recommending action. The review requires all mandatory criteria to be met for a report to pass. The 19.6 % announced by xAI for Grok 4.7 indicates how many tasks cleared that bar: professional requirements are part of the test, even though the model fails many tasks.
The change from GPT-4’s exams is visible in what counts as success. MMLU checked whether the model chose the correct answer; EEBench checks whether the circuit keeps the meter on for the required time; Harvey checks whether the report identifies risks and satisfies the review criteria. Reading, knowledge and decisions must be combined into a verifiable result. The capability being measured is how much of a professional assignment the system can complete while meeting its requirements.
What we expect, and what still needs to be demonstrated
Taken together, these launches show how the role we propose for models has expanded. We started by checking whether they followed a story; moved to asking about their knowledge and posing demanding problems; then gave them tools and assignments requiring several organised steps. Each advance gave us reasons to try a more ambitious task.
That progression does not mean comprehension, knowledge or reasoning are solved stages. The tests coexist and errors persist. Nor does a table chosen by a vendor prove that its model can take on a profession. It does show what kind of capability the vendor considers relevant to try to demonstrate publicly.
The missing word in LAMBADA helped investigate whether the model connected clues in a story. A report on a company acquisition asks whether it can use that comprehension, alongside its knowledge and tools, to detect an issue that affects a decision. The task changes, and so does the criterion by which we judge the response.
That is why the history also lies in the questions we choose to introduce each generation. We have gradually dared to ask for a larger part of the work. Scores must tell us how much of that expectation the model meets; an announcement alone does not turn it into a reliable capability.
Sources
- OpenAI: the first GPT, June 2018 and GPT-2, February 2019. Early tasks and the comprehension results table.
- Paperno and colleagues: LAMBADA and Brown and colleagues: GPT-3, 2020. The camera example and demonstrations in the prompt.
- OpenAI: Codex, August 2021 and Chen and colleagues: Codex and HumanEval report. Generating and checking functions.
- OpenAI: GPT-4 presentation, March 2023 and its technical report. Exams, MMLU and HumanEval.
- Hendrycks and colleagues: MMLU and Cobbe and colleagues: GSM8K. Tasks appearing in the 2023 and 2024 tables.
- Google: Gemini, December 2023 and Anthropic: Claude 3, March 2024. Competition on shared tests.
- OpenAI: o1, September 2024. Saturation of some tests and AIME results; the chart uses Appendix A.
- Anthropic: updated Claude 3.5 Sonnet and computer use, October 2024. SWE-bench Verified and OSWorld.
- Jimenez and colleagues: SWE-bench and SWE-bench Verified dataset. The shift from isolated functions to real issues.
- Google: Gemini 3, November 2025, Andon Labs: Vending-Bench 2 and OpenAI: GPT-5.2, December 2025. Agents, prolonged tasks and professional work.
- xAI: Grok 4.7, September 2026, atopile: EEBench methodology and Harvey: Legal Agent Benchmark. The recent table and its engineering and legal examples.
