A tool can improve the output of a task while weakening the ability that the task was meant to develop.
That tension appears in a recent study of 26,811 Chinese secondary-school students. After adopting generative AI, their homework scores rose by 18% and completion time fell by 30%. Six months later, their results in monthly exams taken without AI had dropped by 20%.
The easy conclusion is that AI harms learning. The useful conclusion is more specific: when we optimise the visible output, we may unknowingly remove the cognitive process that made the task valuable.
What the study actually measured
David Strömberg, Victor Lei, and Yanhui Wu use 30 months of data from students in grades 7 to 12 in one county in central China. The dataset combines homework scores and completion times across nine subjects with monthly closed-book exams and high-school and university entrance exams.
Students adopted AI at different times, which the researchers use in a difference-in-differences design. This was not a randomised trial: students chose whether to use the tools. The analysis tests whether earlier trends were comparable and attempts to isolate the change associated with adoption, but it cannot provide the same causal control as an experiment.
The results are nevertheless difficult to dismiss:
- homework scores rose by 18%;
- average completion time fell by 30%;
- monthly exam scores dropped by 20% within six months;
- entrance-exam scores fell by 18% to 24%, with the full effect appearing after about two years.
The study covers one county and one education system. It is a discussion paper that has not yet completed journal peer review. Its percentages should not be applied mechanically to every country, age group, or form of AI use.
Productivity and learning are not the same metric
In many activities, a better output in less time is productivity. In learning, it may mean the opposite.
An exercise does not exist merely to produce a correct answer. It forces the learner to retrieve knowledge, connect concepts, detect mistakes, try a strategy, and revise it. Some learning happens in that friction. If a tool supplies the answer before the student goes through the process, it improves the deliverable while removing the training.
The problem is that homework grades used to work as an indirect signal of knowledge. AI weakens that relationship. A document can improve even while the person submitting it knows less.
The same break occurs beyond the classroom. A finished report, translation, or program no longer proves that its author can explain the decisions, find a defect, or redo the work when the context changes.
The important qualification is time spent
The most useful finding is not the average but where the loss is concentrated.
The authors identify a pattern consistent with homework outsourcing among roughly 80% of AI users: exceptionally short completion times combined with high scores. Learning losses are concentrated in this group. Students who adopted AI but continued to spend about as much time on homework as non-users experienced small losses.
Time alone does not prove that someone is learning. Here, however, it acts as a signal of how much effort the tool displaced. The meaningful divide may not be simply between using and not using AI, but between using it to work through a problem and using it to avoid the problem.
A chatbot and a tutor do not produce the same behaviour
Another study, published in PNAS in 2025, tests the mechanism with an experimental design.
Nearly one thousand secondary-school mathematics students in Turkey were split between a control group, a group using an interface similar to a general chatbot, and a third group using a GPT-4 tutor designed to provide hints and restrict direct answers.
With the tools available, both assisted groups performed better on practice problems. When access was removed, students who had used the unrestricted chatbot scored 17% below the control group. The guided tutor largely mitigated that loss.
The underlying technology was similar; the interaction was not. An interface optimised to answer makes delegation cheap. An interface optimised to teach can require an attempt, ask an intermediate question, point to an error, or ration help.
This does not make every AI tutor an effective education product. It does show that treating “AI use” as a single intervention hides the decisive variable: the behaviour the product encourages.
Assessment must measure capability, not only output
Banning every tool may temporarily protect an assessment format, but it does not answer which capabilities matter in a world where AI exists. Allowing every use is no better: it confuses obtaining an answer with mastering the knowledge required to judge it.
A stronger design separates at least three goals:
- Unaided capability. What a person must be able to remember, reason through, or execute without the tool.
- Assisted capability. How they prompt, verify, combine, and correct work produced with AI.
- Transfer. Whether practice enables them to solve a new problem, explain a decision, or catch a plausible but incorrect answer.
That changes both the activity and the assessment. It may require process traces, oral defences, annotated iterations, supervised exercises, or new tasks that apply the same principle in another context. The point is not to monitor every click. It is to stop treating the final document as sufficient evidence of learning.
Design AI so it does not remove the useful work
Before introducing AI into a learning process, five questions are worth answering:
- Which effort should be reduced because it is mechanical, and which should remain because it builds a skill?
- Does the tool provide answers or guide decisions?
- Does it require an independent attempt before revealing a solution?
- Can we measure what the person can do afterwards without assistance?
- Do the incentives reward learning, or merely delivery?
AI can be a capable tutor, a patient simulator, and a source of immediate feedback. It can also become a layer that produces correct outputs while hiding deterioration in the underlying capability.
The difference does not lie in the model alone. It lies in the design of the product, the activity, and the assessment. If we measure only the final output, AI will appear to improve the system long before we discover what it has stopped training.
