Cognitive Debt. What the research actually shows.
Tab, tab, accept. What did you just learn?
Atomic Reps sells a retrieval-practice tool, so we have a stake in this reading. Every claim links to its source, and the caveat rides next to its number.
- 01In the only pre-registered randomized coding trial so far, developers who used an AI assistant while learning a new library scored 17 percentage points lower on a comprehension quiz taken right after - roughly 50% vs 67%. One study, 52 people, not yet peer-reviewed.
- 02The sharpest result is not the drop. Inside the AI group, people who asked the model questions learned about as much as the no-AI group, sometimes more. People who let it write the code scored 24-39%.
- 03Perception is unreliable here. In METR's field trial, experienced developers believed AI made them about 20% faster while the measurement said slower. The effect size moved between cohorts; the perception gap replicated.
- 04At codebase level, duplicated code is rising and refactoring is falling across hundreds of millions of changed lines (GitClear commit mining - vendor data, correlational).
- 05The mechanism evidence (an EEG study of essay writers) points at weaker engagement and worse recall with LLM use. It is a contested preprint, and it is not about code.
Fifty-two developers, one new library, one quiz.
Anthropic researchers recruited 52 working developers - a year or more of Python, weekly coding, no prior contact with the Trio async library - and paid them to learn it. Half got a chat assistant (GPT-4o) that could see their code and could write the full solution if asked. Half got documentation and web search, with a no-AI pledge. Everyone then took the same 14-question, 27-point quiz, with no AI allowed.
The AI group scored 4.15 points lower on that 27-point quiz. The paper calls it a 17% score difference, or two grade points; the group averages work out to roughly 50% against 67%. The effect size is large for a study this size (Cohen's d = 0.738, p = 0.01), and the gap survives controlling for warm-up speed. Task time barely moved (p = 0.391): the AI group was not faster, it just understood less of what it had shipped.
The authors attribute the control group's edge to something specific: they hit errors and had to resolve them on their own. The biggest per-topic gap backs that up - debugging questions, where the AI group trailed by about 21 points. That breakdown was exploratory, not pre-registered, so hold it more loosely than the headline.
One library, one 35-minute task, a quiz minutes later, crowdworkers rather than colleagues, and a chat interface. The authors flag that last one in an uncomfortable direction - agentic tools require less human participation than chat, so they read this design as a lower bound for offloading. Nobody has measured the months-long version.
Effect sizes on this page: Cohen's d measures a gap in standard deviations (0.2 small, 0.5 medium, 0.8 large). Redrawn from Figure 6, Shen & Tamkin 2026 (arXiv 2601.20245). The paper prints the gap (4.15 points, d = 0.738, p = 0.01); the arm means shown are Anthropic's published rounding of that figure. Error bars omitted; the paper's Figure 6 carries 95% confidence intervals.
A fair critique. Moore's published review of the trial raises it: the control arm had no code completion and got syntax hints only after pilots showed timeouts, so part of the gap may be tools-vs-no-tools rather than AI-vs-no-AI. The quiz was also a single sitting, which is not how education research usually measures learning.
The drop is not evenly distributed.
The trial's most useful result is hiding inside the AI arm. From screen recordings, the authors coded six ways people actually used the assistant, and the quiz split cleanly in two.
Three patterns scored 24-39%: handing the task to the model, drifting into it over the session, and pasting errors back until they went away. Three patterns scored 65-86%: asking conceptual questions and then writing the code yourself, asking for code with an explanation attached, and generating code first but interrogating it after. The high group sits at or above the no-AI average; the delegators sit far below it.
Hold this loosely: the clusters contain two to seven people each, they were coded after the fact, and nobody was randomized into a usage style. It is a pattern worth acting on, not a proven mechanism. The authors' own summary is careful too - less independent thinking, more cognitive offloading in the low-scoring group.
Vertical line: no-AI arm average (67%)
Redrawn from Figure 11 and Section 6, Shen & Tamkin 2026. The authors coded the clusters from screen recordings after the fact (24 of 26 AI-arm participants), so this is observational: the study randomized AI access, not usage style. The vertical line marks the no-AI arm's average from Figure 6.
Developers cannot feel this happening.
METR ran the field version: 16 experienced open-source maintainers, working real issues on their own large repos, randomized task by task into AI-allowed and AI-forbidden. Before starting, they forecast AI would make them 24% faster. Afterward, they estimated it had made them about 20% faster. The measurement said 19% slower.
The follow-up made the number messier and the lesson clearer. With 57 developers and 800+ tasks, the 47 newly recruited developers measured -4% (likely range -15% to +9%), while the 10 returning originals measured -18%. METR's own reading is not that AI became neutral; it is that the design got contaminated - developers increasingly refused to work without AI, and 30-50% held their AI-suited tasks out of the trial. They are redesigning the study.
So we do not cite METR for a speed number. We cite it for the finding that replicated in both rounds: developers' sense of their own AI speedup did not match the clock. If your team's confidence is the only instrument you have, you are not measuring.
The repos are keeping a ledger too.
If individuals understand less of what they ship, codebases should show it. GitClear mines commit history at scale - 211 million changed lines across 2020-2024 - and tracks what kind of change each line is. Two trends move together across the AI adoption years: moved code (their refactoring proxy) fell from about 25% of changed lines in 2021 to under 10% in 2024, and 2024 was the first year they measured copy-pasted lines outnumbering moved lines.
The duplication numbers are the loud ones. Commits containing a duplicated block of five or more lines: 0.70% in 2020, 0.45% in 2022, 1.80% in 2023, 6.66% in 2024. Their 2026 follow-up (623M analyzed changes, 2023-2026) has the refactor share down to 3.8% and duplication up another 81% since 2023.
Grade this honestly: it is a vendor's report, not peer-reviewed, and correlational - nothing here proves AI caused the shift. The methodology and data definitions are public, which is why we cite it at all. We quote the underlying table rather than the report's own shorthand multipliers, because those multipliers do not share a single stated baseline.
Redrawn from the duplicated-block table, GitClear AI Copilot Code Quality 2025 report (211M changed lines; roughly two-thirds private repos, one-third open source). Commit mining is correlational: the years of AI adoption are also years of everything else.
GitClear, AI Copilot Code Quality report (2025)GitClear, The Maintainability Gap (2026)
Why assisted work sticks worse: the working hypothesis.
Why would assisted output stick worse? The closest thing to mechanism evidence is an MIT Media Lab EEG study - 54 people writing essays over four months, split between an LLM group, a search-engine group, and an unaided group. Minutes after submitting, 83% of the LLM group could not accurately quote their own essay. In each of the other two groups, 11% failed the same test.
The EEG side points the same direction: the authors report brain connectivity scaling down with the amount of external support - strongest networks unaided, intermediate with search, weakest with the LLM. When the researchers switched long-time LLM users back to writing unaided in a final session, the recall deficit persisted.
It is a preprint; the authors say peer review has only started. Stankovic and colleagues' comment, itself an unpublished preprint, disputes parts of the EEG methodology and urges a more conservative read. The task is essay writing, not code. And the authors themselves asked the press to stop saying it shows AI damages brains - it does not. We carry it as a mechanism hypothesis that fits the coding data, which is our call, not the paper's. Their coding-specific follow-up is reportedly in analysis; we will link it when it lands.
Kosmyna, N. et al., arXiv (MIT Media Lab) (2025)Stankovic, M., Hirche, E., Kollatzsch, S. & Doetsch, J. N., arXiv preprint (comment on arXiv:2506.08872) (2026)
We have run the passive experiment before.
We have run the passive-consumption experiment before, at population scale, with short-form video. That literature is further along, and its shape should look familiar.
Nguyen and colleagues' 2025 meta-analysis, pooling 98,299 people across 71 studies, ties heavier short-form video use to weaker attention (r = -.38), weaker inhibitory control (r = -.41), and worse overall cognitive performance (r = -.34). Every one of those is correlational - the authors say plainly that direction is unproven, and the strongest links show up on problematic-use scales rather than raw hours. One popular claim does not survive the source: the effects were no worse for the young; age moderation came out non-significant.
The AI-side parallels are earlier-stage and mostly self-report. Lee and colleagues at Microsoft Research found knowledge workers who trust the output report checking it less. Gerlich's UK survey ties heavier AI use to lower critical-thinking scores through offloading, and it is the weakest study here: self-report, contested constructs, a published correction. Fan and colleagues ran the randomized essay experiment behind the term 'metacognitive laziness' and found ChatGPT users engaged less in monitoring their own work while gaining no more knowledge.
The strongest randomized addition so far is Liu and colleagues' 2026 package: across experiments totaling 1,222 people on math-reasoning and reading tasks, AI assistance lifted immediate performance, and afterwards people performed significantly worse without it and gave up sooner, with the effect appearing after roughly ten minutes of use. A preprint, and not about code.
Students see it in themselves: 62% use AI for homework, 67% believe it harms their thinking. Those are beliefs, not measurements.
None of this is proof about coding. It is a prior: when a tool absorbs the effortful part of a task, the skill under that effort tends to fade, and the people it happens to tend not to notice. The full row-by-row comparison, with each source graded, sits on the hub; the receipts are in the methods table below.
More passive scrolling, measurably weaker attention and self-control
meta-analysis, N=98,299 · Nguyen, L. et al. 2025
Knowledge workers who accept AI output passively report putting less effort into debugging and understanding it
survey, peer-reviewed, N=319 · Lee, H.-P. et al. 2025
The more short-form video people consume, the worse their cognitive performance
meta-analysis, N=98,299 · Nguyen, L. et al. 2025
The more people offload thinking to AI, the lower their critical-thinking scores
survey + interviews, N=666 · Gerlich, M. 2025
Heavier rapid-reward scrolling, weaker sustained attention
meta-analysis, N=98,299 · Nguyen, L. et al. 2025
Leaning on AI cuts the deep processing that learning needs. Fan and colleagues named it "metacognitive laziness."
peer-reviewed study · Fan, Y. et al. 2025
The link holds at every age - the meta-analysis found no age group immune
meta-analysis, N=98,299 · Nguyen, L. et al. 2025
Students see it coming: 62% now use AI for homework, and 67% say using AI for schoolwork harms critical thinking.
youth panel, self-report · RAND American Youth Panel 2026
Habitual GPS use, poorer self-guided spatial memory - the internal map fades where the tool navigates
observational, correlational · Dahmani, L. & Bohbot, V. D. 2020
With the assistant available at test time, performance appears; with it removed, no gain remains (g = 0.76 vs -0.06)
meta-analytic moderator, preprint · Maier et al. 2026
Users don't perceive the decline
Developers still believed they were 20% faster after being measured 19% slower
METR 2025
What has moved since.
The meta-analysis arrived, and it turns on one condition.
A preregistered meta-analysis pooled the programming-learning studies: 10 studies, 1,069 participants, overall learning effect null (g = 0.14, not significant). The moderator is the finding. While the AI stayed available during the test, learners looked far better (g = 0.76); once it was removed, no gain remained (g = -0.06). No gain once AI leaves, not decline - and a preprint. It does not pool the trial above: its literature search closed on 5 December 2025, and that trial posted in January 2026. Two separate readings, pointing the same way.
Maier et al., arXiv preprint (preregistered meta-analysis) (2026)
Interaction design decides the outcome - now causal, in math.
Nearly a thousand high-school students, randomized: unrestricted GPT answers boosted assisted practice by about 48%, then cost about 17% against control once the AI was removed. A guardrailed tutor built on the same model kept the gains and largely eliminated the harm. Mathematics, not code - but it is the causal version of the usage-pattern split above: the harm follows the interaction design, not the tool.
Bastani, H., Bastani, O. et al., field experiment, high-school mathematics (2025)
Offloading's memory cost has an off-switch.
Three lab experiments, 172 participants each: offloading a task lifted immediate performance and cost later memory - and an explicit intention to learn during forced offloading almost eliminated that cost. Intent-to-learn, not retrieval practice, and not programming. It is the mechanism-level case for deliberately entering a learning mode when a tool can do the task for you.
The perception gap reached working engineers - as an experience finding.
A six-month longitudinal survey followed 95 matched professional engineers. 84% still reported improved productivity, while the share who rated at least one developer-experience dimension negatively rose from 14% to 27%. Even so, the only statistically significant aggregate shift was feedback loops improving (33% better vs 18% worse). 82% reported less time writing code. Self-report, continuing AI users, no control group: it measures experience, not competence, and it sits beside METR's finding for that reason.
Vella & Blincoe, arXiv preprint (longitudinal survey) (2026)
Verification decay, measured over six months.
A three-wave longitudinal pilot tracked the same academics as daily AI use rose from 52% to 96%. Performance on the hardest problems fell wave over wave, while confidence held - the belief-performance gap widened to about 35 points. Small convenience sample, preprint; a distinct mechanism (checking ability, not recall) worth watching.
The stakes moved to the labor market - with the caveat that has to travel with them.
Stanford's Canaries payroll study (a working paper on ADP records, summarized by the 2026 AI Index) reports that by June 2026, employment for 22-25-year-olds in the most AI-exposed occupations sat 19% below where it would be had it kept pace with less-exposed peers. Experienced workers show no comparable gap, and the mechanism is reduced hiring, not firing. The honest pair rides along: the same authors find no economy-wide displacement, and unemployment rose across every AI-exposure quintile from 2022 to early 2025 - most for the least-exposed workers. Both research teams call these descriptive facts, not causal estimates. Not a comprehension study, but the roles thinning out are the ones where skill formation used to happen.
The split replicated in a classroom, watched through an eye tracker.
Peer-reviewed ICER 2024 study of 21 novice programmers: students already poised to succeed used GenAI to accelerate, strugglers let it carry them, and nine of ten strugglers shipped working code the researchers judge they could not have produced alone. They finished, in the authors' words, with an illusion of competence. Observational, no learning-outcome measure, and the same accelerate-vs-struggle shape as the usage-pattern clusters above. The full breakdown, including the three new AI-specific metacognitive difficulties, is in the widening-gap analysis.
Prather, J., Reeves, B., Leinonen, J. et al., ICER '24 (2024)
Seniors already believe the gap exists.
In a 77-country survey of 1,569 developers, 16% of seniors said junior engineers fully understand the AI-generated code they submit. A perception survey, nothing more - but it is the industry reading its own pulse.
Every paper, graded.
The grade column is ours. It says how much weight the design can carry, not whether we like the result.
| Paper | N | Design | Task | Grade |
|---|---|---|---|---|
| Shen & Tamkin 2026 | 52 | Pre-registered RCT | Learn Python Trio, 35 min; 27-point quiz | RCT, preprint |
| METR 2025 | 16 devs, 246 issues | Field RCT, own repos | Real OSS issues, ~2h each | Field RCT, unreviewed |
| METR follow-up 2026 | 57 devs, 800+ tasks | Field RCT extension | Real issues, wider repo pool | Selection effects, per METR |
| GitClear 2025 / 2026 | 211M / 623M lines | Commit mining | Industry + OSS repos | Vendor report, correlational |
| Kosmyna et al. 2025 | 54 (18 crossover) | EEG lab study | Essay writing, 4 months | Preprint, methods contested |
| Nguyen et al. 2025 | 98,299 / 71 studies | Meta-analysis | Short-form video use | Peer-reviewed, correlational |
| Gerlich 2025 | 666 | Survey + interviews | AI use vs critical thinking | Self-report, contested |
| Lee et al. 2025 | 319 | Survey, 936 task examples | Knowledge work with GenAI | Peer-reviewed, self-report |
| Fan et al. 2025 | 117 | Randomized experiment | Essay with ChatGPT vs supports | Peer-reviewed, small N |
| Liu et al. 2026 | 1,222 | Randomized experiments | Math + reading with AI, then unaided | Preprint, non-programming |
| RAND Youth Panel 2026 | 1,214 | Panel survey | Self-reported AI use + beliefs | Perception only |
| Maier et al. 2026 | 10 studies, 1,069 (learning arm) | Preregistered meta-analysis | Programming learning, with vs without AI | Preprint; moderator analysis |
| Bastani et al. | ~1,000 students | Randomized field experiment | High-school math with GPT tutor | Causal; different domain |
| Grinschgl et al. 2021 | 3 × 172 | Lab experiments | Offloading with vs without learning intent | Peer-reviewed, non-programming |
| Vella & Blincoe 2026 | 95 matched (158 wave 1) | Two-wave longitudinal survey | Professional engineers, 6 months | Self-report, no control group |
Where this page is weakest.
The headline trial is one hour long.
One library, one 35-minute task, a quiz in the same sitting, 52 people, not yet peer-reviewed. Read the 17% as a direction. Nobody has measured the year-long version of this.
The usage-pattern split is observational.
The six clusters hold 2-7 people each and were coded after the fact. The study randomized AI access, not usage style, so 'ask questions and you keep learning' is a supported pattern, not a proven prescription.
The control group was not tool-neutral.
Moore's review notes the no-AI arm lacked code completion and needed syntax hints added mid-pilot, so some of the gap may be tooling, not AI. The comparison the trial licenses is narrower than the headline suggests.
METR's numbers are unsettled by METR's own account.
The follow-up split by cohort (-4% new recruits, -18% returning originals) and the team attributes the movement to selection effects severe enough that they are redesigning the study. We cite the perception gap, which replicated; we do not cite a speed number.
The mechanism study is contested and not about code.
The EEG work is a preprint with a preprint methods comment against it, the task was essays, and the authors explicitly reject brain-damage framings. Carrying it as the mechanism behind the coding results is our editorial call.
The codebase data is vendor commit mining.
GitClear sells code-quality tooling, the report is not peer-reviewed, and the trends are correlational. We cite its underlying tables, not its shorthand multipliers, because the definitions and data are public.
The precedent literature is correlational end to end.
The short-form-video meta-analysis cannot establish direction, the AI-side surveys are self-report, and the meta-analysis's own moderator test does not support the age claim popular in coverage.
The trial's split says the difference is retrieval: the people who kept learning were the ones pulling understanding out of their own heads. Atomic Reps runs that as a 30-second daily question in Slack. The full case, dose reasoning, and its limits are on the evidence hub.
Read the full case on the evidence hubThe receipts.
The trial and its critique
Field and codebase evidence
Mechanism
The precedent and its parallels
Newer work
- Maier et al., arXiv preprint (preregistered meta-analysis) (2026)
- Bastani, H., Bastani, O. et al., field experiment, high-school mathematics (2025)
- Grinschgl, S., Papenmeier, F. & Meyerhoff, H. S., Quarterly Journal of Experimental Psychology 74(9) (2021)
- Vella & Blincoe, arXiv preprint (longitudinal survey) (2026)
- Three-wave longitudinal study, arXiv preprint (2026)
- Brynjolfsson, E., Chandar, B. & Chen, R., Stanford Digital Economy Lab working paper (ADP payroll records) (2026)
- Prather, J., Reeves, B., Leinonen, J. et al., ICER '24 (2024)
- BairesDev, Dev Barometer Q2 (2026)
The letter
One study at a time, from issue one.
The Retrieval is this page in instalments: one study worth knowing about, one idea worth a name, and one question you answer from memory in about thirty seconds. Everyone starts at issue one, so nothing in it assumes you read the last one.
An email address. Unsubscribe from any issue.
AtomicReps30″Built on the evidence
The tools are fine. The practice atrophy is the problem worth fixing.
Free for one channel.