Every claim below comes from a published paper or an industry report, and we grade each one for the quality of its evidence.
Lead finding · RCT, N=52, pre-registered
Passive AI use erodes comprehension.
Developers randomized to AI assistance scored 17 percentage points lower (roughly 50% vs 67%) on a post-task comprehension quiz. Within the AI group, the paper pins the loss on passive, let-the-model-drive use.
Same 27-point quiz · Anthropic RCT, N=52 · % correct, after the lesson-17pp
Active / unassisted learner67%
AI-assisted learnerarm average, all use patterns50%
This is not an anti-AI result: the same study's question-first use patterns matched or beat unassisted learning. The problem is not the tool. It is what passive use quietly stops your team from building.
Passive AI use erodes understanding quietly. Retrieval practice is the best-established counter in learning science, and we run it as a daily habit in Slack: one sharp question, honest verdicts, thirty seconds. On Team, a weekly recap thread replays what the week's questions covered, with no scores attached.
Each number links to its finding below, caveat included.
01The problem
What passive AI use does to understanding.
This is not an anti-AI page. It shows why a team shipping fast with AI still needs retrieval practice. Five findings, each with its grade, its caveat, and its paper.
Finding 012026
-17pp
comprehension, AI-assisted vs unassisted learners
RCTN=52, pre-registered
Share this finding
17 points lower comprehension
Developers randomized to AI assistance scored 17 percentage points lower (roughly 50% vs 67%) on a post-task comprehension quiz. Within the AI group, the paper pins the loss on passive, let-the-model-drive use.
A 4.15-point gap on a 27-point quiz - roughly 50% vs 67% - N=52 (Cohen's d = 0.738, p = 0.01)
Caveat. arXiv preprint, pre-registered. This is not an anti-AI result: the same study's question-first use patterns matched or beat unassisted learning (small clusters, observational). N=52 on a single quiz: read the drop as directional, not a fixed number. Which figure is the headline, stated once: our -17pp is the distance between the arm means Anthropic publishes, 67% and 50%. The paper's own 4.15-point difference on the 27-point quiz is 15.4 points, and the gap between the two figures is the rounding of those arm means.
With AI on the test, g = 0.76. Without it, g = -0.06.
A preregistered meta-analysis pooled the programming-learning studies. Overall, AI-assisted learners gained nothing reliable. The moderator analysis says why. While the assistant stayed available during the test, AI-assisted learners looked far better; once it was removed, the advantage vanished. The gains live in the tool, not the learner.
10-study learning arm, N=1,069: pooled learning effect g = 0.14 (not significant); AI available during the test g = 0.76; AI removed g = -0.06
Caveat. arXiv preprint. The pooled learning effect is null (g = 0.14, CI -0.18 to 0.47, 10 studies, N=1,069); the availability split is a moderator analysis inside that arm. Read it as no gain once AI is removed, not as decline. The often-quoted 23 studies are the whole meta-analysis: 14 of them sit in the productivity pool, 10 in this learning arm, one in both.
With AI, each developer writes far more code. Over the same window, the refactor rate fell and duplicated blocks climbed. Output got faster; the codebase got worse.
623M changes analyzed; block duplication per million changed lines rose from 40.3 (2023) to 73.0 (2026 year to date), while moved code - their refactoring proxy - fell from 13% of changed lines to 3.8%
Caveat. Industry repo-mining report, not peer-reviewed. Methodology and raw data are public. Correlational, not a controlled trial. We quote the report's underlying series (40.3 to 73.0) rather than a bare multiplier. GitClear's earlier 2025 report measured a different window on a different metric (211M lines, 2020 to 2024, commits carrying a 5+-line duplicate block: 0.70% to 6.66%); the two series are not interchangeable.
An EEG study found weaker brain engagement in heavy LLM users, and most failed to accurately quote work they had produced minutes earlier. This is the proposed mechanism behind the skill-atrophy headlines.
83% of LLM users failed to accurately quote their own essay minutes later (vs 11% in each control group)
Caveat. Preprint, N=54, essay task not coding; a preprint methods comment urges conservative reads. We think the mechanism generalizes - that call is ours; the specific neural numbers may not.
They expected to be 24% faster. The first trial measured them 19% slower.
Experienced developers on their own repos felt faster with AI while the trial measured them slower. The gap between felt and measured speed is the durable finding, not the exact number.
Caveat. Unreviewed field RCT from a research nonprofit (METR). Their larger follow-up measured -4% (likely range -15% to +9%) on 47 newly recruited devs, while the 10 returning originals measured -18%; METR itself flags selection effects and is redesigning the study. That leaves the size, and even the direction, unsettled, so we lead with the perception finding, not the number.
MeasuredFig. 01 - the perception gap, as an instrumentthree readings from one trial · the needle tells them in order
An instrument gauge from 30% slower to 30% faster. The needle reads three values in order: developers forecast being 24% faster, believed afterward they had been 20% faster, and the trial measured them 19% slower. Ghost needles mark the two beliefs; the live needle rests on the measured value.
Effect sizes on this page: Hedges' g and Cohen's d both measure a gap in standard deviations (0.2 small, 0.5 medium, 0.8 large). METR 2025 field RCT, 16 experienced open-source developers on their own repos: pre-task forecast +24%, post-task self-belief +20%, measured outcome -19%. METR's larger 2026 follow-up measured -4% (likely range -15% to +9%) and flags selection effects, so the size and even the direction stay unsettled; the durable finding is the gap between felt and measured speed, not the exact number.
MeasuredFig. 02 - what you lose, and what wins it back% correct, after the lesson
The problem-17pp
Same 27-point quiz · Anthropic RCT, N=52
Active / unassisted learner67%
AI-assisted learnerarm average, all use patterns50%
050100
The fix · different study+13pp
Same chapter exams · Roediger et al. 2011 classroom experiments
Brief retrieval quizzes94%
Restudy / no quizzesstandard teaching81%
050100
Two different studies, each with its own control group: unassisted learners on the left, no-quiz classes on the right. The two halves are not one continuous scale, so read each side against its own baseline. The left pair is a single 27-point quiz (Shen & Tamkin, Anthropic, N=52, pre-registered RCT); the paper prints a 4.15-point gap, and the arm means shown are Anthropic's published rounding of its Figure 6. The right pair comes from classroom experiments with within-class controls, where brief retrieval quizzes scored 94% vs 81% on the same multiple-choice chapter exams (Roediger, Agarwal, McDaniel & McDermott 2011). Same direction both times: passive use erodes, retrieval rebuilds. The left half's direction now has meta-analytic backing: across a preregistered 10-study learning arm, learners looked far better while AI stayed available during the test (g = 0.76) and showed no remaining gain once it was removed (g = -0.06) (Maier et al. 2026, Exhibit 02). The faint vertical line marks 50%.
Model · illustrativeFig. 03 - cognitive debt, drawn over timea model, not a measurement
A schematic line chart over 24 months of AI-era work. A dashed line, how it feels, rises steadily. A solid line, measured understanding under passive use, erodes. A gold line, with retrieval practice, compounds above both. The widening gap between felt and measured is hatched and labeled the debt.
A model, not a measurement: the curve shapes are illustrative. The pinned anchors are measured: Shen & Tamkin, Anthropic 2026 (-17pp comprehension, pre-registered RCT, N=52) · METR 2025 (developers felt +20% faster while the trial measured them slower) · Adesope et al. 2017 (retrieval vs re-reading, g = 0.51 across 272 comparisons). Hover to scrub the timeline after the draw.
02The widening gap
Whoever is still learning loses the most.
9 of 10
struggling novices reached a working solution anyway
2018, without AI: 11 of 31 could not finish, and every one knew it
Lab observationN=21, eye tracking
Watched closely in the lab, a novice class split in two. Students who already had a plan used AI to build it faster. Strugglers let generated code carry them past problems they never understood, and most finished believing otherwise. The researchers' phrase is an illusion of competence. In the same lab's pre-AI study, failure was at least legible: the students who could not finish knew it. Observed in think-aloud and eye-tracking data, not measured by a learning-outcome test.
feel they should produce more, because AI makes it possible
Syntax poll, N=3,593 · self-selected, self-reported · often or daily
SurveyN=3,593, self-report
In a Syntax poll of 3,593 developers, 71% said they feel they should be producing more because AI makes it possible, and 57% said they enjoy coding less than before. Grade it before you quote it: self-selected audience, everything self-reported, so it measures how these developers say it feels, not the population. The counterweight stays printed: DORA's 2025 survey of 4,867 technology professionals measured burnout directly and found no relationship with AI adoption.
Retrieval practice is one of the most-replicated findings in learning science.
Pulling an answer out of memory, instead of re-reading it, is what makes it stick (Roediger & Karpicke, 2006). First, play the mechanism. Then the size of the effect, and a precedent that should worry you.
Model · illustrativeFig. 04 - the forgetting curve, interruptedplayable · click the chart to place a retrieval on any day
An interactive chart of memory retention over 30 days. A dashed baseline falls steeply after a single exposure and levels off at a low floor. An expanding retrieval schedule plays in automatically; placing retrievals resets the gold retention curve to full, slows each later decay and raises the floor. A readout reports the modeled day-30 retention for the schedule you build.
Model calibrated to the classic savings anchors - roughly a third retained after a day, roughly a fifth after a month (Ebbinghaus 1885; modern replication: Murre & Dros 2015). Savings measure relearning effort, not recall, so the axis stays a modeled retention index. Expanding schedule after the spacing literature (Cepeda et al. 2006). The dashed line is one exposure and no retrieval; the gold line is the same memory with retrievals on the days you choose. Illustrative model, not a measurement.
g = 0.51
effect size vs re-reading
Retrieval vs re-reading
Pulling it from memory beats re-reading by a reliable, well-replicated margin.
Adesope et al. 2017 · 272 comparisons across 118 articles
d = 0.40
effect size on novel problems
Carries to new problems
The skill still shows up on problems they have never seen.
Pan & Rickard 2018 · 122 experiments, N=10,382
2 of 10
study techniques, ranked
Rated the best use of your time
Of ten study techniques Dunlosky's review graded, only two reached the top tier. This is one.
Dunlosky et al. 2013 · retrieval + spacing rated high
Effect-size scale (Hedges' g / Cohen's d): 0.2 small · 0.5 medium · 0.8 large. The transfer caveat is in §06, The limits.
The cost of offloading is not new. Short-form video ran this experiment at scale; GPS navigation ran it on spatial memory. If the pattern holds, AI-assisted work is running it again, on your team. That is a precedent, not a demonstrated mechanism - the two columns carry different grades of evidence, and the tag under each cell says which.
Then
The precedent · offloaded habits
Now
The hypothesis · offloading thinking to AI
More passive scrolling, measurably weaker attention and self-control
That was the gain. Now the loss: in the Anthropic trial, how much lower did learners score when AI assisted the work?
Pick one. Even a wrong guess makes the answer stick.
Speed, you can borrow.
Understanding, you have to earn.
AI lends you the first. Practice is how you build the second.
Speed, you can borrow.
Understanding, you have to earn.
AI lends you the first. Practice is how you build the second.
05The dose
Why even 30 seconds helps.
It compounds. No single study tested this exact dose. The bounds below set its shape: spaced, low-stakes, brief.
Spacing beats cramming.
Cepeda et al., 2006 · 317 experiments
Across 839 separate measurements, short, frequent sessions beat one long one. A daily rhythm re-surfaces each topic over days and weeks, squarely inside the range Cepeda's spacing meta-analysis found best.
It works on professionals, not just students.
Larsen, Butler & Roediger, 2009 · medical RCT
Six months later, on a delayed exam, residents in pediatrics and the ER remembered the topics they'd been quizzed and spaced on better than the ones they only restudied. Retrieval is how expert knowledge stays live.
Short prompts respect working-memory limits.
Zheng, Sun & Liu, npj Science of Learning, 2023 · N=30
Retrieval has a cost. In a small lab study, Zheng and colleagues found the benefit only when the task left working-memory capacity to spare. One short question keeps the rep inside those limits by design, and still adds up to hours of practice a year.
Owning the limits is the point. The biggest one first:
The engineering-specific trial is still owed. No one has published one that we can find. Including us.
Transfer is smaller than recall.Retrieval helps you carry a skill to new problems, but the lift is smaller there (about 0.40) than on new formats of the same material (0.58). Both numbers come from Pan & Rickard's meta-analysis. An advantage, not a free lunch.
Spacing has a sweet spot, not a magic interval.The best gap depends on how long you need to remember (Cepeda et al.).
The 30-second dose is our design choice.No study tested exactly one 30-second question a day. It is assembled from the literature's bounds - how far apart to space it (Cepeda), how brief and low-stakes a quiz can stay (Roediger & Agarwal's classroom program), what retrieval costs in working memory (Zheng), and whether it holds up for working professionals (Larsen). Not a study result.
GitClear is repo-mining, not a trial.The +81% figure comes from GitClear's own published series, and GitClear is an industry vendor. It's not peer-reviewed, and it's correlational, not causal. We cite it because the method and raw data are public, and we call it what it is: a report, not a trial.
The meta-analytic learning arm is a preprint, and its pooled effect is null.Maier et al.'s learning arm pools 10 studies (N=1,069) to g = 0.14, not significant. What we quote is its availability moderator: g = 0.76 with AI still on the test, g = -0.06 without. That is no gain once AI is removed, not proof of decline. The split is a moderator analysis, not a randomized comparison of its own.
The EEG study is an essay task.It's a preprint, N=54, and the task was essay writing, not code. We think the atrophy mechanism carries to engineering work, though that call is ours, not the paper's. The specific neural numbers may not travel.
METR's follow-up weakened the number.The larger cohort (N=57) moved the estimate from -19% to -4% (likely range: -15% to +9%). The follow-up leaves the size, and even the direction, unsettled, so we lead with the perception finding, not the number.
The professional evidence is medical.The professional evidence comes from residents on a 6-month exam, not software engineers. We have not seen a coding-task RCT for retrieval practice, and we do not claim one.
And in fairness to the tools
What we are not claiming about AI.
We sell practice, not fear. Three findings that cut against an anti-AI reading, kept on the page on purpose.
AI delivers real speed.
Three randomized field experiments on GitHub Copilot, run inside Microsoft, Accenture and a Fortune 100, across 4,867 developers, measured a 26% lift in completed pull requests among the developers who took it up (SE 10.3%, roughly a 6% to 46% interval; granting access to everyone measured +6.24%). Execution accelerates. That part is real.
DORA 2025 finds AI amplifies what is already there: strong teams get stronger, weak teams get exposed. Tooling is a multiplier, not a substitute for craft.
A Harvard RCT shows an AI tutor can match or beat an actively taught class. The tutor was built to keep students working rather than watching; we read that active ingredient as retrieval - that call is ours, not the paper's.
A daily practice habit built from real answers: one sharp question a day in the channel, honest verdicts on every answer, and a weekly recap of what the team practiced.
A habit, not another platform
The course platform is already paid for. Nobody remembers to log in.
One sharp question a day in the channel the team already lives in. Thirty seconds, in the flow of work. Nothing new to open, nothing to chase.
Honest verdicts, instantly
Real understanding has become hard to tell from AI doing the thinking.
Every answer gets an instant verdict and an explanation of exactly why, so people practice on the topics they actually need.
A record each engineer owns
New hires ship before they actually understand the system.
Daily retrieval builds each engineer a practice record of what they’ve actually retained, week by week from their first answer. It belongs to them; what you notice is sharper questions in review.
A team habit you can rally
Lunch-and-learns are episodic. Whoever showed up retained whatever they retained.
One rep a day lands in the channel the whole team already reads, and live games make practice something the team does together, not homework.
08Versus the alternatives
What you're really weighing this against.
The real decision isn't app-vs-app. It's a choice between habits: do nothing, ban the tools, or lean on the code review and lunch-and-learns you already run.
The alternativeWhat it missesHow Atomic Reps differs
The code review process itself
Existing engineering process· common default
Reactive: the weak spot shows up inside a PR after AI already shipped it, and it burns senior reviewers’ attention.
A leading indicator: daily practice surfaces weak spots before review has to absorb them.
Internal lunch-and-learns / brown bags
Calendar time
Episodic. No measurement. Whoever showed up retained whatever they retained.
Continuous. One rep a day in the channel, so the habit compounds instead of evaporating.
Status quo / do nothing
Zero spend· common default
You find out the worst way: a production incident, or an onboarding that quietly stalls.
A daily habit that keeps the fundamentals warm before an incident tests them.
Banning Copilot / Cursor / Claude Code
Policy + enforcement· common default
Punitive. Retention risk. Kills the productivity gains. Misreads the problem: the tools are fine; the practice atrophy isn’t.
Keeps the team on the tools, and protects the human judgment around them.
Weighing budget lines instead - a catalog seat, a senior hire, an assessment tool? That comparison lives on the pricing page.
LAST ONE: NAME THE METHOD
What's the learning method Atomic Reps runs, 30 seconds a day, in Slack, on your team?
Pick one. Even a wrong guess makes the answer stick.
The Retrieval is this page in instalments: one study worth knowing about, one idea worth a name, and one question you answer from memory in about thirty seconds. Everyone starts at issue one, so nothing in it assumes you read the last one.
