Each request re-sends the whole conversation

A lesson from Working With AI: Context, Tools, Agents and Evals. Play it above, or read it through below.

• The API keeps no conversation. Each request re-sends the whole thread.

  The common read: The conversation is stored on their server and I am adding to it, so my twelfth message costs about what my first one did.

  The Messages API keeps no conversation between requests. The Messages API reference says the endpoint serves "either single queries or stateless multi-turn conversations", and the key word is "stateless": the server keeps no session, no thread id, and nothing to append to. It looks like a conversation because your client re-sends the entire history as an array on every request, and the reference says so in the same paragraph.

  Look at the request body, not the chat interface. You specify the prior conversational turns with the messages parameter, and the model generates the next message from that array, together with the tools and system text this lesson accounts for later. The array is your data structure, held by your process, serialised and posted whole each time. Twelve turns is twelve requests, and the twelfth one carries every byte of the previous eleven along with the new question.

  Loop has been talking to the model for an hour and feels understood. Loop has been paying, on every turn, to re-send everything Loop has said so far. The model stores none of it between turns.

  A common symptom: a team's request count is flat, its token count has tripled, and someone reports that "the model got more expensive this month". The cause is long threads, because a thread's total input tokens grow with the square of its turn count. Estimate a thread's cost before you start it: the cost is the sum of every prefix, so a fifty-turn session costs roughly thirteen hundred times one turn, not fifty times.

• Asking the model to forget a message does not remove it

  The common read: Telling the model to forget something takes that something out of what gets sent, so a correction is a repair rather than an addition.

  A correction is another message. The array grows by one entry, the entry says to ignore an earlier entry, and the earlier entry is still in the array in full, byte for byte, on every request from then on. Nothing in the API removes a message from an array your process owns, and the reference is explicit that consecutive same-role turns are merged rather than replaced.

  The correction appears to work at first, and that is why people keep believing it removed something. Recency is not in the protocol, and it is not a rule the API enforces; it is the observed behaviour of models on inputs that contradict themselves. A recent, specific, unambiguous instruction usually takes precedence over an older contradicting one, and only for as long as it stays recent.

  Twenty turns later the instruction and the text it corrects are both in the middle of the thread, both old, and the model generates an answer from two pieces of text that contradict each other and carry no metadata separating them.

  A common symptom: an engineer told the assistant three times to drop an approach, then reports that it "keeps going back to the old approach" as a memory problem in the model. Only deleting the message does what people expect a correction to do. Edit the array or start a new one, and treat every "actually, ignore that" as two contradicting facts in the input, where position, not intent, decides which one the model follows.

• Every request sends three parts, in one fixed order

  The common read: Tool definitions are registered with the model once at the start of a session, so a big tool surface is a setup cost rather than a per-request cost.

  Name the three parts of the prompt in the order they are assembled, then read on. The order is tools, system, messages. The prompt-caching page states that order as a hierarchy where each level builds on the previous ones, and the order is the useful part: everything you think of as configuration comes BEFORE everything you think of as the conversation, and all of it is text the model reads on every request.

  The size of that first segment is easy to overlook. Anthropic's engineering post introducing advanced tool use (published 2025-11-24) reports fifty-eight tools consuming approximately 55K tokens before a conversation starts, and 134K tokens measured internally before optimisation. The same post introduces the fix: a tool marked defer_loading: true stays out of the prompt until the model searches for it. Claude Code defers MCP tools by default. A client without deferred loading sends every definition on every request.

  A common symptom: a team reports that the agent has become "slow and stupid since we added the integrations" and blames the model version, when the only change was two more MCP servers in a config file. Count the tokens in your prefix (tools plus system) before you edit the wording of your prompt. The tools and the system text are the part of the input you can measure exactly, they are identical on every request, and they are the cheapest part of the input to cut.

• A bigger window did not make the model read better

  The common read: Anything that fits in the context window is equally available to the model, and filling the window costs nothing until you exceed its limit.

  Predict what happens to recall as a context window fills, then read on. Recall falls. Anthropic's context-engineering write-up names this context rot and states the finding: as the number of tokens in the context window increases, the model's ability to accurately recall information from that context decreases. The same post gives the architectural reason, that every token attends to every other token, so n tokens carry n squared pairwise relationships.

  The post reports a gradual decline: models stay highly capable at long contexts but may retrieve and reason less precisely than on short ones. Anthropic published this about its own models on 2025-09-29. It matters most in this lesson because a gradual decline raises no error or warning, so you do not notice it in the long sessions where it happens.

  A common symptom: someone says a session "was going great and then it started making things up", and nobody files a bug, because the thread hit no limit and raised no warning. Act on Anthropic's own formulation: find the smallest possible set of high-signal tokens that maximise the likelihood of the outcome you want. The same post notes that minimal does not mean short; the goal is the most useful information per token.

• Four ways to change the input, and only one is free

  The common read: A summary keeps everything that mattered and drops only the noise, so compaction is strictly better than starting over.

  Compaction always loses information. When it summarises, a model chooses what to lose. Anthropic states the trade-off: the difficulty is choosing what to keep and what to discard, and over-aggressive compaction loses subtle context whose importance only becomes apparent later. Anthropic also states elsewhere that compaction is not sufficient on its own for long-running work. That loss is not a bug: every summary discards something.

  There are four ways to change what is in the input, and each costs something different. Starting a fresh thread costs no tokens, takes effect at once and clears everything, so you lose whatever you have not written down. Compaction keeps the thread going, and you lose details you cannot predict. Writing the durable facts into a file the agent reads costs the work of keeping the file up to date, and every future session reads it.

  Referencing a path instead of pasting a body costs a tool call, and the model reads the current file instead of an old copy.

  A common symptom: someone who has used one thread since Tuesday makes a fourth attempt at a task, gets a worse result than the first, and reports that "it used to be better at this". Externalise before you compact: write the facts that must survive into a file. The real decision is which facts must survive a summary. Once you can name them, the cheapest correct option is to write them to a file the agent reads, which no summary can change, and then start a new thread.

• The first thing Nightshift needs is a file

  The common read: Getting good at this is about writing better prompts, so the work is in the wording of the request I am about to send.

  Before reading on, name the part of the request this lesson did not cover. It was the wording of your prompt. Everything here was about the rest of the input: the tool definitions, the system instructions, the eleven earlier turns, and the file you pasted on Tuesday. Anthropic describes the shift this way: building with language models is becoming less about finding the right words for prompts and more about what configuration of context is most likely to produce the behaviour you want.

  This course builds one thing across six levels. It is called Nightshift, it watches your Downloads folder, and at 06:00 it writes ~/nightshift/brief-YYYY-MM-DD.md with three sections: what arrived, what needs you, and what it did. It has no code yet, and that is deliberate. Its first artifact is a file that says what a good verdict on one downloaded item looks like, because that file persists across every session, and the prompt around it is written again each time.

  One kind of failure is never reported: a thread goes well, the engineer learns something true about their own system in it, closes the tab, and starts the next session by explaining the same thing again. Write down the facts that a summary must not lose. Writing them down costs one file, and every technique in the next five levels answers the question of what belongs in that file.