Skip to content
AtomicReps

The bug that is not on the screen

A lesson from The Model You Stopped Building. Play it above, or read it through below.

  • Both approvals were right. The run disagrees with both of them.

    The common read: Each file is correct on its own, so the behaviour the two of them produce together is correct.

    Neither approval was a mistake. cache.mjs is correct for the contract its own docstring states, and pricing.mjs reads the currency from request context, validates it, and refuses a currency it cannot price. Point at the line to change in either file. There is no such line. The five numbers came out of the relationship between the two, a shape you have met once, in one file. Here it lives elsewhere: the line that creates it sits in a third file, and neither review opened it.

    memoize builds its key from the argument list it was handed. quote takes one argument and reads two inputs, so the key names the SKU and says nothing about the currency, and the first request for seat-pro arrived in USD, ran quote once, and put 4900 into the Map under ["seat-pro"], where every later request for that SKU found it and was answered from the Map.

    Nothing on that run is an error, and nothing about the output looks unusual: five plausible prices for five requests. The ticket that eventually arrives says currency conversion is wrong for European customers, and the engineer who picks it up spends the first hour inside pricing.mjs, which is the one file in the service that converts currency correctly, on every input it is ever given, and which has not changed since the day it was written.

  • A cache key is a claim about what the function reads.

    The common read: A function's inputs are the values in its parameter list, so a cache keyed on the arguments is keyed on everything the function reads.

    Name every input quote reads, then count how many of them appear in its signature. Two are read and one appears. sku selects the list price and arrives as a parameter. currency selects the rate and arrives through the store currentRequest reads, which the caller set for this request and passed to nothing. The pull request that added the wrapper carried one review comment, asking whether the Map should be bounded, filed against the util's memory and not against the input the key never names.

    The signature is not a list of inputs. It is a list of the inputs somebody decided to pass, and the ones nobody passes are the ones every generic wrapper is blind to: request context, module-level state, the environment, the clock, the locale, and whichever tenant is current. Any wrapper keyed on an argument list inherits that blindness and cannot detect it, because a function that reads something ambient has the same signature and the same type as one that does not.

    The docstring on memoize says pure function. That is true of the contract memoize states and false of the function it was handed: quote does no I/O, allocates nothing and reads no database, and it still returns two different values for one argument. The comment that would have caught this fits on one line, and it asks what the function reads outside its parameter list. Ambient input is this course's word for the rest, and it is ours rather than documented vocabulary.

  • An issue stream is a list of what threw. It is not a list of what went wrong.

    The throw landed on the request that had nothing to sell. seat-pro came back at 4900 for a currency the service cannot price and seat-team threw, and the only difference between them is which SKU somebody had already asked for. Capture is a subscription: a tracker's client subscribes to thrown exceptions and rejected promises, so an event exists when a throw exists, and a wrong number that nothing threw over is not a smaller event than a crash, it is no event at all.

    Five requests, five prices, and a counter on every throw handler reads zero. The dashboard was accurate all week, and that is the part worth keeping. It reported what threw, and nothing threw. The postmortem line says the error rate stayed flat through the incident, filed under gaps in monitoring coverage, with an action item to add alerting on the pricing path. There was no error rate to move: the service answered every request it received, in well-formed JSON, with a number that looked like a price.

  • The diff that shipped this is one line, and it is in neither file you read.

    The common read: The diff is the change, so a careful reading of the diff is a careful reading of the change.

    Read this diff and name the behaviour it changes. It adds an import, a comment and one binding, and it swaps a call for another call of the same name. Nothing in it mentions currency, request context, validation or correctness, and every claim it makes is true: pricing is hot on that path, the table does change at most daily, and memoize returns the same value for the same arguments. Reviewing this diff carefully finds nothing. Reviewing it carelessly finds nothing.

    The behaviour change is not in the diff. A wrapper's effect lives in the function it wraps, so the shorter the wiring line, the further the behaviour moved from the text under review, and the two files that would price the risk are untouched and absent from it. The incident doc records that the change was generated rather than typed, which is true and explains nothing, since the identical line typed by hand behaves identically. The wiring diff is this course's name for the shape, not documented vocabulary.

  • Every dimension you record is identical on both workers.

    The common read: Two processes of one build are interchangeable, so a price that differs between them means the data differs.

    The workers differ in one value and it is not a value anybody records. The release is equal, the environment is equal, the commit is equal, the rate table is equal, the request is equal, and the user is one person reloading a page. What separates 4508 from 4900 is which currency asked first, which is a fact about the order of arrivals into one process and has never been written into a field on anything.

    An event carries the dimensions somebody named, and you can filter on those and on nothing else. Naming them is a decision taken before the incident, by someone who did not know which incident it would be, which is why the dimensions that keep earning their place are the ones describing inputs rather than outcomes: the currency in scope, the tenant in scope, the flag state, the process that answered. A dashboard cannot group by a value that never left the process.

    The team splits the dashboard by release, sees both workers on the same release, and closes the question: the deploy is clean. The deploy is clean. The review comment on the eventual fix asks for more logging, and more logging of the same fields would have produced twice as much evidence for the same wrong conclusion, because the field that separates the two workers is not among the fields anyone is logging.

  • When it does throw, you get the frame that noticed.

    The common read: The top of the stack is where the fault is, so the file named on the issue is the file to fix.

    Name the files you expect on the stack when the nightly re-pricing job refuses to settle a row it believes is mispriced. The job re-prices every charged row from the rate table and throws on a disagreement, and the throw carries settlement.mjs, then reconcile.mjs, then Node's own module loader, while cache.mjs, pricing.mjs and handlers.mjs appear on none of the frames and the call that produced 4900 returned before the job printed the ledger it is now checking.

    That is not a defective stack trace. A stack lists the calls open when the throw happened, and the call that produced the number returned long before anything noticed. Anything that carries a value to a distant place does this: a cache, a queue, a scheduled job. The distance in the trace measures how far the value travelled, never who was wrong, and the issue opens against the one file in the chain that checked its work.

  • Grouping is a local reading, mechanised.

    The common read: One issue is one cause, so the issue count is the count of things wrong.

    Predict how many issues a week of that settlement job opens when three currencies are mispriced across two SKUs. Default grouping keys on the exception type and the top frames of the stack, and every one of those rows throws AmountMismatch from the same line of settlement.mjs, so the type matches and the frames match, and they land in one bucket carrying one title and a count. The count is the only thing that moves.

    The bucket is a correct answer to the question grouping asks, which is whether two events look alike where the reader is standing. It is the same local reading the two file reviews performed at the start, run at machine speed over every event, and it is right for the same reasons and blind in the same place. The currency is not on the issue unless somebody attached it, so the title reports that an amount disagreed and the count reports how often.

    Key name arguments. Function read more than arguments. Key lie, quietly. Two file good. Two file together, not good. Nobody write that down. Stack say who notice. Stack never say who decide. Rule stay same: input you read, input you name. In key. On event.

  • The question that finds it fits on one line.

    The common read: This is a memoize problem, so not memoizing avoids it.

    Wrap a function whose inputs you have not enumerated, and the wrapper's key becomes a claim you never checked. The retro's action item is a lint rule banning memoize on the checkout path, which bans the wrapper and leaves the key unexamined. The question that finds this one is what the function reads that is not in its parameter list, and it is answerable in the time it takes to open the file and look at what is in scope. quote answers it on its first line.

    Two shapes are worth carrying, and both are relationships rather than defects. A cache, memo, dedupe or batch key built from an argument list, over a function reading request context, module state, the environment, the clock or the current tenant: whichever caller arrives first decides the value everyone after them receives. And any change whose diff is one line of wiring, because the diff is short precisely when the behaviour moved somewhere the diff does not touch.

    Naming the input is the whole repair. Giving quote a currency parameter and having the handler read it from context and pass it puts both values in the key, and the same five requests price at 4900, 4508, 769300, 2668 and 2900, with the guard firing for the currency the table does not hold. The cost is one parameter and one line in the caller, paid on every wrapper you write, against a defect that no local reading of either file can find.

  • Nothing in this lesson was a defect in a file.

    The common read: These numbers were wrong, so there is a defective file to find.

    Name the file to fix and the fix that goes inside it. There is no such file, and every reading this lesson performed on one file at a time was correct, the first screen's two included. The incident doc names pricing.mjs as the cause. What changed is what you can predict: what the second currency receives, that a guard stops running once something is cached, that two workers of one build answer one request with two numbers, and that the issue names the frame that noticed.