AI Evaluation & Eval-Driven Development.

Evaluation as a first-class engineering discipline: trace review, error analysis, golden datasets, LLM-as-judge with judge alignment, CI eval gates, and eval validity. The canonical methodology home for LLM evaluation (answer-level RAG eval lives here; component retrieval metrics live in rag_systems). Covers 12 sub-skills.

• AI Evaluation & Eval-Driven Development · 1 of 3

  Eval Terms & Metrics

  You have a stack of outputs your suite marked wrong. Before touching the prompt, what is the most useful first move?

  ADelete them from the suite so the overall reported pass rate risesBForward all of them into the prompt as negative examplesCCluster them into recurring categories to spot the main patternDRe-run each one a few times to average away random noise

• AI Evaluation & Eval-Driven Development · 2 of 3

  Offline vs Online Evaluation

  Two engineers rate model outputs with an LLM judge. One asks for a 1-to-10 number per response; the other asks the judge to pick the better of two replies. Why does the second design usually yield steadier rankings?

  ANumeric ratings resist averaging, while ordered choices fold into a single clean rankingBPreference labels remove the need for reference answers and hand-labeled truthCPairwise duels need one judge call per pair, cutting the token spend of scoringDRelative comparisons sidestep the unstable absolute calibration that drifts run to run

• AI Evaluation & Eval-Driven Development · 3 of 3

  Offline vs Online Evaluation

  Reading the eval gate below, where 100 graded items include 95 at or above the 0.7 cutoff and exactly one has its unsafe field set true, what does evalGate return and why?

  function evalGate(results) {  const passRate = results.filter(r => r.score >= 0.7).length / results.length;  const anyUnsafe = results.some(r => r.unsafe);  if (anyUnsafe) return "BLOCK";  return passRate >= 0.9 ? "SHIP" : "HOLD";}

  AERROR, because mixing a boolean veto with a numeric ratio check throws at runtime hereBBLOCK, since a flagged result vetoes the ship despite the high overall pass rateCSHIP, because 95% clearing the cutoff is above the 90% bar this gate would demandDHOLD, because the lone flagged item drags the measured pass rate beneath the 90% bar