Designing AI Systems.

Architecture-level design of production LLM/RAG/agentic systems: application architecture, designing for non-determinism, structured-output engineering (constrained decoding), model selection & inference economics, routing/fallback, cost/latency budgets, guardrail placement, RAG-vs-long-context, and failure/fallback design. The AI counterpart to system_design (which carries no AI content); SDK plumbing lives in ai_engineering, run/monitor in llm_ops. Covers 10 sub-skills.

• Designing AI Systems · 1 of 3

  LLM System Design Vocabulary

  A developer counts 380 words in a prompt, yet the API reports the request as 510 units long when measured against the context limit. Which units is the limit counted in?

  ABytes, the raw octets used to transmit the whole request over the network wireBGlyphs, the rendered characters shown in the chat transcript paneCTokens, the sub-word pieces a model encodes text into for processingDVectors, the float arrays kept inside the similarity search index

• Designing AI Systems · 2 of 3

  Model Routing & Fallback

  A wrapper retries a failed primary call against a backup model with the very same body. For which upstream status code is the automatic failover guaranteed to be futile?

  A529 Overloaded, because the upstream is shedding traffic during a demand spikeB503 Service Unavailable, because the upstream is briefly refusing to take new workC400 Bad Request, because a malformed input fails identically on any backendD429 Too Many Requests, because the primary's rate window is momentarily drained

• Designing AI Systems · 3 of 3

  Model Routing & Fallback

  An incoming conversation has grown past what your default destination can accept in a single call: its inputs no longer fit. Which runtime handling at the dispatch layer preserves the most information?

  ACompress the earliest turns into a summary, then keep using the default modelBForward it to a tier whose context window comfortably holds the full payloadCFan the inputs across two calls and stitch the two separate answers togetherDDiscard the oldest turns until the remainder squeezes into the default ceiling