KV Cache-Friendly Design

Before examining the example, consider the intuition behind KV Cache. Every time the model generates a token, it must refer back to the intermediate computation results of the preceding tokens. Recomputing those results from scratch on every round would become increasingly expensive as the context grows. KV Cache stores the intermediate key-value states so later computation can reuse them. The prerequisite is that the prefix stays completely unchanged: alter a single character in it, and the cache for that prefix can no longer be reused; the model must recompute from the changed point onward. A note on terminology: when this section discusses “cache hits” across requests, API providers usually call this Prompt Cache—a cross-request cache built on top of the inference engine’s KV Cache. The two levels are distinguished at the end of this section.

With that intuition in mind, consider a production incident. A team’s customer service Agent handled 100,000 conversations a day, and the system was running normally. Then an engineer, wanting the Agent to have access to the current time, added a line Current time: {{now}} to the system prompt, injecting the timestamp in real time. The next day, monitoring alerts fired: TTFT for every conversation increased from 0.5 seconds to 3–5 seconds, and the monthly inference bill nearly doubled. The code looked correct and the model had not changed. The issue was in the context.

That one timestamp line invalidated the KV Cache on every request. The system prompt was now different each time, forcing the model to recompute the key-value pairs for the prefix from scratch (here, “Key” and “Value” are two types of vectors in the attention mechanism; Experiment 2-2 below visually demonstrates their roles). This kind of invisible cost appears repeatedly in Agent systems: a seemingly harmless line of code can slow down the entire inference pipeline by an order of magnitude. This section explains how to avoid these pitfalls.

Principles and Constraints of KV Cache

To understand the value of KV Cache, first consider what happens without it. Suppose an Agent has reached the sixth conversation round and accumulated 2,000 context tokens. Without caching, each new token requires the model to recalculate the K and V vectors for the entire prefix. Although the first five rounds are unchanged, the sixth round still recomputes them, and the longer prefix makes this round more expensive than the first. Without caching, the attention computation in the prefill phase (the stage where the model processes all input tokens before generating a response) grows quadratically with context length, causing latency and cost to rise rapidly as the conversation deepens. This is especially problematic for Agent tasks that require many tool calls.

Understanding KV Cache with a simple example. Suppose the context has 4 tokens [A, B, C, D], and the model is about to generate the fifth token, E. The core attention operation compares E’s Query vector with the Key vectors of the existing tokens to calculate match scores. It then uses those scores to compute a weighted sum of the Value vectors, producing E’s output representation.

Without KV Cache, every time a new token is generated, the K and V vectors of all preceding tokens must be recalculated from scratch: generating E requires computing 5 sets of K and V, generating the sixth token requires computing 6 sets… and by the Nth token, N sets must be computed, with the total computation proportional to N².

With KV Cache, the K and V vectors of A, B, C, and D are cached after being computed once. When generating E, only E’s own K and V need to be computed, and then the attention calculation is performed using these along with the 4 cached sets. Note that KV Cache saves the recomputation of the K and V projections for historical tokens, so each decoding step does not need to recompute the entire prefix; however, the attention calculation for each new token still needs to traverse all cached K and V values, with computation growing linearly with context length — this is why long-context decoding becomes increasingly slow, and KV Cache’s memory and bandwidth become the inference bottleneck.

Why does modifying the prefix invalidate the cache? Large language models are composed of stacked Transformer layers (modern LLMs typically have dozens to hundreds of layers), and each layer produces its own K and V cache. These layers are connected in sequence: the output of layer 1 becomes the input to layer 2, the output of layer 2 becomes the input to layer 3, and so on. When processing each word, layer 1 considers that word and all preceding words, then outputs an intermediate representation; layer 2 takes that representation and processes it further. If an early token changes (for example, one character in the system prompt), the output of layer 1 changes, the input to layer 2 changes, and the difference propagates through the subsequent layers. The cached states after that change must be recomputed. The cost is significant: previously processed tokens may need to be recomputed and billed again, and latency can increase substantially (this chapter’s experiments measured severalfold increases). This is why the book repeatedly emphasizes: once the system prompt is set, do not change it.

KV Cache and Prompt Cache: Two Levels of Caching

Before proceeding, it is useful to distinguish two easily confused concepts. KV Cache is an optimization inside model inference: during a single inference pass, it caches the key-value states of already processed tokens to avoid redundant computation. Prompt Cache is an API service-layer optimization: it reuses cached computation for identical prefixes across multiple API requests. Both rely on prefix stability, but they operate at different levels. KV Cache accelerates token generation within a request; Prompt Cache reduces redundant prefix computation across requests. In practice, the API provider matches the request prefix. If multiple requests share the same prefix (for example, the system prompt and tool definitions remain unchanged), the provider can reuse cached prefix computation instead of recomputing those tokens. Reading from the cache costs far less than computing fresh—about one-tenth the price at Anthropic and DeepSeek, and likewise about one-tenth for OpenAI’s GPT-5 family (the earlier GPT-4o generation was half price; starting with GPT-5.6, cache writes additionally carry a 1.25× surcharge). How caching is enabled and billed differs by provider: Anthropic requires explicit cache_control breakpoints, charges a markup for cache writes, enforces a minimum cacheable length (e.g., 1024 tokens), and applies a TTL limit (about 5 minutes by default); OpenAI uses automatic prefix caching without explicit declaration.

When designing context, both levels of caching require a stable prefix—but Prompt Cache has a greater economic impact because it directly affects API billing.

Caching as an Architectural Constraint

The following section covers architectural details of production-grade Agents. First-time readers may skip it and return when building an Agent.

In production-grade Agent systems, caching is not merely a performance optimization—it is an architectural constraint that dictates many seemingly unrelated design decisions throughout the system.

Claude Code illustrates a broader pattern: when Prompt Cache has significant economic value, cache consistency can shape architectural choices across the system. Several design decisions reflect this constraint:

Prompt structure is shaped by cache boundaries. The system prompt is split by a cache boundary marker: content before the marker can be globally cached across users and sessions, while content after the marker contains user- and session-specific information. This means prompt ordering is driven primarily by caching economics and only secondarily by semantic logic. Each runtime condition placed before the cache boundary (OS type, current mode, user preferences, etc.) increases the number of cache-key variants. If each condition is binary, N conditions produce 2^N combinations. For example, 3 binary conditions (macOS/Linux, normal/debug mode, Chinese/English) produce 2×2×2 = 8 cache keys. Prompt fragments are therefore typed as either “cacheable” or “cache-breaking,” with explicit warning markers for the latter.

Sub-agents must be byte-aligned with the parent Agent. When the main Agent spawns a sub-agent or performs a side query, the sub-agent’s prompt, tool definitions, model configuration, message prefix, and reasoning configuration must match the parent Agent’s cache key byte-for-byte. The reason is that if the API request initiated by the sub-agent has a prefix identical to the parent Agent’s request, it can hit the API provider’s Prompt Cache, thereby reducing billing and latency. This constraint propagates upward from the caching layer, influencing how Agents are generated and how parameters are passed.

Replacement strings for tool results are frozen upon first occurrence. When large tool outputs are replaced with summary previews, the replacement string is persisted. Even after a session restarts, the system reuses exactly the same replacement string so that the restored message sequence remains byte-identical to the cached stream.

The core insight is that caching economics is not a post-hoc optimization but an upfront architectural constraint. If your Agent system uses Prompt Caching, the requirement for cache key consistency will permeate prompt design, multi-agent coordination, session restoration, and other layers. The earlier this constraint is incorporated into the architecture, the lower the subsequent engineering cost.

KV Cache Is Not Necessarily One-Shot: Editable, Composable “Notes”

(The following is optional advanced material from current research. It can be skipped on first reading without affecting the rest of this chapter; the three practical conclusions above are the foundation.)

So far, this section has assumed a strict rule: change one byte in the prefix, and the subsequent cache is invalidated. This rule holds in today’s inference engines, but it may not be inevitable. A recent line of research starts from a counterintuitive observation: during the prefill phase, the model behaves as if it is “taking notes.” When it reads a field in the context (e.g., “User’s city: Beijing”), it does not simply cache that field verbatim. Instead, it writes downstream representations of the conclusion—what this field means—into later KV states. Measurements show that the KV states of the field’s own tokens often contribute less than 1% to the final decision; what influences the output more are the downstream “notes” left by that field.

This discovery suggests two operations that were previously considered impractical. The first is Editing: since the conclusion has already been written into downstream notes, a changed field can propagate through cached reasoning when the model has an explicit chain of thought (CoT), producing results close to full recomputation with about 1% of the compute. Conversely, without CoT, an isolated field change may be ignored because the conclusion is already embedded downstream without a reasoning path to update it. The second is Composition: a precomputed “skill” cache can be relocated using Rotary Position Embedding (RoPE) and spliced into another context without recomputing attention. In this framing, assembling a long context from modular cache blocks drops from O(L²) recomputation to O(L) splicing, with output quality close to full recomputation.

The margin-note analogy is useful here. When reading a long document, one does not reread the entire document every time a fact changes; instead, one updates the note that records what the fact implies. The idea of KV Cache as notes is similar: if the cached states already encode the inference of a fact, then changing the fact may require correcting the downstream note rather than recomputing everything. Because the notes are represented in a portable form, a block of notes from one problem can also be repositioned (via RoPE relocation) and reused in another. The paper implemented this idea on vLLM, speeding up p90 time to first token by factors ranging from tens to hundreds, with a prefix cache hit rate of about 98.5% and outputs close to token-by-token recomputation (across 12 models, logit cosine similarity 0.90–0.999).

For Agents, the implication is that long contexts may not always need to be torn down and rebuilt when tools, memory fields, or runtime state change. In principle, this could make context mutable while preserving some caching benefits, turning context assembly from O(L²) recomputation into O(L) note splicing. This is still research-stage work; the three practical conclusions earlier in this section remain the default principles for current production systems.

Now that we understand how context is processed and cached, the next question is how to design the content itself. The following sections discuss what belongs in context and how to organize it, along three related threads:

• Prompt Engineering, Prompt Injection, and Dynamic Prompts (Agent Skills): How to write the system prompt and what to include. This is the most direct part of context engineering. Tool definitions, another static component alongside the system prompt, also directly affect the accuracy of the Agent’s tool use. This chapter provides the core principles, and Chapter 4 expands on them in detail. The next issue is security: when external content attempts to hijack a carefully designed context, how should the system defend itself at the context level? As prompts grow longer and cover more scenarios, placing everything into a single system prompt becomes impractical: it wastes tokens and dilutes attention. This leads naturally to the progressive disclosure mechanism of Agent Skills, where knowledge is loaded on demand rather than included all at once.
• Agent Status Bar: An independent mechanism that injects dynamic meta-information (task progress, environment status, tool call count, etc.) at the end of the context, compensating for the model’s inability to actively summarize implicit states. Analogous to the time, battery, and network signal shown at the top of a phone screen, the Agent Status Bar lets the model access the current runtime state at any time.
• Context Compression Strategies: Addressing the problem of ever-expanding context—when to compress, how to compress, and how compression coexists with KV Cache.

Engineering Practice

For Anthropic models, each step calls markPrefixCacheBoundary() before appending transient reminders, placing up to 4 ephemeral breakpoints and reading back the longest-hit prefix. Multiple anchors keep the cache useful even when the tail jumps or compaction rewrote part of the prefix: the system block (stable for the whole task), the rolling-summary anchor at the front (markTaskContextAnchor), the current turn boundary, and the tail just before the reminders. Compaction’s summaries and tool-result replacements are content-addressed and byte-stable across turns, delivering “frozen replacement strings” and “byte-identical session resume.” See compaction.

Related reading

• Prompt Engineering — what to put in the static prefix and how to lay it out so the cache holds
• Compaction — how compaction coexists with the cache — append-only, content-addressed, deterministically replayable