92% Cached: The Economics That Let One Mac Serve 12x More Coding Agents

Inference Economics

One public benchmark — 92% of tokens served from cache — is all it takes to reframe local LLM sizing. Here is why agent traffic is the most cacheable workload there is, what a two-node Mac mini cluster already does right, where it leaks hit rate, and the math that turns a hit rate into a request budget.

hellodk24 Aug 2026 prefix cachingKV cacheApple Siliconcoding agents

92%

of agent tokens served from prefix cache in the referenced benchmark

12.5x

cheaper per token than a general-purpose workload (1 ÷ 0.08)

~1.9 → ~22

req/min sustained on one node: no cache vs the tweet’s 92% workload

3

independent ceilings — prefill, decode, KV-RAM — that bound throughput

01 · The Tweet That Reframed Cluster Sizing

Most LLM cost discussions start at model price. This one starts with a counter.

dx

@thdxr · dax

Oct 20, 2025

“some interesting data from our big pickle test — 92% of tokens were cached, this means our GPUs only had to process 8% of tokens. coding agents produce highly cacheable workloads – so our costs are 12.5x cheaper than a general purpose one”

The arithmetic is the point: 12.5x = 1 ÷ 0.08. Only the uncached 8% ever pays full prefill cost.

The claim isn’t about a smarter model or faster silicon. It is a statement about workload shape: if nearly every token a client sends has been seen before, then nearly every token is free to process — you pay only for the delta. Cost stops scaling with conversation length and starts scaling with novelty. For anyone running inference — cloud GPUs billed per token, or a home lab of Mac minis — that ratio changes what hardware can absorb.

02 · Why Coding Agents Are So Cacheable

An agent turn doesn’t send a question. It re-sends a world state, plus a tiny append.

Every turn of a coding-agent session ships the same growing payload: the system prompt, the full tool definitions, the accumulated file context, and the entire conversation history — then appends a small delta (one tool result, one model reply). Turn N contains turn N−1 as a strict prefix. A prefix cache turns that structure into an asset:

• System prompt + tool definitions — identical across every call of every session using that agent.
• File context and history — grows by append, never rewritten, so earlier blocks stay valid forever.
• The delta — often a few hundred tokens against a 100k-token prefix: a few percent of the bill.

Bar widths are schematic. The pattern is real: each turn re-sends everything and adds a sliver.

A general-purpose workload — unrelated chat sessions landing cold — looks nothing like this. Same GPUs, different traffic shape, 12.5x different economics. Which raises the practical question for a self-hosted setup: does my inference stack actually capture this?

03 · Case Study: What oMLX Already Does Right

oMLX is Apache-2.0 local LLM inference for Apple Silicon. I run it on a personal two-node Mac mini cluster linked over WiFi + Thunderbolt bridge — and its cache design already earns a large slice of the tweet’s economics.

Capability	Where	What it buys
Chained content-addressed block hashes	omlx/cache/paged_cache.py	vLLM-style paged blocks (256 tokens) hashed and chained — a prompt matches wherever its blocks match, so mid-prompt prefix reuse works without exact full-prompt hits.
Two-tier cache	RAM hot tier + SSD cold tier	Paged SSD tier under a 100GB cap keeps evicted prefixes restorable — warm context survives longer than RAM alone would allow.
Cluster-level cache-aware router	omlx/cluster/cache_aware_scheduler.py	Scores nodes by prefix-hash overlap, so requests drift toward whichever node already holds their context.
Cache observability	PrefixCacheStats	tokens_saved, exact hits, partial-block skips — the raw material for exactly the measurement behind the tweet.
Scheduling depth	max_num_seqs = 256	Chunked prefill, decode-fairness scheduling, memory-guard watermarks — batch machinery ready to exploit whatever the cache saves.

This is not a toy: content-addressed chaining, tiering, overlap routing and stats are the same architectural moves as production vLLM deployments. But auditing it against agent workloads surfaced six places where hit rate still leaks.

04 · Six Gaps Between Here and 92%

The improvement backlog, ordered roughly by leverage on a small cluster.

1

No session stickiness

New conversations round-robin across nodes. The next turn has a ~50% chance of landing on the node without the warm prefix — on a 2-node cluster this is the single biggest lever, and it halves effective hit rate before any other tuning matters.

Fix: TTL-based session→node pinning, overlap scoring as tiebreak for new sessions.

2

FCFS admission ignores the cache

The queue admits strictly first-come-first-served. Sorting by prefix overlap before admission batches requests that share prefixes together and raises hit rate for free — this is standard vLLM v1 behavior.

Fix: opt-in CACHE_AWARE scheduling policy ordering admission by matched-block count.

3

256-token blocks waste up to 255 tokens per branch point

Agent conversations branch constantly — parallel tool calls fork from one shared prefix — and every fork discards the unfilled tail of its last shared block. At 256 tokens that is up to 255 wasted prefill tokens per branch.

Fix: selectable block-size profiles; default agent/coding profiles to 64-token blocks.

4

Overlap-only routing ignores load

The router picks max overlap regardless of queue depth. Under bursty traffic the best-overlap node is often the busiest, trading a cache hit for queue delay.

Fix: score = overlap minus a load penalty (queue depth, active seqs), with a tunable weight.

5

No per-client cache economics

tokens_saved exists but isn’t surfaced per API key or session. The tweet is fundamentally a measurement — you can’t tune routing, block size, or pricing per workload when cacheability is only visible as one global counter.

Fix: per-key hit-rate and tokens-saved time series in the admin dashboard, plus a cluster rollup.

6

fp16 KV in the hot tier

Resident prefix capacity is bounded by KV bytes. Dropping hot-tier KV to 8-bit doubles the number of resident 100k-token agent contexts on the same RAM.

Fix: opt-in 8-bit KV quantization with a documented quality delta.

05 · The Three Ceilings: From Hit Rate to Request Budget

Hit rate is an input. Throughput is a minimum. Sustained request rate R is capped by whichever ceiling binds first.

Ceiling	Inequality	Meaning
Prefill	R × P × (1−h) ≤ PP_batched	Uncached prompt tokens must fit in the batched prefill budget. P = prompt tokens, h = hit rate. This is the ceiling caching attacks directly.
Decode	R × O ≤ TG_batched	Every request still generates O output tokens one at a time. Caching does nothing here — decode is paid in full on every request.
KV-RAM	streams ≤ KV_budget ÷ bytes/token	Concurrent streams must fit resident KV. On memory-constrained machines this binds before compute does.

Worked example — reference bench fixture

44GB model on a 128GB machine; batched prefill PP = 3,100 tok/s; decode TG = 180 tok/s at batch 4. Agent turn shape: P = 100,000 prompt tokens, O = 500 output tokens.

Scenario	Hit Rate	Sustained Rate	Binding Ceiling
No cache	0%	~1.9 req/min	Prefill — 3,100 tok/s ÷ 100k tokens/request
Tweet workload	92%	~22 req/min	Decode + prefill co-bound — both land near 22
Tuned: 95% hit + larger decode batch	95%	~30–48 req/min	Decode aggregate — the remaining wall

Raising hit rate pushes R off the prefill ceiling — until decode becomes the wall.

The punchline: caching doesn’t speed up decode. It removes prefill as the bottleneck — so the same silicon absorbs ~12x more agent traffic, and the binding constraint migrates to decode throughput.

Memory-constrained footnote: on a 16GB Mac mini the KV-RAM ceiling binds first — concurrent streams run out of resident KV long before compute saturates. There, session stickiness and 8-bit KV quantization are worth more than raw batch scaling.

06 · Takeaways

• Coding agents are a cache-native workload. Append-only prompts make 90%+ hit rates normal, not exceptional — the 92% figure is a property of the traffic, not the vendor.
• Cost models should be built on miss rate, not token count. Price follows the 8% that misses, not the 100% that flows.
• Local clusters compete on cache architecture. Content-addressed block hashing, two-tier storage, and overlap routing put a two-node Mac mini fleet in the same design space as production vLLM farms.
• The backlog is specific: stickiness, cache-aware admission, smaller blocks for branching workloads, load-weighted routing, per-client economics, and KV quantization — six concrete levers between a working cache and the tweet’s numbers.
• Measure like the tweet measures. Hit rate per workload is the number that decides whether your cluster serves 2 agents or 24.