Token Thrift

A context window is not a chat history. It is what you send to the model for one dispatch — one call, built fresh, and thrown away. Your durable record of what happened lives in your stores and can be as large as you want. A common mistake is conflating the two: append everything to the window until it fills, then run an LLM "compaction" turn to squeeze it back down. That's reactive compaction — a panic move performed under pressure, in place, on your only copy of the recent turns, at the exact moment the window is already full and the model has the least room to get the rewrite right. It spends tokens to grow the problem, then spends a model call to guess at shrinking it, and the guess is lossy in the one way you can't audit: you no longer hold the verbose original to check the summary against. That's a different move from deliberately summarizing content you still hold, on your own schedule, as a focusing tool — more on that distinction below, where it matters most.

Token thrift is one lever of a larger discipline, not the whole of it. Getting the right context into a dispatch is necessary but not sufficient — a small model still needs mechanisms that keep it on-contract turn to turn (Behavioral Rails), and when it still goes sideways you need a way to find out why without guessing (Read the Wire). This page is the first lever: what you send.

The opposite stance, and the one the ADK's primitives are built for: hold a large working set — everything you could send, kept around between dispatches — and per dispatch subtract it down to the high-signal slice this call needs. You don't lose context. You focus it. Punching Above Its Weights builds this live on a real on-device model — see the subtractive pass for the mechanics on real weights; this page is the why, the evidence, and the boundaries. The primitives underneath — Tokenizable.estimateTokens, the overflow exception's per-bucket breakdown, SpooledArtifact, ToolCall.inline — are documented on Budgets.

Three goals, and all three point the same way

Token-thrift is not one goal wearing three hats. It is three distinct wins that happen to share a method: send the smallest high-signal slice that produces the result.

1. Budget — never overflow the window. A dispatch that exceeds the active model's window fails outright. The subtractive pass guarantees the dispatched set fits, with the battery's E_*_CONTEXT_OVERFLOW exception (and its per-bucket breakdown) as the deterministic backstop. This is the hard floor: don't break.

2. Cost and quality — never drag in the irrelevant. Trimming context is not only cheaper; irrelevant context is actively harmful. It is one of the primary drivers of hallucination, and a focused dispatch is not a budget compromise on a better-but-bigger prompt — the focused dispatch is the better prompt. More context is not better output. Higher-signal context is.

3. Speed — a model answers faster on less input. Prefill — the phase that builds the KV cache from your prompt before the first token — is compute-bound and scales with the number of input tokens. Fewer tokens in, lower time-to-first-token. This bites hardest exactly where on-device and self-hosted agents live, where you are bound by your own compute rather than hiding behind a provider's fleet.

The evidence: this is not a vibe

Every claim above is in the literature. Don't take the assertions on faith; the papers are linked.

Irrelevant context degrades output. Shi et al., "Large Language Models Can Be Easily Distracted by Irrelevant Context" (ICML 2023): model performance drops sharply when irrelevant information is added to an otherwise-sufficient prompt. This is the foundational result behind "don't drag unrelated elements in."

More context is not better; position matters. Liu et al., "Lost in the Middle: How Language Models Use Long Contexts" (TACL 2024): a U-shaped curve where relevant information in the middle of a long context is used worst, and performance degrades as the context grows — even for models built for long context. Filling the window does not help the model find the answer; it buries it.

A model doesn't want to hallucinate — bad input makes it inevitable

Hallucination is not a malfunction you can prompt your way out of. It has a theoretical lower bound: a calibrated language model fit to a distribution must hallucinate at some rate, independent of architecture (Kalai & Vempala, "Calibrated Language Models Must Hallucinate"; Kalai et al., "Why Language Models Hallucinate", 2025). Feed a model the wrong or irrelevant context and confident errors follow by construction — the attention mechanism aggregates every token indiscriminately and cannot tell signal from distractor. You cannot beg this away. What you can control is the input, and the lever that matters is input quality, not the temperature knob: prompt and input design dominate factuality while temperature barely moves accuracy across the studied range (Renze & Guven, "The Effect of Sampling Temperature on Problem Solving"). Token-thrift is hallucination mitigation you can actually action: fix what you send, not how hot you sample.

Subtractive, not additive

Keep your full record wherever you keep it. Build the window per dispatch: load everything reasonable into a large working set — messages, relevant memories, standing instructions, prior thoughts, tool results — then focus it down to the slice this call needs. The working set can be huge; the dispatched set is small and on-target. You start wide and subtract, rather than grow toward a cliff and panic-compact at the edge. Nothing is lost — it is still in the record; you simply did not ship it on this call.

How you hydrate the working set is your choice. The ADK does not impose a strategy. The subtractive pass — the one the showcase builds, and shipped as a Context Battery via subtractToFit (fit-to-budget shedding) and selectRelevantTurns / scaledRelevanceFloor (relevance-scored turn selection) — is how we do it: one effective, efficient way to get dynamic, per-model dispatches, built on ADK primitives, recommended but not mandated. The mechanism that makes it natural is the pipeline shape: you get trim points before the model ever sees a token (a per-turn input pipeline, and a per-dispatch input pipeline that re-runs every iteration), operating on mutable working-set collections where every primitive has a mutate/delete seam. "Trim and mutate before dispatch" is the literal API, not a metaphor.

Cut by evidence, not by guesswork

When a narrowed dispatch still overflows, the battery throws E_*_CONTEXT_OVERFLOW carrying a per-bucket token breakdown. Your error path reads which bucket is heavy and sheds that — the lowest-reranked retrievable tail, a fat tool result demoted to a handle, the oldest turns — then retries deliberately. You cut by measured evidence. Contrast the failure mode to avoid: unverifiable, whole-window compaction, a weak model rewriting the entire transcript in one pass where a flipped negation or a dropped date is undetectable. Dropping is lossy but predictable; blind summarizing hands a weak model one more chance to lie.

A dense summary is better input — not a lossy compromise

Summarization is not the enemy; unverifiable summarization of truth-bearing records is. A token-dense summary that preserves the facts and intent of a primitive is genuinely better input than a human's verbose rambling or a model's chain-of-thought wandering — fewer distractor tokens, higher signal density. The research is direct: Xu, Shi & Choi, "RECOMP" (ICLR 2024), summarize retrieved documents before injection at roughly a tenth of the tokens with minimal quality loss, and report it "relieves the burden of LMs to identify relevant information"; Jiang et al., "LLMLingua" (EMNLP 2023), compress prompts up to 20× with compressed prompts sometimes matching or beating the verbose originals on reasoning. In plain terms: a good summary of something you can re-derive isn't a worse prompt, it's often a better one.

So subtractive does not mean only deletion. Between drop and keep-whole sits mutation: compact an old Thought to its intent and conclusions, demote a fat artifact to a queryable handle, trim a message. The boundary that keeps it safe: summarize the things you control and can re-derive — reasoning, verbose prose, retrieved corpus — where a dense rewrite is a net gain; never blind-rewrite the truth-bearing records (what the user actually said, exact figures, tool outputs of record) where an unverifiable flip is unrecoverable. Dense summary of supporting context: good. Mangling the record: the failure mode.

When NOT to thrift

Token-thrift is relevance density, not minimalism for its own sake, and it is not right for every workload. The sharpest counterexample is a coding agent: it wants every token it can get, because holding as much of the relevant codebase in context as possible is the job — call sites, type definitions, neighboring modules, the failing test. There our recommendation inverts: a big window kept deliberately full of high-relevance code, not an aggressive subtract. The principle still holds (it is relevant code you want, not random files — irrelevant context still hurts), but the policy inverts: maximize relevant coverage rather than minimize token count, and reach for a large-window model on purpose.

The rule: thrift hardest when the signal is sparse and scattered across a long history (chat, RAG Q&A, tool-heavy turns); thrift least when the task needs broad, dense, simultaneous coverage (code comprehension, long-document synthesis, multi-file refactors). Same primitives, opposite dial. Don't cargo-cult the subtractive pass onto a workload that needs breadth — that is how you starve a coding agent and then blame the model.

Should you even build a coding agent?

Our answer: no — unless coding tooling is your core business. The existing coding agents are mature, actively maintained, and have absorbed an enormous amount of work you would be re-spending from zero. They are also increasingly portable, converging on open protocols instead of locking you in — the Agent Client Protocol (ACP) on the editor↔agent side, Model Context Protocol (MCP) on the tool side — so switching agents no longer means switching editors or tools. Our bias is to lean on those and spend ADK effort where you actually have an edge — your domain, your data, your workflow. The ADK is for building agents that are your product, not for re-deriving a category that is already well served.

Where to go next

• Punching Above Its Weights — the live demo: Gemma-4 E2B in a browser tab with a window you drag, the subtractive pass on real weights, and the build's field notes. Start from see it: one model, a window you control.
• Behavioral Rails — the gate cascade and the planner book-end: what keeps a small model on-contract once the window itself is right.
• Read the Wire — how to debug it when a small model still goes sideways: read the wire before you theorize.
• Budgets — the primitives this is built on: Tokenizable.estimateTokens, the SpooledArtifact range API, ToolCall.inline, and the overflow exception's per-bucket breakdown.
• Thought — why reasoning is a high-leverage compaction target, and the separate-authority reasons it is its own primitive.
• Context Batteries — the subtractive pass and relevance selection as a shipped API: subtractToFit, selectRelevantTurns, scaledRelevanceFloor, and friends.