---
url: 'https://adk.nht.io/the-loop/token-thrift.md'
description: >-
  The context window is not a chat history — it is what you send for one
  dispatch. Build it subtractively: hold everything, send the focused slice. Why
  it matters (overflow, hallucination, speed), what the research says, when to
  do it, and the one case where you shouldn't.
---

# Token Thrift

## LLM summary — Token Thrift (concept/recipe)

* A context window is **what you send for one dispatch**, not your conversation history. Keep the full record in your stores; build each dispatch fresh and small. Token-thrift **focuses** context, it does not lose it. Reactive, in-place, whole-window compaction under pressure is what this page argues against; deliberately summarizing content you still hold, on your own schedule, is a different move and a good one (see below). The live build is [Punching Above Its Weights](/showcase/punching-above-its-weights); the primitives are [Budgets](/the-loop/budgets).
* Token-thrift is one of three siblings in the same discipline — **context discipline**: this page (what to send), [Behavioral Rails](/the-loop/behavioral-rails) (how the harness keeps a small model on-contract turn to turn), and [Read the Wire](/the-loop/read-the-wire) (how you debug it when a small model still goes sideways). Same showcase, three levers.
* **Three goals**: (1) never overflow the window; (2) cost *and quality* — irrelevant context is detrimental and a leading cause of hallucination, so a focused dispatch is the *better* prompt, not a cheaper compromise; (3) speed — shorter input = less prefill = faster time-to-first-token, decisive on self-hosted / edge.
* **Subtractive, not additive**: hold a large working set (messages, memories, retrievables, thoughts, tools), then trim/mutate per dispatch via the pipeline. ADK does not force this — how you hydrate is your choice; the subtractive pass is one effective default built on ADK primitives.
* **Hallucination is structural**: a calibrated model *must* hallucinate at some rate (Kalai & Vempala; Kalai et al. 2025); bad input makes it inevitable (Shi et al. 2023, "easily distracted by irrelevant context"). Input quality dominates the temperature knob (Renze & Guven). The lever you control is what you send.
* **A dense, fact-preserving summary is *better* input** than verbose raw text (RECOMP; LLMLingua) — a focusing tool, not a lossy compromise. The boundary: summarize controllable/re-derivable context (reasoning, prose, retrieved corpus); never blind-rewrite truth-bearing records.
* **When NOT to thrift**: it is relevance density, not minimalism. Coding agents, long-document synthesis, and multi-file refactors need *broad, dense coverage* — keep a big window full of relevant context; the policy inverts. And building your own coding agent is usually a poor use of time unless it is your core business; the established tools are mature and converging on open standards (Agent Client Protocol + Model Context Protocol).

A context window is not a chat history. It is **what you send to the model for one dispatch** — one call, built
fresh, and thrown away. Your durable record of what happened lives in your stores and can be as large as you
want. A common mistake is conflating the two: append everything to the window until it fills, then run an LLM
"compaction" turn to squeeze it back down. That's *reactive* compaction — a panic move performed under pressure,
in place, on your only copy of the recent turns, at the exact moment the window is already full and the model has
the least room to get the rewrite right. It spends tokens to grow the problem, then spends a model call to
*guess* at shrinking it, and the guess is lossy in the one way you can't audit: you no longer hold the verbose
original to check the summary against. That's a different move from *deliberately* summarizing content you still
hold, on your own schedule, as a focusing tool — more on that distinction below, where it matters most.

Token thrift is one lever of a larger discipline, not the whole of it. Getting the *right context* into a
dispatch is necessary but not sufficient — a small model still needs mechanisms that keep it on-contract turn to
turn ([Behavioral Rails](/the-loop/behavioral-rails)), and when it still goes sideways you need a way to find out
why without guessing ([Read the Wire](/the-loop/read-the-wire)). This page is the first lever: what you send.

The opposite stance, and the one the ADK's primitives are built for: hold a large **working set** — everything
you *could* send, kept around between dispatches — and per dispatch **subtract** it down to the high-signal
slice this call needs. You don't lose context. You focus it.
[Punching Above Its Weights](/showcase/punching-above-its-weights) builds this live on a real on-device model —
see [the subtractive pass](/showcase/punching-above-its-weights#the-subtractive-pass-shrink-to-fit-every-dispatch)
for the mechanics on real weights; this page is the why, the evidence, and the boundaries. The primitives
underneath — [`Tokenizable.estimateTokens`](https://adk.nht.io/api/@nhtio/adk/common/classes/Tokenizable#property-estimatetokens), the overflow exception's per-bucket breakdown,
[`SpooledArtifact`](https://adk.nht.io/api/@nhtio/adk/spooled_artifact/classes/SpooledArtifact), [`ToolCall.inline`](https://adk.nht.io/api/@nhtio/adk/forge/classes/ToolCall#property-inline) — are documented on [Budgets](/the-loop/budgets).

## Three goals, and all three point the same way

Token-thrift is not one goal wearing three hats. It is three distinct wins that happen to share a method: send
the smallest high-signal slice that produces the result.

1. **Budget — never overflow the window.** A dispatch that exceeds the active model's window fails outright. The
   subtractive pass guarantees the dispatched set fits, with the battery's `E_*_CONTEXT_OVERFLOW` exception (and
   its per-bucket breakdown) as the deterministic backstop. This is the hard floor: don't break.

2. **Cost *and quality* — never drag in the irrelevant.** Trimming context is
   not only cheaper; irrelevant context is **actively harmful**. It is one of the primary drivers of
   hallucination, and a focused dispatch is not a budget compromise on a better-but-bigger prompt — the focused
   dispatch *is* the better prompt. More context is not better output. Higher-signal context is.

3. **Speed — a model answers faster on less input.** Prefill — the phase that builds the KV cache from your
   prompt before the first token — is compute-bound and scales with the number of input tokens. Fewer tokens
   in, lower time-to-first-token. This bites hardest exactly where on-device and self-hosted agents live, where
   you are bound by your own compute rather than hiding behind a provider's fleet.

## The evidence: this is not a vibe

Every claim above is in the literature. Don't take the assertions on faith; the papers are linked.

**Irrelevant context degrades output.** Shi et al., *["Large Language Models Can Be Easily Distracted by
Irrelevant Context"](https://arxiv.org/abs/2302.00093)* (ICML 2023): model performance drops sharply when
irrelevant information is added to an otherwise-sufficient prompt. This is the foundational result behind "don't
drag unrelated elements in."

**More context is not better; position matters.** Liu et al., *["Lost in the Middle: How Language Models Use
Long Contexts"](https://arxiv.org/abs/2307.03172)* (TACL 2024): a U-shaped curve where relevant information in
the *middle* of a long context is used worst, and performance degrades as the context grows — even for models
built for long context. Filling the window does not help the model find the answer; it buries it.

::: danger A model doesn't *want* to hallucinate — bad input makes it inevitable
Hallucination is not a malfunction you can prompt your way out of. It has a *theoretical lower bound*: a
calibrated language model fit to a distribution **must** hallucinate at some rate, independent of architecture
(Kalai & Vempala, *["Calibrated Language Models Must Hallucinate"](https://arxiv.org/abs/2311.14648)*; Kalai et
al., *["Why Language Models Hallucinate"](https://arxiv.org/abs/2509.04664)*, 2025). Feed a model the wrong or
irrelevant context and confident errors follow *by construction* — the attention mechanism aggregates every
token indiscriminately and cannot tell signal from distractor. You cannot beg this away. What you *can* control
is the input, and the lever that matters is **input quality, not the temperature knob**: prompt and input design
dominate factuality while temperature barely moves accuracy across the studied range (Renze & Guven, *["The
Effect of Sampling Temperature on Problem Solving"](https://arxiv.org/abs/2402.05201)*). Token-thrift is
hallucination mitigation you can actually action: fix what you send, not how hot you sample.
:::

## Subtractive, not additive

Keep your full record wherever you keep it. Build the window *per dispatch*: load everything reasonable into a
large working set — messages, relevant memories, standing instructions, prior thoughts, tool results — then
**focus it down** to the slice this call needs. The working set can be huge; the dispatched set is small and
on-target. You start wide and subtract, rather than grow toward a cliff and panic-compact at the edge. Nothing
is lost — it is still in the record; you simply did not ship it on this call.

**How you hydrate the working set is your choice.** The ADK does not impose a strategy. The subtractive pass —
the one [the showcase](/showcase/punching-above-its-weights#the-subtractive-pass-shrink-to-fit-every-dispatch)
builds, and shipped as a Context Battery via [`subtractToFit`](https://adk.nht.io/api/@nhtio/adk/batteries/context/thrift/subtractive_pass/functions/subtractToFit) (fit-to-budget shedding) and
[`selectRelevantTurns`](https://adk.nht.io/api/@nhtio/adk/batteries/context/thrift/relevance/functions/selectRelevantTurns) / [`scaledRelevanceFloor`](https://adk.nht.io/api/@nhtio/adk/batteries/context/thrift/relevance/functions/scaledRelevanceFloor) (relevance-scored turn selection) — is how *we* do it: one
effective, efficient way to get dynamic, per-model dispatches, built on ADK primitives, recommended but not
mandated. The mechanism that makes it natural is the pipeline shape: you get trim points before the model ever
sees a token (a per-turn input pipeline, and a per-dispatch input pipeline that re-runs every iteration),
operating on mutable working-set collections where every primitive has a `mutate`/`delete` seam. "Trim and
mutate before dispatch" is the literal API, not a metaphor.

### Cut by evidence, not by guesswork

When a narrowed dispatch still overflows, the battery throws `E_*_CONTEXT_OVERFLOW` carrying a **per-bucket token
breakdown**. Your error path reads which bucket is heavy and sheds *that* — the lowest-reranked retrievable tail,
a fat tool result demoted to a handle, the oldest turns — then retries deliberately. You cut by measured
evidence. Contrast the failure mode to avoid: **unverifiable, whole-window compaction**, a weak model rewriting
the entire transcript in one pass where a flipped negation or a dropped date is undetectable. Dropping is lossy
but *predictable*; blind summarizing hands a weak model one more chance to lie.

## A dense summary is *better* input — not a lossy compromise

Summarization is not the enemy; *unverifiable*
summarization of *truth-bearing records* is. A token-dense summary that preserves the facts and intent of a
primitive is genuinely **better input** than a human's verbose rambling or a model's chain-of-thought wandering
— fewer distractor tokens, higher signal density. The research is direct: Xu, Shi & Choi,
*["RECOMP"](https://arxiv.org/abs/2310.04408)* (ICLR 2024), summarize retrieved documents before injection at
roughly a tenth of the tokens with minimal quality loss, and report it *"relieves the burden of LMs to identify
relevant information"*; Jiang et al., *["LLMLingua"](https://arxiv.org/abs/2310.05736)* (EMNLP 2023), compress
prompts up to 20× with compressed prompts sometimes *matching or beating* the verbose originals on reasoning. In
plain terms: a good summary of something you can re-derive isn't a worse prompt, it's often a better one.

So *subtractive* does not mean *only deletion*. Between drop and keep-whole sits **mutation**: compact an old
[Thought](/the-loop/primitives/thought) to its intent and conclusions, demote a fat artifact to a queryable
handle, trim a message. The boundary that keeps it safe: summarize the things you *control and can re-derive* —
reasoning, verbose prose, retrieved corpus — where a dense rewrite is a net gain; never blind-rewrite the
**truth-bearing records** (what the user actually said, exact figures, tool outputs of record) where an
unverifiable flip is unrecoverable. Dense summary of supporting context: good. Mangling the record: the failure
mode.

## When NOT to thrift

Token-thrift is **relevance density, not minimalism for its own sake**, and it is not right for every workload.
The sharpest counterexample is a **coding agent**: it wants every token it can get, because holding as much of
the *relevant* codebase in context as possible is the job — call sites, type definitions, neighboring modules,
the failing test. There our recommendation inverts: a big window kept deliberately full of high-relevance code,
not an aggressive subtract. The principle still holds (it is *relevant* code you want, not random files —
irrelevant context still hurts), but the policy inverts: maximize relevant coverage rather than minimize token
count, and reach for a large-window model on purpose.

The rule: **thrift hardest when the signal is sparse and scattered across a long history** (chat,
RAG Q\&A, tool-heavy turns); **thrift least when the task needs broad, dense, simultaneous coverage** (code
comprehension, long-document synthesis, multi-file refactors). Same primitives, opposite dial. Don't cargo-cult
the subtractive pass onto a workload that needs breadth — that is how you starve a coding agent and then blame
the model.

::: tip Should you even build a coding agent?
Our answer: no — unless coding tooling is your core business. The existing coding agents are mature, actively
maintained, and have absorbed an enormous amount of work you would be re-spending from zero. They are also
increasingly **portable**, converging on open protocols instead of locking you in — the Agent Client Protocol
(ACP) on the editor↔agent side, Model Context Protocol (MCP) on the tool side — so switching agents no longer
means switching editors or tools. Our bias is to lean on those and spend ADK effort where you actually have an
edge — your domain, your data, your workflow. The ADK is for building agents that are *your product*, not for
re-deriving a category that is already well served.
:::

## Where to go next

* [Punching Above Its Weights](/showcase/punching-above-its-weights) — the live demo: Gemma-4 E2B in a browser
  tab with a window you drag, [the subtractive pass on real weights](/showcase/punching-above-its-weights#the-subtractive-pass-shrink-to-fit-every-dispatch),
  and the build's field notes. Start from [see it: one model, a window you control](/showcase/punching-above-its-weights#see-it-one-model-a-window-you-control).
* [Behavioral Rails](/the-loop/behavioral-rails) — the gate cascade and the planner book-end: what keeps a small
  model on-contract once the window itself is right.
* [Read the Wire](/the-loop/read-the-wire) — how to debug it when a small model still goes sideways: read the
  wire before you theorize.
* [Budgets](/the-loop/budgets) — the primitives this is built on: `Tokenizable.estimateTokens`, the
  `SpooledArtifact` range API, `ToolCall.inline`, and the overflow exception's per-bucket breakdown.
* [Thought](/the-loop/primitives/thought) — why reasoning is a high-leverage compaction target, and the
  separate-authority reasons it is its own primitive.
* [Context Batteries](/batteries/context/) — the subtractive pass and relevance selection as a shipped API:
  `subtractToFit`, `selectRelevantTurns`, `scaledRelevanceFloor`, and friends.