---
url: 'https://adk.nht.io/the-loop/read-the-wire.md'
description: >-
  Evidence-directed agent debugging. Read what was actually sent and received
  before forming a theory — the model is the only non-deterministic component;
  everything else is inspectable.
---

# Read the Wire

## LLM summary — Read the Wire

* **The doctrine**: when an agent misbehaves, read what was actually sent and received *before* forming a
  theory. The model is the one genuinely non-deterministic component in the whole stack; everything around it —
  the assembled prompt, the pipeline that built it, the parser that read the response — is inspectable code you
  can step through. Inspect it before you blame the model.
* Most "the model is dumb" symptoms are actually your context: the tool result that was in the prompt the whole
  time and got ignored; the pass that shed the exact result the model just asked for; the contradictory
  directive that made a "slow model" take 19 dispatches to do a 7-dispatch job.
* **Re-run triage**: re-run an identical failing dispatch N times. Reproduces identically every time →
  deterministic, the context itself forces the failure — go find the contradiction. Scatters across runs →
  sampling variance — wire the recovery path and move on; there's no static bug to chase. Receipts: an 8-of-8
  identical reproduction found a real bug; a 1-in-5 scatter on a different failure confirmed a recovery path was
  sufficient.
* **Bisect on the real model, not in your head.** The self-apology loop: 3 of 6 runs reproduced with accumulated
  apologies left in context; 0 of 6 reproduced once they were stripped to the newest one. That's an ablation
  result, not a guess — theorize *after* you've isolated the variable, not before.
* **Verify your instrumentation before trusting its finding.** A capped ring buffer made every turn look like
  exactly 12 iterations — nearly reported as "systematic over-iteration" before reading the uncapped wire showed
  a healthy 7/11/5/17/10 spread. If your measurement tool has a cap, you found the cap, not the phenomenon.
* **The shipped seams**: `onPromptAssembled` and `onRawGeneration` are the LLM batteries' observer hooks for
  exactly this — tap the wire in your own code instead of hand-rolling logging inside a fork of the battery.
* Deep links: [the showcase](/showcase/punching-above-its-weights) for the live build these findings came from;
  [Behavioral Rails](/the-loop/behavioral-rails) for the gates these findings produced.

Your agent just did something inexplicable — answered a question it clearly had the data for, or looped on a
tool it already called, or apologized for a mistake it didn't make. You're about to guess why. Our discipline
on this build: don't, not yet. The easy move is to theorize: "the model must not understand X," "it's probably
a prompt-wording issue," "small models just aren't reliable at this." All three might be true. None of them
earned acting on until we'd looked at **the wire** — the assembled prompt that actually went out, and the raw
generation that actually came back — because in an agent built from inspectable pipelines, the model is the
*only* piece you can't step through. Everything feeding it is your code. Read that first.

## The doctrine

The model is the one non-deterministic component in this stack, by construction — that's what sampling from a
distribution means, and no amount of temperature-lowering changes what it fundamentally is. Everything *around*
the model is not: the subtractive pass that built this **dispatch**'s context (one model call and its response —
the unit a turn is built out of), the parser that read its tool call, the gate cascade that decided whether to
ack — all of it is ordinary deterministic code with ordinary bugs, and all of it is inspectable. When a turn goes
wrong, the prior for "it's a bug in the deterministic code around the model" is much higher than the prior for
"the model is fundamentally incapable," and the deterministic code is the part you can actually fix. So inspect
the deterministic parts first. Reading the wire is how you find out which one you're looking at, instead of
theorizing about the one part that's rolling dice.

In practice, most "the model is dumb" symptoms turn out to be about the context the model was handed, not about
the model:

* **The result was in the prompt the whole time.** The model produced a correct tool call, got a correct result
  back, and then failed to use it — not because it can't reason from a tool result, but because the subtractive
  pass or a prior gate shed the exact turn that made the result legible before the next generation ran.
* **The pass shed the thing the model just asked for.** A model calls a tool, the result comes back, and the
  *next* iteration's context-shedding pass — running its own budget logic with no visibility into what the model
  just requested — evicts that exact result to make room. The model isn't confused; its own answer was taken
  away from it between dispatches.
* **The contradictory directive that looked like a slow model.** A turn that should take one planning dispatch
  and a handful of tool calls instead grinds through nineteen dispatches. Nothing about the model changed between
  the 19-dispatch run and the 7-dispatch run that followed the fix — one contradictory instruction was removed
  from the assembled context. "The model is slow at this task" and "the model is being asked to do two
  incompatible things" produce the identical external symptom. Only the wire tells them apart.

## Re-run triage: fixable vs. irreducible

The fastest way to tell a real bug from ordinary sampling variance is the cheapest experiment available: re-run
the identical failing dispatch, same context, several times, and watch what comes back. In plain terms, there are
only two outcomes: either the same context produces the same failure every time (a real bug, worth chasing), or
it produces a mix of failures and successes (bad luck on the sample, not worth chasing).

* **Reproduces identically, every time.** The failure isn't the model rolling badly — the context in front of it
  *forces* the bad output deterministically. Go find the contradiction, the missing result, the malformed
  instruction. This is a real, fixable bug, and it will still be there on run six because it isn't luck. A
  self-handle loop case reproduced identically eight times out of eight — a genuine contradiction was sitting in
  the context, findable and fixable.
* **Scatters — different failures or different successes across runs.** This is sampling variance, not a static
  defect. A leaked-envelope case re-run five times produced one leak and four clean (differently phrased) runs.
  There is no single line of context to "fix" here; the correct response is to make sure the recovery path
  handles the bad roll gracefully — which the bounded-gate design in
  [Behavioral Rails](/the-loop/behavioral-rails) already does — and move on. Chasing a deterministic root cause
  for a genuinely stochastic outcome burns time for nothing.

The rule of thumb: if you can't reproduce it on demand, you can't fix it in the deterministic code, and trying to
will just churn the prompt until the symptom happens to go away in whatever sample you're staring at.

::: tip Don't cross-reference different runs to build one theory
A theory has to be built from one run's actual inputs and outputs, not from stitching together the interesting
parts of several unrelated runs. If run 3 showed the leak and run 5 showed the recovery, that's two separate data
points about the *distribution* of outcomes — not evidence about what happened in run 3. Diagnose one run at a
time.
:::

## Bisect on the real model, not in your head

Re-run triage tells you *whether* something is deterministic. Bisection tells you *what*, when the "what" isn't
obvious from staring at one transcript. The move is the same one you'd use on any flaky test: hold everything
constant except one variable — an **ablation**, in the sense of removing exactly one thing and observing the
effect — and check whether the failure follows the variable.

The self-apology loop is the clean example. A model that apologizes for a formatting mistake, sees its own
apology land in context, and on the next turn apologizes for *that*, spiraling into an apology about an apology.
The suspect variable was "how many prior apologies are sitting in context." Bisected directly on the real model
at temperature 1.0, not reasoned about in the abstract: with three accumulated apologies left in context, 3 of 6
runs produced a fourth apology. With the same starting point but the accumulated apologies stripped down to just
the newest one, 0 of 6 reproduced — and a follow-up batch of 8 came back clean 8 of 8. That's not a plausible
theory about apology accumulation. That's a measured, ablated result, and the fix (strip stale corrective
apologies before they can compound) follows directly from it instead of from a guess about what "should" help.

The ordering matters: ablate first, theorize second. A theory formed before the ablation is just a hypothesis
wearing the costume of a finding.

## Verify the instrumentation before trusting the finding

A subtler trap: the tooling you're using to observe the agent can itself be the thing lying to you, and the false
signal it produces looks exactly like a real one.

The concrete case: a dispatch-count metric was backed by a ring buffer capped at the last twelve entries. Every
turn, no matter how long it actually ran, reported as exactly twelve iterations — because the buffer silently
dropped anything older than the cap, not because every turn coincidentally took twelve dispatches. That number
was on the verge of being written up as "systematic over-iteration: every turn burns exactly 12 dispatches,"
which would have sent the investigation looking for a bug that produces suspiciously *uniform* behavior — a much
harder (and wrong) thing to find, since no such bug existed. Reading the *uncapped* wire instead showed a healthy,
unremarkable spread — 7, 11, 5, 17, 10 — turn to turn. The uniformity was the instrument, not the agent.

If a measurement is suspiciously clean, check whether it's capped, truncated, or sampled before you trust what it
seems to be telling you. A cap doesn't announce itself; it just makes the truth invisible past the edge and
substitutes something that looks like a stable, wrong, number.

## The shipped seams

None of the above requires hand-rolled logging bolted onto a fork of a battery. The LLM batteries ship two
**observer hooks** — callbacks the harness itself calls at a fixed point, you never call them — for exactly this
purpose:

* **`onPromptAssembled`**, a [`PromptAssembledObserverFn`](https://adk.nht.io/api/@nhtio/adk/batteries/type-aliases/PromptAssembledObserverFn) — fires once per terminal generation, the instant
  assembly completes and *before* the engine dispatch, with an observation object: the rendered `preface`, the
  per-turn `messages`, and the rendered `tools` block (API batteries get a `requestBody` instead). This is the
  wire *to* the model: what the subtractive pass actually kept, in the actual shape the model will see it, not an
  approximation reconstructed after the fact.
* **`onRawGeneration`**, a [`RawGenerationObserverFn`](https://adk.nht.io/api/@nhtio/adk/batteries/type-aliases/RawGenerationObserverFn) — fires once per terminal generation, after
  envelope-stripping and reasoning/tool-call parsing but before the result is persisted, with an observation
  object: the complete `rawText`, the residual `cleanedText`, the extracted `reasoning`, and the extracted
  `toolCalls`. This is the wire *from* the model — the same text the parsers actually saw, plus what they made of
  it, before gate classification touches any of it.

Both hooks hand you a structured observation, not a bare string, so wiring either to a JSONL dump means picking
the field you want off the object. Every LLM battery accepts both hooks as constructor options — here wired on
[`LiteRtLmAdapter`](https://adk.nht.io/api/@nhtio/adk/batteries/llm/litert_lm/adapter/classes/LiteRtLmAdapter):

```ts
// A trivial one-line-per-call appender; swap in whatever sink you actually use.
declare function appendJsonl(file: string, entry: unknown): void

const adapter = new LiteRtLmAdapter({
  // ...
  onPromptAssembled: (observation) =>
    appendJsonl('dispatch.jsonl', { at: 'prompt', ...observation }),
  onRawGeneration: (observation) =>
    appendJsonl('dispatch.jsonl', { at: 'generation', ...observation }),
})
```

That JSONL file is the wire. Re-run triage is grepping it for one dispatch and replaying that exact prompt N
times. Bisection is diffing two dispatches and changing one field. Instrumentation-cap checking is asking whether
anything upstream of the dump truncated before you saw it. All three techniques on this page operate directly on
what these two hooks give you — read it before you theorize.

## Where this leads

The findings this doctrine produces don't stay findings — they become the bounded gates and own-voice nudges
covered in [Behavioral Rails](/the-loop/behavioral-rails): the false-abstention gate, the fake-citation detector,
the parroting self-echo exemption, and the mutual-exclusion rule all trace back to a wire read first and a theory
formed second. For the live build these receipts came from, see
[Punching Above Its Weights](/showcase/punching-above-its-weights) — in particular
[the gate cascade](/showcase/punching-above-its-weights#the-gate-cascade-owns-the-turn) and
[the shape of the turn](/showcase/punching-above-its-weights#the-shape-of-the-turn), which show the pipeline these
hooks tap into.