The Shared Contract

The overview ends the wire-format-vs-native split on a bleak note: the two on-device batteries hand back text, and "there is no structured-output contract — there is a decoder and a hope." This page is where that hope stops being hope.

LiteRT-LM and transformers.js are two genuinely different runtimes — ONNX on one side, Google's .litertlm engine on the other — but they are not two unrelated batteries that happen to both emit text. They share a deliberate common contract, the chat_common layer, so that the parts of "run a local model turn" that have nothing to do with the runtime are written once and behave identically across both. It has three pillars: the parser layer, a portable generation vocabulary, and a lifecycle hook surface. (The wire-format batteries reuse parts of this where it makes sense — WebLLM and the embeddings battery share the lifecycle surface — but the full three-pillar contract is what makes LiteRT-LM and transformers.js interchangeable.)

Pillar 1 — the parser layer

The shared parser layer knows the real emitted formats of the small open-weight families — hermes, gemma, gpt_oss, phi, pythonic, llama3_json, mistral, qwen3_coder — plus an auto driver that tries them in a fixed precedence and takes the first confident match (the exact order is in Assembly → LLM batteries). Both toolCallParser and reasoningParser default to 'auto', so the common case needs no configuration. The parsers were not written by reading tokenizer templates — they were written against real generations, because the template tells you what the model trained on and the decoder tells you what actually comes out, and those are not the same string. (That archaeology, and the times it bit us, is the on-device build showcase.)

Pillar 2 — a portable generation vocabulary

Both runtimes have their own spelling for the same knobs: transformers.js calls the output budget maxNewTokens, LiteRT calls it maxOutputTokens; LiteRT's sampler is a numeric enum, transformers.js's is something else again. You should not have to relearn that to swap one battery for the other. So both accept a single canonical ChatGenerationOptions vocabulary — maxTokens, sampler ('greedy'|'top-k'|'top-p'), temperature, topK, topP, seed, enableThinking, and multimodal: { image, audio } — and each adapter maps it onto its own runtime API. The precedence is canonical-wins: if you set both maxTokens and the native maxNewTokens, maxTokens is honored and the native field is the fallback consulted only when the canonical one is absent. The defaults are identical across both batteries and chosen for reproducibility — sampler: 'greedy', enableThinking: false (many reasoning templates default thinking on and burn your token budget before the answer; this turns it off unless you ask). Prefer the canonical names; the native fields still work, but portable config is the point.

Pillar 3 — a lifecycle hook surface

On-device batteries spend real wall-clock time before the first token: pulling weights (a .litertlm model is ~2 GB) and then booting the WebGPU/wasm runtime — engine creation and the opaque shader/graph compilation that is often the slowest part of a cold start. The shared layer normalizes that into one opt-in BatteryLifecycleHooks surface across all of the on-device batteries: an onLifecycle firehose plus per-phase hooks (onLoading → onCompiling → onReady → onGenerating → onComplete, or onError), each handed a normalized BatteryLifecycleReport with progress in 0..1 during loading when the provider reports it. It is purely additive — omit the hooks and behavior is byte-for-byte unchanged — and a throwing hook can never abort a load or a turn. The full report shape and the per-phase semantics are in Assembly → LLM batteries.

Pillar 4 — raw-generation observability

The parser layer is the seam between what the model emitted and what the battery extracted — and on a small on-device model those two diverge more than you would like. A 2B emits a tool call in a shape no parser recognizes, the call silently falls through into the assistant prose, and the symptom you see is "the agent ignored its tools" or "it abstained on something that is clearly documented." The cause is invisible unless you can see the raw text the parser actually received.

So both on-device batteries (transformers.js and LiteRT-LM) expose an opt-in RawGenerationObserverFn via onRawGeneration. It fires once per completed generation — after envelope-token stripping and reasoning/tool-call parsing, but before anything is persisted — with the model's complete rawText, the residual cleanedText that becomes the assistant message, and the extracted reasoning / toolCalls. A call the model made but the parser declined shows up as a non-empty rawText whose toolCalls is empty and whose text is still sitting in cleanedText: the leak, made visible.

new TransformersJsAdapter({
  model: 'onnx-community/gemma-4-E2B-it-ONNX',
  onRawGeneration: ({ rawText, cleanedText, toolCalls }) => {
    if (toolCalls.length === 0 && /\bcall:|<\|tool_call>/.test(rawText)) {
      console.warn('model emitted a tool call the parser did not catch:', rawText)
    }
  },
})

Like every other hook here it is purely observational — the return value is ignored, a throwing observer can never corrupt the generation path, and omitting it is byte-for-byte unchanged. It is the supported seam for bringing a parser up against a new model, capturing ground-truth fixtures, and answering "why did it do that?" live — the job that otherwise tempts you into patching a temporary hook into the adapter.

The same leak on the native-API batteries — localToolCallParser

The on-device batteries parse tool calls out of text unconditionally, because those runtimes never return a structured call — the parser layer is how a call is found. The native-API batteries (Ollama, OpenAI-compatible Chat Completions, and WebLLM) are the other way around: the provider parses tool calls server-side and hands back a structured message.tool_calls, so those batteries trust it and do not text-parse.

That is correct until you point a small model at one of them. Gemma, Phi, and friends routinely emit a call in a surface form the provider's chat template does not recognize — <call:name{…}, a fenced ```json block, <tool_code…>, bare name\nkey: value — and when the template misses it the call lands in plain content with tool_calls empty. Same leak as Pillar 4, different runtime: the agent looks like it "ignored its tools." This is a cross-model, cross-weight reality, not one provider's quirk.

So the native-API batteries take an opt-in localToolCallParser ('auto', a family name like 'gemma', or a custom ToolCallParserFn — the exact same chat_common parser layer). It is consulted only when the provider returned zero structured calls, so native tool-calling always wins; recovered calls execute exactly like native ones and show up on onRawGeneration. Absent = disabled = today's native-only behaviour, byte-for- byte. It is the native-side mirror of the on-device toolCallParser — the difference is that on-device it is always on (there is nothing else), and on the native path it is a fallback behind the provider's own parse.

new OllamaAdapter({
  model: 'gemma4:e2b-it-qat',
  // recover a call the Ollama template did not lift into message.tool_calls
  localToolCallParser: 'gemma',
})

The parser does not get a vote on authorization

This one surprises people: a tool-call parser surfaces every call the model emits — including a call to a tool that does not exist. It does not silently drop the unknown one. That looks wrong until you see the loop it serves: the dispatch layer answers a bad call with Tool not found: <name>. Available tools: …, and the model corrects itself on the next turn. A parser that ate the unknown call would hide both the request and the feedback that fixes it. Authorization — is this tool allowed — is the consumer's decision, made in the dispatch layer where the registry actually lives, not a silent veto buried in string parsing. The parser's only job is structural: did the model ask for a tool, and which one.