---
url: 'https://adk.nht.io/showcase/building-on-device-batteries.md'
description: >-
  The on-device LLM batteries do tool-calling, reasoning, and multimodal on
  360M–4B models in Node and the browser — no API. This is how they were built
  and tested: the parser archaeology, the false greens, the crashes, and the one
  wall we couldn't engineer around — each traced to a real run.
---

# Tool-calling on a 0.6B, in a browser tab, proven against real weights

## LLM summary — Building the on-device batteries

* ADK ships three on-device LLM batteries. transformers.js runs in Node AND the browser; WebLLM and LiteRT-LM are browser/WebGPU. Across them the tested matrix covers tool-calling, reasoning, and multimodal paths on 360M–4B open-weight models, no API (the 360M row is a text/streaming baseline, not every capability on every model). This page is the build/test narrative, grounded in `tests/_fixtures/model_matrix.ts` + real runs.
* **No structured-OUTPUT contract is the design constraint.** transformers.js + LiteRT-LM accept rich multimodal INPUT but their chat models emit plain TEXT; tool calls + reasoning are parsed back out (`chat_common` parser layer). LiteRT's `.d.ts` even types `tool_calls`/`channels` on output — the runtime never populates them. Trust types for input; trust a real run for output.
* **Parser archaeology**: you cannot trust the chat template, only a real generation. Gemma emits decoder-stripped `call:NAME{k:v}` at runtime (not the template's `<\|tool_call>`); Phi = `functools[...]`; Granite = Hermes (no new parser). Validate against the real tokenizer output.
* **Authorization isn't the parser's job**: a parser surfaces an unknown-tool call; dispatch replies "Tool not found" and the model self-corrects. Dropping it would hide the request + the loop.
* **The walls we hit (each traced to a real run)**: DeepSeek has no thinking off-switch (so we surface reasoning, never suppress); a sine-tone audio fixture passed a false-green test (fixed with real speech); `image.rgb is not a function` from a positional processor-arg bug; `read_audio` needs `AudioContext` (browser-only → we hand-decode PCM WAV); q4f16 crashes on several graphs (→ dtype downgrade); the Qwen2.5-VL upstream preprocessing bug; the WebGPU per-cell OOM (fixed with `dispose()`); and the headline — **LiteRT browser is Gemma-only**, proven at the file-format level (`tf_lite_prefill_decode` vs `tf_lite_artisan_text_decoder`), not conquerable with the public toolchain.
* **Multi-model composition**: separate specialist models (Whisper audio, Gemma vision-caption, Tesseract OCR) each turn their modality into text, feeding a text-only LLM — giving a blind/deaf reasoner senses by composition.

The industry's answer to "can a small model call tools reliably?" is to wait for a bigger model with the
capability baked in. Our answer is three batteries that do tool-calling, reasoning, and multimodal on
360-million-to-4-billion-parameter open-weight models — in a Node process and in a browser tab, with no API
and no key — and a test suite that proves it against real weights instead of vibes. A 0.6B model emitting a
parseable `get_weather` call on your CPU is not supposed to be a solved problem at this size. It is. This page
is how, and every place it went wrong on the way, because the wrong turns are where the actual engineering is.

::: tip TL;DR
On-device LLMs hand you a string, not a tool call. The whole battery is the machinery that turns that string
back into structured intent reliably enough to dispatch — parsers grounded in *real generations* (not chat
templates, which lie), a test matrix that runs every model for real and dumps the output when it's wrong, and
a set of hard-won corrections to the things that quietly pass while being completely broken. The model was
never the hard part. The hard part was finding out what the model actually does, as opposed to what its
documentation claims it does — and those are different strings.
:::

::: warning What we could NOT make work

* **LiteRT-LM in the browser runs Gemma and only Gemma.** Not a preference — a runtime allow-list plus a
  non-public export path. Converting a non-Gemma model to a browser-runnable `.litertlm` is a dead end with
  today's public toolchain. Proven below, at the file-format level.
* **q4f16 crashes several ONNX graphs** (DeepSeek-R1-Distill, Qwen2.5-Coder): ONNX Cast-node errors. The fix
  is a dtype downgrade to q4, not a code change — the crash is upstream.
* **Qwen2.5-VL at q4f16 throws during image preprocessing** (`Tensor size != data length`) — an upstream
  transformers.js bug. It's in the matrix as a probe; a failure there is recorded as data, not pretended away.

:::

## No structured-output contract: the constraint everything else serves

A cloud model behind the OpenAI wire hands you a `tool_calls` array. An on-device ONNX or `.litertlm` model
hands you a string. That is the entire difference, and it dictates the shape of these batteries: there is no
structured-output contract, so somewhere in that string is `<tool_call>{"name":...}</tool_call>` or
`call:get_weather{city:Paris}` or `functools[...]` or a `<think>` block, and the battery's job is to find it
and turn it back into intent.

"Text-out" is easy to over-read, so be precise about it. The transformers.js battery accepts genuinely rich
**input** — its Gemma-4-E2B-ONNX path perceives image and audio through the processor, transcribes speech,
names colours. (LiteRT-LM exposes the same modality flags, but its browser image path is still a probe.) What
they lack is structured **output**. And the LiteRT runtime makes the trap vivid: its
TypeScript declarations type `tool_calls` and `channels` on the output message — but a real generation never
populates them. We dumped the raw chunks from a live `gemma-4-E2B-web` run: 14 chunks, every one a
`{type:'text'}` content item, `tool_calls` and `channels` perpetually null. **Trust the types for what you
send in; trust a real run for what comes out.** The published docs lag the library, and the types
over-promise on the output side.

## Parser archaeology: the chat template is not the decoder

The seductive mistake is to read a model's tokenizer chat template, see it defines a tool-call format, and
write your parser against that. The template tells you what the model was *trained* on. It does not tell you
what the decoder actually *emits* at runtime — and the special tokens that make the difference are stripped on
decode.

The canonical example, and the one that set the rule: Gemma 4's template wraps tool calls in
`<|tool_call>call:…<tool_call|>`. The model, at runtime, emits `call:get_weather{city:Paris}` — the bare inner
form, special-token wrapper gone, scalars unquoted. A parser written against the template would decline a
valid call on every single turn. We only found it by running the model and printing what came back. The
`gemma` parser now matches the real runtime form.

Gemma wasn't a one-off. Phi-4's format is `functools[...]`, not the `<|tool_calls|>` you might
assume. Granite 4.0 turned out to emit Hermes-style `<tool_call>{json}</tool_call>` — so it needs *no new
parser at all*, just the existing `hermes` one. Each came from running the real tokenizer and reading what it
emitted. The big-only families with no small on-device model (`qwen3_coder`'s 480B XML, `gpt_oss`'s
Harmony, `mistral`'s `[TOOL_CALLS]`) get the same treatment via captured real outputs from the hosted
versions — never hand-written synthetic fixtures, because a synthetic fixture is just a guess wearing a test's
clothes.

## Authorization is not the parser's job

When a model emits a call to a tool that doesn't exist, the parser reports it anyway. It does not silently
drop the unknown call. That feels wrong until you see the loop: the dispatch
layer answers with `Tool not found: <name>. Available tools: …`, and the model corrects on the next turn. A
parser that swallowed the unknown call would hide both the request and the feedback that repairs it.
Authorization — *is this tool allowed* — belongs in the dispatch layer where the registry lives, not in a
silent veto buried in string parsing. The parser answers one question: *did the model ask for a tool, and
which.* We tightened this deliberately after an early version over-policed.

## The walls — every one a real run

Each of these passed some earlier, dumber version of the test, or failed in a way that looked like our bug and
wasn't.

**DeepSeek has no thinking off-switch.** We assumed `enableThinking: false` would silence reasoning on every
model. It doesn't: the DeepSeek-R1-Distill template has no `enable_thinking` parameter at all — it thinks
regardless. Suppressing the output would discard tokens already generated and hide what the model actually
did; the fix was to *surface* reasoning as Thoughts and only guarantee that no `<think>` markup leaks into the
visible answer.

**The audio test that passed while the model refused.** The first audio matrix entry used a 1-second 400 Hz
sine tone and asserted the prose matched `['tone','beep',…,'audio']`. It passed — green check, looked done.
Except the model had *refused*: "Please provide the audio so I can describe it." The word "audio" in the
refusal matched the assertion. A featureless stimulus plus a permissive matcher is a test that proves nothing.
The fix: a real speech fixture ("the quick brown fox…") and an assertion requiring distinctive content words a
refusal cannot contain. Now it's a true grounding test. So in our matrix, we capture the real generation
*before* writing the assertion — the alternative is testing your imagination.

**`image.rgb is not a function`.** The Gemma-4 processor is positional: `_call(text, images, audio)`. Audio
lives in the third slot. When we passed audio-only, it collapsed into the *images* slot and the image
processor choked. The fix is to fill the images slot with `null` when there's audio but no image — obvious in
hindsight, invisible until a real audio run threw.

**`read_audio` needs a browser.** transformers.js's audio helper decodes through the Web Audio API
(`AudioContext`), which doesn't exist in Node — so audio input worked in the browser and threw in Node, in a
battery whose whole selling point is running identically in both. We now decode uncompressed PCM WAV ourselves
with a `DataView`: dependency-free, env-neutral, and it returns *before* importing the heavy ONNX peer (which
matters — eagerly importing that peer hangs the browser test runner).

**The WebGPU matrix that ran out of memory.** The full browser matrix builds a fresh adapter per scenario
cell — about a dozen model loads per model in one browser session. Without freeing each, the ONNX Runtime
sessions accumulated until the heap exhausted, surfacing late as `Can't create a session … memory copy` on
models that had loaded fine minutes earlier. Proof it was cumulative and not a code bug: a failing model
passed 11/11 in isolation. The fix was a real `dispose()` that releases the model's ONNX sessions between
cells — because `reset()` only nulled the JavaScript reference and left the native memory alive. (A
type-checks-but-fails-at-runtime gap fell out of the same work: a new lifecycle phase was wired everywhere
*except* the validation schema, so the build passed `tsc` and threw at construction. Preflight is four gates,
not one.)

### The wall we couldn't engineer around: LiteRT is Gemma-only in the browser

The others were bugs to fix. This one is a fact, and the bytes settle it.

The `@litert-lm/core` web runtime runs exactly two models: `gemma-4-E2B-it-web` and `gemma-4-E4B-it-web`. We
added a real community `SmolLM2-360M.litertlm` and ran it on a real GPU: it
**loaded** — read its template, built the engine, reached generation — then died with
`Streaming kTfLitePrefillDecode models is not supported yet`. So loadability was never the wall; the
prefill/decode kernel is.

Then we tried to convert our way out. Installed the real `litert-torch` toolchain (the public CLI has no
`--web` flag, contrary to a stale internal script that claimed one), converted SmolLM2 successfully, and
inspected the output file. The discriminator is right there in the bytes: the working Gemma build is
`model_type: tf_lite_artisan_text_decoder`; the public CLI emits `tf_lite_prefill_decode` — the exact type the
web runtime rejects. "Artisan" is a non-public Google export path. Google's own Web API doc agrees: it supports
"only" those two Gemma files, "working toward" general support.

So: **you cannot put a non-Gemma model in a browser via LiteRT-LM with the public toolchain today.** Not "it's
hard" — the export CLI emits the wrong model architecture and the runtime allow-lists the rest out. For
non-Gemma on-device-in-a-tab, [transformers.js](/batteries/llm/transformers-js) (ONNX, broad family support)
is the answer.

## Multi-model composition: senses by committee

A text-only model can still "see" and "hear" — by composition. We proved a pipeline of *separate specialist
models*, each turning its modality into text for a downstream text-only LLM: Whisper transcribes audio, a
Gemma vision pass captions an image, Tesseract OCRs a document — and the reasoning model consumes the text.
Three genuinely different models in front of one, each blind to what the others do, giving a blind-and-deaf
reasoner senses it doesn't natively have. It's the multi-model counterpart to the single unified-model
multimodal path, for when you'd rather compose narrow specialists than load one model that does everything.

## How to run it yourself

The matrix is real and you can run it. `pnpm run test:matrix` drives the Node half (transformers.js + embeddings)
against real weights; `pnpm run test:matrix:browser` runs the headed WebGPU half on your own GPU. It's gated on
`TEST_MODEL_MATRIX=1` so normal test runs skip the multi-GB downloads. The mechanics — and why there's no
convert script — are on [The Real-Model Matrix](/developer-tools/model-conversion).

## What it costs

Running models on-device, tested against real weights, is a tax you pay so your users don't pay a different
one. You own the parser archaeology (what does this family *actually* emit?), the dtype roulette (which quant
crashes which graph?), the per-runtime kernel walls (LiteRT is Gemma-only and no amount of cleverness changes
that), and the memory hygiene (`dispose()` or die). The payoff is knowing exactly which model works where, and
why, instead of shipping a `models[]` array you copied from a catalog and never ran. The
small model was never the bottleneck. Finding out what it truly does, instead of trusting what its
documentation claimed, was the work — and now it's done, written down, and tested on every push.