Tool-calling on a 0.6B, in a browser tab, proven against real weights

The industry's answer to "can a small model call tools reliably?" is to wait for a bigger model with the capability baked in. Our answer is three batteries that do tool-calling, reasoning, and multimodal on 360-million-to-4-billion-parameter open-weight models — in a Node process and in a browser tab, with no API and no key — and a test suite that proves it against real weights instead of vibes. A 0.6B model emitting a parseable get_weather call on your CPU is not supposed to be a solved problem at this size. It is. This page is how, and every place it went wrong on the way, because the wrong turns are where the actual engineering is.

TL;DR

On-device LLMs hand you a string, not a tool call. The whole battery is the machinery that turns that string back into structured intent reliably enough to dispatch — parsers grounded in real generations (not chat templates, which lie), a test matrix that runs every model for real and dumps the output when it's wrong, and a set of hard-won corrections to the things that quietly pass while being completely broken. The model was never the hard part. The hard part was finding out what the model actually does, as opposed to what its documentation claims it does — and those are different strings.

What we could NOT make work

• LiteRT-LM in the browser runs Gemma and only Gemma. Not a preference — a runtime allow-list plus a non-public export path. Converting a non-Gemma model to a browser-runnable .litertlm is a dead end with today's public toolchain. Proven below, at the file-format level.
• q4f16 crashes several ONNX graphs (DeepSeek-R1-Distill, Qwen2.5-Coder): ONNX Cast-node errors. The fix is a dtype downgrade to q4, not a code change — the crash is upstream.
• Qwen2.5-VL at q4f16 throws during image preprocessing (Tensor size != data length) — an upstream transformers.js bug. It's in the matrix as a probe; a failure there is recorded as data, not pretended away.

No structured-output contract: the constraint everything else serves

A cloud model behind the OpenAI wire hands you a tool_calls array. An on-device ONNX or .litertlm model hands you a string. That is the entire difference, and it dictates the shape of these batteries: there is no structured-output contract, so somewhere in that string is <tool_call>{"name":...}</tool_call> or call:get_weather{city:Paris} or functools[...] or a <think> block, and the battery's job is to find it and turn it back into intent.

"Text-out" is easy to over-read, so be precise about it. The transformers.js battery accepts genuinely rich input — its Gemma-4-E2B-ONNX path perceives image and audio through the processor, transcribes speech, names colours. (LiteRT-LM exposes the same modality flags, but its browser image path is still a probe.) What they lack is structured output. And the LiteRT runtime makes the trap vivid: its TypeScript declarations type tool_calls and channels on the output message — but a real generation never populates them. We dumped the raw chunks from a live gemma-4-E2B-web run: 14 chunks, every one a {type:'text'} content item, tool_calls and channels perpetually null. Trust the types for what you send in; trust a real run for what comes out. The published docs lag the library, and the types over-promise on the output side.

Parser archaeology: the chat template is not the decoder

The seductive mistake is to read a model's tokenizer chat template, see it defines a tool-call format, and write your parser against that. The template tells you what the model was trained on. It does not tell you what the decoder actually emits at runtime — and the special tokens that make the difference are stripped on decode.

The canonical example, and the one that set the rule: Gemma 4's template wraps tool calls in <|tool_call>call:…<tool_call|>. The model, at runtime, emits call:get_weather{city:Paris} — the bare inner form, special-token wrapper gone, scalars unquoted. A parser written against the template would decline a valid call on every single turn. We only found it by running the model and printing what came back. The gemma parser now matches the real runtime form.

Gemma wasn't a one-off. Phi-4's format is functools[...], not the <|tool_calls|> you might assume. Granite 4.0 turned out to emit Hermes-style <tool_call>{json}</tool_call> — so it needs no new parser at all, just the existing hermes one. Each came from running the real tokenizer and reading what it emitted. The big-only families with no small on-device model (qwen3_coder's 480B XML, gpt_oss's Harmony, mistral's [TOOL_CALLS]) get the same treatment via captured real outputs from the hosted versions — never hand-written synthetic fixtures, because a synthetic fixture is just a guess wearing a test's clothes.

Authorization is not the parser's job

When a model emits a call to a tool that doesn't exist, the parser reports it anyway. It does not silently drop the unknown call. That feels wrong until you see the loop: the dispatch layer answers with Tool not found: <name>. Available tools: …, and the model corrects on the next turn. A parser that swallowed the unknown call would hide both the request and the feedback that repairs it. Authorization — is this tool allowed — belongs in the dispatch layer where the registry lives, not in a silent veto buried in string parsing. The parser answers one question: did the model ask for a tool, and which. We tightened this deliberately after an early version over-policed.

The walls — every one a real run

Each of these passed some earlier, dumber version of the test, or failed in a way that looked like our bug and wasn't.

DeepSeek has no thinking off-switch. We assumed enableThinking: false would silence reasoning on every model. It doesn't: the DeepSeek-R1-Distill template has no enable_thinking parameter at all — it thinks regardless. Suppressing the output would discard tokens already generated and hide what the model actually did; the fix was to surface reasoning as Thoughts and only guarantee that no <think> markup leaks into the visible answer.

The audio test that passed while the model refused. The first audio matrix entry used a 1-second 400 Hz sine tone and asserted the prose matched ['tone','beep',…,'audio']. It passed — green check, looked done. Except the model had refused: "Please provide the audio so I can describe it." The word "audio" in the refusal matched the assertion. A featureless stimulus plus a permissive matcher is a test that proves nothing. The fix: a real speech fixture ("the quick brown fox…") and an assertion requiring distinctive content words a refusal cannot contain. Now it's a true grounding test. So in our matrix, we capture the real generation before writing the assertion — the alternative is testing your imagination.

image.rgb is not a function. The Gemma-4 processor is positional: _call(text, images, audio). Audio lives in the third slot. When we passed audio-only, it collapsed into the images slot and the image processor choked. The fix is to fill the images slot with null when there's audio but no image — obvious in hindsight, invisible until a real audio run threw.

read_audio needs a browser. transformers.js's audio helper decodes through the Web Audio API (AudioContext), which doesn't exist in Node — so audio input worked in the browser and threw in Node, in a battery whose whole selling point is running identically in both. We now decode uncompressed PCM WAV ourselves with a DataView: dependency-free, env-neutral, and it returns before importing the heavy ONNX peer (which matters — eagerly importing that peer hangs the browser test runner).

The WebGPU matrix that ran out of memory. The full browser matrix builds a fresh adapter per scenario cell — about a dozen model loads per model in one browser session. Without freeing each, the ONNX Runtime sessions accumulated until the heap exhausted, surfacing late as Can't create a session … memory copy on models that had loaded fine minutes earlier. Proof it was cumulative and not a code bug: a failing model passed 11/11 in isolation. The fix was a real dispose() that releases the model's ONNX sessions between cells — because reset() only nulled the JavaScript reference and left the native memory alive. (A type-checks-but-fails-at-runtime gap fell out of the same work: a new lifecycle phase was wired everywhere except the validation schema, so the build passed tsc and threw at construction. Preflight is four gates, not one.)

The wall we couldn't engineer around: LiteRT is Gemma-only in the browser

The others were bugs to fix. This one is a fact, and the bytes settle it.

The @litert-lm/core web runtime runs exactly two models: gemma-4-E2B-it-web and gemma-4-E4B-it-web. We added a real community SmolLM2-360M.litertlm and ran it on a real GPU: it loaded — read its template, built the engine, reached generation — then died with Streaming kTfLitePrefillDecode models is not supported yet. So loadability was never the wall; the prefill/decode kernel is.

Then we tried to convert our way out. Installed the real litert-torch toolchain (the public CLI has no --web flag, contrary to a stale internal script that claimed one), converted SmolLM2 successfully, and inspected the output file. The discriminator is right there in the bytes: the working Gemma build is model_type: tf_lite_artisan_text_decoder; the public CLI emits tf_lite_prefill_decode — the exact type the web runtime rejects. "Artisan" is a non-public Google export path. Google's own Web API doc agrees: it supports "only" those two Gemma files, "working toward" general support.

So: you cannot put a non-Gemma model in a browser via LiteRT-LM with the public toolchain today. Not "it's hard" — the export CLI emits the wrong model architecture and the runtime allow-lists the rest out. For non-Gemma on-device-in-a-tab, transformers.js (ONNX, broad family support) is the answer.

Multi-model composition: senses by committee

A text-only model can still "see" and "hear" — by composition. We proved a pipeline of separate specialist models, each turning its modality into text for a downstream text-only LLM: Whisper transcribes audio, a Gemma vision pass captions an image, Tesseract OCRs a document — and the reasoning model consumes the text. Three genuinely different models in front of one, each blind to what the others do, giving a blind-and-deaf reasoner senses it doesn't natively have. It's the multi-model counterpart to the single unified-model multimodal path, for when you'd rather compose narrow specialists than load one model that does everything.

How to run it yourself

The matrix is real and you can run it. pnpm run test:matrix drives the Node half (transformers.js + embeddings) against real weights; pnpm run test:matrix:browser runs the headed WebGPU half on your own GPU. It's gated on TEST_MODEL_MATRIX=1 so normal test runs skip the multi-GB downloads. The mechanics — and why there's no convert script — are on The Real-Model Matrix.

What it costs

Running models on-device, tested against real weights, is a tax you pay so your users don't pay a different one. You own the parser archaeology (what does this family actually emit?), the dtype roulette (which quant crashes which graph?), the per-runtime kernel walls (LiteRT is Gemma-only and no amount of cleverness changes that), and the memory hygiene (dispose() or die). The payoff is knowing exactly which model works where, and why, instead of shipping a models[] array you copied from a catalog and never ran. The small model was never the bottleneck. Finding out what it truly does, instead of trusting what its documentation claimed, was the work — and now it's done, written down, and tested on every push.