The Real-Model Matrix

The two text-out on-device LLM batteries — transformers.js and LiteRT-LM (WebLLM, the third on-device battery, is wire-format and hands back structured calls) — inject tool definitions into the chat template and then parse the model's raw text back into structured tool calls and reasoning, because that's all a local ONNX/.litertlm model gives you. Which means the whole thing hinges on one fragile assumption: that the model emits the format we wrote the parser for. The chat template tells you what the model was trained on. It does not tell you what the decoder actually hands back at runtime — and those are not always the same string.

So we don't guess. We run the model.

The real-model matrix

tests/_fixtures/model_matrix.ts is the authoritative manifest: one entry per model, each pinning a real HF repo (ONNX) or .litertlm URL and its expected behavior. For a tool/reasoning LLM entry that's the parser family + a prompt crafted to elicit it; for an embedding entry it's a determinism/unit-norm check; for a baseline or probe entry it's just "does it load and produce output." Each entry runs the scenarios its declared capabilities satisfy — single-tool, multi-tool, reasoning-then-tool, multi-turn, streaming, no-tool, thinking-off, media — in stream and batch modes, and the runner checks the expected tool call / reasoning / multimodal grounding was extracted, dumping the raw model output on a miss, because the miss is the point.

# Node matrix (transformers.js LLM + embeddings). Downloads real weights on first run.
pnpm run test:matrix

# One family only.
TEST_MODEL_MATRIX=1 TEST_MODEL_MATRIX_ONLY=tjs-phi-4-mini pnpm run test:node

# Browser WebGPU matrix (LiteRT + transformers.js-webgpu) — LOCAL ONLY (needs a real GPU).
pnpm run test:matrix:browser

It earns its keep. The first real run caught that Gemma 4 E2B emits call:get_weather{city:Paris} — the bare, decoder-stripped inner form — and not the <|tool_call>call:…<tool_call|> wrapper its tokenizer template literally contains. The gemma parser now accepts both. We would never have found that by reading the template; the special tokens are stripped on decode. The on-device build showcase is the full account of what the matrix surfaced — the false greens, the OOM, the crashes.

The matrix is gated, and the browser half is local-only

TEST_MODEL_MATRIX is unset in normal test runs, so the matrix specs skip cleanly (they pull multi-GB weights). The browser-WebGPU matrix runs on your machine via pnpm run test:matrix:browser — shared CI runners have no GPU, so requestAdapter() returns null and a .litertlm engine can't start. A scheduled manual CI job runs the Node half only.

Why there's no convert script

You might expect a convert_model helper here. There isn't one, and that's a deliberate, evidence-backed decision — not an omission.

ONNX (transformers.js): every model in the matrix is consumed as-is from onnx-community / Xenova / Snowflake. The community ONNX exports are exactly what the battery loads; there was never a need to convert our own, so a convert wrapper would be a maintenance surface with no consumer.

LiteRT-LM (.litertlm): browser conversion is a dead end with the public toolchain, proven at the file-format level (see the LiteRT-LM page for the full account). The short version: the @litert-lm/core web runtime only runs models whose model_type is tf_lite_artisan_text_decoder — the two Gemma -web builds Google ships. The public litert-torch export_hf CLI emits tf_lite_prefill_decode instead, which the web runtime rejects (kTfLitePrefillDecode … not supported). Converting a non-Gemma model to a browser-runnable .litertlm is not possible today, so the matrix consumes the community .litertlm builds directly and a convert script would only promise something it can't deliver.

capture_tool_outputs.ts — real output for the big-only families

Some parser families have no small live entry in the matrix — the qwen3_coder, gpt_oss, and mistral formats only show up in big models that won't run on-device. Hand-written synthetic fixtures for those are a guess, and we've been burned by guesses. So instead we capture the real raw output from the hosted versions through an OpenAI-compatible proxy and bake it into committed fixtures the parser tests run against offline.

pnpm run capture:tools -- --model qwen3-coder-next --family qwen3_coder \
  --base-url "$CAPTURE_BASE_URL" --api-key "$CAPTURE_API_KEY"

pnpm run capture:tools -- --model gpt-oss --family gpt_oss --capture reasoning

No URL or key is hard-coded — pass --base-url/--api-key or set CAPTURE_BASE_URL/CAPTURE_API_KEY. This is a dev tool to refresh fixtures; the committed JSON in tests/_fixtures/captured_tool_outputs/ is what the deterministic, offline parser tests actually consume. It never runs in CI.