Transformers.js

This is the only battery in the section that runs the same code in Node and the browser. ONNX Runtime auto-selects its backend — onnxruntime-node (native, CPU, no GPU required) in Node, onnxruntime-web (WASM + WebGPU) in the browser — and the battery does not gate on navigator.gpu. isAvailable() returns true whenever the @huggingface/transformers peer imports, full stop. That env-neutrality is the headline: the agent you test in Node CI is the agent that runs in the user's tab.

import { TransformersJsAdapter } from '@nhtio/adk/batteries/llm/transformers_js'

const executor = new TransformersJsAdapter({
  model: 'onnx-community/gemma-4-E2B-it-ONNX',
  dtype: 'q4',
})

It is text-out, like LiteRT-LM: the model emits a string, and tool calls + reasoning are parsed back out by the shared chat_common layer (toolCallParser / reasoningParser, both default 'auto' — try-all, first confident match wins). Streaming v1 reports prose deltas live, then stops streaming prose the moment a tool-call or reasoning start-marker appears; the authoritative clean message is persisted after generation completes (so the persisted message is always correct — only live prose preceding a tool call is truncated).

Tested models

The broadest table in this section, because env-neutral Node execution makes it cheap to run real weights in CI. Every row is a real matrix entry; every quirk traces to a real generation, not a guess.

Model	Parser	Notes (the real quirk)
Gemma 4 E2B (onnx-community/gemma-4-E2B-it-ONNX, q4)	gemma	Text + image + audio + the combined image-and-audio turn. Emits decoder-stripped call:NAME{k:v} at runtime, not the template's <|tool_call>.
Qwen3 0.6B (q4f16)	hermes	Reasons before the call; at a tight budget the <think> eats the tokens and the call truncates mid-JSON — needs ~320-token budget so the hermes JSON completes.
DeepSeek-R1-Distill-Qwen 1.5B (q4)	think-tag	Reasoning-only; q4f16 crashes this graph (ONNX Cast-node error) → use q4. May spend its whole budget reasoning without concluding.
Llama 3.2 1B (q4f16)	auto→llama3_json	Emits bare {"name","parameters"} JSON (not pythonic) — the auto driver claims it.
SmolLM2 360M (q4)	—	Text/streaming baseline; no tool/reasoning assumptions.
Phi-4-mini (q4)	phi	The q4 quant declines tool calls — text baseline only here; the phi parser (functools[...]) is proven via the capture tier.
Granite 4.0 350M	hermes	Emits Hermes-style <tool_call>{json}</tool_call> — no Granite-specific parser needed.
Qwen2.5-Coder 0.5B (q4)	auto → llama3_json	Emits fenced ```json {name,arguments} ``` (not <tool_call>); the llama3_json parser un-fences and claims it. q4f16 crashes → q4.
Nemotron 3 Nano 4B	qwen3_coder	Mamba-hybrid LOAD-PROBE; emits qwen3_coder XML + a stray </think> (orphan-recovered).
Qwen2.5-VL 3B (image, q4f16)	—	PROBE: q4f16 VL preprocessing throws a Tensor size != data-length error upstream in transformers.js — a run failure here is DATA, not our bug.

Embeddings (separate transformers_js_embed battery, same runtime): all-MiniLM-L6-v2, BGE small en v1.5, Snowflake Arctic embed S (asymmetric query/document prefixes), and EmbeddingGemma 300M — all verified unit-norm + deterministic.

Multimodal input

"Text-out" is about output structure, not input. transformers.js genuinely perceives image and audio: a multimodal model's processor is called positionally as _call(text, images, audio), and the model consumes them. Gemma-4-E2B transcribes speech, names an image's dominant colour, and does both in a single turn.

One env-neutrality detail worth knowing: transformers.js's own read_audio decodes via the Web Audio API (AudioContext), which does not exist in Node. So for the common case — uncompressed PCM WAV at 16 kHz, exactly what speech models want — the battery decodes the RIFF itself with a DataView, dependency-free, in Node and the browser alike, before it ever imports the heavy peer. Compressed containers (mp3/flac/ogg) fall back to read_audio (browser-only, by that path's nature).

Media output

Media output is wired but opt-in. By default the battery is text-out (the tested chat checkpoints emit text), so it attaches nothing — byte-for-byte unchanged. Supply an extractMediaOutputs hook and a wrapped media-emitting model's generated audio/image is surfaced as an assistant Message.attachments entry:

const executor = new TransformersJsAdapter({
  model: 'onnx-community/Llama-3.2-1B-Instruct-q4f16',
  // Turn the model's text answer into spoken audio and attach it to the turn.
  extractMediaOutputs: async (rawResult) => {
    const wavBytes = await synthesizeSpeechFromResult(rawResult) // your TTS
    return [{ kind: 'audio', mimeType: 'audio/wav', bytes: wavBytes }]
  },
})

The adapter persists each output via ctx.storeMediaBytes, builds a first-party Media.toolGenerated(...), and attaches it to the assistant message (a media-only turn — empty text + attachment — is legitimate). The hook receives the raw generation result; what it does with it is yours. This is the same seam LiteRT-LM exposes. (A dedicated text-to-audio/image generation battery — emitting media as the model's own task — is a separate, future thing; this hook is for an LLM turn that produces media alongside or instead of text.)

Call dispose() in long-lived browser sessions

reset() only nulls the cached pipeline reference — it does NOT free the underlying ONNX Runtime sessions or the WebGPU/wasm device memory they hold. Loading many models back-to-back in one browser session (a model picker, a test matrix) without disposing accumulates those sessions until the heap is exhausted, surfacing as Can't create a session … memory copy. await adapter.dispose() releases the model's ONNX sessions; the next dispatch re-resolves a fresh pipeline. This is exactly the bug the build showcase recounts hitting in the WebGPU matrix.

Errors

Typed E_TRANSFORMERS_JS_* family — context overflow, stream error, invalid tool-call args, tool-parse failed — plus E_INVALID_TRANSFORMERS_JS_OPTIONS and E_UNSUPPORTED_MEDIA_MODALITY. Full option + exception detail in Assembly → LLM batteries.