Specialist Batteries

An agent that can only read text is blind and deaf to everything else a user hands it — a voice memo, a scanned form, a photo. The obvious fix is a multimodal LLM, and that's a fine answer when you have one. But it is not the only answer, and it is not always the right one: multimodal weights are bigger, harder to run on-device, and not every model family ships one. This domain exists for the other answer — a small, narrow model that turns one modality into text, handed to whatever text-only LLM you're already running.

The thesis

1. Multimodality doesn't have to come from a multimodal LLM. A speech-to-text model, an OCR engine, and an image captioner are each far smaller and far more mature at their one job than a general-purpose multimodal chat model is. Chain one in front of a text-only LLM and the agent perceives that modality — the LLM never needs to have been trained on pixels or waveforms at all.

2. Runtime-matched. If your LLM runs in the browser, these run in that same browser tab — no separate service, no second deployment target. All three adapters are proven in Node (native ONNX / WASM Tesseract) and in a real, headed Chromium session on real WebGPU (tests/functional/batteries/specialists/specialists.webgpu.spec.ts). That's not an aspiration — it's the same cross-environment property the Transformers.js LLM battery and the embeddings batteries already commit to.

3. On-device only, ever — and not merely for data sovereignty. Every batteries domain in this section that touches the network (the LLM batteries, the Vector Storage adapters) does so because there's a real, converged wire contract worth abstracting: an OpenAI-compatible chat endpoint, a vector-database query API. The cloud speech-to-text and vision landscape has no such convergence — every vendor's STT/OCR/vision API has its own auth model, its own request shape, its own SDK, and none of them agree on enough to be worth a shared abstraction. So this domain draws the boundary at on-device, full stop: no cloud engine, no HTTP client, no API-key option, on any of the three adapters. That's also what makes this the cost-less path to a fully local agent — zero network calls, zero third-party API bills, for the multimodal half of your agent's perception.

4. No agent integration ships, deliberately. Same posture as the embeddings batteries: the adapter is the whole product. There is no SpecialistTool class, no auto-wiring into TurnRunnerConfig, no forged tool the runner mints on your behalf. You decide whether a specialist becomes a tool the model calls, a step in your own pipeline the model never sees, or a courtesy stash entry another battery reads. The recipes below cover all three.

The three adapters, at a glance

Modality	Class	Subpath	Engine	Verified model(s)
Speech-to-text	TransformersJsSttAdapter	@nhtio/adk/batteries/specialists/stt/transformers_js	@huggingface/transformers automatic-speech-recognition	onnx-community/whisper-base (Node), onnx-community/whisper-tiny.en (browser)
OCR	TesseractJsOcrAdapter	@nhtio/adk/batteries/specialists/ocr/tesseract_js	tesseract.js (WASM, no native binary)	eng language pack
Image captioning	TransformersJsCaptionAdapter	@nhtio/adk/batteries/specialists/caption/transformers_js	@huggingface/transformers image-to-text	Xenova/vit-gpt2-image-captioning

All three: construct-once, preload()/reset()/dispose() lifecycle, eager .unknown(false) option validation, static + instance isAvailable() (always true — none of these have a WebGPU or platform requirement), and the shared BatteryLifecycleHooks boot-progress contract. See each adapter's own page for its full option table, method signature, and engine-specific notes.

Agent-integration recipes

Because no agent wiring ships, here are the three ways a real deployment actually uses one of these adapters.

(a) A BYO tool over the adapter

Wrap the adapter in a Tool (the Bring your own tools pattern) so the model decides when to invoke it — e.g. the model was handed an audio attachment and chooses to transcribe it before answering. This mirrors the media battery's own forge pattern for resolving a media_id to a Media value already visible in the turn:

import { Tool, Media } from '@nhtio/adk'
import { validator } from '@nhtio/validation'
import { TransformersJsSttAdapter } from '@nhtio/adk/batteries/specialists/stt/transformers_js'
import type { DispatchContext } from '@nhtio/adk/types'

const stt = new TransformersJsSttAdapter({ model: 'onnx-community/whisper-base' })

// The same default-resolver convention `forgeMediaTools` uses: scan turn attachments, then
// prior tool-call results, for a Media with the given id.
const resolveMedia = (ctx: DispatchContext, mediaId: string): Media | undefined => {
  for (const message of ctx.turnMessages) {
    for (const attachment of message.attachments ?? []) {
      if (attachment.id === mediaId) return attachment
    }
  }
  for (const toolCall of ctx.turnToolCalls) {
    const results = toolCall.results
    if (Media.isMedia(results) && results.id === mediaId) return results
  }
  return undefined
}

const transcribeAudio = (ctx: DispatchContext) =>
  new Tool({
    name: 'transcribe_audio',
    description: 'Transcribes a previously attached audio clip to text.',
    inputSchema: validator.object({
      media_id: validator.string().description('The id of an audio attachment in this turn').required(),
    }),
    async handler({ media_id }) {
      const media = resolveMedia(ctx, media_id)
      if (!media) return `Error (MEDIA_NOT_FOUND): no audio attachment with id "${media_id}" in this turn.`
      try {
        const { text } = await stt.transcribe(media) // Media satisfies SpecialistMediaLike directly
        return text
      } catch (err) {
        return `Error (STT_FAILED): ${err instanceof Error ? err.message : String(err)}`
      }
    },
  })

The readable Error (CODE): … string convention matches the media battery's forged tools — a failure the model can read and, where sensible, react to (ask the user to re-record, try a different attachment) rather than a thrown exception that aborts the turn.

(b) Direct use outside the tool-call loop

Not every use needs the model in the loop at all. Call the adapter directly from your own dispatchInputPipeline middleware, or from app code before the turn even starts — e.g. transcribe every incoming voice message up front and hand the LLM plain text from the start, with no tool round-trip:

import { TransformersJsSttAdapter } from '@nhtio/adk/batteries/specialists/stt/transformers_js'

const stt = new TransformersJsSttAdapter({ model: 'onnx-community/whisper-base' })

async function ingestVoiceMessage(bytes: Uint8Array): Promise<string> {
  const { text } = await stt.transcribe({ bytes, mimeType: 'audio/wav' })
  return text // hand this straight to your text-only LLM as the user's message content
}

This is the right call whenever the model doesn't need to decide whether to transcribe — the answer is always yes, so skip the round-trip.

(c) A courtesy write to Media.stash

Media.stash is a free-form register (Record<string, MediaStashEntry>, no framework-reserved keys) that typically holds a text description/transcript/caption for a piece of media. The LLM batteries' own UnsupportedMediaPolicy ('fallback-stash' mode) walks a default key list — ['text:transcript', 'text:caption', 'text:description'] — and renders whatever it finds there as the model's substitute for media it can't natively consume:

import { Media } from '@nhtio/adk'
import { TransformersJsCaptionAdapter } from '@nhtio/adk/batteries/specialists/caption/transformers_js'

const caption = new TransformersJsCaptionAdapter({ model: 'Xenova/vit-gpt2-image-captioning' })

async function attachWithCaption(bytes: Uint8Array, mimeType: string): Promise<Media> {
  const { text } = await caption.describe({ bytes, mimeType })
  const media = Media.userAttachment({ kind: 'image', mimeType, filename: 'photo.png', /* … */ })
  media.stash.set('text:caption', { value: text, trustTier: 'first-party', derivedFromMedia: media.id })
  return media
}

media.stash is a Registry instance, not a plain object — it's a controlled-mutation, dot-path key-value store (deep-clone-isolated on every read/write), so entries are written with .set(key, entry), never by assigning or spreading stash itself. Do this once, when the media is first attached, and any downstream text-only battery configured with unsupportedMediaPolicy: 'fallback-stash' picks up the caption automatically — no per-turn tool call, no model decision. See the shared LLM contract for the exact UnsupportedMediaPolicy shapes.

Model choice

• STT: whisper-tiny/whisper-base are the practical browser picks — small enough for a one-time download, proven in the headed WebGPU matrix. In the browser, pin dtype: 'fp32': the auto-selected quantization fails ONNX session creation for these small Whisper graphs (Missing required scale … DequantizeLinear, from MatMulNBits weights) — fp32 is the known cross-execution-provider combination that works, at the cost of a larger download (tiny/base run ≈150-500MB at fp32 vs. much smaller quantized).
• Caption: Xenova/vit-gpt2-image-captioning — around 85MB at 8-bit quantization, or noticeably larger (fp32-only in practice for the browser, same reason as STT above) on first download. Produces short, generic captions; see the caption page for why Florence-2-class unified vision models are explicitly out of scope for this adapter's single-purpose describe(): { text } contract, and the createPipeline escape hatch if you need one anyway.
• OCR: tesseract.js language packs download on first use per language — languages has no default so a deployment never silently fetches a pack it didn't plan for.

Composition: what the pattern is for

A specialist adapter's job ends at text. What that text is for is composition: hand it to a separate, text-only model as ordinary conversation input, and that model grounds its answer on the specialist's output despite never having seen the original bytes. This isn't hypothetical — it's a proven receipt (tests/functional/batteries/specialists/specialist_compose.node.spec.ts):

• Whisper transcribes speech.wav → "The quick brown fox jumps over the lazy dog."
• Tesseract OCRs sample_ocr.png → "HELLO OCR\n123"
• A separate text-only Llama-3.2-1B, given only the transcript and asked "what animal is mentioned?", answers "Fox" — it never touched the audio; the specialist's text is the only channel it had.

This is the same pattern the on-device batteries showcase calls "senses by committee": three genuinely different models, each blind to what the others do, giving a blind-and-deaf reasoner senses it doesn't natively have. The specialists domain is the dedicated-adapter version of that same idea — narrower, purpose-built classes instead of driving the media pipeline's engines directly.

Where to go next

• Speech-to-Text — TransformersJsSttAdapter full reference.
• OCR — TesseractJsOcrAdapter full reference.
• Image Captioning — TransformersJsCaptionAdapter full reference.
• Bring your own tools — the Tool construction pattern used in recipe (a).
• Media Pipeline — the bundled multi-engine media pipeline these adapters complement (not replace) when you need broader file-format coverage than one specialist model.
• Embeddings batteries — the sibling domain this one mirrors in shape: same construct-once/no-agent-integration posture, different job.
• Assembly → Specialist batteries — the full API-surface reference.