Speech-to-Text (STT)

TransformersJsSttAdapter turns spoken audio into text with a Whisper-family model, running on-device through @huggingface/transformers. Construct it once, call TransformersJsSttAdapter.transcribe as many times as you have audio clips. There is no network hop and no API key — the whole job runs against local compute, in Node (onnxruntime-node) or the browser (onnxruntime-web, WebGPU when available), auto-selected the same way the Transformers.js LLM battery selects its backend.

import { TransformersJsSttAdapter } from '@nhtio/adk/batteries/specialists/stt/transformers_js'

const stt = new TransformersJsSttAdapter({ model: 'onnx-community/whisper-base' })

const { text } = await stt.transcribe({ bytes: wavBytes, mimeType: 'audio/wav' })
console.log(text) // "The quick brown fox jumps over the lazy dog."

await stt.dispose() // release the ONNX session when you're done with this adapter

Input forms

transcribe accepts audio in whichever of these shapes is convenient:

• Already-decoded PCM — { pcm: Float32Array, sampleRate: number }. Skips container decoding entirely; useful if you're capturing raw samples from a microphone stream.
• Raw bytes — a bare Uint8Array (MIME type unknown).
• Bytes + MIME — { bytes: Uint8Array, mimeType?: string }.
• A Media-like value — anything with { mimeType: string, asBytes(): Promise<Uint8Array> }. A real @nhtio/adk Media instance satisfies this structurally, so you can hand one straight through without this battery ever importing the class.

Whichever form you pass, the adapter normalizes it to 16 kHz mono PCM — the shape Whisper-family models expect — before the pipeline call.

Why 16 kHz mono, specifically

Whisper's own preprocessing was trained against 16 kHz mono. Passing pre-decoded PCM at a different sample rate (say, 44.1 kHz from a microphone capture) is fine — resampleTo (a plain linear-interpolation resample, no peer dependency) brings it to 16 kHz, and is a no-op when the input is already at that rate. Encoded containers (WAV/MP3/FLAC/…) go through an injectable DecodeAudioFn first — default: a lazy audio-decode peer import plus a downmix to mono — then the same resample.

Method and options

async transcribe(input: SpecialistAudioInput, opts?: TranscribeOptions): Promise<TranscribeResult>

Option	Type	Default	Effect
language	string	auto-detect	Named source language (e.g. 'english') improves accuracy when known.
translate	boolean	false	Translate the recognized speech to English instead of transcribing verbatim.
timestamps	boolean	false	Populate TranscribeResult.segments with per-chunk start/end times.

TranscribeResult is { text: string, segments?: SttSegment[] }. Each SttSegment is { start: number, end: number | null, text: string } — end is null when the engine can't bound the chunk.

const { text, segments } = await stt.transcribe(
  { bytes: wavBytes, mimeType: 'audio/wav' },
  { timestamps: true }
)
for (const seg of segments ?? []) {
  console.log(`[${seg.start.toFixed(1)}s] ${seg.text}`)
}

Constructor options (TransformersJsSttAdapterOptions):

Option	Type	Default	Notes
model	string	— (required)	The Whisper (or compatible) ASR model id, e.g. onnx-community/whisper-base. No default — Whisper-family models are multi-hundred-megabyte downloads and this battery never triggers one silently.
pipeline	TransformersJsSttPipeline	—	A pre-built pipeline; when set, skips lazy creation entirely.
createPipeline	CreateTransformersJsSttPipeline	dynamic import + pipeline('automatic-speech-recognition', …)	Override the pipeline factory (test doubles, custom loading).
device	'auto' | 'webgpu' | 'wasm' | 'cpu' | 'gpu' | …	environment default	Forwarded to pipeline().
dtype	'auto' | 'fp32' | 'fp16' | 'q8' | 'q4' | …	environment default	Quantization/precision. See the browser note below.
chunkLengthS	number	30	Length of audio chunks the pipeline processes at once, in seconds.
decodeAudio	DecodeAudioFn	defaultDecodeAudio	Override the container-decode step.
modelSource	TransformersJsSttModelSource	HuggingFace Hub	Serve model files from OPFS, a bundled source, or elsewhere.
onInitProgress	ProgressCallback	—	Raw model-load progress reports.
isAvailable	() => boolean	true	Override the availability probe.
lifecycle hooks	BatteryLifecycleHooks	—	onLifecycle/onLoading/onCompiling/onReady/onGenerating/onComplete/onError — the same normalized phase machine (loading → compiling → ready → generating → complete/error) documented on the shared contract page.

Lifecycle

preload() eagerly loads and caches the pipeline (idempotent, single-flight — concurrent calls share one load). reset() drops the cached JS reference only. dispose() awaits the pipeline's own dispose() — this is what actually frees the ONNX Runtime sessions and any WebGPU/wasm device memory — then calls reset().

Call dispose() in long-lived browser sessions

Same trap as the Transformers.js LLM battery: reset() alone leaves the underlying ONNX session (and any GPU/wasm buffers) alive. Loading several Whisper models back-to-back in one browser tab without disposing accumulates sessions until memory runs out.

Browser: pin dtype: 'fp32'

The browser's auto-selected quantization fails ONNX session creation for whisper-tiny/-base with an error like Missing required scale ... DequantizeLinear (from the model's MatMulNBits weights). fp32 is the known-good cross-execution-provider combination for these small models — tiny/base sit around 150-500MB at fp32, a reasonable one-time download for a browser tab:

const stt = new TransformersJsSttAdapter({
  model: 'onnx-community/whisper-tiny.en',
  dtype: 'fp32' as never, // browser: auto-quant fails session creation for whisper
})

Errors

E_INVALID_TRANSFORMERS_JS_STT_OPTIONS (fatal, bad constructor options) and E_TRANSFORMERS_JS_STT_ENGINE_ERROR (non-fatal, pipeline load/runtime failure — including a malformed pipeline result). Both follow the same two-category exception scheme documented across every bundled battery.

A complete example

import { readFile } from 'node:fs/promises'
import { TransformersJsSttAdapter } from '@nhtio/adk/batteries/specialists/stt/transformers_js'

const stt = new TransformersJsSttAdapter({ model: 'onnx-community/whisper-base' })

try {
  const bytes = new Uint8Array(await readFile('meeting-clip.wav'))
  const { text } = await stt.transcribe({ bytes, mimeType: 'audio/wav' })
  console.log(text)
} finally {
  await stt.dispose()
}

Where to go next

• Specialist batteries — overview — the thesis, the three adapters at a glance, and how to wire one into an agent as a tool.
• OCR and Caption — the other two specialists.
• Bring your own tools — the pattern for wrapping this adapter in a transcribe_audio Tool.
• Assembly → Specialist batteries — the full option/exception surface.