Image Captioning

TransformersJsCaptionAdapter describes what's in an image — a plain-language caption, not glyphs read off it (that's OCR's job) — using transformers.js's image-to-text pipeline. Construct it once, call TransformersJsCaptionAdapter.describe per image.

import { TransformersJsCaptionAdapter } from '@nhtio/adk/batteries/specialists/caption/transformers_js'

const caption = new TransformersJsCaptionAdapter({ model: 'Xenova/vit-gpt2-image-captioning' })

const { text } = await caption.describe({ bytes: pngBytes, mimeType: 'image/png' })
console.log(text) // e.g. "a red square on a white background"

await caption.dispose()

Input forms

Same three shapes as the other two specialists — a bare Uint8Array, { bytes, mimeType? }, or a Media-like value ({ mimeType, asBytes() }). The adapter normalizes whichever you pass to bytes, then wraps them in a plain Blob.

Why Blob and not RawImage

transformers.js's own RawImage class is the "native" way to hand it an image, but constructing one requires importing the peer up front. A Blob is a cross-environment global (Node 18+, every browser) that the image-to-text pipeline's ImageInput union accepts directly — verified against the installed @huggingface/transformers 4.2.0 type declarations. Building the input from a Blob keeps this adapter's hot path peer-free until the pipeline itself is resolved, and means a fake-pipeline unit test never needs to load @huggingface/transformers at all.

Method and options

async describe(input: SpecialistImageInput, opts?: DescribeOptions): Promise<DescribeResult>

DescribeOptions.maxNewTokens?: number is forwarded to the pipeline as max_new_tokens (omitted entirely when unset, so the model's own default budget applies). DescribeResult is { text: string }.

An empty caption is an engine error, not a valid result

If the pipeline returns nothing usable — missing or empty generated_text, in any of the single-result, flat-batch, or nested-batch shapes the pipeline can return — describe throws E_TRANSFORMERS_JS_CAPTION_ENGINE_ERROR rather than resolving with { text: '' }. A captioner that produces no text didn't do its job; that's a failure to surface, not a result to hand your downstream model.

Constructor options (TransformersJsCaptionAdapterOptions):

Option	Type	Default	Notes
model	string	— (required)	The image-to-text model id. No default; the documented reference model is Xenova/vit-gpt2-image-captioning.
pipeline	TransformersJsCaptionPipeline	—	A pre-built pipeline; when set, skips lazy creation.
createPipeline	CreateTransformersJsCaptionPipeline	dynamic import + pipeline('image-to-text', …)	Override the pipeline factory.
device	'auto' | 'webgpu' | 'wasm' | 'cpu' | 'gpu' | …	environment default	Forwarded to pipeline().
dtype	'auto' | 'fp32' | 'fp16' | 'q8' | 'q4' | …	environment default	Quantization/precision. See the browser note below.
modelSource	TransformersJsCaptionModelSource	HuggingFace Hub	Serve model files from OPFS, a bundled source, or elsewhere.
onInitProgress	ProgressCallback	—	Raw model-load progress reports.
isAvailable	() => boolean	true	Override the availability probe.
lifecycle hooks	BatteryLifecycleHooks	—	onLifecycle/onLoading/onCompiling/onReady/onGenerating/onComplete/onError.

Lifecycle

preload() eagerly loads and caches the pipeline (idempotent, single-flight). reset() drops the cached JS reference. dispose() awaits the pipeline's own dispose() — freeing ONNX sessions and any GPU/wasm buffers — then calls reset(). Same trade-off as the STT battery: don't skip dispose() in a long-lived browser session loading multiple captioners.

Browser: pin dtype: 'fp32'

The same session-creation failure that affects Whisper in the browser applies here — pin dtype: 'fp32' for vit-gpt2-sized captioning models:

const caption = new TransformersJsCaptionAdapter({
  model: 'Xenova/vit-gpt2-image-captioning',
  dtype: 'fp32' as never,
})

Model choice, and what's out of scope

Xenova/vit-gpt2-image-captioning is the documented reference model — a small ViT-encoder/GPT2-decoder captioner, around 85MB at 8-bit quantization (larger, and fp32-only in practice for the browser per the note above). It produces short, generic captions — "a red square," "a dog running in a field" — which is exactly the single-purpose shape this adapter's describe(): { text } return commits to.

Florence-2 and other unified vision models are deliberately out of scope

Florence-2-class models handle captioning as one of many tasks selected by a prefix string in the prompt (<CAPTION>, <DETAILED_CAPTION>, <OD> for object detection, …), and their outputs range from plain text to structured bounding-box records depending on which task you asked for. That doesn't fit a single-purpose describe(image) → { text } contract — modeling it honestly would mean either a much wider options surface tailored to one model family, or silently discarding the structured outputs those models are actually good at. Neither is this adapter's job. If you need a Florence-2-class model, use the createPipeline override to wire one in directly (you own the task-prefix + output-shape logic), or reach for the Transformers.js LLM battery's multimodal input path if a full multimodal chat model is a better fit for what you're building.

Errors

E_INVALID_TRANSFORMERS_JS_CAPTION_OPTIONS (fatal, bad constructor options) and E_TRANSFORMERS_JS_CAPTION_ENGINE_ERROR (non-fatal — pipeline load/runtime failure, or an empty/missing caption).

A complete example

import { readFile } from 'node:fs/promises'
import { TransformersJsCaptionAdapter } from '@nhtio/adk/batteries/specialists/caption/transformers_js'

const caption = new TransformersJsCaptionAdapter({ model: 'Xenova/vit-gpt2-image-captioning' })

try {
  const bytes = new Uint8Array(await readFile('photo.png'))
  const { text } = await caption.describe({ bytes, mimeType: 'image/png' }, { maxNewTokens: 32 })
  console.log(text)
} finally {
  await caption.dispose()
}

Where to go next

• Specialist batteries — overview — the thesis, the three adapters at a glance, and how to wire one into an agent as a tool.
• STT and OCR — the other two specialists.
• Bring your own tools — the pattern for wrapping this adapter in a tool.
• Assembly → Specialist batteries — the full option/exception surface.