---
url: 'https://adk.nht.io/batteries/specialists/caption.md'
description: >-
  TransformersJsCaptionAdapter — on-device image-to-text captioning, Node and
  browser alike. Bytes/Media in, a caption out. No cloud endpoint, no API key.
---

# Image Captioning

## LLM summary — Caption specialist battery

* [`TransformersJsCaptionAdapter`](https://adk.nht.io/api/@nhtio/adk/batteries/specialists/caption/transformers_js/adapter/classes/TransformersJsCaptionAdapter) from `@nhtio/adk/batteries/specialists/caption/transformers_js`. Image-to-text captioning via `@huggingface/transformers`'s `image-to-text` pipeline (OPTIONAL peer; reference model `Xenova/vit-gpt2-image-captioning`). ENVIRONMENT-NEUTRAL — Node (`onnxruntime-node`) and browser (`onnxruntime-web`/WebGPU); `isAvailable()` is always `true`.
* Method: `async describe(input: SpecialistImageInput, opts?: DescribeOptions): Promise<DescribeResult>`. `DescribeResult = { text: string }`. `DescribeOptions.maxNewTokens?: number` forwarded as `max_new_tokens` (omitted when unset).
* `SpecialistImageInput` = `Uint8Array` | `{ bytes, mimeType? }` | duck-typed `{ mimeType, asBytes() }`.
* Image is built as a plain `Blob` (a cross-env global — Node 18+ and every browser), NOT a peer-imported `RawImage` — the `image-to-text` pipeline's `ImageInput` union accepts `Blob` directly, so the hot path stays peer-free until the pipeline itself resolves.
* An empty/missing `generated_text` from the pipeline is treated as an ENGINE FAILURE (`E_TRANSFORMERS_JS_CAPTION_ENGINE_ERROR`), not a valid empty caption — "a captioner that produces no text didn't do its job."
* `TransformersJsCaptionAdapterOptions`: `model: string` (REQUIRED, no default), `pipeline?`, `createPipeline?`, `device?`, `dtype?`, `modelSource?`, `onInitProgress?`, `isAvailable?`, plus the shared `BatteryLifecycleHooks`.
* **Out of scope**: no Florence-2 or other unified vision-task model — those want task-prefix strings, bounding-box outputs, and structured returns this single-purpose `describe(): {text}` shape doesn't fit. Reach for `createPipeline`/a custom adapter (or the LLM battery's own multimodal input, per the Transformers.js LLM page) if you need that.
* Browser note: same fp32 dtype pin as STT — auto-quant fails ONNX session creation for vit-gpt2-sized models.
* Exceptions: `E_INVALID_TRANSFORMERS_JS_CAPTION_OPTIONS` (529, fatal), `E_TRANSFORMERS_JS_CAPTION_ENGINE_ERROR` (502, non-fatal).

[`TransformersJsCaptionAdapter`](https://adk.nht.io/api/@nhtio/adk/batteries/specialists/caption/transformers_js/adapter/classes/TransformersJsCaptionAdapter) describes what's in an image — a plain-language caption, not glyphs read
off it (that's [OCR](/batteries/specialists/ocr)'s job) — using transformers.js's `image-to-text` pipeline.
Construct it once, call [`TransformersJsCaptionAdapter.describe`](https://adk.nht.io/api/@nhtio/adk/batteries/specialists/caption/transformers_js/adapter/classes/TransformersJsCaptionAdapter#describe) per image.

```ts
import { TransformersJsCaptionAdapter } from '@nhtio/adk/batteries/specialists/caption/transformers_js'

const caption = new TransformersJsCaptionAdapter({ model: 'Xenova/vit-gpt2-image-captioning' })

const { text } = await caption.describe({ bytes: pngBytes, mimeType: 'image/png' })
console.log(text) // e.g. "a red square on a white background"

await caption.dispose()
```

## Input forms

Same three shapes as the other two specialists — a bare `Uint8Array`, `{ bytes, mimeType? }`, or a
`Media`-like value (`{ mimeType, asBytes() }`). The adapter normalizes whichever you pass to bytes, then wraps
them in a plain `Blob`.

::: tip Why `Blob` and not `RawImage`
transformers.js's own `RawImage` class is the "native" way to hand it an image, but constructing one requires
importing the peer up front. A `Blob` is a cross-environment global (Node 18+, every browser) that the
`image-to-text` pipeline's `ImageInput` union accepts directly — verified against the installed
`@huggingface/transformers` 4.2.0 type declarations. Building the input from a `Blob` keeps this adapter's hot
path peer-free until the pipeline itself is resolved, and means a fake-pipeline unit test never needs to load
`@huggingface/transformers` at all.
:::

## Method and options

```ts
async describe(input: SpecialistImageInput, opts?: DescribeOptions): Promise<DescribeResult>
```

`DescribeOptions.maxNewTokens?: number` is forwarded to the pipeline as `max_new_tokens` (omitted entirely when
unset, so the model's own default budget applies). `DescribeResult` is `{ text: string }`.

::: warning An empty caption is an engine error, not a valid result
If the pipeline returns nothing usable — missing or empty `generated_text`, in any of the single-result,
flat-batch, or nested-batch shapes the pipeline can return — `describe` throws
`E_TRANSFORMERS_JS_CAPTION_ENGINE_ERROR` rather than resolving with `{ text: '' }`. A captioner that produces
no text didn't do its job; that's a failure to surface, not a result to hand your downstream model.
:::

Constructor options (`TransformersJsCaptionAdapterOptions`):

| Option | Type | Default | Notes |
| --- | --- | --- | --- |
| `model` | `string` | — (required) | The image-to-text model id. No default; the documented reference model is `Xenova/vit-gpt2-image-captioning`. |
| `pipeline` | `TransformersJsCaptionPipeline` | — | A pre-built pipeline; when set, skips lazy creation. |
| `createPipeline` | `CreateTransformersJsCaptionPipeline` | dynamic import + `pipeline('image-to-text', …)` | Override the pipeline factory. |
| `device` | `'auto' \| 'webgpu' \| 'wasm' \| 'cpu' \| 'gpu' \| …` | environment default | Forwarded to `pipeline()`. |
| `dtype` | `'auto' \| 'fp32' \| 'fp16' \| 'q8' \| 'q4' \| …` | environment default | Quantization/precision. See the browser note below. |
| `modelSource` | `TransformersJsCaptionModelSource` | HuggingFace Hub | Serve model files from OPFS, a bundled source, or elsewhere. |
| `onInitProgress` | `ProgressCallback` | — | Raw model-load progress reports. |
| `isAvailable` | `() => boolean` | `true` | Override the availability probe. |
| lifecycle hooks | `BatteryLifecycleHooks` | — | `onLifecycle`/`onLoading`/`onCompiling`/`onReady`/`onGenerating`/`onComplete`/`onError`. |

## Lifecycle

`preload()` eagerly loads and caches the pipeline (idempotent, single-flight). `reset()` drops the cached JS
reference. `dispose()` awaits the pipeline's own `dispose()` — freeing ONNX sessions and any GPU/wasm buffers —
then calls `reset()`. Same trade-off as the [STT battery](/batteries/specialists/stt#lifecycle): don't skip
`dispose()` in a long-lived browser session loading multiple captioners.

::: tip Browser: pin `dtype: 'fp32'`
The same session-creation failure that affects Whisper in the browser applies here — pin `dtype: 'fp32'` for
vit-gpt2-sized captioning models:

```ts
const caption = new TransformersJsCaptionAdapter({
  model: 'Xenova/vit-gpt2-image-captioning',
  dtype: 'fp32' as never,
})
```

:::

## Model choice, and what's out of scope

`Xenova/vit-gpt2-image-captioning` is the documented reference model — a small ViT-encoder/GPT2-decoder
captioner, around 85MB at 8-bit quantization (larger, and fp32-only in practice for the browser per the note
above). It produces short, generic captions — "a red square," "a dog running in a field" — which is exactly
the single-purpose shape this adapter's `describe(): { text }` return commits to.

::: info Florence-2 and other unified vision models are deliberately out of scope
Florence-2-class models handle captioning as one of many *tasks* selected by a prefix string in the prompt
(`<CAPTION>`, `<DETAILED_CAPTION>`, `<OD>` for object detection, …), and their outputs range from plain text to
structured bounding-box records depending on which task you asked for. That doesn't fit a single-purpose
`describe(image) → { text }` contract — modeling it honestly would mean either a much wider options surface
tailored to one model family, or silently discarding the structured outputs those models are actually good at.
Neither is this adapter's job. If you need a Florence-2-class model, use the `createPipeline` override to wire
one in directly (you own the task-prefix + output-shape logic), or reach for the
[Transformers.js LLM battery](/batteries/llm/transformers-js)'s multimodal input path if a full multimodal chat
model is a better fit for what you're building.
:::

## Errors

`E_INVALID_TRANSFORMERS_JS_CAPTION_OPTIONS` (fatal, bad constructor options) and
`E_TRANSFORMERS_JS_CAPTION_ENGINE_ERROR` (non-fatal — pipeline load/runtime failure, or an empty/missing
caption).

## A complete example

```ts
import { readFile } from 'node:fs/promises'
import { TransformersJsCaptionAdapter } from '@nhtio/adk/batteries/specialists/caption/transformers_js'

const caption = new TransformersJsCaptionAdapter({ model: 'Xenova/vit-gpt2-image-captioning' })

try {
  const bytes = new Uint8Array(await readFile('photo.png'))
  const { text } = await caption.describe({ bytes, mimeType: 'image/png' }, { maxNewTokens: 32 })
  console.log(text)
} finally {
  await caption.dispose()
}
```

## Where to go next

* [Specialist batteries — overview](/batteries/specialists/) — the thesis, the three adapters at a glance,
  and how to wire one into an agent as a tool.
* [STT](/batteries/specialists/stt) and [OCR](/batteries/specialists/ocr) — the other two specialists.
* [Bring your own tools](/assembly/byo-tools) — the pattern for wrapping this adapter in a tool.
* [Assembly → Specialist batteries](/assembly/batteries-specialists) — the full option/exception surface.