Media Pipeline

This is a featured battery

The media pipeline is large enough to warrant its own section. This page is the hub; the spokes cover the DSL, plans and errors, text/data operations, engines and dispatch, the bundled fleet, media generation, the agent glue, and the BYO runtime in depth.

We built this battery because the industry's current answer to "my agent needs to work with a PDF" is to ship your user's PDF to somebody else's servers and call it a feature. Most agent stacks give you exactly two options for media: send the bytes to an external API and hope their retention policy means what you think it means (it means what their lawyers think it means), or inline the bytes into the model's context window and watch your token budget evaporate one base64 chunk at a time. Those aren't two options. They're two ways to lose.

We are about to state an opinion

What follows is not a fact, not a benchmark, and not something we can back with receipts. It is a belief we hold strongly enough to build a battery around, and you are free to disagree with it:

Your data should stay in your application and your infrastructure unless there is genuinely no other option — or you deliberately, explicitly choose to send it out.

If you don't share that belief, nothing here punishes you for it — the engine seams let you wire an external service in. We just won't do it for you by default, and we won't pretend a default isn't a decision.

This battery is the third option: a contract that lets you process media in-process wherever a capable engine exists, and gives you the choice of where to process when one doesn't. Redact a contract, split a PDF, fix a spreadsheet, transcribe a voice note, create a workbook from nothing — the bundled engines do all of that locally, in memory, in your process. When a bundled engine can't cover a format (document conversion has no mature cross-environment equivalent — that's the honest exception, and it's why engines/soffice exists), the engine seam is the same shape whether you fill it with a local binary, a remote service, or a BYO implementation. The point isn't "everything must be local." The point is that local is the default where it's possible, and external is a decision you make out loud, at the composition site, where your code reviewer can see it — not a default we quietly made on your behalf.

What it is

One declarative plan over every media operation, with three front-ends that compile to it:

import { createMediaPipeline } from '@nhtio/adk/batteries/media'

const mp = await createMediaPipeline({
  engines: [
    // an ordered array of self-declaring engines: resolvers run eagerly at
    // construction; each engine's heavy peer loads lazily on first actual use
    () => import('@nhtio/adk/batteries/media/engines/jimp').then((m) => m.jimpEngine()),
  ],
})

// 1. The chainable builder — implementor DX, knex-style: thenable, immutable, no .execute()
const redacted = await mp(payload)
  .select({ pages: [2, 3, 4, 5] })
  .redact({ match: [/\d{3}-\d{2}-\d{4}/], replace: '[SSN]' })

// 2. The pipe DSL — the same plan, as a statement (this is what models write)
const result = await mp.query(payload, 'select pages=2-5 | redact match=/\\d{3}-\\d{2}-\\d{4}/ replace="[SSN]"')

// 3. JSON ops — the same plan, structured
const viaOps = await mp.ops(payload, [
  { verb: 'select', args: { pages: [2, 3, 4, 5] } },
  { verb: 'redact', args: { match: [{ source: '\\d{3}-\\d{2}-\\d{4}', flags: '' }], replace: '[SSN]' } },
])

A payload is { bytes, mimeType, filename }. Chains run as an @nhtio/middleware onion — one stage per step, bytes streaming between steps in memory, with your interceptors wrapping every step. That's where DLP scanning, caching, and byte limits live, if you want them. We ship the seam, not the policy, and we'll say the quiet part: if you feed untrusted files into this pipeline without a scanning interceptor, the absence of a scanner is a choice you made, not one we made for you.

The verb surface spans documents, spreadsheets, slides, images, audio, and plain text/data: convert, select/split/merge/reorder, redact, update_text, diff/apply_patch, append and the data.* path operations, extraction, chunking, the ten sheet.* ops, the eight slides.* ops, image transforms, and transcription. What's advertised always narrows to the engines you configured — a deployment with no convert engine doesn't show models a convert verb at all. Lying to a model about what it can do is how you get a retry loop; the grammar a model sees is the truth about your deployment, nothing more.

And agents don't only process files they were handed — they create them. empty:xlsx | sheet update_cells … builds a workbook from nothing and populates it in one statement; Generating Media has the whole story.

Where to go next

In reading order, each a focused ~5 minutes:

1. Plans & Self-Repair — the neutral plan IR that everything compiles to; three front-ends, one plan; round-trip guarantees; the error contract that lets models fix their own statements.
2. The Pipe DSL — the grammar models write to produce those plans, and why a human should care about a grammar they'll never type.
3. Text, Data & Patches — append, the data.* path ops, both apply_patch dialects, and exactly what redact does (and doesn't do) per format.
4. The Capability Model — convert/mutate/edit, why mutate and edit are different capabilities, declarations as plain data.
5. Engine Dispatch — ordered supply, the one selection rule, multi-hop pathfinding, selection middleware.
6. The Bundled Fleet — every bundled engine with its trade-offs stated plainly, including the two-spreadsheet-engines story.
7. Generating Media — EMPTY_MIME, the empty:<format> sentinel, and deterministic vs model-based semantic generation.
8. Agent Tools — both tool surfaces, media referencing by id, the gate seam, failure strings.
9. BYO Runtime & Custom Engines — typed options, BinaryExecutor/ScratchWorkspace, writing your own engine.