Agent Tools

The pipeline core never imports anything agent-shaped — you can use it in a build script, a worker, a CLI. The forge is the layer that knows the ADK loop: it mints Tool instances over a configured pipeline, wires media in by reference and out as first-class Media, and renders every failure as a string a model can act on.

Two surfaces, your choice

import { createMediaPipeline } from '@nhtio/adk/batteries/media'
import { forgeMediaTools } from '@nhtio/adk/batteries/media/forge'

const mp = await createMediaPipeline({ engines: [/* … */] })

const tools = forgeMediaTools(mp, { surface: 'composite' })
// → { list_media, media_query }

const runner = new TurnRunner({
  ...storageCallbacks,
  executorCallback,
  tools: Object.values(tools),
})

composite is one media_query tool whose args are { media_id, q } — the model writes pipe statements. Multi-step work costs one round-trip, the tool list stays small, and the description embeds the deployment's actual grammar: only the verbs your engines support, with few-shot examples generated from real plans via toPipe() (so they can never drift from the parser — including showing regexes in their JSON-escaped form, which is how the model must write them).

Field note: the first live model, cold

The first time a real model touched this surface — live API, no fine-tuning, nothing but the grammar in the tool description — it read the inline [media id: …] marker on the attachment and wrote redact match=/\d{3}-\d{2}-\d{4}/ replace="[SSN]" on the first attempt: regex literal, named args, quoted value, correct verb. Asked to find a document with no id in its prompt, it called list_media, read the JSON, and drove media_query with the discovered id. Asked to convert in a deployment with zero engines, it attempted once, read the do-not-retry directive, and fell back gracefully. One model through one gateway is a data point, not a proof — but the whole battery is a bet that pipes-plus-key=value is the one idiom every model already knows, and the first time the bet was actually placed, it paid. The gated live suite (forge_real_llm.node.spec.ts, TEST_OPENAI_* env) re-places it on demand.

granular mints one narrow tool per available verb — select, chunk, sheet_update_cells, and so on, with schemas generated from the same verb table. Better for small models that handle flat tools more reliably than statement composition; costs a bigger tool list and one round-trip per step. We recommend composite and we built the whole grammar so we could — but "small model, flat tools" is a real constraint, not a failure of taste, so the granular surface is a first-class citizen rather than a grudging one.

Either way, tools follow engine supply. No convert engine ⇒ no convert tool, no convert verb in the grammar text. A tool list that advertises capabilities the deployment can't deliver is lying to the model, and a model that's been lied to retries forever. The returned record is keyed by tool name, so what you got is inspectable at the call site.

How the model references media

Models can't pass bytes; they pass ids. This sounds obvious until you notice most stacks never actually show the model an id — the file goes into the context as an opaque content block and the model is left gesturing at "the PDF." Two mechanisms keep ids discoverable here:

1. Inline markers. Every LLM battery renders [media id: 01906c2e… | report.pdf] immediately before each media content block — attachments and tool results alike. The marker is harness-authored from the harness-controlled Media.id, renders outside the untrusted envelope, and carries no authority; it's a structural reference, nothing more.
2. list_media. The guaranteed fallback: enumerates every media visible in the turn (user attachments and prior tool results) with id, filename, kind, MIME, and origin.

resolveMedia turns those ids back into bytes — the default scans the turn's messages and tool calls, and you can supply your own for external media stores. The same resolution serves @id refs inside statements (merge with=@…, diff with=@…), so multi-file operations work with ids the model already has. A miss doesn't just say no — it says no and enumerates what the model could have asked for, because a bare MEDIA_NOT_FOUND invites the model to hallucinate another id and try again:

Error (MEDIA_NOT_FOUND): no media with id "doc-7".
Visible media ids: 01906c2e… (report.pdf), 01906c2f… (data.xlsx)
To create NEW media instead, pass media_id "empty:<format>" (available: xlsx, csv, png, …).

That last line only appears when creation is actually possible: media_id has one reserved namespace, empty:<format>, which creates a brand-new blank file and runs the statement against it — creation and population in one round-trip. The sentinel, the reachability rule behind what's creatable, and the defaults the model is told all live on Generating Media; from this page's perspective it's one more way a media_id resolves, and every bit of its advertisement (the description section, the schema hint, the MEDIA_NOT_FOUND exemplar above) disappears when the deployment can't create anything.

What comes back

Mutation chains return first-party Media — persisted through ctx.storeMediaBytes, trust-tiered first-party, source-tagged tool:media_query — which lands on ToolCall.results and renders natively (with its own id marker, so the model can chain operations on the result). split/extract assets return Media[]. Reads (extract text, chunk, diff, transcribe) return strings.

Every processing failure — DSL errors included — comes back as a readable Error (CODE): … string rather than a thrown tool error, because the model is expected to repair and retry: fix the verb it typo'd, quote the sheet name, pick a supported format, or stop calling a verb the deployment doesn't run (Do not retry this verb here).

The gate seam

File mutation is a side effect, and "the model redacted the wrong contract" is not an error message you want to compose for the first time in an incident channel. The gates page is blunt about whose job safety is — yours — and forgeMediaTools wires the seam in:

const tools = forgeMediaTools(mp, {
  surface: 'composite',
  gate: async (ctx, call) => {
    const verdict = await ctx.waitFor<{ approved: boolean }>({
      reason: 'tool_approval',
      payload: call, // { tool: 'media_query', args: { media_id, q } }
    })
    if (!verdict.approved) throw new Error('operator denied the operation')
  },
})

The gate runs before any bytes move; a throw aborts through the standard tool-error path (E_TOOL_DOWNSTREAM_ERROR with your denial as cause). No gate configured means zero overhead and zero behavior change — which also means that if you skip it, nothing will remind you that you skipped it. The same optional gate exists on the SearXNG and Scrapper factories — network-side-effecting tools deserve the same seam.

Overrides

overrides renames or re-describes minted tools, keyed by default name:

forgeMediaTools(mp, {
  surface: 'composite',
  overrides: { media_query: { name: 'process_file' } },
})
// → { list_media, process_file }