Generation — Recipes

These are complete, working shapes for the ways this domain is meant to be wired in practice. None of them require touching adapter source — every seam here is a public constructor option or a public method.

Recipe A: a BYO Tool that generates and returns Media

The adapter returns plain bytes (GeneratedMediaOutput[]); turning those into a persisted, trust-tiered Media that a ToolCall result carries is your handler's job, using the same storeMediaBytes + Media.toolGenerated pair the BYO Tools guide already documents for a locally-rendered chart:

import { Tool, Media } from '@nhtio/adk'
import { validator } from '@nhtio/validation'
import { OpenAIGenerationAdapter } from '@nhtio/adk/batteries/generation/openai'
import type { DispatchContext } from '@nhtio/adk/types'

const generator = new OpenAIGenerationAdapter({
  model: 'gpt-image-1',
  apiKey: process.env.OPENAI_API_KEY,
})

const generateImage = (ctx: DispatchContext) =>
  new Tool({
    name: 'generate_image',
    description: 'Generates an image from a text prompt.',
    inputSchema: validator.object({
      prompt: validator.string().description('What to draw').required(),
    }),
    async handler({ prompt }) {
      let outputs
      try {
        outputs = await generator.generate(prompt)
      } catch (err) {
        return `Error (GENERATION_FAILED): ${err instanceof Error ? err.message : String(err)}`
      }
      const [output] = outputs
      const reader = await ctx.storeMediaBytes(output.filename ?? 'generated.png', output.bytes)
      return Media.toolGenerated({
        kind: output.kind,
        mimeType: output.mimeType,
        filename: output.filename ?? 'generated.png',
        reader,
      })
    },
  })

ctx.storeMediaBytes(id, bytes) persists the bytes and resolves a MediaReader — it does not itself touch ctx.turnMessages/ctx.turnToolCalls, build a ToolCall, or fire any hooks. Your handler simply returns the Media instance; from there the executor (the Chat Completions battery, or whatever drives your tool loop) is what invokes the handler, takes that returned Media as the tool result, and persists the corresponding ToolCall via ctx.storeToolCall(). The Media lands on ToolCall.results unwrapped — no SpooledArtifact — exactly as byo-tools.md documents for locally-produced media. The adapter and this handler own producing the bytes and the Media; putting that result into a persisted ToolCall is the executor's job, not something storeMediaBytes or the return does on its own.

Recipe B: an edit-tool consuming an inbound Media attachment

GenerationImageInput accepts anything satisfying the GenerationMediaLike duck ({ mimeType, asBytes() }) — a real Media instance qualifies without either battery importing the other's types. That means an edit tool can resolve a Media the same way the specialists recipes resolve one for transcription, and hand it straight to adapter.edit():

import { Tool, Media } from '@nhtio/adk'
import { validator } from '@nhtio/validation'
import { GeminiGenerationAdapter } from '@nhtio/adk/batteries/generation/gemini'
import type { DispatchContext } from '@nhtio/adk/types'

const generator = new GeminiGenerationAdapter({
  model: 'gemini-2.5-flash-image',
  apiKey: process.env.GEMINI_API_KEY,
})

const resolveMedia = (ctx: DispatchContext, mediaId: string): Media | undefined => {
  for (const message of ctx.turnMessages) {
    for (const attachment of message.attachments ?? []) {
      if (attachment.id === mediaId) return attachment
    }
  }
  for (const toolCall of ctx.turnToolCalls) {
    const results = toolCall.results
    if (Media.isMedia(results) && results.id === mediaId) return results
  }
  return undefined
}

const editImage = (ctx: DispatchContext) =>
  new Tool({
    name: 'edit_image',
    description: 'Edits a previously attached image per a text instruction.',
    inputSchema: validator.object({
      media_id: validator.string().description('The id of an image attachment in this turn').required(),
      instruction: validator.string().description('What to change').required(),
    }),
    async handler({ media_id, instruction }) {
      const media = resolveMedia(ctx, media_id)
      if (!media) return `Error (MEDIA_NOT_FOUND): no image attachment with id "${media_id}" in this turn.`
      let outputs
      try {
        outputs = await generator.edit([media], instruction) // Media satisfies GenerationMediaLike directly
      } catch (err) {
        return `Error (EDIT_FAILED): ${err instanceof Error ? err.message : String(err)}`
      }
      const [output] = outputs
      const reader = await ctx.storeMediaBytes(`${media_id}-edited.png`, output.bytes)
      return Media.toolGenerated({
        kind: output.kind,
        mimeType: output.mimeType,
        filename: `${media_id}-edited.png`,
        reader,
      })
    },
  })

No adapter-side conversion is needed — generator.edit([media], instruction) passes media straight through toBytes/toGenerationBytes, which reads media.mimeType and calls media.asBytes().

Recipe C: a courtesy Media.stash caption

After generating an image, recording the prompt that produced it as a caption on the Media itself lets a later turn (or a different tool) recover why the image exists without re-reading the tool call history — the same pattern the specialists battery documents for a transcription result:

const reader = await ctx.storeMediaBytes(filename, output.bytes)
const media = Media.toolGenerated({ kind: output.kind, mimeType: output.mimeType, filename, reader })
media.stash.set('text:caption', {
  value: prompt,
  trustTier: 'first-party',
  derivedFromMedia: media.id,
})
return media

media.stash is a Registry instance, not a plain object — write entries with .set(key, entry), never by assigning or spreading stash itself. 'text:caption' is one of the default keys the LLM batteries' 'fallback-stash' UnsupportedMediaPolicy mode already walks (alongside 'text:transcript'/ 'text:description'), so a model that can't natively see images can still be told what one depicts.

Recipe D: running the on-device engine behind isolation

The transformers.js engine's ~2GB download and minutes-per-image latency are exactly what the isolation battery exists to move off your main process. The wiring mirrors the isolation battery's own embeddings recipe verbatim — construct the real, unmodified adapter inside the guest; expose a thin facade host-side:

// generation_isolation_spec.ts
import { defineIsolatedService, method } from '@nhtio/adk/batteries/isolation'
import type { GeneratedMediaOutput } from '@nhtio/adk/batteries/generation/transformers_js'

export const GenerationService = defineIsolatedService({
  name: 'generation',
  methods: {
    generate: method<[prompt: string], GeneratedMediaOutput[]>(),
  },
})

// generation_guest.ts
import { serveIsolated } from '@nhtio/adk/batteries/isolation'
import { TransformersJsGenerationAdapter } from '@nhtio/adk/batteries/generation/transformers_js'
import { GenerationService } from './generation_isolation_spec'

const adapter = new TransformersJsGenerationAdapter({ model: 'onnx-community/Janus-Pro-1B-ONNX' })

serveIsolated(GenerationService, () => ({
  generate: (prompt) => adapter.generate(prompt),
}))

// generation_isolated_pipeline.ts
import { forkIsolated } from '@nhtio/adk/batteries/isolation/child_process'
import { GenerationService } from './generation_isolation_spec'

export const createIsolatedGenerationPipeline = () => {
  const service = forkIsolated(GenerationService, { modulePath: './generation_guest.js' })
  return {
    generate: (prompt: string) => service.api.generate(prompt),
    dispose: () => service.dispose(),
  }
}

The model download, the ONNX runtime, and every byte of compute live entirely in the child process — a crash there never touches the host, and service.onCrash(...)/autoRespawn apply exactly as they do for any other isolated service. edit() is deliberately omitted from GenerationService's method surface here since this engine always throws on it — expose it only if your guest facade wraps a different engine.

Same seam, no adapter changes

As with the isolation battery's embeddings recipe, the isolation boundary sits entirely outside TransformersJsGenerationAdapter's own code — new TransformersJsGenerationAdapter(options) then .generate() is unchanged whether it runs in-process or inside a forked child.

Recipe E: live-testing against real engines

The domain's own .cross.spec.ts live specs are gated on environment variables (see .env.test.example):

Variable	Engine
TEST_GENERATION_OPENAI_API_KEY / _BASE_URL / _MODEL	OpenAI-shaped engine
TEST_GENERATION_GEMINI_API_KEY / _BASE_URL / _MODEL	Gemini engine

LB polyglot routing

Both live specs point _BASE_URL at this repo's own LLM-LB gateway rather than the vendor's API directly, and both request a Gemini-family model (gemini-2.5-flash-image) even through the OpenAI-shaped spec — the gateway translates the OpenAI-shaped request into whatever the underlying model needs. Auth in that topology goes through headers: { Authorization: 'Bearer ' + apiKey } rather than each adapter's own apiKey option (x-goog-api-key for Gemini, Authorization: Bearer natively for OpenAI) — the LB expects one uniform Bearer scheme regardless of which adapter is calling it.

The OpenAI-shaped edit() gateway gap

The OpenAI live spec only exercises generate() — probing this gateway's /v1/images/edits route returned a 404 (the gateway does not implement OpenAI-shaped image edits, only generations). The Gemini live spec exercises both generate() and edit() successfully through the same gateway. If you need edits through a gateway, verify the OpenAI-shaped route directly against your gateway before depending on it, or route edits through the Gemini engine instead.

Recipe F: a BYO Tool backed by the local-diffusion engine

The local-diffusion engine is Node-only and drives a BYO inference subprocess over a stdin/stdout line protocol, so — unlike the cloud engines — it has a real process lifecycle to manage: you spawn a backend (a Python diffusers script, DiffusionBee-style server, etc.), warm it once, and tear it down. The generate()/edit() → GeneratedMediaOutput[] contract and the storeMediaBytes + Media.toolGenerated return shape are identical to Recipe A; the only additions are lifecycle (preload/dispose), streamed per-step progress (onGenerating), and per-call cancellation.

import { Tool, Media } from '@nhtio/adk'
import { validator } from '@nhtio/validation'
import { LocalDiffusionGenerationAdapter } from '@nhtio/adk/batteries/generation/local_diffusion'
import type { DispatchContext } from '@nhtio/adk/types'

const checkpoint = '/models/v1-5-pruned-emaonly.safetensors'

// `command` and `model` are BOTH required. The default spawner runs `spawn(command, args)` and does NOT
// forward `model` into the args — pass the checkpoint to your backend yourself via `args`.
const generator = new LocalDiffusionGenerationAdapter({
  command: 'python',
  args: ['local-diffusion-backend.py', '--model', checkpoint],
  model: checkpoint,
  // Optional: fires once when a request begins (no `progress`), then once per `dnpr` frame with a
  // normalized 0..1 `progress` — hence the guard.
  onGenerating: (report) => {
    if (typeof report.progress === 'number') console.log(`step progress: ${report.progress}`)
  },
})

// Warm the backend once up front so the first tool call doesn't pay startup + rejects fast if the
// backend can't launch. Optional — the first generate() will preload on demand otherwise.
await generator.preload()

const generateLocally = (ctx: DispatchContext) =>
  new Tool({
    name: 'generate_image_local',
    description: 'Generates an image from a text prompt using a local diffusion model.',
    inputSchema: validator.object({
      prompt: validator.string().description('What to draw').required(),
    }),
    async handler({ prompt }) {
      let outputs
      try {
        // Per-call SD knobs + cancellation. `ctx.abortSignal` is the turn's AbortSignal — wiring it in
        // lets a cancelled turn abort the generation. One in-flight call at a time: a concurrent call
        // throws E_LOCAL_DIFFUSION_BUSY; an aborted call throws E_LOCAL_DIFFUSION_ABORTED.
        outputs = await generator.generate(prompt, { steps: 20, cfgScale: 7, signal: ctx.abortSignal })
      } catch (err) {
        return `Error (GENERATION_FAILED): ${err instanceof Error ? err.message : String(err)}`
      }
      const [output] = outputs
      const reader = await ctx.storeMediaBytes(output.filename ?? 'generated.png', output.bytes)
      return Media.toolGenerated({
        kind: output.kind,
        mimeType: output.mimeType,
        filename: output.filename ?? 'generated.png',
        reader,
      })
    },
  })

The adapter is single-flight (one in-flight generate/edit; a second concurrent call throws E_LOCAL_DIFFUSION_BUSY) and cancellation is best-effort — passing the turn's ctx.abortSignal writes an advisory __stop__ and rejects promptly with E_LOCAL_DIFFUSION_ABORTED. When you're done with the adapter (process shutdown, test teardown), call await generator.dispose() to gracefully stop and kill the backend; reset() is the non-terminal hard stop if you want to re-preload() later. edit() works the same way — pass a Media/GenerationImageInput exactly as in Recipe B; the input bytes are sent to the backend as canonical base64.

Bring your own backend

The ADK ships the protocol and the process driver, not an inference runtime. See the Local Diffusion page for the full wire protocol and a runnable Python reference backend — any process that speaks the protocol works.

Where to go next

• Index — the shared contract and engine matrix.
• OpenAI / Gemini / Transformers.js / Local Diffusion — full per-engine reference.
• BYO Tools — the Tool construction contract these recipes build on.
• Isolation battery — the wrapper Recipe D depends on.
• Assembly → Generation batteries — full option/exception surface, all four engines.