Recipes

Recipe: the LiteRT-LM worker pair, refit

The flagship agent's WebGPU engine already runs off the main thread — but it does so by hand, in a 582-line pair of files (litert_lm_worker.ts + litert_lm_worker_proxy.ts) that hand-roll a message protocol, a correlation scheme, and a crash-escalation ladder (gpu_loss_policy.ts's GpuLossPolicy) specific to this one engine. Here is the same shape, on this battery's substrate.

The LiteRT-LM adapter's engine-injection seam is LiteRtLmAdapterOptions.createEngine, typed as:

type CreateLiteRtLmEngine = (input: {
  engineSettings: EngineSettings
  onInitProgress?: (report: LiteRtLmInitProgressReport) => void
}) => Promise<LiteRtLmEngine>

Anything structurally assignable to that type can be handed to the adapter — including a thin facade over a forked/spawned isolated worker. Declare the spec:

// litert_isolation_spec.ts
import { defineIsolatedService, method } from '@nhtio/adk/batteries/isolation'
import type { EngineSettings } from '@litert-lm/core'

export const LiteRtWorkerService = defineIsolatedService({
  name: 'litert-engine',
  methods: {
    load: method<[settings: EngineSettings], void>(),
    generate: method<[prompt: string], string>(),
  },
})

Implement it guest-side — this is the direct replacement for litert_lm_worker.ts:

// litert_worker_guest.ts
import { serveIsolated } from '@nhtio/adk/batteries/isolation'
import { LiteRtWorkerService } from './litert_isolation_spec'

let engine: import('@litert-lm/core').Engine | undefined

serveIsolated(LiteRtWorkerService, () => ({
  load: async (settings) => {
    const { Engine } = await import('@litert-lm/core')
    engine = await Engine.create(settings)
  },
  generate: async (prompt) => {
    if (!engine) throw new Error('engine not loaded')
    return engine.generate(prompt) // illustrative — see the real Engine API for the exact call shape
  },
}))

And a host-side factory satisfying CreateLiteRtLmEngine — this replaces litert_lm_worker_proxy.ts:

// litert_isolated_engine.ts
import { spawnIsolated, createCrashPolicy } from '@nhtio/adk/batteries/isolation'
import { LiteRtWorkerService } from './litert_isolation_spec'
import type { CreateLiteRtLmEngine, LiteRtLmEngine } from '@nhtio/adk/batteries/llm/litert_lm'

export const createIsolatedLiteRtEngine: CreateLiteRtLmEngine = async ({ engineSettings }) => {
  const worker = spawnIsolated(LiteRtWorkerService, {
    worker: new URL('./litert_worker_guest.ts', import.meta.url),
    autoRespawn: { policy: createCrashPolicy() }, // same window/maxCrashes defaults as GpuLossPolicy
  })
  await worker.api.load(engineSettings)
  // Adapt the isolated facade to whatever surface LiteRtLmEngine actually exposes for generation.
  return {
    generate: (prompt: string) => worker.api.generate(prompt),
    dispose: () => worker.dispose(),
  } as unknown as LiteRtLmEngine
}

import { LiteRtLmAdapter } from '@nhtio/adk/batteries/llm/litert_lm'
import { createIsolatedLiteRtEngine } from './litert_isolated_engine'

const llm = new LiteRtLmAdapter({
  model: 'https://example.com/gemma-2b.litertlm',
  createEngine: createIsolatedLiteRtEngine,
})

autoRespawn: { policy: createCrashPolicy() } reproduces GpuLossPolicy's exact sliding-window shape (the same windowMs/maxCrashes defaults) — a wedged WebGPU device inside the worker surfaces as a crash, recycle()s a fresh worker automatically, and only gives up once the crash rate outpaces the window. See the showcase's Surviving the GPU section for the original hand-rolled version of this exact ladder.

What this buys you over the hand-rolled pair

No bespoke correlation-id scheme, no hand-written crash-detection wiring, and the crash policy is the same battery-provided createCrashPolicy every other isolated service uses — rather than a one-off GpuLossPolicy class that only this engine understands. The 582 lines of protocol plumbing collapse to a spec file, a guest file that's mostly business logic, and a thin adapter shim.

This exact refit is proven in the repo's own test suite: a CreateLiteRtLmEngine-typed factory over forkIsolated lives in tests/_fixtures/isolation/litert_refit_host_factory.ts (typed CreateLiteRtLmEngine at its own declaration site, so a type error there would mean the refit does NOT structurally fit the real adapter contract), exercised end-to-end against a LiteRT-shaped guest in tests/functional/batteries/isolation/litert_refit.node.spec.ts and, over the Web Worker transport, in tests/unit/batteries/isolation/litert_refit.browser.spec.ts.

Recipe: isolated embeddings, zero adapter changes

TransformersJsEmbeddingsAdapter (src/batteries/embeddings/transformers_js/adapter.ts) is an ordinary class: new TransformersJsEmbeddingsAdapter(options), then preload()/embed()/embedMany()/dispose(). Nothing about it knows or needs to know it's running inside a forked child — the isolation boundary is entirely outside the adapter's own code:

// embeddings_isolation_spec.ts
import { defineIsolatedService, method } from '@nhtio/adk/batteries/isolation'

export const EmbeddingsService = defineIsolatedService({
  name: 'embeddings',
  methods: {
    embed: method<[text: string], number[]>(),
    embedMany: method<[texts: string[]], number[][]>(),
  },
})

// embeddings_guest.ts
import { serveIsolated } from '@nhtio/adk/batteries/isolation'
import { TransformersJsEmbeddingsAdapter } from '@nhtio/adk/batteries/embeddings/transformers_js'
import { EmbeddingsService } from './embeddings_isolation_spec'

const adapter = new TransformersJsEmbeddingsAdapter({ model: 'Xenova/all-MiniLM-L6-v2' })

serveIsolated(EmbeddingsService, () => ({
  embed: (text) => adapter.embed(text),
  embedMany: (texts) => adapter.embedMany(texts),
}))

// embeddings_isolated_pipeline.ts — a createPipeline-shaped host factory, mirroring the specialists'
// createPipeline seams the same way the LiteRT recipe mirrors CreateLiteRtLmEngine
import { forkIsolated } from '@nhtio/adk/batteries/isolation/child_process'
import { EmbeddingsService } from './embeddings_isolation_spec'

export const createIsolatedEmbeddingsPipeline = () => {
  const service = forkIsolated(EmbeddingsService, { modulePath: './embeddings_guest.js' })
  return {
    embed: (text: string) => service.api.embed(text),
    embedMany: (texts: string[]) => service.api.embedMany(texts),
    dispose: () => service.dispose(),
  }
}

The heavy ONNX runtime, its wasm/WebGPU buffers, and every model-load cost live entirely in the child process — a crash there (an OOM kill loading a large model, a native ONNX Runtime fault) never touches the host, and service.onCrash(...)/autoRespawn apply exactly as they do for any other isolated service.

A close variant of this recipe is proven in the repo's test suite: tests/functional/batteries/isolation/embeddings_pipeline.node.spec.ts hands the REAL, unmodified TransformersJsEmbeddingsAdapter a createPipeline that forwards each call over forkIsolated to an out-of-process pipeline — the isolation boundary sits at the adapter's own public injection seam, with zero changes to the adapter.

Recipe: custom classes across the wire

Ordinary values, and the opaque containers (Date/RegExp/Map/Set/ArrayBuffer/typed arrays), cross for free at the raw codec tier. A custom class instance needs to opt in to @nhtio/encoder's round-trip protocol on BOTH sides:

// shared/money.ts
import { ENCODE_METHOD, DECODE_METHOD, registerClass } from '@nhtio/encoder'

export class Money {
  constructor(public readonly cents: number, public readonly currency: string) {}

  [ENCODE_METHOD]() {
    return { cents: this.cents, currency: this.currency }
  }

  static [DECODE_METHOD](data: { cents: number; currency: string }): Money {
    return new Money(data.cents, data.currency)
  }
}

registerClass(Money)

Import ./shared/money.ts (which self-registers via the top-level registerClass(Money) call) on BOTH the guest and host entry points, before the isolated service is constructed/served — or pass it through the encodables option instead, which is sugar over the exact same registerClass call:

forkIsolated(MySpec, {
  modulePath: './guest.js',
  encodables: [Money], // registered lazily — only touches the encoder peer when this array is non-empty
})

// guest.ts
serveIsolated(MySpec, () => ({ /* ... */ }), { encodables: [Money] })

A method whose argument or return value contains a Money instance now round-trips correctly — the codec detects it as an exotic leaf, escalates just that leaf to the encoded tier, and the far side reconstructs a real Money instance via [DECODE_METHOD].

The Error subclass caveat

Registering a class doesn't happen automatically for Error subclasses you didn't explicitly register. @nhtio/encoder's DEFAULT Error handling round-trips message/name/stack faithfully, but a thrown class MyCustomError extends Error {} that never called registerClass(MyCustomError) on both sides crosses the wire as a generic Error on arrival — instanceof MyCustomError is lost. If a guest-thrown custom error type needs to be distinguishable host-side (e.g. to branch recovery logic on err instanceof RateLimitError), register it explicitly the same way as Money above.

Recipe: observability

Every phase in the isolation lifecycle — spawn, dispose, recycle, crash, call, stream, wire traffic, and codec escalation — reports through the same IsolationObservabilityHooks shape, passed to createIsolatedService/spawnIsolated/forkIsolated (host side) and serveIsolated (guest side):

import { createCrashPolicy } from '@nhtio/adk/batteries/isolation'
import { forkIsolated } from '@nhtio/adk/batteries/isolation/child_process'
import { CounterService } from './spec'

const counter = forkIsolated(CounterService, {
  modulePath: './guest.js',
  autoRespawn: { policy: createCrashPolicy() },
  onCall: (report) => {
    if (report.phase === 'call:settle') {
      console.log(`${report.method} settled in ${report.durationMs}ms (ok=${report.ok})`)
    }
  },
  onCodecEscalate: (report) => {
    console.warn(
      `${report.argPath} escalated past the raw codec tier (${report.escalateReason}) — this call paid the @nhtio/encoder cost`
    )
  },
  onCrashReport: (report) => {
    console.error(`crash: ${report.reason} (${report.inFlight} calls in flight)`)
  },
})

onCodecEscalate (phase codec:escalate) is the "why is this call slower than expected" signal: every argument, result, or event payload that could not ship at the free raw tier fires it with the argument-relative argPath and an escalateReason ('function', 'error', 'custom-encodable', …) — a dashboard built on this hook surfaces exactly which call sites are paying the encoder tax, without needing to inspect wire bytes by hand.

onWire (phases wire:out/wire:in) is the firehose underneath that — every envelope crossing the wire, tagged with its discriminant kind and (when known) which codec tier carried it. It is deliberately verbose (every single call/result/delta), which is why it's its own opt-in hook rather than folded into onCall/onStream.

debugPayloads is a separate opt-in, on top of the hooks

Wiring onCall/onWire alone gets you metadata (method names, durations, envelope kinds, byte-size estimates) but not payload bodies. Set debugPayloads: true to additionally attach the raw payload to call:*/wire:* reports (IsolationReport.payload) — kept separate because payloads can be large or carry sensitive data, so even a caller who wants call-level tracing doesn't get bodies by default.

The firehose onIsolation hook fires on every phase (in addition to whichever per-phase-group hook also matches) — reach for it when building a single unified event stream rather than wiring each phase group individually.

Where to go next

• Browser — the Web Worker transport this recipe's LiteRT/embeddings guests can run over instead of a forked child.
• Node — forkIsolated's full option surface and the execa ChildResolver hazards.
• Assembly → Isolation batteries — the full option/exception surface for every entry point used in these recipes.