Web Retrieval Glue

What this module is

This module is a deliberately thin seam between web-tool payloads and retrieval state. The three converter functions are pure: they read a normalized payload and return plain RawRetrievable objects. They do not instantiate Retrievable, SpooledArtifact, or any other core class, so importing the module has no runtime core-class coupling. Artifact types travel as caller-supplied open resolver functions.

The separate storeRetrievables helper accepts the Retrievable constructor through deps.retrievable. It stores each constructed record and returns the instance returned by the context, which may be a post-spool replacement.

Producers

SearXNG results

searxngResultsToRetrievables makes one record per normalized result, using its URL as source and clamping its score. Search snippets are short by construction, so they normally remain inline. A spool hook is nevertheless offered for consistency with the other converters and for deployments that want a uniform artifact pipeline.

Scrapper articles

scrapperArticleToRetrievable is the converter that should usually use spool: article text can be large, and a reader-backed artifact lets the model query it without placing the entire body in the prompt. See the Scrapper article example for the surrounding tool flow.

Set asMarkdown: true when the selected article content is markdown. If recommend.markdown is supplied, the converter copies that resolver to artifactConstructor, allowing storage-time auto-spooling to select the markdown artifact. If it is omitted, the record honestly has no format resolver and auto-spooling uses the base artifact; the converter cannot import a core class at runtime to guarantee otherwise.

Scrapper links

scrapperLinksToRetrievables has the same opt-in spooling behavior as the other converters. This is useful for unusually large link sets or for a pipeline that treats every web record uniformly. Because ordinary anchor text is short, the converter sets inline: true by default. That default affects rendering only: a stored record can still be auto-spooled, while its short text is materialized inline. Pass inline: false explicitly when a deliberately spooled link should render as a handle.

The spool hook contract

The hook is called only when the converter has a recommendation to offer. Its arguments are the stable record id, the complete text, and an ArtifactConstructorResolver. The resolver is an open recommendation, not a string name; call it to obtain the artifact constructor your consumer has supplied. Return a reader-backed SpooledArtifact to spool the content, or return undefined to keep the plain string. The hook is responsible for writing bytes to its own byte store and constructing the artifact. If no hook or recommendation is provided, the converter preserves plain inline-string output.

Spooling in a converter and storage-time auto-spooling compose: an already-spooled artifact is not replaced by the generic fallback. inline is independent of storage. false (the core default) means handle-mode for spooled content; true means full inline rendering.

You must wire this yourself

The SearXNG and Scrapper factories return tools; they do not call these converters or store retrieval records. An assembly must connect those steps explicitly:

const raw = scrapperArticleToRetrievable(article, {
  asMarkdown: true,
  spool: (id, text, recommended) => {
    const reader = ctx.storeRetrievableBytes(id, text)
    return new (recommended())(reader)
  },
}, { markdown: () => MarkdownArtifact })

const [stored] = await storeRetrievables(ctx, [raw], { retrievable: Retrievable })

For search, the equivalent flow is search output → searxngResultsToRetrievables → storeRetrievables. For links, use the optional third recommend argument when the link records should be spooled.

Related contracts

The core behavior of inline, storage-time auto-spooling, dynamic content, and direct content materialization is documented in Retrievable. For token fit and rendering consequences, read budgets. The vector-store adapter is documented in Retrievable Glue.