---
url: 'https://adk.nht.io/batteries/tools/web_retrieval.md'
description: >-
  Pure converters for turning SearXNG and Scrapper payloads into Retrievable
  records, with opt-in spooling and explicit rendering controls.
---

# Web Retrieval Glue

## LLM summary — web retrieval glue

* `@nhtio/adk/batteries/tools/web_retrieval` contains pure payload-to-`RawRetrievable` converters, not `Tool` instances, and has zero runtime coupling to core classes.
* `searxngResultsToRetrievables(payload, opts?, recommend?)` converts short search snippets. It offers `spool` for API consistency, though snippets are short by construction.
* `scrapperArticleToRetrievable(article, opts?, recommend?)` is the usual candidate for spooling. With `asMarkdown: true`, `recommend.markdown` is passed through as `artifactConstructor`; the caller must supply that resolver for markdown-aware auto-spooling.
* `scrapperLinksToRetrievables(payload, opts?, recommend?)` also supports `spool`. Its `inline` value defaults to `true`; pass `inline: false` when intentionally using handles.
* `storeRetrievables(ctx, raws, { retrievable })` is the only helper that constructs core records and returns the post-store instances.
* A `spool` hook receives `(id, text, recommendedResolver)` and returns a `SpooledArtifact` or `undefined` to retain inline text. The hook owns persistence and the concrete artifact class.
* SearXNG and Scrapper tool factories never call these converters. Wire tool output to a converter and then to `ctx.storeRetrievable` yourself.
* See [Retrievable](../../the-loop/primitives/retrievable), [retrievable glue](../vector/retrievable), and [budgets](../../the-loop/budgets).

## What this module is

This module is a deliberately thin seam between web-tool payloads and retrieval state. The three converter functions are pure: they read a normalized payload and return plain `RawRetrievable` objects. They do not instantiate `Retrievable`, `SpooledArtifact`, or any other core class, so importing the module has no runtime core-class coupling. Artifact types travel as caller-supplied open resolver functions.

The separate `storeRetrievables` helper accepts the `Retrievable` constructor through `deps.retrievable`. It stores each constructed record and returns the instance returned by the context, which may be a post-spool replacement.

## Producers

### SearXNG results

`searxngResultsToRetrievables` makes one record per normalized result, using its URL as `source` and clamping its score. Search snippets are short by construction, so they normally remain inline. A `spool` hook is nevertheless offered for consistency with the other converters and for deployments that want a uniform artifact pipeline.

### Scrapper articles

`scrapperArticleToRetrievable` is the converter that should usually use `spool`: article text can be large, and a reader-backed artifact lets the model query it without placing the entire body in the prompt. See the [Scrapper article example](./scrapper#output-shape-and-artifact) for the surrounding tool flow.

Set `asMarkdown: true` when the selected article content is markdown. If `recommend.markdown` is supplied, the converter copies that resolver to `artifactConstructor`, allowing storage-time auto-spooling to select the markdown artifact. If it is omitted, the record honestly has no format resolver and auto-spooling uses the base artifact; the converter cannot import a core class at runtime to guarantee otherwise.

### Scrapper links

`scrapperLinksToRetrievables` has the same opt-in spooling behavior as the other converters. This is useful for unusually large link sets or for a pipeline that treats every web record uniformly. Because ordinary anchor text is short, the converter sets `inline: true` by default. That default affects rendering only: a stored record can still be auto-spooled, while its short text is materialized inline. Pass `inline: false` explicitly when a deliberately spooled link should render as a handle.

## The spool hook contract

The hook is called only when the converter has a recommendation to offer. Its arguments are the stable record id, the complete text, and an `ArtifactConstructorResolver`. The resolver is an open recommendation, not a string name; call it to obtain the artifact constructor your consumer has supplied. Return a reader-backed `SpooledArtifact` to spool the content, or return `undefined` to keep the plain string. The hook is responsible for writing bytes to its own byte store and constructing the artifact. If no hook or recommendation is provided, the converter preserves plain inline-string output.

Spooling in a converter and storage-time auto-spooling compose: an already-spooled artifact is not replaced by the generic fallback. `inline` is independent of storage. `false` (the core default) means handle-mode for spooled content; `true` means full inline rendering.

## You must wire this yourself

The SearXNG and Scrapper factories return tools; they do not call these converters or store retrieval records. An assembly must connect those steps explicitly:

```ts
const raw = scrapperArticleToRetrievable(article, {
  asMarkdown: true,
  spool: (id, text, recommended) => {
    const reader = ctx.storeRetrievableBytes(id, text)
    return new (recommended())(reader)
  },
}, { markdown: () => MarkdownArtifact })

const [stored] = await storeRetrievables(ctx, [raw], { retrievable: Retrievable })
```

For search, the equivalent flow is `search output → searxngResultsToRetrievables → storeRetrievables`. For links, use the optional third `recommend` argument when the link records should be spooled.

## Related contracts

The core behavior of `inline`, storage-time auto-spooling, dynamic content, and direct content materialization is documented in [Retrievable](../../the-loop/primitives/retrievable). For token fit and rendering consequences, read [budgets](../../the-loop/budgets). The vector-store adapter is documented in [Retrievable Glue](../vector/retrievable).