Function: contentTokens()

function contentTokens(text: string): Set<string>;

Defined in: src/batteries/context/thrift/relevance.ts:102

Extract the "content word" tokens of a string for lexical overlap scoring: lowercased alphanumeric runs of length >= 4, deduplicated into a set. Deliberately crude (no stemming, no stopword list beyond the length-4 floor) — this is a cheap, zero-model-call, zero-dependency relevance signal, not a semantic one; the calibration in RELEVANCE_FLOOR_MIN/RELEVANCE_FLOOR_MAX was run against exactly this scoring function, so changing it invalidates those constants.

Parameters

Parameter	Type	Description
text	string	Text to extract content-word tokens from.

Returns

Set<string>

The distinct content words found, lowercased.