Type Alias: TokenEstimatorFn

type TokenEstimatorFn = (value: string) => number;

Defined in: src/lib/classes/tokenizable.ts:112

A custom token-count function registered for a non-built-in TokenEncodingId via registerTokenEstimator.

Parameters

Parameter	Type	Description
value	string	The already-RESOLVED text to count (the same string countFor measures for the built-in backends — i.e. render(ctx)'s output, never the raw evaluator or ctx).

Returns

number

The estimated token count for value.

Remarks

Synchronous, matching the built-in estimation backends: every backend behind the TokenEncoding switch (js-tiktoken, @lenml/tokenizer-gemini, llama-tokenizer-js, the Claude heuristic) resolves synchronously, and Tokenizable.estimateTokens itself returns a plain number — not a Promise<number>. A custom estimator therefore must not be async either, so a registered encoding stays a drop-in peer of the built-ins (same call shape, no await threaded through estimateTokens for some encodings and not others). Wrap an inherently-async tokenizer in a synchronous cache (measure ahead of time / memoize a warmed instance) before registering it.