Function: registerTokenEstimator()

function registerTokenEstimator(
  encoding: string,
  estimator: TokenEstimatorFn,
): void;

Defined in: src/lib/classes/tokenizable.ts:173

Register (or replace) a TokenEstimatorFn for a custom TokenEncodingId, so Tokenizable.estimateTokens can measure it without a core code change.

Parameters

Parameter	Type	Description
encoding	string	The custom encoding identifier this estimator handles. Must not be one of the built-in TokenEncoding values.
estimator	TokenEstimatorFn	The synchronous token-count function to call for encoding.

Returns

void

Remarks

This is the additive escape hatch for the closed TokenEncoding set. Say a context-management battery needs to measure tokens for a model that core has no built-in backend for — it simply calls this once, at startup, for that model's identifier. From then on the encoding is measurable, with no core code change (a new case in Tokenizable.estimateTokens's switch) ever required. Mirrors the @nhtio/adk!registerMediaReaderResolver / @nhtio/adk!registerSpoolReaderResolver registry idiom used for reader handles — register a factory once, core (or the primitive) consults it, nobody imports a battery from core.

Resolution order inside Tokenizable.estimateTokens: the built-in switch is tried FIRST (so the closed set's fast path — including its degrade-to-heuristic-on-encoder-failure contract — is completely unaffected by the registry existing), and only an encoding the switch doesn't recognise falls through to this registry.

Idempotent-by-overwrite: registering the same encoding twice replaces the previous estimator — useful for hot-swapping a warmed tokenizer instance without restarting. Registering a name that shadows a BUILT-IN TokenEncoding (e.g. 'gemma', 'cl100k_base') is rejected: the built-ins are canonical and must always resolve to their core backend, never to a consumer override.

Throws

@nhtio/adk!E_TOKEN_ESTIMATOR_SHADOWS_BUILTIN when encoding names a built-in.