Skip to content
1 min read · 103 words

Variable: E_TOKEN_ESTIMATOR_SHADOWS_BUILTIN

ts
const E_TOKEN_ESTIMATOR_SHADOWS_BUILTIN: CreatedException<[string]>;

Defined in: src/lib/classes/tokenizable.ts:126

Thrown by registerTokenEstimator when the given encoding identifier names a built-in TokenEncoding rather than a genuinely custom one.

Remarks

The built-in encodings are canonical: 'gemma', 'cl100k_base', and friends must always resolve to their core backend (js-tiktoken, @lenml/tokenizer-gemini, llama-tokenizer-js, the Claude heuristic), never to a consumer-supplied override. Allowing a shadow registration would let one battery silently change another's token counts for a shared built-in name — a correctness hazard that is cheap to prevent at registration time. Fatal: this is a programming error in the caller (pick a distinct identifier), not a runtime condition to recover from.