Class: Tokenizable

Defined in: src/lib/classes/tokenizable.ts:279

A mutable string with a built-in token counter.

Remarks

The wrapped string can be read via the standard coercion protocol and updated at any time via Tokenizable.set. Token counts are computed lazily on first access per encoding and cached until the value changes, avoiding redundant encoder invocations when the same content is measured multiple times across a pipeline.

Estimation is dispatched by encoding identifier — see TokenEncoding for the full list of built-in backends and their accuracy characteristics, and registerTokenEstimator for adding more without a core change. An encoding that is neither a built-in nor registered resolves to undefined (see Tokenizable.estimateTokens) — this pre-dates the registry and is unchanged by it. Separately, a built-in encoder that THROWS while measuring (as opposed to an unrecognised name) degrades to a ceil(length / 3.5) character heuristic inside a runner execution — see degradeOrThrow and utils/estimation_context.

The class implements the standard JS value-coercion protocol (toString, valueOf, toJSON, toLocaleString, Symbol.for('nodejs.util.inspect.custom')) so instances behave transparently as strings in most contexts.

Constructors

Constructor

new Tokenizable(value:
  | string
  | TokenizableEvaluator): Tokenizable;

Defined in: src/lib/classes/tokenizable.ts:346

Parameters

Parameter	Type	Description
value	| string | TokenizableEvaluator	The initial value to wrap: a plain string (static) or a TokenizableEvaluator evaluated at assembly time (dynamic).

Returns

Tokenizable

Properties

Property	Modifier	Type	Default value	Description	Defined in
dynamic	readonly	boolean	undefined	Whether the current wrapped value is evaluator-backed rather than a static string.	src/lib/classes/tokenizable.ts:318
estimateTokens	public	(encoding: TokenEncodingId, ctx?: DispatchContext) => number	undefined	Estimate the token count under the given TokenEncodingId of the string this Tokenizable resolves to for the OPTIONAL context — i.e. of render(ctx). Passing the same ctx assembly uses keeps the budget count honest for dynamic content (it measures exactly what will ship). Accepts both a built-in TokenEncoding and any encoding registered via registerTokenEstimator.	src/lib/classes/tokenizable.ts:327
render	public	(ctx?: DispatchContext) => string	undefined	Resolve the value against an optional dispatch context.	src/lib/classes/tokenizable.ts:320
set	public	(value: | string | TokenizableEvaluator) => void	undefined	Replace the wrapped value (string or evaluator) and invalidate the cached token estimates.	src/lib/classes/tokenizable.ts:310
toJSON	public	() => string	undefined	-	src/lib/classes/tokenizable.ts:305
toLocaleString	public	() => string	undefined	-	src/lib/classes/tokenizable.ts:308
toString	public	() => string	undefined	-	src/lib/classes/tokenizable.ts:306
valueOf	public	() => string	undefined	-	src/lib/classes/tokenizable.ts:307
emptyableSchema	static	AlternativesSchema<any>	emptyableStringOrTokenizableSchema	Variant of Tokenizable.schema that additionally accepts the EMPTY string. Remarks For fields where "present but empty" is a legitimate state rather than a mistake — e.g. @nhtio/adk!Thought.content in opaque-replay mode, where the meaning lives in the vendor payload and the prose is only kept for token-accounting and observer inspection. Do NOT reach for this by default. Tokenizable.schema stays strict precisely because an empty system prompt or a blank standing instruction is a bug worth failing on.	src/lib/classes/tokenizable.ts:303
schema	static	AlternativesSchema<any>	stringOrTokenizableSchema	Validator schema that accepts a plain string or a Tokenizable instance. Remarks Reusable fragment for any schema that wants to accept either form — for example, systemPrompt and each item in standingInstructions in turnContextSchema.	src/lib/classes/tokenizable.ts:290
TokenEncoding	static	readonly ["gpt2", "r50k_base", "p50k_base", "p50k_edit", "cl100k_base", "o200k_base", "gemini", "gemma", "llama2", "claude"]	TokenEncoding	The set of supported token-encoding identifiers, re-exposed as a static for convenience.	src/lib/classes/tokenizable.ts:281

Methods

[ENCODE_METHOD]()

ENCODE_METHOD: unknown;

Defined in: src/lib/classes/tokenizable.ts:627

Serialise this Tokenizable into an @nhtio/encoder snapshot.

Returns

unknown

The wrapped string, or the evaluator function for a dynamic value.

Remarks

The wrapped VALUE is the entire state; the token-count caches are derived and deliberately not encoded (they rebuild lazily after decode). For a STATIC value the snapshot is the string. For a DYNAMIC value the snapshot is the EVALUATOR FUNCTION itself — @nhtio/encoder serialises functions (source + explicit bindings), so a dynamic Tokenizable round-trips its evaluator and stays dynamic, re-evaluating live on the next assembly (it does NOT downgrade to a frozen string). Evaluators must therefore stay serializer-friendly: capture only module-level refs / explicit bindings, not live per-turn state. Round-trips via Tokenizable.[DECODE_METHOD].

---

[DECODE_METHOD]()

static DECODE_METHOD: Tokenizable;

Defined in: src/lib/classes/tokenizable.ts:638

Reconstruct a Tokenizable from an Tokenizable.[ENCODE_METHOD] snapshot.

Parameters

Parameter	Type	Description
data	unknown	The wrapped string (static) or evaluator function (dynamic) produced by Tokenizable.[ENCODE_METHOD].

Returns

Tokenizable

A fresh Tokenizable over the same value.

---

estimateTokens()

static estimateTokens(
   value:
  | string
  | TokenizableEvaluator,
   encoding: TokenEncodingId,
   ctx?: DispatchContext): number;

Defined in: src/lib/classes/tokenizable.ts:590

Convenience overload for one-off token counting without managing a Tokenizable instance.

Parameters

Parameter	Type	Description
value	| string | TokenizableEvaluator	The string (or TokenizableEvaluator) to count tokens for.
encoding	TokenEncodingId	The encoding identifier to use for counting — a built-in TokenEncoding or any encoding registered via registerTokenEstimator.
ctx?	DispatchContext	Optional dispatch context; for a dynamic value it selects which resolved string is counted (so the count matches what assembly ships). Ignored for a static string.

Returns

number

The estimated number of tokens.

Remarks

Creates a temporary instance and immediately discards it — no caching benefit. Use the instance method when you need to count the same value under multiple encodings or when the value may change over time.

---

isTokenizable()

static isTokenizable(value: unknown): value is Tokenizable;

Defined in: src/lib/classes/tokenizable.ts:609

Returns true if value is a Tokenizable instance.

Parameters

Parameter	Type	Description
value	unknown	The value to test.

Returns

value is Tokenizable

true when value is a Tokenizable instance.

Remarks

Uses @nhtio/adk!isInstanceOf for cross-realm safety — instanceof would fail for instances created in a different module copy or VM context.