Class: Tokenizable

Defined in: src/lib/classes/tokenizable.ts:256

A mutable string with a built-in token counter.

Remarks

The wrapped string can be read via the standard coercion protocol and updated at any time via Tokenizable.set. Token counts are computed lazily on first access per encoding and cached until the value changes, avoiding redundant encoder invocations when the same content is measured multiple times across a pipeline.

Estimation is dispatched by encoding identifier — see TokenEncoding for the full list of built-in backends and their accuracy characteristics, and registerTokenEstimator for adding more without a core change. An encoding that is neither a built-in nor registered resolves to undefined (see Tokenizable.estimateTokens) — this pre-dates the registry and is unchanged by it. Separately, a built-in encoder that THROWS while measuring (as opposed to an unrecognised name) degrades to a ceil(length / 3.5) character heuristic inside a runner execution — see degradeOrThrow and utils/estimation_context.

The class implements the standard JS value-coercion protocol (toString, valueOf, toJSON, toLocaleString, Symbol.for('nodejs.util.inspect.custom')) so instances behave transparently as strings in most contexts.

Constructors

Constructor

new Tokenizable(value:
  | string
  | TokenizableEvaluator): Tokenizable;

Defined in: src/lib/classes/tokenizable.ts:307

Parameters

Parameter	Type	Description
value	| string | TokenizableEvaluator	The initial value to wrap: a plain string (static) or a TokenizableEvaluator evaluated at assembly time (dynamic).

Returns

Tokenizable

Properties

Property	Modifier	Type	Default value	Description	Defined in
estimateTokens	public	(encoding: TokenEncodingId, ctx?: DispatchContext) => number	undefined	Estimate the token count under the given TokenEncodingId of the string this Tokenizable resolves to for the OPTIONAL context — i.e. of render(ctx). Passing the same ctx assembly uses keeps the budget count honest for dynamic content (it measures exactly what will ship). Accepts both a built-in TokenEncoding and any encoding registered via registerTokenEstimator.	src/lib/classes/tokenizable.ts:288
render	public	(ctx?: DispatchContext) => string	undefined	Resolve the wrapped content against an OPTIONAL dispatch context and return the string. For a static value the context is ignored. For a dynamic (evaluatable) value the evaluator is invoked with ctx (TokenizableEvaluator); assembly passes the live context so the content matches the dispatch it ships in, while a no-context call returns the evaluator's undefined-branch fallback.	src/lib/classes/tokenizable.ts:281
set	public	(value: | string | TokenizableEvaluator) => void	undefined	Replace the wrapped value (string or evaluator) and invalidate the cached token estimates.	src/lib/classes/tokenizable.ts:274
toJSON	public	() => string	undefined	-	src/lib/classes/tokenizable.ts:269
toLocaleString	public	() => string	undefined	-	src/lib/classes/tokenizable.ts:272
toString	public	() => string	undefined	-	src/lib/classes/tokenizable.ts:270
valueOf	public	() => string	undefined	-	src/lib/classes/tokenizable.ts:271
schema	static	AlternativesSchema<any>	stringOrTokenizableSchema	Validator schema that accepts a plain string or a Tokenizable instance. Remarks Reusable fragment for any schema that wants to accept either form — for example, systemPrompt and each item in standingInstructions in turnContextSchema.	src/lib/classes/tokenizable.ts:267
TokenEncoding	static	readonly ["gpt2", "r50k_base", "p50k_base", "p50k_edit", "cl100k_base", "o200k_base", "gemini", "gemma", "llama2", "claude"]	TokenEncoding	The set of supported token-encoding identifiers, re-exposed as a static for convenience.	src/lib/classes/tokenizable.ts:258

Methods

[ENCODE_METHOD]()

ENCODE_METHOD: unknown;

Defined in: src/lib/classes/tokenizable.ts:583

Serialise this Tokenizable into an @nhtio/encoder snapshot.

Returns

unknown

The wrapped string, or the evaluator function for a dynamic value.

Remarks

The wrapped VALUE is the entire state; the token-count caches are derived and deliberately not encoded (they rebuild lazily after decode). For a STATIC value the snapshot is the string. For a DYNAMIC value the snapshot is the EVALUATOR FUNCTION itself — @nhtio/encoder serialises functions (source + explicit bindings), so a dynamic Tokenizable round-trips its evaluator and stays dynamic, re-evaluating live on the next assembly (it does NOT downgrade to a frozen string). Evaluators must therefore stay serializer-friendly: capture only module-level refs / explicit bindings, not live per-turn state. Round-trips via Tokenizable.[DECODE_METHOD].

---

[DECODE_METHOD]()

static DECODE_METHOD: Tokenizable;

Defined in: src/lib/classes/tokenizable.ts:594

Reconstruct a Tokenizable from an Tokenizable.[ENCODE_METHOD] snapshot.

Parameters

Parameter	Type	Description
data	unknown	The wrapped string (static) or evaluator function (dynamic) produced by Tokenizable.[ENCODE_METHOD].

Returns

Tokenizable

A fresh Tokenizable over the same value.

---

estimateTokens()

static estimateTokens(
   value:
  | string
  | TokenizableEvaluator,
   encoding: TokenEncodingId,
   ctx?: DispatchContext): number;

Defined in: src/lib/classes/tokenizable.ts:546

Convenience overload for one-off token counting without managing a Tokenizable instance.

Parameters

Parameter	Type	Description
value	| string | TokenizableEvaluator	The string (or TokenizableEvaluator) to count tokens for.
encoding	TokenEncodingId	The encoding identifier to use for counting — a built-in TokenEncoding or any encoding registered via registerTokenEstimator.
ctx?	DispatchContext	Optional dispatch context; for a dynamic value it selects which resolved string is counted (so the count matches what assembly ships). Ignored for a static string.

Returns

number

The estimated number of tokens.

Remarks

Creates a temporary instance and immediately discards it — no caching benefit. Use the instance method when you need to count the same value under multiple encodings or when the value may change over time.

---

isTokenizable()

static isTokenizable(value: unknown): value is Tokenizable;

Defined in: src/lib/classes/tokenizable.ts:565

Returns true if value is a Tokenizable instance.

Parameters

Parameter	Type	Description
value	unknown	The value to test.

Returns

value is Tokenizable

true when value is a Tokenizable instance.

Remarks

Uses @nhtio/adk!isInstanceOf for cross-realm safety — instanceof would fail for instances created in a different module copy or VM context.