Class: Tokenizable

A mutable string with a built-in token counter.

Remarks

The wrapped string can be read via the standard coercion protocol and updated at any time via Tokenizable.set. Token counts are computed lazily on first access per encoding and cached until the value changes, avoiding redundant encoder invocations when the same content is measured multiple times across a pipeline.

Estimation is dispatched by encoding identifier — see TokenEncoding for the full list of supported backends and their accuracy characteristics. Unrecognised encodings fall back to a ceil(length / 4) character heuristic.

The class implements the standard JS value-coercion protocol (toString, valueOf, toJSON, toLocaleString, Symbol.for('nodejs.util.inspect.custom')) so instances behave transparently as strings in most contexts.

Constructors

Constructor

new Tokenizable(value: string): Tokenizable;

Parameters

Parameter	Type	Description
value	string	The initial string value to wrap.

Returns

Tokenizable

Properties

Property	Modifier	Type	Default value	Description
estimateTokens	public	(encoding: | "gpt2" | "r50k_base" | "p50k_base" | "p50k_edit" | "cl100k_base" | "o200k_base" | "gemini" | "llama2" | "claude") => number	undefined	-
set	public	(value: string) => void	undefined	-
toJSON	public	() => string	undefined	-
toLocaleString	public	() => string	undefined	-
toString	public	() => string	undefined	-
valueOf	public	() => string	undefined	-
schema	static	AlternativesSchema<any>	stringOrTokenizableSchema	Validator schema that accepts a plain string or a Tokenizable instance. Remarks Reusable fragment for any schema that wants to accept either form — for example, systemPrompt and each item in standingInstructions in turnContextSchema.
TokenEncoding	static	readonly ["gpt2", "r50k_base", "p50k_base", "p50k_edit", "cl100k_base", "o200k_base", "gemini", "llama2", "claude"]	TokenEncoding	-

Methods

estimateTokens()

static estimateTokens(value: string, encoding:
  | "gpt2"
  | "r50k_base"
  | "p50k_base"
  | "p50k_edit"
  | "cl100k_base"
  | "o200k_base"
  | "gemini"
  | "llama2"
  | "claude"): number;

Convenience overload for one-off token counting without managing a Tokenizable instance.

Parameters

Parameter	Type	Description
value	string	The string to count tokens for.
encoding	| "gpt2" | "r50k_base" | "p50k_base" | "p50k_edit" | "cl100k_base" | "o200k_base" | "gemini" | "llama2" | "claude"	The encoding identifier to use for counting.

Returns

number

The estimated number of tokens.

Remarks

Creates a temporary instance and immediately discards it — no caching benefit. Use the instance method when you need to count the same value under multiple encodings or when the value may change over time.

---

isTokenizable()

static isTokenizable(value: unknown): value is Tokenizable;

Returns true if value is a Tokenizable instance.

Parameters

Parameter	Type	Description
value	unknown	The value to test.

Returns

value is Tokenizable

true when value is a Tokenizable instance.

Remarks

Uses @nhtio/adk!isInstanceOf for cross-realm safety — instanceof would fail for instances created in a different module copy or VM context.