Function: stripEnvelopeSpecialTokens()

function stripEnvelopeSpecialTokens(text: string): string;

Defined in: src/batteries/llm/chat_common/helpers.ts:109

Strip the non-semantic envelope/turn-boundary special tokens (see ENVELOPE_SPECIAL_TOKEN_RE) from decoded model text before it reaches the parser layer.

Parameters

Parameter	Type
text	string

Returns

string

Remarks

Why this exists. The transformers.js streaming path decodes with skip_special_tokens:false (it must — the live prose-stop gate watches for tool/think markers, which ARE special tokens). That leaves envelope tokens like Llama's <|python_tag|>{json}<|eom_id|> or ChatML's trailing <|im_end|> in the accumulated text, so the JSON tool-call parsers see <|python_tag|>{…} and decline — even though the NON-streaming path (which decodes with skip_special_tokens:true) parses the identical call fine. Normalising here makes the stream and batch paths parse equivalent text. Surfaced by the deep model matrix (Llama-3.2-1B + Qwen2.5-Coder tool calls passed on batch, failed on stream).