Token Protocol

Semantic tokens are the contract between the trusted runtime and the untrusted model. This page is the deep reference for how a token is formed, why it cannot be forged, and how the runtime restores it even when the model mangles it.

Token format

The canonical form is {{type:id:cap}}; on the wire the runtime appends a third segment, a per-token capability cap:

{{email:e_001:5873cc5722d8aa11bb22cc33dd44ee55}}
   │      │        │
   │      │        └─ cap: a per-token capability that binds this token to one entry in one session
   │      └─ id: type prefix + counter (e_001), or a corpus-scoped hash under corpus_id
   └─ type: the entity type (email, person, iban, custom:<name>, ...)

• type names the entity. Built-in types (email, phone, iban, person, ...) and per-request custom types ({{custom:asset_tag:...}}) share the format.
• id is stable within a session. In the default per-request scheme it is prefix_NNN; under a shared corpus_id it is a deterministic corpus-scoped hash so the same value maps to the same id across every document and query (see RAG pipeline).
• cap (capability cap) ties the token to a specific session entry. A token that does not carry the right cap is not resolved, so a model cannot invent {{iban:ib_001:a7a60e9fe667aa11bb22cc33dd44ee55}} and have it restored to a real value. Restoration values come only from the session map, never from the token text.

None of the three segments encodes the entity's attributes. The id and cap are opaque handles. Safe metadata such as gender, formality, and language is not embedded in the token; it travels beside the tokens in the transform response entity_context and reaches the model only when policy permits.

Deterministic assignment

Token ids are assigned deterministically. Detected entities are sorted by a fixed key (start position, then end, then type, then value) before ids are allocated, so the same input always produces the same token ids regardless of detection order. The full token also carries a cap, and in the default per-request scheme that cap is minted randomly per token, so two independent sessions emit different full tokens for the same value: full-token identity holds only within one session. Under a shared corpus_id both the id and the cap are derived deterministically from the value, so two independent calls agree there. Deterministic id assignment is what makes a round trip reproducible.

Identity is type-aware for per-request tokens: when deciding whether two detected values are the same entity and should share an id, the runtime canonicalizes free-text values case and Unicode insensitively, so the same person matches regardless of case, while opaque identifier types such as customer_id and url, along with custom entity types other than the built-in ones (money, de_tax_id, and the rest, which stay free-text), are compared by exact value after invisible formatting is stripped, so distinct opaque values keep distinct tokens. Under a shared corpus_id, token identity uses a frozen, type-blind canon rather than this per-type rule.

The trust boundary in one line

Raw values live inside the trusted zone (the runtime and, when deployed, the Python NER sidecar). What crosses to the model is {{type:id:cap}} plus optional non-identifying metadata (gender, formality, language) chosen by policy. The reverse direction restores values only for tokens that resolve against the session map.

Restoration and three-stage repair

A model rarely returns tokens verbatim: it changes case, drops a brace, or truncates an id. The rehydrator repairs before it resolves, in three stages, and never guesses:

1. Strict: accept only exact {{type:id:cap}} syntax and resolve it.
2. Repair: normalize common model mutations before re-parsing. It restores braces ({type:id:cap} to {{type:id:cap}}), strips whitespace the model inserted inside the braces, lowercases an uppercased type ({{Person:p_001:...}} to {{person:p_001:...}}), and normalizes an id that lost its underscore or its zero padding ({{person:p1:...}} to {{person:p_001:...}}). Repair only rewrites the syntax around a cap segment and carries that segment through unchanged. It never synthesizes one, so a genuinely capless token stays capless and still does not resolve.
3. Fuzzy: resolve a remaining near-miss only when there is a single candidate. Ambiguity is never resolved to a guess.

Anything still unresolved after all three stages is left in place and flagged, not fabricated. A token that resolves to nothing in the session map is treated as a model hallucination and never turned into a value. This is the fail-closed rule: the runtime would rather return an unresolved token than invent PII.

Why this matters

• No forged restores. The capability cap plus session-map-only resolution means a model cannot make the runtime emit a value it was never given.
• Robust to real LLM output. Case changes, dropped braces, and truncated ids are repaired, so practical model noise does not break a round trip.
• Deterministic and joinable. Stable, sorted id assignment makes round trips reproducible and, with corpus_id, lets tokens from different requests refer to the same entity.

See ADR 0003: Semantic tokens for the decision record and Guarantees for the security properties.