Token Protocol

Semantic tokens are the contract between the trusted runtime and the untrusted model. This page is the deep reference for how a token is formed, why it cannot be forged, and how the runtime restores it even when the model mangles it.

Token format

The canonical form is {{type:id:cap}}; on the wire the runtime appends a third segment, a per-token capability cap:

{{email:e_001:5873cc5722d8}}
   │      │        │
   │      │        └─ cap: a per-token capability that binds this token to one entry in one session
   │      └─ id: type prefix + counter (e_001), or a corpus-scoped hash under corpus_id
   └─ type: the entity type (email, person, iban, custom:<name>, ...)

• type names the entity. Built-in types (email, phone, iban, person, ...) and per-request custom types ({{custom:asset_tag:...}}) share the format.
• id is stable within a session. In the default per-request scheme it is prefix_NNN; under a shared corpus_id it is a deterministic corpus-scoped hash so the same value maps to the same id across every document and query (see RAG pipeline).
• cap (capability cap) ties the token to a specific session entry. A token that does not carry the right cap is not resolved, so a model cannot invent {{iban:ib_001:a7a60e9fe667}} and have it restored to a real value. Restoration values come only from the session map, never from the token text.

Deterministic assignment

Token ids are assigned deterministically. Detected entities are sorted by a fixed key (start position, then end, then type, then value) before ids are allocated, so the same input always produces the same tokens regardless of detection order. This is what makes a round trip reproducible and lets two independent calls agree under a shared corpus_id.

The trust boundary in one line

Raw values live only inside the runtime. What crosses to the model is {{type:id:cap}} plus optional non-identifying metadata (gender, formality, language) chosen by policy. The reverse direction restores values only for tokens that resolve against the session map.

Restoration and three-stage repair

A model rarely returns tokens verbatim: it changes case, drops a brace, or truncates an id. The rehydrator repairs before it resolves, in three stages, and never guesses:

1. Strict — accept only exact {{type:id:cap}} syntax and resolve it.
2. Repair — normalize common model mutations before re-parsing: uppercase type ({{Type:id}} → {{type:id:cap}}), missing underscore ({{type:id1}} → {{type:id_001}}), and single-brace tokens ({type:id} → {{type:id:cap}}).
3. Fuzzy — resolve a remaining near-miss only when there is a single candidate. Ambiguity is never resolved to a guess.

Anything still unresolved after all three stages is left in place and flagged, not fabricated. A token that resolves to nothing in the session map is treated as a model hallucination and never turned into a value. This is the fail-closed rule: the runtime would rather return an unresolved token than invent PII.

Why this matters

• No forged restores. The capability cap plus session-map-only resolution means a model cannot make the runtime emit a value it was never given.
• Robust to real LLM output. Case changes, dropped braces, and truncated ids are repaired, so practical model noise does not break a round trip.
• Deterministic and joinable. Stable, sorted id assignment makes round trips reproducible and, with corpus_id, lets tokens from different requests refer to the same entity.

See ADR 0003: Semantic tokens for the decision record and Guarantees for the security properties.