ADR 0003: Semantic Tokens Over Redaction

Status: Accepted

Context

The established PII tools redact: they replace a sensitive value with [REDACTED], ****, or a fixed mask, and the original is gone. That is correct for log sanitization and compliance scanning, but it breaks two things OGuardAI needs. First, a redacted prompt gives the model nothing to reason about, so the output is generic and impersonal. Second, redaction is one-way: there is no way to put the real value back into the model's answer for the end user.

Decision

We replace each detected value with a typed semantic token of the form {{type:id:cap}}, for example {{person:p_001:ad4f97591c16}} or {{email:e_001:1bcaef1a4aff}}, and carry safe metadata alongside it.

• Typed and deterministic. crates/tokenizer assigns sequential per-type ids from EntityType::id_prefix() (p_001, p_002, e_001), so the same value maps to a stable token within a session. crates/core/src/token.rs is the single source of truth for rendering the display token.
• Carries context, not the value. The token is accompanied by safe metadata (type, and where known gender, formality, language). The model uses this to produce grammatically correct, gender-aware, formality-appropriate text around the token. A bare [REDACTED] cannot support that.
• Reversible by design. Because the token is typed and mapped, crates/rehydrate can restore the original on output under policy, in one of six restore modes (full, partial, masked, formatted, abstract, none). Redaction cannot do this because the value was destroyed.
• Repairable. Models sometimes damage token syntax (extra spaces, changed brackets, altered case). crates/token-robustness runs a 3-stage repair (strict parse, deterministic repair, fuzzy resolve) before restoration. Detection-only tools never face this problem because they never restore.

Consequences

• The model produces personalized, correct output while working from tokens, which is the core reason to use OGuardAI over a redaction tool.
• Round-trip restoration becomes possible and policy-controlled per output channel (see ADR 0005).
• The system takes on a problem redaction tools do not have, LLM-damaged tokens, and pays for it with the token-repair pipeline.
• The token format is a protocol invariant: it is fixed in code, not configurable, because both the transform and rehydrate sides and the repair pipeline depend on it. See ADR 0001.