Extending Entities

Overview

OGuardAI recognizes entity types in three groups. The built-in engine detects the named regex formats (Email, Phone, CustomerId, Order, Address, Iban, Ssn, Ip, Url, CreditCard, DateOfBirth, Passport, VatId, HealthId) and a set of built-in custom patterns (money amounts, the German tax id, the German social-security number, quote ids, case references). Person, Company, and Location come from the Python NER sidecar. None of these are a fixed ceiling: a deployment adds its own types in policy YAML, and the detector floor itself is extensible.

You almost never touch any of that to add your own type. A deployment defines its own domain identifiers, MRN, Aktenzeichen, NPI, loyalty number, clearance code, contract number, entirely in policy YAML with custom_patterns. No code, no fork, no recompile. That is the supported path and the one this guide leads with. Editing Rust is reserved for contributing a detector to the shipped built-in floor (the last section).

Add a custom entity type in policy YAML (no code)

custom_patterns declares a domain identifier. Each match becomes {{custom:<type>:id}} and is governed by a normal policy rule. It is additive: a custom pattern can only ADD detections, never disable or weaken a built-in one.

# policies/acme/policy.yaml
custom_patterns:
  - entity_type: "mrn"                                  # your type name, lowercase_with_underscores
    pattern: '\bMRN[-:\s]?\d{6,10}\b'                   # Rust regex; linear-time engine, no catastrophic backtracking, validated at load
    confidence: 0.9
    context_words: ["mrn", "medical record", "patient"] # nearby words raise confidence

rules:
  - entity_type: "mrn"            # reference it by the same bare name
    protection_level: 1
    action: "tokenize"
    restore_mode: "masked"
    conditions: []

That is the whole change. Restart the server and MRN-04417829 tokenizes as {{custom:mrn:x_001:42f5f25fb8cc}}; the model never sees the value, and it restores per the rule's restore_mode and the resolved output channel.

Tokenize only part of the match (value_group)

By default the whole match is the entity. To match on a label but protect ONLY the identifier, set value_group to the capture group holding the value. This is how you catch a bare identifier (a bank account, a routing number) that a plain regex would otherwise over-match on every long number: anchor on the label, tokenize just the group.

custom_patterns:
  - entity_type: "bank_account"
    pattern: '(?i)\b(?:account|acct|konto)[ \t:#.-]{0,4}(\d{8,17})\b'
    value_group: 1                # tokenize only capture group 1 (the number, not the label)
    context_words: ["account", "acct", "konto"]

account 0532013000 tokenizes only 0532013000 (the word account stays for model context), and a bare number with no label is not matched. value_group is validated against the pattern's capture-group count at startup (out of range aborts the load, fail closed); omit it to tokenize the whole match.

Require a nearby label (context_required)

value_group embeds the label in the regex. When you would rather keep a broad, readable pattern (an ICD-10 or SWIFT/BIC shape) but still avoid matching every lookalike token, set context_required: true. A match is then kept ONLY when one of context_words appears in the context_window_chars immediately before it; otherwise it is dropped. It turns the confidence boost into a gate.

custom_patterns:
  - entity_type: "icd10"
    pattern: '\b[A-TV-Z][0-9][0-9AB](?:\.[0-9A-TV-Z]{1,4})?\b'
    context_words: ["icd", "diagnosis", "diagnose", "dx"]
    context_required: true          # keep a code only near a diagnosis label
    context_window_chars: 40        # optional; defaults to the built-in window

Diagnose E11.9 und I10 tokenizes both codes (the list stays inside the window), while a bare A12 in an unrelated sentence is left alone. This is opt-in: the default (context_required: false) leaves matching exactly as before, and the gate only ever narrows one of your own custom patterns. It never applies to a built-in type, so it can never weaken the email, IBAN, or SSN floor. context_required: true with no context_words, or a window outside [1, 4096], aborts the load (fail closed).

Every entity_type you reference must be a built-in type or a custom type declared in custom_patterns of the same policy; an unknown label is treated as a typo and aborts startup (fail closed), so a mistyped ssn can never silently leave SSNs unprotected. To reference a custom type detected by a deployment-level detector (not declared in this policy), write it as custom:<name> to confirm intent and skip the typo check.

Working examples ship for every vertical: policies/templates/healthcare-hipaa (mrn, npi, dea), policies/financial (bank_account, routing_number), policies/government (clearance_id, facility_code), policies/hr (employee_id), policies/legal-privilege (matter_id).

Restore your custom type (no code)

Custom entities restore through the same six modes (full, partial, masked, formatted, abstract, none) as built-ins, plus config-driven restore strategies, all in YAML:

• Partial with a strategy. For last-N restoration (a card, an account, an MRN) or numeric generalization (a salary band), add a restore_strategies block. It applies only when the entity resolves at partial for the channel, so it can never exceed the channel ceiling, and masks the whole value if it cannot produce a bounded result (fail closed).

restore_strategies:
  credit_card:
    kind: reveal_last_n   # **** **** **** 4821
    n: 4
  salary:
    kind: bucket          # generalize a number into a coarse band
    ranges:
      - {max: 50000, label: "band: under 50k"}
      - {label: "band: 50k and up"}

• Formatted / abstract labels come from the restore-template floor (crates/rehydrate/data/restore_floor.yaml, 30 languages) and per-deployment overrides via restore_templates / GUARDAI_RESTORE_TEMPLATES: config, not code. A custom type with no specific label falls back to a safe generic label.

Semantic types regex cannot catch (NER labels, no code)

Identifiers that are not a fixed format (a diagnosis, a medication, a matter subject) are caught by asking the NER sidecar for extra zero-shot labels, declared per policy:

detection:
  required_for: ["diagnosis"]          # fail closed if NER is down
  ner_labels: ["medical condition"]    # extra labels to detect
  ner_label_map:
    "medical condition": "diagnosis"   # report the label as this entity type

required_for makes the type fail closed: if NER is required but the sidecar is unavailable, the request errors instead of under-detecting. Regex custom_patterns (an MRN, a contract number) do not depend on the sidecar.

What this does and does not protect (topical sensitivity)

ner_labels catches a sensitive span the model can name, such as "cancer diagnosis" or "criminal charge". It does not, on its own, protect a secret that is not a span: diffuse context spread across several sentences, a combination of otherwise-safe facts, or an implication with no keyword. The honest boundary is that OGuardAI removes the identifiers and the topics you declare, not that entity tokenization guarantees full subject-matter confidentiality. For a fixed set of sensitive phrases, declare them as ner_labels (semantic), custom_patterns (literal identifier), or topical_redaction (below, when you need the whole surrounding sentence removed, not just the phrase).

Redact a sensitive topic, not just an identifier (topical_redaction)

custom_patterns and ner_labels tokenize a matched value and keep the rest of the sentence intact. Some matter is sensitive as a topic: the diagnosis, the criminal charge, the trade secret is the secret even when every name around it is tokenized. topical_redaction is for that. Each rule declares literal phrases and/or regex patterns; a match redacts the surrounding sentence (or a character window) to a generic sentinel [REDACTED:sensitive_topic] BEFORE detection and tokenization, so the raw matter never enters the token map, the model, or rehydrate. It is non-restorable by design (a topical secret has no safe partial form).

topical_redaction:
  - name: "clinical_topic"
    phrases: ["cancer diagnosis", "HIV status"]   # matched case-insensitively, UTF-8 safe
    scope: "sentence"                              # redact the whole containing sentence
    on_match: "continue"                           # redact and keep processing (default)
  - name: "matter_code"
    patterns: ['\bMATTER-\d{4,6}\b']               # linear-time regex, compiled at load
    scope: "window"
    window_chars: 30                               # redact 30 chars on each side of the match
  - name: "classified_block"
    phrases: ["classified"]
    on_match: "reject"                             # deny the whole request (403) instead

It is additive, fail-closed, and deterministic:

• Additive / tighten-only. A rule can only ADD redaction; it never weakens the built-in entity floor or an entity rule. Inheritance unions rules across the chain (a child policy may add rules, never remove a parent's).
• Every ingress. The pass runs on text / JSON / chat transform, /v1/detect, batch, RAG (the full document is redacted before chunking, so a phrase cannot be split across a chunk boundary), and image OCR (overlapping words are blacked out), plus a post-rehydrate pass that strips topical matter the model itself generated.
• Fail-closed. A rule with no phrase or pattern, an empty phrase, an invalid regex, an out-of-range window_chars, a duplicate name, or an unknown field aborts startup. on_match: reject returns 403 GUARDAI_POLICY_DENIED before any LLM-facing output.
• Generic sentinel. The replacement never names the rule, because the topic label can itself disclose the matter.

Working example: policies/topical-demo.

What it does NOT do: guarantee full subject-matter confidentiality for diffuse facts spread across sentences, rare combinations of otherwise-safe facts, or an undeclared topic with no keyword. It removes the identifiers plus the topics you declare. For those residual contextual cases, keep the material out of the prompt at the application layer.

Transparent proxy note. topical_redaction is enforced on the direct API, the SDKs, and MCP. The transparent proxy (oguardai-proxy) has several independent content paths and does not yet apply topical redaction, so it fails closed at startup if its policy declares topical rules. Use the direct API or SDK for topical enforcement.

Protect a whole CSV column (document_columns)

A regex or NER detector catches values by their shape or meaning. A free-text column, an internal note, or a bespoke reference code often has no shape a detector can match, yet the whole column is sensitive. document_columns maps a CSV column, by header, to an entity type: every cell in that column becomes an entity of that type, so the column is protected even when nothing else matches it.

document_columns:
  - format: "csv"
    header: "employee_notes"   # matched case-insensitively, whitespace-normalized
    entity_type: "employee_note"

rules:
  - entity_type: "employee_note"  # a normal rule governs action + restore
    protection_level: 2
    action: "tokenize"
    restore_mode: "masked"

It is detection only and reuses the whole policy engine:

• The normal rule governs. The mapped entity_type is declared like a custom_patterns type; its ordinary policy rule decides action, restore mode, and channel behavior. A column mapped to a built-in type (e.g. email) is governed by that built-in's rule.
• Additive, built-in wins overlap. A column rule only ADDs detection. Where a cell also contains a built-in PII value (an email in a notes cell), the built-in detection wins that span and the rest of the cell is covered by the column type, so no bytes are left raw and the email keeps its own (restorable) protection.
• CSV file uploads only. It applies to POST /v1/transform/file (and /v1/rehydrate/file) for CSV, the format that exposes stable per-cell offsets. It has no effect on text/JSON/chat transform, and DOCX/HTML/Markdown/PDF are not covered (they flatten to text without reliable cell boundaries). Working example: policies/csv-columns-demo.
• Fail-closed. An invalid format, a blank header, an unknown field, or too many rules aborts startup; a CSV whose offsets cannot be built fails the request.

Compatibility matrix

Every row is policy or config. There is no "needs custom code" path for adding a deployment's own entity type.

End-to-end (live)

curl -X POST http://localhost:3000/v1/transform \
  -H "Content-Type: application/json" \
  -H "X-API-Key: $GUARDAI_API_KEY" \
  -d '{"input": "Patient MRN-04417829 admitted.", "policy": "acme"}'
# -> safe_text: "Patient {{custom:mrn:x_001:42f5f25fb8cc}} admitted."

Advanced: contribute a built-in detector (requires a code change)

You only edit Rust to promote a detector into the shipped built-in floor (so every deployment gets it without configuring custom_patterns), to give a type its own token id prefix, or to hand-write a formatted/abstract restore beyond the template floor. This is a contributor path, not a deployment path.

• Add a DetectionPattern to all_patterns() in crates/detector-builtins/src/patterns.rs. This is how built-in custom detectors like money and the German tax id ship.
• For a distinct token prefix, add a named variant to EntityType in crates/core/src/types.rs with its own id_prefix() arm (policy-defined custom types otherwise share the x_ prefix).
• For a type-specific formatted/abstract restore beyond the template floor, add a match arm in crates/rehydrate/src/restore.rs.
• Add a pattern test alongside the others and run cargo test --workspace.

Limitations

• Regex or NER. A custom type is detected by a regex pattern or by an NER label; a format-free type needs the sidecar via ner_labels.
• Built-in wins an overlap. A custom pattern adds detection; where its match overlaps a built-in detector (for example a money-shaped value), the built-in type wins. Custom patterns are additive and never weaken a built-in.
• Shared token prefix. Policy-defined custom types share the {{custom:<type>:id}} form and the x_ id prefix; a distinct prefix needs the enum change in the advanced section.