Operations
Failure Modes
How OGuardAI behaves when components fail, with recovery procedures for each scenario
OGuardAI fails closed by default: when protection cannot be enforced, the request fails rather
than returning unprotected text. The one deliberate exception is both mode, which is explicitly
best-effort: on an NER outage it falls back to builtin regex detection and continues with reduced
coverage. advanced mode fails closed instead, rejecting the request when NER is unavailable. See
the matrix below for the exact behavior per failure.
Failure Matrix
| Component Failure | OGuardAI Behavior | Fail Mode | Recovery |
|---|---|---|---|
| NER sidecar down (both mode) | Falls back to builtin regex detection. Person, company, and location entities are not detected. | Fail-open (reduced coverage) | Restart NER sidecar. |
| NER sidecar down (advanced mode) | Request fails with GUARDAI_DETECTION_FAILED. NER is required in advanced mode. | Fail-closed | Restart NER sidecar or switch to detector.mode: both. |
NER sidecar slow (> detector.timeout_secs) | Request times out waiting for NER. In both mode, falls back to builtin. In advanced mode, request fails. | Mode-dependent | Scale the NER sidecar, or tune detector.timeout_secs (env GUARDAI_DETECTOR_TIMEOUT_SECS, default 5s, max 120s). |
Redis unavailable (revocation_backend: redis or replay_backend: redis) | At startup the server refuses to start until Redis is reachable. At runtime a Redis error on the replay check fails closed: the continuation is rejected with 503 GUARDAI_REPLAY_STORE_UNAVAILABLE, never accepted. The default memory backends do not use Redis. The Redis session backend behaves the same way: the server refuses to start until Redis is reachable. | Fail-closed (startup blocked; runtime rejects) | Restore Redis, or switch the affected backend to memory. |
| Disk full | File-backed revocation writes fail. Audit log writes fail. Server continues processing requests but revocation state may not persist. | Fail-closed (revocation writes rejected) | Free disk space. Revocation state in memory remains consistent until restart. |
| Config file invalid | Server refuses to start. Validation errors printed to stderr with line numbers. | Fail-closed (startup blocked) | Fix config. Run oguardai config validate before deploying. |
| No policies loaded | In non-dev mode the server refuses to start (no valid policies found). In dev mode it starts but logs that protection is EFFECTIVELY DISABLED (nothing is tokenized or blocked). A request naming an unknown or missing policy is rejected with an unknown-policy error. | Fail-closed in non-dev (startup blocked); dev starts unprotected with a loud warning | Add policy files to the policy directory. Run oguardai config validate to check policy references. |
| Session blob expired | Clean error returned: GUARDAI_SESSION_EXPIRED. No data is leaked. Client must re-transform the original text to get a new session. | Fail-closed (request rejected) | Re-submit the original text through /v1/transform. Increase session.ttl_seconds if expirations are frequent. |
| Session blob tampered | AES-256-GCM authentication tag verification fails. Request rejected with GUARDAI_SESSION_EXPIRED. No data is leaked. | Fail-closed (request rejected) | Client must re-transform from original text. Investigate source of tampering. |
| Session blob rolled back (server-side store) | With a server-side Redis session backend, a per-session high-water marker rejects a load whose counter is below the highest ever stored, or whose marker is missing: 409 GUARDAI_SESSION_ROLLBACK. This catches an attacker who overwrites only the blob with a captured older same-id blob. | Fail-closed (request rejected) | Run Redis with maxmemory-policy noeviction and ACLs restricting who may write the guardai:sess: keyspace. Re-transform the original text for a fresh session. |
| Rate limit exceeded | HTTP 429 returned with Retry-After header. No data processed. | Fail-closed (request rejected) | Wait for rate limit window to reset. Increase rate_limit.requests_per_second or add instances. |
| Output guard detects new PII | Depending on config: mask replaces new PII with type labels, block rejects the entire response, warn passes through with warning flag. | Configurable | Review LLM output patterns. Adjust output guard sensitivity or action mode. |
| Session key rotated | With a multi-key session.keyring, blobs sealed under a retained kid keep unsealing until that key is removed or their TTL expires; a single-key secret change invalidates existing blobs. New transforms seal under current_kid. | Fail-closed (blobs under a removed key rejected) | Keep the prior key for one TTL for zero-downtime rotation; remove it to force-invalidate. |
| Tesseract missing (image endpoints) | /v1/transform/image and /v1/redact/image reject with 503 GUARDAI_OCR_UNAVAILABLE. Text and JSON paths are unaffected. Happens on the proxy image or a from-source build without the tesseract-ocr package (the official server image bundles it). | Fail-closed (image endpoints only) | Install Tesseract, or use the official server image which bundles eng/deu/ara. |
| Prompt injection detected | Request is rejected or the injected content is neutralized, depending on prompt_security.action config. | Fail-closed (default) | Review rejected input. Adjust prompt security sensitivity if false positives occur. |
Degradation Flow
Degradation Matrix
What still works when a component is unavailable:
| Component Down | Transform | Rehydrate | Detect | Revoke | Output Guard |
|---|---|---|---|---|---|
| NER sidecar | Partial (builtin only) | Full | Partial (builtin only) | Full | Full |
Redis (when revocation_backend: redis) | Full | Full | Full | Unavailable (server won't start until Redis returns) | Full |
| Disk | Full | Full | Full | Rejected (write fails, revoke rolled back, 500) | Full |
| Policy files | Full (loaded policies retained) | Full | Full | Full | Full |
"Partial" means the operation succeeds but with reduced entity coverage: the NER-only types (person, company, location) drop out, while the builtin regex and custom-pattern types still resolve. "Degraded" means the operation succeeds but with reduced durability or consistency.
Design Principles
- Never return unprotected text. If detection cannot run, the request fails rather than passing raw PII through.
- Cryptographic verification on every unseal. Tampered or expired session blobs are always rejected.
- Stateless by default. The sealed session model has no external dependencies for a single instance. Redis and file backends are optional there, but a multi-replica deployment requires shared Redis revocation (
revocation_backend=redis) so a value revoked on one replica is refused by all. - NER failure behavior depends on mode. In
bothmode, NER unavailability falls back to builtin (fail-open). Inadvancedmode, NER is required and failures block the request (fail-closed). MonitorGUARDAI_DETECTION_FAILEDerrors for sidecar outages. - Revoked values are permanently suppressed. Even if the session blob is valid, revoked entities always return
[DELETED].
Monitoring Recommendations
- Poll
/v1/capabilitiesperiodically. Thener_activefield reflects the configured detector mode (not live sidecar status). Monitorguardai_errors_totalforGUARDAI_DETECTION_FAILEDto detect sidecar outages. - Track
guardai_errors_totalby error code. Spike inSESSION_EXPIREDmay indicate TTL misconfiguration. - Track
guardai_rate_limit_rejections_total. Sustained rejections indicate capacity issues. - Track
guardai_output_guard_triggers_total. High trigger rate may indicate LLM is generating PII.
Observability
How OGuardAI emits structured logs and audit events, how to set the log level and format, the OpenTelemetry span export it ships, and the liveness and readiness endpoints for the runtime and the NER sidecar.
Service Level Objectives
Availability, latency, throughput, and security SLOs for OGuardAI deployments