Failure Modes
How OGuardAI behaves when components fail, with recovery procedures for each scenario
OGuardAI fails closed by default: when protection cannot be enforced, the request fails rather
than returning unprotected text. The one deliberate exception is both mode, which is explicitly
best-effort: on an NER outage it falls back to builtin regex detection and continues with reduced
coverage. advanced mode fails closed instead, rejecting the request when NER is unavailable. See
the matrix below for the exact behavior per failure.
Failure Matrix
| Component Failure | OGuardAI Behavior | Fail Mode | Recovery |
|---|---|---|---|
| NER sidecar down (both mode) | Falls back to builtin regex detection. Person, company, and location entities are not detected. | Fail-open (reduced coverage) | Restart NER sidecar. |
| NER sidecar down (advanced mode) | Request fails closed with GUARDAI_DETECTION_UNAVAILABLE (503, retriable). NER is required in advanced mode. | Fail-closed | Restart NER sidecar or switch to detector.mode: both. |
NER sidecar slow (> detector.timeout_secs) | Request times out waiting for NER. In both mode, falls back to builtin. In advanced mode, request fails. | Mode-dependent | Scale the NER sidecar, or tune detector.timeout_secs (env GUARDAI_DETECTOR_TIMEOUT_SECS, default 5s, max 120s). |
Redis unavailable (revocation_backend: redis or replay_backend: redis) | At startup the server refuses to start until Redis is reachable. At runtime a Redis error on the replay check fails closed: the continuation is rejected with 503 GUARDAI_REPLAY_STORE_UNAVAILABLE, never accepted. The default memory backends do not use Redis. The Redis session backend behaves the same way: the server refuses to start until Redis is reachable. | Fail-closed (startup blocked; runtime rejects) | Restore Redis, or switch the affected backend to memory. |
| Disk full | File-backed revocation writes fail. With strict audit, every affected request whose audit record cannot be durably written returns 503 GUARDAI_AUDIT_UNAVAILABLE; it is not acknowledged as unaudited success. | Fail-closed for revocation writes and strict audit | Free disk space. Treat a failed mutating request as outcome-unknown until domain state and the audit chain agree; retry with the same Idempotency-Key when supported. |
| Audit file custody invalid | The file sink rejects a symlink, non-regular file, wrong owner, multi-link live file, incompatible mode, POSIX ACL grant, or symlinked or non-directory parent traversal. The default accepts any non-volatile filesystem and warns for remote or unverified families; memory-backed filesystems (tmpfs, ramfs, hugetlbfs) are always refused for the active log and exports. audit.require_certified_filesystem: true restores the strict ext4, XFS, Btrfs, APFS allowlist. Offline verification and export reject unsafe sources without mutation. | Fail-closed (startup blocked or strict request rejected) | Restore a stable trusted parent and a valid private regular file on durable storage. Do not relax the custody checks. See Compliance Controls. |
| Config file invalid | Server refuses to start. Validation errors printed to stderr with line numbers. | Fail-closed (startup blocked) | Fix config. Run oguardai config validate before deploying. |
| No policies loaded | In non-dev mode the server refuses to start (no valid policies found). In dev mode it starts but logs that protection is EFFECTIVELY DISABLED (nothing is tokenized or blocked). A request naming an unknown or missing policy is rejected with an unknown-policy error. | Fail-closed in non-dev (startup blocked); dev starts unprotected with a loud warning | Add policy files to the policy directory. Run oguardai config validate to check policy references. |
| Session blob expired | Clean error returned: GUARDAI_SESSION_EXPIRED. No data is leaked. Client must re-transform the original text to get a new session. | Fail-closed (request rejected) | Re-submit the original text through /v1/transform. Increase session.ttl_seconds if expirations are frequent. |
| Session blob tampered | AES-256-GCM authentication tag verification fails. Request rejected with GUARDAI_SESSION_EXPIRED. No data is leaked. | Fail-closed (request rejected) | Client must re-transform from original text. Investigate source of tampering. |
| Session blob rolled back (server-side store) | With a server-side Redis session backend, a per-session high-water marker rejects a load whose counter is below the highest ever stored, or whose marker is missing: 409 GUARDAI_SESSION_ROLLBACK. This catches an attacker who overwrites only the blob with a captured older same-id blob. | Fail-closed (request rejected) | Run Redis with maxmemory-policy noeviction and ACLs restricting who may write the guardai:sess: keyspace; the bundled Docker Compose stacks ship this hardening by default. Re-transform the original text for a fresh session. |
| Rate limit exceeded | HTTP 429 returned with Retry-After header. No data processed. | Fail-closed (request rejected) | Wait for rate limit window to reset. Increase rate_limit.requests_per_second or add instances. |
Redis unavailable (rate_limit.backend: redis) | The shared tenant quota lives in Redis so every replica charges one bucket. At startup the server refuses to start until Redis is reachable. At runtime a Redis error on the quota charge fails closed: the request is rejected with 503 GUARDAI_RATE_LIMIT_UNAVAILABLE, never admitted, and never falls back to a per-replica limiter. The default local backend does not use Redis. | Fail-closed (startup blocked; runtime rejects) | Restore Redis, or switch rate_limit.backend to local on a single replica. |
| Output guard detects new PII | Depending on config: mask replaces new PII with type labels, block rejects the entire response with 422 GUARDAI_OUTPUT_BLOCKED, warn passes through with warning flag. The content WAS judged and refused, so retrying the same content will not help. | Configurable | Review LLM output patterns. Adjust output guard sensitivity or action mode. |
| Output guard cannot run | 503 GUARDAI_OUTPUT_GUARD_UNAVAILABLE. The scan itself failed, so the content was NEVER judged. Distinct from a block on purpose: this is a deployment fault, not a verdict on the response, and the request is retriable once the scanner is reachable. | Fail-closed (request rejected, retriable) | Restore the output NER scanner. A policy whose detection.required_for names an NER-backed type needs detector.scan_output_with_ner on with detector.mode both or advanced, and a reachable NER detector; otherwise EVERY response under that policy fails closed regardless of content. |
| Multipart form has an unknown, nameless, duplicate, or oversized control field (upload endpoints) | 400 GUARDAI_BAD_REQUEST before entitlement, extraction, or OCR runs. Covers /v1/transform/file, /v1/transform/image, and /v1/redact/image. A typo'd control name is refused rather than silently ignored (which would have applied the default policy), a second occurrence of any declared field is refused, and a control field over 255 bytes is refused. The client-supplied field name is never echoed. | Fail-closed (request rejected) | Send exactly the documented fields, each once, with non-oversized control values. policy and language follow the JSON body's semantics. |
Document is unsupported or malformed (/v1/transform/file) | 400 GUARDAI_BAD_REQUEST. The caller's input is at fault. | Fail-closed (request rejected) | Fix or convert the document. Retrying the same bytes fails the same way. |
| Document refused by a safety ceiling | 413 GUARDAI_PAYLOAD_TOO_LARGE: the upload size limit, the PDF extraction wall clock, the sandbox address-space cap, or extracted text over the budget. The document may be perfectly valid. | Fail-closed (request rejected) | Raise file_upload.max_size_bytes or the limits.pdf_extract_* keys within their built-in ceilings, or split the document. See PDF Support. |
| Document extraction cannot run | 503 GUARDAI_DOCUMENT_EXTRACTION_UNAVAILABLE: the extraction sandbox could not be started or its pipes were unavailable. The document is not implicated. | Fail-closed (request rejected, retriable) | Retriable. The fixed 503 body carries no detail; read the recorded cause from the server log. |
| Session key rotated | With a multi-key session.keyring, blobs sealed under a retained kid keep unsealing until that key is removed or their TTL expires; a single-key secret change invalidates existing blobs. New transforms seal under current_kid. | Fail-closed (blobs under a removed key rejected) | Keep the prior key for one TTL for zero-downtime rotation; remove it to force-invalidate. |
| Tesseract missing (image endpoints) | /v1/transform/image and /v1/redact/image reject with 503 GUARDAI_OCR_UNAVAILABLE. Text and JSON paths are unaffected. Happens on the proxy image or a native binary install without the tesseract-ocr package (the official server image bundles it). | Fail-closed (image endpoints only) | Install Tesseract, or use the official server image which bundles eng/deu/ara. |
| Image OCR result cannot be verified (image endpoints) | /v1/transform/image and /v1/redact/image reject with 422 GUARDAI_OCR_UNVERIFIABLE when OCR reads no text region, or any read region falls below the confidence floor. A text-bearing image that cannot be read reliably cannot be proven free of PII, so it is refused rather than returned as clean. Distinct from 503 GUARDAI_OCR_UNAVAILABLE (engine missing) and from a 400 malformed image. | Fail-closed (image endpoints only) | Not retriable as-is: supply a clearer source image or adjust the OCR configuration. The fixed body carries no image content. |
| Prompt injection detected | Request is rejected or the injected content is neutralized, depending on prompt_security.action config. | Fail-closed (default) | Review rejected input. Adjust prompt security sensitivity if false positives occur. |
Degradation Flow
Degradation Matrix
What still works when a component is unavailable:
| Component Down | Transform | Rehydrate | Detect | Revoke | Output Guard |
|---|---|---|---|---|---|
| NER sidecar | Partial (builtin only) | Full | Partial (builtin only) | Full | Full |
Redis (when revocation_backend: redis) | Full | Full | Full | Unavailable (server won't start until Redis returns) | Full |
| Disk | Full | Full | Full | Rejected (write fails, revoke rolled back, 500) | Full |
| Policy files | Full (loaded policies retained) | Full | Full | Full | Full |
"Partial" means the operation succeeds but with reduced entity coverage: the NER-only types (person, company, location) drop out, while the builtin regex and custom-pattern types still resolve. "Degraded" means the operation succeeds but with reduced durability or consistency.
Design Principles
- Never return unprotected text. If detection cannot run, the request fails rather than passing raw PII through.
- Cryptographic verification on every unseal. Tampered or expired session blobs are always rejected.
- Stateless by default. The sealed session model has no external dependencies for a single instance. Redis and file backends are optional there, but a multi-replica deployment requires shared Redis revocation (
revocation_backend=redis) so a value revoked on one replica is refused by all. - NER failure behavior depends on mode. In
bothmode, NER unavailability falls back to builtin (fail-open), unless the policy requires an NER-backed type, in which case the request fails closed withGUARDAI_DETECTION_UNAVAILABLE(503, retriable). Inadvancedmode, NER is required: a sidecar outage (unreachable, transport error, timeout, or a degraded response) fails closed with the sameGUARDAI_DETECTION_UNAVAILABLE(503, retriable), whileGUARDAI_DETECTION_FAILED(500) is reserved for a genuine capability rejection (the sidecar responded but cannot serve the required labels). MonitorGUARDAI_DETECTION_UNAVAILABLEfor sidecar outages. - Revoked values are permanently suppressed. Even if the session blob is valid, revoked entities always return
[DELETED].
Monitoring Recommendations
- Poll
/v1/capabilitiesperiodically. Thener_activefield reflects the configured detector mode (not live sidecar status). For sidecar outages, useguardai_ner_degraded_totaltogether with theerror_code = GUARDAI_DETECTION_UNAVAILABLElog events (anadvanced-mode detector error logsGUARDAI_DETECTION_FAILEDinstead). See Reading the NER degradation counter below for exactly what the counter does and does not prove. guardai_errors_totalis a single unlabelled counter and carries no per-code breakdown, so use it only as the coarse "is anything failing" signal. For a specific code, query the structured application log or the audit stream: each failed request emits onerequest failedevent carryingerror_code,status, andrequest_id. A spike inerror_code = GUARDAI_SESSION_EXPIREDmay indicate TTL misconfiguration.- Track
guardai_rate_limit_rejections_total. Sustained rejections indicate capacity issues. - Track
guardai_output_guard_triggers_total. High trigger rate may indicate the LLM is generating PII or echoing raw session values instead of their tokens.
Reading the NER degradation counter
guardai_ner_degraded_total records a DETECTION outcome, not a request outcome. It is incremented once per protected operation whose detection produced regex-only coverage while the deployment has NER configured, which happens in two cases: the both-mode fallback when the sidecar is unreachable, and the benign case of a request that explicitly selects only builtin_regex. It is not a count of completed requests, and it is not disjoint from GUARDAI_DETECTION_UNAVAILABLE at request granularity.
What holds on every path: an advanced-mode outage, or an explicit python_ner selection that could not run, fails inside detection itself, before any path reaches the counter. So a GUARDAI_DETECTION_UNAVAILABLE (a sidecar outage) or GUARDAI_DETECTION_FAILED (a capability rejection) raised by the DETECTOR never coincides with an increment.
What does not hold globally is the counter implying a completed request. Each path places the increment at a different point relative to its remaining fail-closed steps:
| Path | Counter position | Can still fail after the increment |
|---|---|---|
/v1/transform (text), /v1/transform/image | Last step, after the policy NER requirement, the transform and the strict audit | Nothing. Here the increment does mean the request completed |
/v1/rag/ingest | After the strict audit | Nothing |
/v1/transform (input_type: json / chat_messages) | After the policy NER requirement, before the response is assembled | GUARDAI_INTERNAL_ERROR (serialization or shape check), GUARDAI_AUDIT_UNAVAILABLE (strict audit) |
/v1/detect | After the policy NER requirement | GUARDAI_AUDIT_UNAVAILABLE |
/v1/rag/query | Immediately after detection and transform | Session sealing errors, GUARDAI_AUDIT_UNAVAILABLE |
/v1/rag/context | After the chunk loop, before the anti-downgrade floor and the reseal | GUARDAI_POLICY_DENIED, sealing errors, GUARDAI_AUDIT_UNAVAILABLE |
/v1/redact/image | BEFORE the policy NER requirement is enforced | GUARDAI_DETECTION_UNAVAILABLE, GUARDAI_REDACTION_FAILED, GUARDAI_AUDIT_UNAVAILABLE |
/v1/batch/transform, /v1/batch/detect | Per successful item, inside the loop | Any other item, plus the request-level GUARDAI_AUDIT_UNAVAILABLE |
Two consequences for alerting:
/v1/redact/imageis the one path where a single request both increments the counter and returnsGUARDAI_DETECTION_UNAVAILABLE. Aboth-mode sidecar outage under a policy whosedetection.required_fornames an NER-backed type increments first, then fails the policy check. Batch reaches the same result across items: one item increments while another reportsGUARDAI_DETECTION_UNAVAILABLEin its per-itemerror.- On the structured, detect, RAG query, RAG context and batch paths the increment precedes later fail-closed steps, so the counter can move for a request that ends in
GUARDAI_POLICY_DENIED,GUARDAI_AUDIT_UNAVAILABLEorGUARDAI_INTERNAL_ERROR.
Alert on the counter as "NER coverage was lost for this much traffic", not as "this many requests succeeded without NER". To attribute an outage, correlate it with the detector mode and the per-request error_code in the structured log rather than assuming the two signals partition the traffic.
Streaming (/v1/transform/stream) and file ingestion route through the transform pipeline, so they inherit whichever row above matches their input type. /v1/rehydrate and /v1/rag/answer run no detection and never touch the counter.
Observability
How OGuardAI emits structured logs and audit events, how to set the log level and format, the OpenTelemetry span export it ships, and the liveness and readiness endpoints for the runtime and the NER sidecar.
Service Level Objectives
How to set availability, latency, and throughput objectives for an OGuardAI deployment, and the hard limits and security properties the runtime enforces