OGuardAI
Operations

Failure Modes

How OGuardAI behaves when components fail, with recovery procedures for each scenario

OGuardAI fails closed by default: when protection cannot be enforced, the request fails rather than returning unprotected text. The one deliberate exception is both mode, which is explicitly best-effort: on an NER outage it falls back to builtin regex detection and continues with reduced coverage. advanced mode fails closed instead, rejecting the request when NER is unavailable. See the matrix below for the exact behavior per failure.

Failure Matrix

Component FailureOGuardAI BehaviorFail ModeRecovery
NER sidecar down (both mode)Falls back to builtin regex detection. Person, company, and location entities are not detected.Fail-open (reduced coverage)Restart NER sidecar.
NER sidecar down (advanced mode)Request fails closed with GUARDAI_DETECTION_UNAVAILABLE (503, retriable). NER is required in advanced mode.Fail-closedRestart NER sidecar or switch to detector.mode: both.
NER sidecar slow (> detector.timeout_secs)Request times out waiting for NER. In both mode, falls back to builtin. In advanced mode, request fails.Mode-dependentScale the NER sidecar, or tune detector.timeout_secs (env GUARDAI_DETECTOR_TIMEOUT_SECS, default 5s, max 120s).
Redis unavailable (revocation_backend: redis or replay_backend: redis)At startup the server refuses to start until Redis is reachable. At runtime a Redis error on the replay check fails closed: the continuation is rejected with 503 GUARDAI_REPLAY_STORE_UNAVAILABLE, never accepted. The default memory backends do not use Redis. The Redis session backend behaves the same way: the server refuses to start until Redis is reachable.Fail-closed (startup blocked; runtime rejects)Restore Redis, or switch the affected backend to memory.
Disk fullFile-backed revocation writes fail. With strict audit, every affected request whose audit record cannot be durably written returns 503 GUARDAI_AUDIT_UNAVAILABLE; it is not acknowledged as unaudited success.Fail-closed for revocation writes and strict auditFree disk space. Treat a failed mutating request as outcome-unknown until domain state and the audit chain agree; retry with the same Idempotency-Key when supported.
Audit file custody invalidThe file sink rejects a symlink, non-regular file, wrong owner, multi-link live file, incompatible mode, POSIX ACL grant, or symlinked or non-directory parent traversal. The default accepts any non-volatile filesystem and warns for remote or unverified families; memory-backed filesystems (tmpfs, ramfs, hugetlbfs) are always refused for the active log and exports. audit.require_certified_filesystem: true restores the strict ext4, XFS, Btrfs, APFS allowlist. Offline verification and export reject unsafe sources without mutation.Fail-closed (startup blocked or strict request rejected)Restore a stable trusted parent and a valid private regular file on durable storage. Do not relax the custody checks. See Compliance Controls.
Config file invalidServer refuses to start. Validation errors printed to stderr with line numbers.Fail-closed (startup blocked)Fix config. Run oguardai config validate before deploying.
No policies loadedIn non-dev mode the server refuses to start (no valid policies found). In dev mode it starts but logs that protection is EFFECTIVELY DISABLED (nothing is tokenized or blocked). A request naming an unknown or missing policy is rejected with an unknown-policy error.Fail-closed in non-dev (startup blocked); dev starts unprotected with a loud warningAdd policy files to the policy directory. Run oguardai config validate to check policy references.
Session blob expiredClean error returned: GUARDAI_SESSION_EXPIRED. No data is leaked. Client must re-transform the original text to get a new session.Fail-closed (request rejected)Re-submit the original text through /v1/transform. Increase session.ttl_seconds if expirations are frequent.
Session blob tamperedAES-256-GCM authentication tag verification fails. Request rejected with GUARDAI_SESSION_EXPIRED. No data is leaked.Fail-closed (request rejected)Client must re-transform from original text. Investigate source of tampering.
Session blob rolled back (server-side store)With a server-side Redis session backend, a per-session high-water marker rejects a load whose counter is below the highest ever stored, or whose marker is missing: 409 GUARDAI_SESSION_ROLLBACK. This catches an attacker who overwrites only the blob with a captured older same-id blob.Fail-closed (request rejected)Run Redis with maxmemory-policy noeviction and ACLs restricting who may write the guardai:sess: keyspace; the bundled Docker Compose stacks ship this hardening by default. Re-transform the original text for a fresh session.
Rate limit exceededHTTP 429 returned with Retry-After header. No data processed.Fail-closed (request rejected)Wait for rate limit window to reset. Increase rate_limit.requests_per_second or add instances.
Redis unavailable (rate_limit.backend: redis)The shared tenant quota lives in Redis so every replica charges one bucket. At startup the server refuses to start until Redis is reachable. At runtime a Redis error on the quota charge fails closed: the request is rejected with 503 GUARDAI_RATE_LIMIT_UNAVAILABLE, never admitted, and never falls back to a per-replica limiter. The default local backend does not use Redis.Fail-closed (startup blocked; runtime rejects)Restore Redis, or switch rate_limit.backend to local on a single replica.
Output guard detects new PIIDepending on config: mask replaces new PII with type labels, block rejects the entire response with 422 GUARDAI_OUTPUT_BLOCKED, warn passes through with warning flag. The content WAS judged and refused, so retrying the same content will not help.ConfigurableReview LLM output patterns. Adjust output guard sensitivity or action mode.
Output guard cannot run503 GUARDAI_OUTPUT_GUARD_UNAVAILABLE. The scan itself failed, so the content was NEVER judged. Distinct from a block on purpose: this is a deployment fault, not a verdict on the response, and the request is retriable once the scanner is reachable.Fail-closed (request rejected, retriable)Restore the output NER scanner. A policy whose detection.required_for names an NER-backed type needs detector.scan_output_with_ner on with detector.mode both or advanced, and a reachable NER detector; otherwise EVERY response under that policy fails closed regardless of content.
Multipart form has an unknown, nameless, duplicate, or oversized control field (upload endpoints)400 GUARDAI_BAD_REQUEST before entitlement, extraction, or OCR runs. Covers /v1/transform/file, /v1/transform/image, and /v1/redact/image. A typo'd control name is refused rather than silently ignored (which would have applied the default policy), a second occurrence of any declared field is refused, and a control field over 255 bytes is refused. The client-supplied field name is never echoed.Fail-closed (request rejected)Send exactly the documented fields, each once, with non-oversized control values. policy and language follow the JSON body's semantics.
Document is unsupported or malformed (/v1/transform/file)400 GUARDAI_BAD_REQUEST. The caller's input is at fault.Fail-closed (request rejected)Fix or convert the document. Retrying the same bytes fails the same way.
Document refused by a safety ceiling413 GUARDAI_PAYLOAD_TOO_LARGE: the upload size limit, the PDF extraction wall clock, the sandbox address-space cap, or extracted text over the budget. The document may be perfectly valid.Fail-closed (request rejected)Raise file_upload.max_size_bytes or the limits.pdf_extract_* keys within their built-in ceilings, or split the document. See PDF Support.
Document extraction cannot run503 GUARDAI_DOCUMENT_EXTRACTION_UNAVAILABLE: the extraction sandbox could not be started or its pipes were unavailable. The document is not implicated.Fail-closed (request rejected, retriable)Retriable. The fixed 503 body carries no detail; read the recorded cause from the server log.
Session key rotatedWith a multi-key session.keyring, blobs sealed under a retained kid keep unsealing until that key is removed or their TTL expires; a single-key secret change invalidates existing blobs. New transforms seal under current_kid.Fail-closed (blobs under a removed key rejected)Keep the prior key for one TTL for zero-downtime rotation; remove it to force-invalidate.
Tesseract missing (image endpoints)/v1/transform/image and /v1/redact/image reject with 503 GUARDAI_OCR_UNAVAILABLE. Text and JSON paths are unaffected. Happens on the proxy image or a native binary install without the tesseract-ocr package (the official server image bundles it).Fail-closed (image endpoints only)Install Tesseract, or use the official server image which bundles eng/deu/ara.
Image OCR result cannot be verified (image endpoints)/v1/transform/image and /v1/redact/image reject with 422 GUARDAI_OCR_UNVERIFIABLE when OCR reads no text region, or any read region falls below the confidence floor. A text-bearing image that cannot be read reliably cannot be proven free of PII, so it is refused rather than returned as clean. Distinct from 503 GUARDAI_OCR_UNAVAILABLE (engine missing) and from a 400 malformed image.Fail-closed (image endpoints only)Not retriable as-is: supply a clearer source image or adjust the OCR configuration. The fixed body carries no image content.
Prompt injection detectedRequest is rejected or the injected content is neutralized, depending on prompt_security.action config.Fail-closed (default)Review rejected input. Adjust prompt security sensitivity if false positives occur.

Degradation Flow

Degradation Matrix

What still works when a component is unavailable:

Component DownTransformRehydrateDetectRevokeOutput Guard
NER sidecarPartial (builtin only)FullPartial (builtin only)FullFull
Redis (when revocation_backend: redis)FullFullFullUnavailable (server won't start until Redis returns)Full
DiskFullFullFullRejected (write fails, revoke rolled back, 500)Full
Policy filesFull (loaded policies retained)FullFullFullFull

"Partial" means the operation succeeds but with reduced entity coverage: the NER-only types (person, company, location) drop out, while the builtin regex and custom-pattern types still resolve. "Degraded" means the operation succeeds but with reduced durability or consistency.

Design Principles

  1. Never return unprotected text. If detection cannot run, the request fails rather than passing raw PII through.
  2. Cryptographic verification on every unseal. Tampered or expired session blobs are always rejected.
  3. Stateless by default. The sealed session model has no external dependencies for a single instance. Redis and file backends are optional there, but a multi-replica deployment requires shared Redis revocation (revocation_backend=redis) so a value revoked on one replica is refused by all.
  4. NER failure behavior depends on mode. In both mode, NER unavailability falls back to builtin (fail-open), unless the policy requires an NER-backed type, in which case the request fails closed with GUARDAI_DETECTION_UNAVAILABLE (503, retriable). In advanced mode, NER is required: a sidecar outage (unreachable, transport error, timeout, or a degraded response) fails closed with the same GUARDAI_DETECTION_UNAVAILABLE (503, retriable), while GUARDAI_DETECTION_FAILED (500) is reserved for a genuine capability rejection (the sidecar responded but cannot serve the required labels). Monitor GUARDAI_DETECTION_UNAVAILABLE for sidecar outages.
  5. Revoked values are permanently suppressed. Even if the session blob is valid, revoked entities always return [DELETED].

Monitoring Recommendations

  • Poll /v1/capabilities periodically. The ner_active field reflects the configured detector mode (not live sidecar status). For sidecar outages, use guardai_ner_degraded_total together with the error_code = GUARDAI_DETECTION_UNAVAILABLE log events (an advanced-mode detector error logs GUARDAI_DETECTION_FAILED instead). See Reading the NER degradation counter below for exactly what the counter does and does not prove.
  • guardai_errors_total is a single unlabelled counter and carries no per-code breakdown, so use it only as the coarse "is anything failing" signal. For a specific code, query the structured application log or the audit stream: each failed request emits one request failed event carrying error_code, status, and request_id. A spike in error_code = GUARDAI_SESSION_EXPIRED may indicate TTL misconfiguration.
  • Track guardai_rate_limit_rejections_total. Sustained rejections indicate capacity issues.
  • Track guardai_output_guard_triggers_total. High trigger rate may indicate the LLM is generating PII or echoing raw session values instead of their tokens.

Reading the NER degradation counter

guardai_ner_degraded_total records a DETECTION outcome, not a request outcome. It is incremented once per protected operation whose detection produced regex-only coverage while the deployment has NER configured, which happens in two cases: the both-mode fallback when the sidecar is unreachable, and the benign case of a request that explicitly selects only builtin_regex. It is not a count of completed requests, and it is not disjoint from GUARDAI_DETECTION_UNAVAILABLE at request granularity.

What holds on every path: an advanced-mode outage, or an explicit python_ner selection that could not run, fails inside detection itself, before any path reaches the counter. So a GUARDAI_DETECTION_UNAVAILABLE (a sidecar outage) or GUARDAI_DETECTION_FAILED (a capability rejection) raised by the DETECTOR never coincides with an increment.

What does not hold globally is the counter implying a completed request. Each path places the increment at a different point relative to its remaining fail-closed steps:

PathCounter positionCan still fail after the increment
/v1/transform (text), /v1/transform/imageLast step, after the policy NER requirement, the transform and the strict auditNothing. Here the increment does mean the request completed
/v1/rag/ingestAfter the strict auditNothing
/v1/transform (input_type: json / chat_messages)After the policy NER requirement, before the response is assembledGUARDAI_INTERNAL_ERROR (serialization or shape check), GUARDAI_AUDIT_UNAVAILABLE (strict audit)
/v1/detectAfter the policy NER requirementGUARDAI_AUDIT_UNAVAILABLE
/v1/rag/queryImmediately after detection and transformSession sealing errors, GUARDAI_AUDIT_UNAVAILABLE
/v1/rag/contextAfter the chunk loop, before the anti-downgrade floor and the resealGUARDAI_POLICY_DENIED, sealing errors, GUARDAI_AUDIT_UNAVAILABLE
/v1/redact/imageBEFORE the policy NER requirement is enforcedGUARDAI_DETECTION_UNAVAILABLE, GUARDAI_REDACTION_FAILED, GUARDAI_AUDIT_UNAVAILABLE
/v1/batch/transform, /v1/batch/detectPer successful item, inside the loopAny other item, plus the request-level GUARDAI_AUDIT_UNAVAILABLE

Two consequences for alerting:

  1. /v1/redact/image is the one path where a single request both increments the counter and returns GUARDAI_DETECTION_UNAVAILABLE. A both-mode sidecar outage under a policy whose detection.required_for names an NER-backed type increments first, then fails the policy check. Batch reaches the same result across items: one item increments while another reports GUARDAI_DETECTION_UNAVAILABLE in its per-item error.
  2. On the structured, detect, RAG query, RAG context and batch paths the increment precedes later fail-closed steps, so the counter can move for a request that ends in GUARDAI_POLICY_DENIED, GUARDAI_AUDIT_UNAVAILABLE or GUARDAI_INTERNAL_ERROR.

Alert on the counter as "NER coverage was lost for this much traffic", not as "this many requests succeeded without NER". To attribute an outage, correlate it with the detector mode and the per-request error_code in the structured log rather than assuming the two signals partition the traffic.

Streaming (/v1/transform/stream) and file ingestion route through the transform pipeline, so they inherit whichever row above matches their input type. /v1/rehydrate and /v1/rag/answer run no detection and never touch the counter.