OGuardAI
Operations

Deployment Guide

Deploy OGuardAI from local development to production Kubernetes clusters

This guide covers every deployment mode from local development to production Kubernetes clusters.

Note: The current runtime uses sealed sessions (client-carried encrypted blobs), so no shared session store is required, and each instance is stateless for session handling. A multi-replica deployment still needs a shared revocation backend (redis), because a value revoked on one replica must be refused by all.


Table of Contents

  1. Prerequisites
  2. Quick Start (Docker)
  3. Development (Local Build)
  4. Production (Docker Compose)
  5. Kubernetes (Helm)
  6. Configuration Reference
  7. Session Backend Selection
  8. Key Rotation Procedure
  9. Monitoring and Alerting
  10. Troubleshooting
  11. Rate Limiting in Multi-Instance Deployments
  12. Metrics in Multi-Instance Deployments
  13. Detector Mode Selection

Prerequisites

RequirementVersionPurpose
Docker24+Container runtime
Docker Compose2.20+Multi-service orchestration
Helm3.12+Kubernetes deployment (optional)
Kubernetes1.27+Container orchestration (optional)
Rust1.88+Building from source (optional)
Python3.10+NER detector sidecar (optional)
Node.js20+SDK and MCP server (optional)

Quick Start (Docker)

Run OGuardAI with a single command using the pre-built Docker image:

docker run -p 3000:3000 \
  -e GUARDAI_SESSION_SECRET=$(openssl rand -base64 32) \
  ghcr.io/oronts/oronts-guardai/oguardai-server:latest

This starts OGuardAI with:

  • Builtin regex detectors (30+ patterns)
  • Sealed session backend (client-carried encrypted blobs)
  • Dev auth mode (all requests accepted, not for production)
  • Default policy
  • Bundled Tesseract OCR (English, German, Arabic), so the /v1/transform/image and /v1/redact/image endpoints work out of the box

Verifying image provenance

Published images are Trivy-scanned before push, signed with cosign (keyless, via Sigstore Fulcio/Rekor), and carry an SPDX SBOM attestation. Verify the signature and provenance before running in production (cosign and jq required):

IMAGE=ghcr.io/oronts/oronts-guardai/oguardai-server:latest
IDENTITY='https://github.com/oronts/.*/.github/workflows/docker.yml@.*'
ISSUER=https://token.actions.githubusercontent.com

# Verify the keyless signature (identity = the release workflow's OIDC subject).
cosign verify "$IMAGE" --certificate-identity-regexp "$IDENTITY" --certificate-oidc-issuer "$ISSUER"

# Verify and extract the SPDX SBOM attestation.
cosign verify-attestation "$IMAGE" --type spdxjson \
  --certificate-identity-regexp "$IDENTITY" --certificate-oidc-issuer "$ISSUER" \
  | jq -r '.payload | @base64d | fromjson | .predicate' > sbom.spdx.json

A failed cosign verify means the image was not built by the official pipeline; do not run it.

Verify the server is running:

curl http://localhost:3000/v1/health
# {"status":"healthy","version":"0.1.0","uptime_seconds":1.2}

Test a transform:

curl -X POST http://localhost:3000/v1/transform \
  -H "Content-Type: application/json" \
  -d '{"input": "Contact julia@example.com for help"}'

Expected response:

{
  "safe_text": "Contact {{email:e_001:7b1068662ab5}} for help",
  "session_id": "...",
  "session_state": "...",
  "entities": [{"token": "{{email:e_001:7b1068662ab5}}", "type": "email"}]
}

The third token segment is a random per-token capability suffix; rehydrate resolves a token only on an exact {{type:id:cap}} match (fail closed), so LLM output must be passed back verbatim.

Image OCR: The official server image bundles Tesseract with English, German, and Arabic language data, so /v1/transform/image and /v1/redact/image work without extra setup. Only from-source builds need Tesseract installed on the host, and the separate proxy image does not include it (see PDF Support).


Development (Local Build)

Build from Source

# Clone the repository
git clone https://github.com/oronts/oguardai.git
cd oguardai

# Build all Rust crates
cargo build --workspace

# Run all tests
cargo test --workspace

# Start the server with defaults
cargo run -p oguardai-server -- --config oguardai.yaml

Minimal Development Config

Create oguardai.yaml for local development:

server:
  host: "127.0.0.1"
  port: 3000

auth:
  mode: dev

session:
  backend: sealed
  secret: "dev-only-secret-change-in-prod!!"
  ttl_seconds: 3600

detector:
  mode: builtin

policy:
  default: default
  directory: policies

transform:
  context_strategy: full
  max_context_tokens: 4096

Running with Python Detector (Development)

# Terminal 1: Start Python NER detector
cd apps/detector-py
uv sync
uv run uvicorn guardai_detector_service.main:app --host 0.0.0.0 --port 9090

# Terminal 2: Start Rust server with detector URL
GUARDAI_DETECTOR_URL=http://localhost:9090 \
  cargo run -p oguardai-server -- --config oguardai.yaml

Production (Docker Compose)

Docker Compose deploys three services: the OGuardAI server, the Python NER detector, and a Redis instance for durable revocation state. Sessions use the sealed backend (client-carried encrypted blobs), so no external session store is required; Redis here holds only revocation digests, not session state.

Environment Setup

Create a .env file (never commit this to version control):

# Required: 32-byte session encryption secret
GUARDAI_SESSION_SECRET=$(openssl rand -base64 32)

# Optional: log level
GUARDAI_LOG_LEVEL=guardai_server=info,tower_http=info

Start the Full Stack

# Required environment variables:
export GUARDAI_SESSION_SECRET="$(openssl rand -base64 32)"
export GUARDAI_API_KEY_1="$(openssl rand -hex 32)"

docker compose -f deploy/docker/docker-compose.yml up --build

# Test with authenticated request:
# curl -H "X-API-Key: $GUARDAI_API_KEY_1" http://localhost:3000/v1/health

This starts:

ServicePortDescription
server3000OGuardAI Rust server
detector9090 (internal)Python NER detector (GLiNER/spaCy), reachable only on the compose network
redis6379 (internal)Revocation store (redis:7-alpine, append-only)

Production Docker Compose Configuration

The production docker-compose.yml at deploy/docker/docker-compose.yml includes:

  • Health checks on all three services (HTTP readiness for server and detector, redis-cli ping for Redis).
  • Restart policy: unless-stopped for automatic recovery.
  • Volume mounts: Policies directory mounted read-only; a named redis-data volume for revocation persistence.
  • Dependency ordering: The server waits for both the detector and Redis to be healthy before starting.
  • Auth: The server runs in api_key mode (GUARDAI_AUTH_MODE=api_key) and requires GUARDAI_API_KEY_1 to be set before the stack starts.
  • Durable revocation: The server sets GUARDAI_REVOCATION_BACKEND=redis and GUARDAI_REDIS_URL=redis://redis:6379. Redis persists only HMAC-SHA-256 digests of revoked (type, value) pairs (append-only), never raw PII, so a value revoked once survives a restart.
  • Durable, fail-closed audit trail: The server sets GUARDAI_AUDIT_BACKEND=file, GUARDAI_AUDIT_FILE_PATH=/var/lib/guardai/audit/audit.log (on the named guardai-audit volume), and GUARDAI_AUDIT_STRICT=true. A non-dev start requires this durable, strict trail; set GUARDAI_ALLOW_EPHEMERAL_AUDIT=1 only to explicitly accept an ephemeral one.
  • Detector auth: The detector is not published on a host port and runs with GUARDAI_DETECTOR_ALLOW_INSECURE=true because it is reachable only on the internal compose network. If you expose it, set GUARDAI_DETECTOR_API_KEY (the X-Detector-API-Key shared secret) on both the detector and the server instead.

Minimal Docker Compose (Server Only)

For deployments that do not need advanced NER or shared sessions:

docker compose -f deploy/docker/docker-compose.minimal.yml up --build

This starts only the OGuardAI server with builtin detectors and sealed sessions.


Kubernetes (Helm)

Install the Helm Chart

# Add the OGuardAI Helm repository (when published)
helm repo add oguardai https://oronts.github.io/oguardai/charts
helm repo update

# Install with default values
helm install oguardai oguardai/oguardai \
  --namespace oguardai \
  --create-namespace \
  --set session.secret="$(openssl rand -base64 32)" \
  --set auth.apiKeys[0].name=default \
  --set auth.apiKeys[0].key="$(openssl rand -hex 32)" \
  --set audit.backend=file \
  --set audit.strict=true

# Or install from local chart
helm install oguardai deploy/helm/oguardai \
  --namespace oguardai \
  --create-namespace \
  --set session.secret="$(openssl rand -base64 32)" \
  --set auth.apiKeys[0].name=default \
  --set auth.apiKeys[0].key="$(openssl rand -hex 32)" \
  --set audit.backend=file \
  --set audit.strict=true

The chart defaults to auth.mode=api_key, and a non-dev server start requires a durable, strict audit trail (audit.backend=file plus audit.strict=true), so the install commands above set both. The chart mounts a volume at the audit path's parent directory; it is an emptyDir by default, so back it with a PersistentVolume for a trail that survives pod restarts.

Production Helm Values

Create values-production.yaml:

server:
  replicaCount: 3
  resources:
    requests:
      cpu: 500m
      memory: 512Mi
    limits:
      cpu: "2"
      memory: 1Gi

# replicaCount > 1 requires the shared redis backends (revocation + idempotency),
# or the chart refuses to render.
revocation:
  backend: redis
idempotency:
  backend: redis
redis:
  external:
    url: redis://redis:6379

session:
  backend: sealed
  ttlSeconds: 3600
  # Reference an existing Kubernetes secret
  existingSecret: oguardai-session-secret
  existingSecretKey: session-secret

auth:
  mode: api_key
  # API keys must use the name/key format matching the Helm chart template.
  # The template generates env vars GUARDAI_API_KEY_{NAME} from each entry.
  apiKeys:
    - name: my-service
      key: "GENERATE_WITH_openssl_rand_hex_32"  # Replace before deploying

policy:
  default: default
  directory: /app/policies

# Durable, fail-closed audit trail. REQUIRED for a non-dev (api_key/jwt/oidc) start:
# the server refuses to boot with the ephemeral log backend unless
# GUARDAI_ALLOW_EPHEMERAL_AUDIT=1 is set. Back the audit volume with a
# PersistentVolume; the default emptyDir does not survive pod restarts.
audit:
  backend: file
  strict: true
  file:
    path: /audit/audit.log

transform:
  contextStrategy: full
  maxContextTokens: 4096

# Enable Python NER detector
detector:
  enabled: true
  # The detector handles raw PII and fails closed (503) without the shared
  # X-Detector-API-Key secret. When detector.enabled, the chart requires
  # auth.apiKey or auth.existingSecret (or the explicit dev-only opt-out
  # auth.allowInsecure=true).
  auth:
    existingSecret: oguardai-detector-auth
    existingSecretKey: detector-api-key
  resources:
    requests:
      cpu: 500m
      memory: 1Gi
    limits:
      cpu: "2"
      memory: 2Gi

# Enable horizontal pod autoscaler
autoscaling:
  enabled: true
  minReplicas: 3
  maxReplicas: 10
  targetCPUUtilizationPercentage: 70

# Shared revocation is REQUIRED for more than one replica. The chart refuses to
# install when replicaCount > 1 or autoscaling.enabled unless revocation.backend
# is redis with an external URL: the default memory backend is pod-local, so a
# value revoked on one pod would not be refused by the others. Redis stores only
# HMAC-SHA-256 digests of revoked (type, value) pairs, never raw PII.
revocation:
  backend: redis

redis:
  external:
    url: "redis://redis-master.oguardai.svc.cluster.local:6379"

# Pod security
podSecurityContext:
  runAsNonRoot: true
  runAsUser: 1000
  fsGroup: 1000

securityContext:
  readOnlyRootFilesystem: true
  allowPrivilegeEscalation: false
  capabilities:
    drop:
      - ALL

Install with production values:

helm install oguardai deploy/helm/oguardai \
  --namespace oguardai \
  --create-namespace \
  -f values-production.yaml

Using External Secrets

To use an external secret manager (Vault, AWS Secrets Manager) with the Helm chart:

# Create the Kubernetes secret first
kubectl create secret generic oguardai-session-secret \
  --namespace oguardai \
  --from-literal=session-secret="$(openssl rand -base64 32)"

# Reference it in Helm values
helm install oguardai deploy/helm/oguardai \
  --set session.existingSecret=oguardai-session-secret \
  --set session.existingSecretKey=session-secret \
  --set auth.apiKeys[0].name=default \
  --set auth.apiKeys[0].key="$(openssl rand -hex 32)"

Helm Chart Components

The chart deploys the following Kubernetes resources:

ResourcePurpose
DeploymentOGuardAI server pods
ServiceClusterIP service for internal access
ConfigMapoguardai.yaml configuration
SecretSession encryption secret
ServiceAccountPod identity
HPAHorizontal pod autoscaler (optional, autoscaling.enabled)
PodDisruptionBudgetAvailability during voluntary disruptions (auto-enabled when replicaCount > 1, or forced via pdb.enabled)
NetworkPolicyIngress/egress restrictions (optional, networkPolicy.enabled)
IngressExternal HTTP(S) routing (optional, ingress.enabled)

Configuration Reference

The oguardai.yaml file controls all server behavior. Every field has a sensible default.

Server Section

server:
  host: "0.0.0.0"      # Bind address (default: 0.0.0.0)
  port: 3000            # Bind port (default: 3000)
  allowed_origins:      # CORS allowlist enforced in non-dev auth modes; env GUARDAI_CORS_ORIGINS (comma-separated)
    - "https://app.example.com"
  tls:                  # Optional in-process TLS; TLS is usually terminated at the ingress instead
    enabled: false
    cert_path: "/etc/oguardai/tls.crt"   # env GUARDAI_TLS_CERT
    key_path: "/etc/oguardai/tls.key"    # env GUARDAI_TLS_KEY

Auth Section

auth:
  mode: api_key         # dev | api_key | jwt | oidc (default: dev)
  api_keys:             # Only used when mode=api_key
    - key: sk-...
      identity: my-service
      tenant_id: tenant-1  # Optional: multi-tenancy scope
      scopes:           # Optional: restrict to specific scopes
        - transform
        - rehydrate
  # JWT config (only used when mode=jwt)
  jwt:
    secret: "your-hmac-secret"
    issuer: "https://auth.example.com"
    audience: "guardai-api"   # Expected `aud` claim; a token whose audience differs is rejected
  # OIDC config (only used when mode=oidc)
  oidc:
    issuer: "https://issuer.example.com"
    jwks_url: "https://issuer.example.com/.well-known/jwks.json"
    audience: "guardai-api"
    tenant_claim: "tenant_id"   # Claim that carries the tenant identifier (required)
    algorithms: [RS256]         # Asymmetric allowlist: RS256/384/512, ES256/384, PS256/384/512

Note: The api_keys entries use key, identity, tenant_id, and scopes fields.

Session Section

session:
  backend: sealed       # sealed (default: sealed)
  secret: "..."         # 32-byte encryption secret (REQUIRED for production)
  ttl_seconds: 3600     # Session TTL in seconds (default: 3600)

The GUARDAI_SESSION_SECRET environment variable overrides session.secret.

Supported backends: sealed (default, stateless, cross-replica), memory (in-process, dev/test, single-instance), and redis (server-side sessions encrypted at rest in a shared Redis, cross-replica HA; set session.redis_url, replay_backend: redis, and revocation_backend: redis). managed is planned.

Revocation Backend

Revocation lists which entity values must be refused at rehydrate (only HMAC digests are stored, never raw PII).

revocation_backend: memory   # memory (file-backed, per-instance) | redis (shared)
replay_backend: memory       # memory (per-instance) | redis (shared); rejects replayed continuations
session:
  redis_url: "redis://redis:6379"   # required when revocation_backend or replay_backend is redis

GUARDAI_REVOCATION_BACKEND, GUARDAI_REPLAY_BACKEND, and GUARDAI_REDIS_URL override these. Use redis whenever you run more than one instance, so a value revoked on one is refused by all. The provided deploy/docker/docker-compose.ha.yml wires this.

Detector Section

detector:
  mode: builtin         # builtin | advanced | both (default: builtin)
  advanced_url: "http://detector:9090"  # URL of Python NER sidecar
  default_language: null  # ISO 639-1 fallback when a request omits its own language; null falls back to "en" (no auto-detect)
  timeout_secs: 5  # NER request timeout (max 120); env GUARDAI_DETECTOR_TIMEOUT_SECS
ModeBehavior
builtinRust regex patterns only (30+ patterns)
advancedPython NER sidecar only
bothBoth builtin and advanced; merge results

Policy Section

policy:
  default: default      # Default policy name (default: "default")
  directory: policies   # Directory containing policy YAML files (default: "policies")

Transform Section

transform:
  context_strategy: full   # full | type_summary | referenced_only | none (default: full)
  max_context_tokens: 4096 # Max tokens for entity_context (default: 4096)

Output Protection Section

output_protection:
  enabled: true         # Output guard is enabled by default
  mode: strict          # strict | permissive (default: strict)
  default_action: mask  # allow | warn | mask | block (default: mask)
  exempt_types:         # Entity types exempt from output protection
    - greeting

Prompt Security Section

prompt_security:
  enabled: true         # Enable prompt injection scanning (default: true)
  action: strip         # warn | strip | block (default: strip)

File Upload Section

file_upload:
  max_size_bytes: 52428800  # 50MB default

Rate Limit Section

rate_limit:
  enabled: false        # Enable rate limiting (default: false)
  requests_per_second: 100  # Global rate limit (default: 100)
  burst_size: 200       # Burst allowance (default: 200)

Audit Section

audit:
  backend: file         # log (ephemeral, fail-open; the default) | file (durable, hash-chained)
  strict: true          # fail a request closed (503) when its audit event cannot be recorded (default: false)
  file:
    path: /var/lib/guardai/audit/audit.log   # required when backend=file
  allow_ephemeral: false  # explicit waiver of the non-dev durable, strict requirement

A non-dev (api_key/jwt/oidc) start requires the durable, strict trail (backend: file plus strict: true); it refuses to boot otherwise. Set audit.allow_ephemeral: true (or GUARDAI_ALLOW_EPHEMERAL_AUDIT=1) to explicitly accept an ephemeral or non-strict trail. The --compliance preset always forces file + strict and ignores the waiver.

Idempotency Section

idempotency:
  backend: memory              # memory (per-replica) | redis (shared; uses session.redis_url)
  ttl_seconds: 86400           # how long a completed Idempotency-Key result is replayed (default: 86400)
  in_progress_ttl_seconds: 60  # how long an in-flight claim blocks a duplicate before it is reclaimable (default: 60)

Mutating, non-streaming routes (transform, batch transform, RAG ingest/delete, revoke, session delete) accept an Idempotency-Key header so a client can retry without executing the mutation twice.

Tenants Section

tenants:
  acme-corp:
    default_policy: gdpr-strict
    rate_limit:
      requests_per_second: 50
      burst_size: 100
  startup-inc:
    default_policy: default
    rate_limit:
      requests_per_second: 200
      burst_size: 400

Environment Variable Overrides

Environment VariableOverridesExample
GUARDAI_SESSION_SECRETsession.secret32-byte production secret
GUARDAI_DETECTOR_URLdetector.advanced_urlhttp://detector:9090
GUARDAI_DETECTOR_TIMEOUT_SECSdetector.timeout_secs5 (max 120)
GUARDAI_AUTH_MODEauth.modeapi_key
GUARDAI_API_KEY_1 .. GUARDAI_API_KEY_NInjects auth.api_keys entriesopenssl rand -hex 32
GUARDAI_REVOCATION_BACKENDrevocation_backendredis
GUARDAI_REPLAY_BACKENDreplay_backendredis
GUARDAI_REDIS_URLsession.redis_url (revocation/replay/session Redis)redis://redis:6379
GUARDAI_AUDIT_BACKENDaudit.backendfile
GUARDAI_AUDIT_FILE_PATHaudit.file.path/var/lib/guardai/audit/audit.log
GUARDAI_AUDIT_STRICTaudit.stricttrue
GUARDAI_ALLOW_EPHEMERAL_AUDITaudit.allow_ephemeral (waive the non-dev durable, strict audit requirement)1
GUARDAI_DETECTOR_API_KEYShared secret for the server-to-detector hop (X-Detector-API-Key); set on both sidesopenssl rand -hex 32
GUARDAI_IDEMPOTENCY_BACKENDidempotency.backendredis
GUARDAI_IDEMPOTENCY_TTL_SECONDSidempotency.ttl_seconds86400
GUARDAI_IDEMPOTENCY_IN_PROGRESS_TTL_SECONDSidempotency.in_progress_ttl_seconds60
GUARDAI_HOSTserver.host0.0.0.0
GUARDAI_PORTserver.port3000
RUST_LOGLog level filterguardai_server=info,tower_http=info

The provided compose stack relies on GUARDAI_AUTH_MODE, GUARDAI_API_KEY_1, GUARDAI_REVOCATION_BACKEND, GUARDAI_REDIS_URL, and the GUARDAI_AUDIT_* trio above. deploy/systemd/oguardai.env.example is the env-file starting point for systemd deployments; add the GUARDAI_AUDIT_* variables to it, since a non-dev start requires the durable, strict audit trail.


Session Backend Selection

BackendUse CaseStatus
Sealed (default)Stateless deployments, horizontal scaling, edge deploymentsAvailable
MemoryDevelopment, testing, single-instance demosAvailable
RedisMulti-instance production, shared state, server-side session management (encrypted at rest)Shipped

Sealed Backend

The sealed backend is the default session backend. Session state is encrypted (AES-GCM) and returned to the client as an opaque blob. The client must carry this blob between transform and rehydrate calls.

Advantages:

  • No server-side state; horizontal scaling on the session path with no session store or session affinity. A multi-replica deployment still shares one dependency, the revocation store (revocation_backend=redis), so an erased value stays erased across replicas.
  • You want the client to control session lifecycle.

Considerations:

  • Blob size grows with entity count.
  • Client must carry blob between requests.
  • Your entity counts per request should be moderate (< 100 entities).

The memory backend is available for dev/test (in-process, lost on restart, single-instance). The redis session backend ships for multi-instance shared session state: each session is encrypted at rest (AES-256-GCM) so Redis holds only ciphertext. The Redis revocation and replay backends also ship for multi-instance shared state.


Key Rotation Procedure

OGuardAI seals through a key ring indexed by key ID (kid). New blobs seal under current_kid; blobs sealed under any other key still in the ring keep decrypting until that key is removed. This supports both a simple single-key change and a zero-downtime multi-key rotation.

Single-key change (brief disruption)

With no session.keyring block the runtime holds one key derived from session.secret. Replacing that secret invalidates every existing blob, so do it in a low-traffic window.

# Generate a cryptographically random 32-byte secret
openssl rand -base64 32
session:
  secret: "new-production-secret-32-bytes!!"

After deploy, blobs sealed under the old secret fail authentication and return GUARDAI_SESSION_EXPIRED; clients re-transform the original text to get a new blob.

Zero-downtime rotation (multi-key keyring)

Configure a session.keyring and roll the key in stages so in-flight sessions never break:

  1. Add the new key (kid 1) but keep current_kid on the old key, then deploy to every replica so all of them know kid 1 before any pod seals under it:
session:
  keyring:
    current_kid: 0
    keys:
      - kid: 0
        secret: "${OLD_SECRET}"
      - kid: 1
        secret: "${NEW_SECRET}"
  1. Switch current_kid to 1 (keep both keys): new sessions seal under kid 1 while kid 0 blobs still decrypt.
  2. After one session TTL, remove kid 0 from the ring. Blobs sealed under it then fail to decrypt.

See the key rotation runbook for the full procedure and ordering constraints.

Key Rotation Timeline

Single-key change:
  Old key active   |  Deploy new secret  |  New key only
  (all sessions)   |  (old blobs become  |  (clients re-transform
                   |   undecryptable)    |   to get new blobs)

Zero-downtime keyring:
  Seal under kid 0 | Add kid 1, seal kid 0 | Switch current_kid 1 | Remove kid 0
                   |                       | (kid 0 decrypts      | (kid 0 blobs
                   |                       |  until TTL)          |  now invalid)

Monitoring and Alerting

Health Check

The /v1/health endpoint returns the overall system status (requires auth when enabled; use /livez for unauthenticated probes):

curl http://localhost:3000/livez                    # Unauthenticated liveness probe
curl -H "X-API-Key: $GUARDAI_API_KEY" http://localhost:3000/v1/health  # Full health check

Response:

{
  "status": "healthy",
  "version": "0.1.0",
  "uptime_seconds": 3600.5,
  "components": {
    "detector": {"status": "healthy", "message": "builtin_and_ner (full entity detection)"},
    "session": {"status": "healthy", "message": "sealed"},
    "policies": {"status": "healthy", "message": "3 policies loaded"}
  }
}

Status values: healthy, degraded (partial functionality), unhealthy (service failure).

What to Monitor

MetricSourceAlert Threshold
HTTP response codesAccess logs / reverse proxy5xx rate > 1%
Request latency (p95)Structured logs (latency_ms field)> 500ms for transform, > 200ms for rehydrate
Health check status/v1/health endpointStatus != "healthy"
Entity detection rateStructured logs (entity_count field)Sudden drop may indicate detector failure
Session seal/unseal errorsStructured logs (error events)Any GUARDAI_SESSION_EXPIRED spike
Rate limit rejectionsStructured logsSustained 429 responses
Container restartsKubernetes / Docker> 0 in 5-minute window
Memory usageContainer metrics> 80% of limit
CPU usageContainer metricsSustained > 70%
Detector sidecar (NER)/readyz + metrics/readyz returns 503, or guardai_ner_degraded_total increases
# Example Prometheus alerting rules
groups:
  - name: guardai
    rules:
      - alert: OGuardAIUnhealthy
        expr: probe_success{job="guardai-health"} == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "OGuardAI health check failing"

      - alert: OGuardAIHighLatency
        expr: histogram_quantile(0.95, rate(guardai_transform_duration_seconds_bucket{job="guardai"}[5m])) > 0.5
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "OGuardAI p95 latency above 500ms"

      - alert: OGuardAIHighErrorRate
        expr: rate(guardai_errors_total{job="guardai"}[5m]) / (rate(guardai_transforms_total{job="guardai"}[5m]) + rate(guardai_rehydrates_total{job="guardai"}[5m]) + 1) > 0.01
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "OGuardAI 5xx error rate above 1%"

Log Aggregation

OGuardAI emits structured JSON logs compatible with all major SIEM and log aggregation systems:

{
  "timestamp": "2026-04-15T10:30:00Z",
  "level": "INFO",
  "target": "guardai_server::pipeline",
  "message": "transform completed",
  "request_id": "req-abc-123",
  "session_id": "sess-def-456",
  "tenant_id": "acme-corp",
  "entity_count": 3,
  "latency_ms": 12.5,
  "policy": "gdpr-strict"
}

Forward logs to your SIEM using standard log collectors (Fluent Bit, Fluentd, Vector, Filebeat).


Troubleshooting

Server does not start

Symptom: oguardai-server exits immediately.

Causes and fixes:

CauseFix
Port already in useChange server.port in config or stop conflicting process
Invalid YAML configRun oguardai config validate or check YAML syntax
Missing policies directoryCreate policies/ directory or update policy.directory

"session expired" errors on rehydrate

Symptom: Rehydrate returns {"error": "...", "code": "GUARDAI_SESSION_EXPIRED"}.

Causes and fixes:

CauseFix
Session TTL exceededIncrease session.ttl_seconds or call rehydrate sooner
Different session secret between transform and rehydrateEnsure all server instances use the same GUARDAI_SESSION_SECRET
Key rotation mid-flightWith a multi-key session.keyring, rotation is zero-downtime: keep the prior key in the ring so in-flight blobs unseal under their kid until removed or TTL-expired, while new sessions seal under current_kid. A single-key secret change is disruptive; rotate it in a maintenance window. Removing a key invalidates blobs sealed under it

Detector sidecar not connecting

Symptom: /readyz returns 503, or guardai_ner_degraded_total / GUARDAI_DETECTION_FAILED increase. Note that /v1/health always reports the detector healthy even when a configured NER sidecar is unreachable, so use /readyz (which probes the sidecar) and ner_active on /v1/capabilities to detect a downed sidecar.

Causes and fixes:

CauseFix
Sidecar not startedCheck docker compose logs detector
Wrong URLVerify detector.advanced_url matches sidecar address
Sidecar still loading modelsWait for sidecar health check to pass (may take 20-30s on first start)
Network isolationEnsure server and detector are on the same Docker/Kubernetes network

High latency on first request

Symptom: First request takes significantly longer than subsequent requests.

Cause: Regex patterns and detection models are compiled on first use.

Fix: This is expected behavior. Subsequent requests reuse compiled patterns. For consistent latency, send a warmup request after deployment.

Entity not detected

Symptom: Known PII is not being replaced.

Causes and fixes:

CauseFix
Entity type not in policyCheck policy YAML includes the entity type
Below confidence thresholdLower threshold in detect request or policy
Builtin-only modeNames and companies require the Python NER sidecar
Non-standard formatBuiltin patterns cover common formats; unusual formats may need custom patterns

Session seal/unseal failures

Symptom: GUARDAI_SESSION_EXPIRED or seal/unseal errors on rehydrate. The /v1/health session component always reports healthy and only echoes the configured backend name; it does not surface per-request seal failures, so watch the error codes and logs rather than the health status.

Causes and fixes:

CauseFix
Wrong or missing session secretVerify GUARDAI_SESSION_SECRET is set and matches across all instances
Key rotation mid-flightWith a multi-key session.keyring, rotation is zero-downtime: keep the prior key in the ring so in-flight blobs unseal under their kid until removed or TTL-expired, while new sessions seal under current_kid. A single-key secret change is disruptive; rotate it in a maintenance window. Removing a key invalidates blobs sealed under it
Tampered or corrupted blobClient must pass the exact session_state blob received from transform
Session TTL exceededIncrease session.ttl_seconds or call rehydrate sooner

Rate Limiting in Multi-Instance Deployments

OGuardAI's rate limiter operates per-instance. Each server process maintains independent rate limit buckets.

Implications for Kubernetes / multi-instance:

DeploymentEffective Rate LimitExample
Single instance, limit=100/s100/s totalAs configured
3 instances, limit=100/s eachUp to 300/s totalClients hitting different instances get 100/s each
3 instances behind load balancer~100/s per client (sticky) or ~300/s (round-robin)Depends on LB strategy

For strict per-client limits: Use sticky sessions (session affinity) in your load balancer so each client always hits the same instance.

For global limits: Divide the desired global limit by the number of instances:

# 3 instances, want 300 req/s global -> 100/s per instance
rate_limit:
  enabled: true
  requests_per_second: 100
  burst_size: 200

For shared rate limiting (advanced): Deploy a Redis-backed rate limiter in front of OGuardAI (e.g., Kong, Envoy, or API gateway rate limiting). OGuardAI's built-in rate limiter then serves as a secondary defense.


Metrics in Multi-Instance Deployments

Each OGuardAI instance exposes its own /metrics endpoint. Prometheus scrapes all instances independently.

Prometheus scrape config:

# prometheus.yml
scrape_configs:
  - job_name: 'guardai'
    # /metrics requires the `admin` scope whenever auth is enabled. Present an admin
    # credential: an Authorization: Bearer token in jwt/oidc mode, or route the scrape
    # through a proxy that injects the X-API-Key header in api_key mode.
    authorization:
      type: Bearer
      credentials: '<admin-scoped-jwt>'
    kubernetes_sd_configs:
      - role: pod
        selectors:
          - role: pod
            label: app=oguardai-server
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_ip]
        target_label: __address__
        replacement: '$1:3000'

Aggregation in PromQL:

# Total transforms across all instances
sum(guardai_transforms_total)

# Per-instance error rate
rate(guardai_errors_total[5m])

# p95 latency across all instances
histogram_quantile(0.95, sum(rate(guardai_transform_duration_seconds_bucket[5m])) by (le))

Key point:

Per-instance metrics is the standard pattern for Prometheus-based observability. No shared metrics backend needed.


Detector Mode Selection

Quick Decision Guide

Your Use CaseRecommended ModeWhy
Low-latency API protectionbuiltinSub-millisecond, no dependencies
Full PII coverage (names, companies)bothBuiltin + NER gives best coverage
Maximum accuracy, latency acceptableadvancedAll detection via trained models
NER sidecar sometimes unavailablebothGraceful fallback to builtin

How to Check Current Mode

# Health endpoint shows detector status (requires auth when enabled)
curl -H "X-API-Key: $GUARDAI_API_KEY" http://localhost:3000/v1/health | jq '.components.detector'

# Capabilities shows what entity types are available
curl -H "X-API-Key: $GUARDAI_API_KEY" http://localhost:3000/v1/capabilities | jq '.entity_types[].name'

# Diagnostics shows full config (requires admin scope)
curl -H "X-API-Key: $GUARDAI_API_KEY" http://localhost:3000/v1/diagnostics | jq '.detector_mode'

NER Sidecar Availability

When detector.mode: both is configured:

NER StatusHealth ShowsCapabilities ShowsBehavior
Running + healthybuiltin_and_ner (full entity detection)All types incl. person/company/location (exact list depends on loaded policies)Full detection
Not runningSame (optimistic)Same (optimistic)One NER-timeout wait per request (detector.timeout_secs), then fallback to builtin
Not configuredbuiltin_only (person/company/location detection unavailable)Builtin types only (no person/company/location)Builtin only, no timeout

Latency Troubleshooting

If transform latency is high (>1 second p50):

  1. Check detector mode: curl -H "X-API-Key: $GUARDAI_API_KEY" http://localhost:3000/v1/diagnostics | jq .detector_mode
  2. If "Both": Check if NER sidecar is running at the configured URL
  3. If NER is down: Either start the sidecar or switch to detector.mode: builtin
  4. Verify: curl -H "X-API-Key: $GUARDAI_API_KEY" http://localhost:3000/metrics | grep transform_duration_seconds

This is the canonical deployment guide. A stub exists at docs/deployment-guide.md for repository navigation.