Deployment Guide
Deploy OGuardAI from local development to production Kubernetes clusters
This guide covers every deployment mode from local development to production Kubernetes clusters.
Note: The current runtime uses sealed sessions (client-carried encrypted blobs), so no shared session store is required, and each instance is stateless for session handling. A multi-replica deployment still needs a shared revocation backend (redis), because a value revoked on one replica must be refused by all.
Table of Contents
- Prerequisites
- Quick Start (Docker)
- Development (Local Build)
- Production (Docker Compose)
- Kubernetes (Helm)
- Configuration Reference
- Session Backend Selection
- Key Rotation Procedure
- Monitoring and Alerting
- Troubleshooting
- Rate Limiting in Multi-Instance Deployments
- Metrics in Multi-Instance Deployments
- Detector Mode Selection
Prerequisites
| Requirement | Version | Purpose |
|---|---|---|
| Docker | 24+ | Container runtime |
| Docker Compose | 2.20+ | Multi-service orchestration |
| Helm | 3.12+ | Kubernetes deployment (optional) |
| Kubernetes | 1.27+ | Container orchestration (optional) |
| Rust | 1.88+ | Building from source (optional) |
| Python | 3.10+ | NER detector sidecar (optional) |
| Node.js | 20+ | SDK and MCP server (optional) |
Quick Start (Docker)
Run OGuardAI with a single command using the pre-built Docker image:
docker run -p 3000:3000 \
-e GUARDAI_SESSION_SECRET=$(openssl rand -base64 32) \
ghcr.io/oronts/oronts-guardai/oguardai-server:latestThis starts OGuardAI with:
- Builtin regex detectors (30+ patterns)
- Sealed session backend (client-carried encrypted blobs)
- Dev auth mode (all requests accepted, not for production)
- Default policy
- Bundled Tesseract OCR (English, German, Arabic), so the
/v1/transform/imageand/v1/redact/imageendpoints work out of the box
Verifying image provenance
Published images are Trivy-scanned before push, signed with cosign (keyless, via Sigstore Fulcio/Rekor), and carry an SPDX SBOM attestation. Verify the signature and provenance before running in production (cosign and jq required):
IMAGE=ghcr.io/oronts/oronts-guardai/oguardai-server:latest
IDENTITY='https://github.com/oronts/.*/.github/workflows/docker.yml@.*'
ISSUER=https://token.actions.githubusercontent.com
# Verify the keyless signature (identity = the release workflow's OIDC subject).
cosign verify "$IMAGE" --certificate-identity-regexp "$IDENTITY" --certificate-oidc-issuer "$ISSUER"
# Verify and extract the SPDX SBOM attestation.
cosign verify-attestation "$IMAGE" --type spdxjson \
--certificate-identity-regexp "$IDENTITY" --certificate-oidc-issuer "$ISSUER" \
| jq -r '.payload | @base64d | fromjson | .predicate' > sbom.spdx.jsonA failed cosign verify means the image was not built by the official pipeline; do not run it.
Verify the server is running:
curl http://localhost:3000/v1/health
# {"status":"healthy","version":"0.1.0","uptime_seconds":1.2}Test a transform:
curl -X POST http://localhost:3000/v1/transform \
-H "Content-Type: application/json" \
-d '{"input": "Contact julia@example.com for help"}'Expected response:
{
"safe_text": "Contact {{email:e_001:7b1068662ab5}} for help",
"session_id": "...",
"session_state": "...",
"entities": [{"token": "{{email:e_001:7b1068662ab5}}", "type": "email"}]
}The third token segment is a random per-token capability suffix; rehydrate resolves a token only
on an exact {{type:id:cap}} match (fail closed), so LLM output must be passed back verbatim.
Image OCR: The official server image bundles Tesseract with English, German, and Arabic language data, so
/v1/transform/imageand/v1/redact/imagework without extra setup. Only from-source builds need Tesseract installed on the host, and the separate proxy image does not include it (see PDF Support).
Development (Local Build)
Build from Source
# Clone the repository
git clone https://github.com/oronts/oguardai.git
cd oguardai
# Build all Rust crates
cargo build --workspace
# Run all tests
cargo test --workspace
# Start the server with defaults
cargo run -p oguardai-server -- --config oguardai.yamlMinimal Development Config
Create oguardai.yaml for local development:
server:
host: "127.0.0.1"
port: 3000
auth:
mode: dev
session:
backend: sealed
secret: "dev-only-secret-change-in-prod!!"
ttl_seconds: 3600
detector:
mode: builtin
policy:
default: default
directory: policies
transform:
context_strategy: full
max_context_tokens: 4096Running with Python Detector (Development)
# Terminal 1: Start Python NER detector
cd apps/detector-py
uv sync
uv run uvicorn guardai_detector_service.main:app --host 0.0.0.0 --port 9090
# Terminal 2: Start Rust server with detector URL
GUARDAI_DETECTOR_URL=http://localhost:9090 \
cargo run -p oguardai-server -- --config oguardai.yamlProduction (Docker Compose)
Docker Compose deploys three services: the OGuardAI server, the Python NER detector, and a Redis instance for durable revocation state. Sessions use the sealed backend (client-carried encrypted blobs), so no external session store is required; Redis here holds only revocation digests, not session state.
Environment Setup
Create a .env file (never commit this to version control):
# Required: 32-byte session encryption secret
GUARDAI_SESSION_SECRET=$(openssl rand -base64 32)
# Optional: log level
GUARDAI_LOG_LEVEL=guardai_server=info,tower_http=infoStart the Full Stack
# Required environment variables:
export GUARDAI_SESSION_SECRET="$(openssl rand -base64 32)"
export GUARDAI_API_KEY_1="$(openssl rand -hex 32)"
docker compose -f deploy/docker/docker-compose.yml up --build
# Test with authenticated request:
# curl -H "X-API-Key: $GUARDAI_API_KEY_1" http://localhost:3000/v1/healthThis starts:
| Service | Port | Description |
|---|---|---|
server | 3000 | OGuardAI Rust server |
detector | 9090 (internal) | Python NER detector (GLiNER/spaCy), reachable only on the compose network |
redis | 6379 (internal) | Revocation store (redis:7-alpine, append-only) |
Production Docker Compose Configuration
The production docker-compose.yml at deploy/docker/docker-compose.yml includes:
- Health checks on all three services (HTTP readiness for server and detector,
redis-cli pingfor Redis). - Restart policy:
unless-stoppedfor automatic recovery. - Volume mounts: Policies directory mounted read-only; a named
redis-datavolume for revocation persistence. - Dependency ordering: The server waits for both the detector and Redis to be healthy before starting.
- Auth: The server runs in
api_keymode (GUARDAI_AUTH_MODE=api_key) and requiresGUARDAI_API_KEY_1to be set before the stack starts. - Durable revocation: The server sets
GUARDAI_REVOCATION_BACKEND=redisandGUARDAI_REDIS_URL=redis://redis:6379. Redis persists only HMAC-SHA-256 digests of revoked(type, value)pairs (append-only), never raw PII, so a value revoked once survives a restart. - Durable, fail-closed audit trail: The server sets
GUARDAI_AUDIT_BACKEND=file,GUARDAI_AUDIT_FILE_PATH=/var/lib/guardai/audit/audit.log(on the namedguardai-auditvolume), andGUARDAI_AUDIT_STRICT=true. A non-dev start requires this durable, strict trail; setGUARDAI_ALLOW_EPHEMERAL_AUDIT=1only to explicitly accept an ephemeral one. - Detector auth: The detector is not published on a host port and runs with
GUARDAI_DETECTOR_ALLOW_INSECURE=truebecause it is reachable only on the internal compose network. If you expose it, setGUARDAI_DETECTOR_API_KEY(theX-Detector-API-Keyshared secret) on both the detector and the server instead.
Minimal Docker Compose (Server Only)
For deployments that do not need advanced NER or shared sessions:
docker compose -f deploy/docker/docker-compose.minimal.yml up --buildThis starts only the OGuardAI server with builtin detectors and sealed sessions.
Kubernetes (Helm)
Install the Helm Chart
# Add the OGuardAI Helm repository (when published)
helm repo add oguardai https://oronts.github.io/oguardai/charts
helm repo update
# Install with default values
helm install oguardai oguardai/oguardai \
--namespace oguardai \
--create-namespace \
--set session.secret="$(openssl rand -base64 32)" \
--set auth.apiKeys[0].name=default \
--set auth.apiKeys[0].key="$(openssl rand -hex 32)" \
--set audit.backend=file \
--set audit.strict=true
# Or install from local chart
helm install oguardai deploy/helm/oguardai \
--namespace oguardai \
--create-namespace \
--set session.secret="$(openssl rand -base64 32)" \
--set auth.apiKeys[0].name=default \
--set auth.apiKeys[0].key="$(openssl rand -hex 32)" \
--set audit.backend=file \
--set audit.strict=trueThe chart defaults to auth.mode=api_key, and a non-dev server start requires a durable, strict
audit trail (audit.backend=file plus audit.strict=true), so the install commands above set both.
The chart mounts a volume at the audit path's parent directory; it is an emptyDir by default, so back
it with a PersistentVolume for a trail that survives pod restarts.
Production Helm Values
Create values-production.yaml:
server:
replicaCount: 3
resources:
requests:
cpu: 500m
memory: 512Mi
limits:
cpu: "2"
memory: 1Gi
# replicaCount > 1 requires the shared redis backends (revocation + idempotency),
# or the chart refuses to render.
revocation:
backend: redis
idempotency:
backend: redis
redis:
external:
url: redis://redis:6379
session:
backend: sealed
ttlSeconds: 3600
# Reference an existing Kubernetes secret
existingSecret: oguardai-session-secret
existingSecretKey: session-secret
auth:
mode: api_key
# API keys must use the name/key format matching the Helm chart template.
# The template generates env vars GUARDAI_API_KEY_{NAME} from each entry.
apiKeys:
- name: my-service
key: "GENERATE_WITH_openssl_rand_hex_32" # Replace before deploying
policy:
default: default
directory: /app/policies
# Durable, fail-closed audit trail. REQUIRED for a non-dev (api_key/jwt/oidc) start:
# the server refuses to boot with the ephemeral log backend unless
# GUARDAI_ALLOW_EPHEMERAL_AUDIT=1 is set. Back the audit volume with a
# PersistentVolume; the default emptyDir does not survive pod restarts.
audit:
backend: file
strict: true
file:
path: /audit/audit.log
transform:
contextStrategy: full
maxContextTokens: 4096
# Enable Python NER detector
detector:
enabled: true
# The detector handles raw PII and fails closed (503) without the shared
# X-Detector-API-Key secret. When detector.enabled, the chart requires
# auth.apiKey or auth.existingSecret (or the explicit dev-only opt-out
# auth.allowInsecure=true).
auth:
existingSecret: oguardai-detector-auth
existingSecretKey: detector-api-key
resources:
requests:
cpu: 500m
memory: 1Gi
limits:
cpu: "2"
memory: 2Gi
# Enable horizontal pod autoscaler
autoscaling:
enabled: true
minReplicas: 3
maxReplicas: 10
targetCPUUtilizationPercentage: 70
# Shared revocation is REQUIRED for more than one replica. The chart refuses to
# install when replicaCount > 1 or autoscaling.enabled unless revocation.backend
# is redis with an external URL: the default memory backend is pod-local, so a
# value revoked on one pod would not be refused by the others. Redis stores only
# HMAC-SHA-256 digests of revoked (type, value) pairs, never raw PII.
revocation:
backend: redis
redis:
external:
url: "redis://redis-master.oguardai.svc.cluster.local:6379"
# Pod security
podSecurityContext:
runAsNonRoot: true
runAsUser: 1000
fsGroup: 1000
securityContext:
readOnlyRootFilesystem: true
allowPrivilegeEscalation: false
capabilities:
drop:
- ALLInstall with production values:
helm install oguardai deploy/helm/oguardai \
--namespace oguardai \
--create-namespace \
-f values-production.yamlUsing External Secrets
To use an external secret manager (Vault, AWS Secrets Manager) with the Helm chart:
# Create the Kubernetes secret first
kubectl create secret generic oguardai-session-secret \
--namespace oguardai \
--from-literal=session-secret="$(openssl rand -base64 32)"
# Reference it in Helm values
helm install oguardai deploy/helm/oguardai \
--set session.existingSecret=oguardai-session-secret \
--set session.existingSecretKey=session-secret \
--set auth.apiKeys[0].name=default \
--set auth.apiKeys[0].key="$(openssl rand -hex 32)"Helm Chart Components
The chart deploys the following Kubernetes resources:
| Resource | Purpose |
|---|---|
| Deployment | OGuardAI server pods |
| Service | ClusterIP service for internal access |
| ConfigMap | oguardai.yaml configuration |
| Secret | Session encryption secret |
| ServiceAccount | Pod identity |
| HPA | Horizontal pod autoscaler (optional, autoscaling.enabled) |
| PodDisruptionBudget | Availability during voluntary disruptions (auto-enabled when replicaCount > 1, or forced via pdb.enabled) |
| NetworkPolicy | Ingress/egress restrictions (optional, networkPolicy.enabled) |
| Ingress | External HTTP(S) routing (optional, ingress.enabled) |
Configuration Reference
The oguardai.yaml file controls all server behavior. Every field has a sensible default.
Server Section
server:
host: "0.0.0.0" # Bind address (default: 0.0.0.0)
port: 3000 # Bind port (default: 3000)
allowed_origins: # CORS allowlist enforced in non-dev auth modes; env GUARDAI_CORS_ORIGINS (comma-separated)
- "https://app.example.com"
tls: # Optional in-process TLS; TLS is usually terminated at the ingress instead
enabled: false
cert_path: "/etc/oguardai/tls.crt" # env GUARDAI_TLS_CERT
key_path: "/etc/oguardai/tls.key" # env GUARDAI_TLS_KEYAuth Section
auth:
mode: api_key # dev | api_key | jwt | oidc (default: dev)
api_keys: # Only used when mode=api_key
- key: sk-...
identity: my-service
tenant_id: tenant-1 # Optional: multi-tenancy scope
scopes: # Optional: restrict to specific scopes
- transform
- rehydrate
# JWT config (only used when mode=jwt)
jwt:
secret: "your-hmac-secret"
issuer: "https://auth.example.com"
audience: "guardai-api" # Expected `aud` claim; a token whose audience differs is rejected
# OIDC config (only used when mode=oidc)
oidc:
issuer: "https://issuer.example.com"
jwks_url: "https://issuer.example.com/.well-known/jwks.json"
audience: "guardai-api"
tenant_claim: "tenant_id" # Claim that carries the tenant identifier (required)
algorithms: [RS256] # Asymmetric allowlist: RS256/384/512, ES256/384, PS256/384/512Note: The
api_keysentries usekey,identity,tenant_id, andscopesfields.
Session Section
session:
backend: sealed # sealed (default: sealed)
secret: "..." # 32-byte encryption secret (REQUIRED for production)
ttl_seconds: 3600 # Session TTL in seconds (default: 3600)The GUARDAI_SESSION_SECRET environment variable overrides session.secret.
Supported backends:
sealed(default, stateless, cross-replica),memory(in-process, dev/test, single-instance), andredis(server-side sessions encrypted at rest in a shared Redis, cross-replica HA; setsession.redis_url,replay_backend: redis, andrevocation_backend: redis).managedis planned.
Revocation Backend
Revocation lists which entity values must be refused at rehydrate (only HMAC digests are stored, never raw PII).
revocation_backend: memory # memory (file-backed, per-instance) | redis (shared)
replay_backend: memory # memory (per-instance) | redis (shared); rejects replayed continuations
session:
redis_url: "redis://redis:6379" # required when revocation_backend or replay_backend is redisGUARDAI_REVOCATION_BACKEND, GUARDAI_REPLAY_BACKEND, and GUARDAI_REDIS_URL override these. Use redis
whenever you run more than one instance, so a value revoked on one is refused by
all. The provided deploy/docker/docker-compose.ha.yml wires this.
Detector Section
detector:
mode: builtin # builtin | advanced | both (default: builtin)
advanced_url: "http://detector:9090" # URL of Python NER sidecar
default_language: null # ISO 639-1 fallback when a request omits its own language; null falls back to "en" (no auto-detect)
timeout_secs: 5 # NER request timeout (max 120); env GUARDAI_DETECTOR_TIMEOUT_SECS| Mode | Behavior |
|---|---|
builtin | Rust regex patterns only (30+ patterns) |
advanced | Python NER sidecar only |
both | Both builtin and advanced; merge results |
Policy Section
policy:
default: default # Default policy name (default: "default")
directory: policies # Directory containing policy YAML files (default: "policies")Transform Section
transform:
context_strategy: full # full | type_summary | referenced_only | none (default: full)
max_context_tokens: 4096 # Max tokens for entity_context (default: 4096)Output Protection Section
output_protection:
enabled: true # Output guard is enabled by default
mode: strict # strict | permissive (default: strict)
default_action: mask # allow | warn | mask | block (default: mask)
exempt_types: # Entity types exempt from output protection
- greetingPrompt Security Section
prompt_security:
enabled: true # Enable prompt injection scanning (default: true)
action: strip # warn | strip | block (default: strip)File Upload Section
file_upload:
max_size_bytes: 52428800 # 50MB defaultRate Limit Section
rate_limit:
enabled: false # Enable rate limiting (default: false)
requests_per_second: 100 # Global rate limit (default: 100)
burst_size: 200 # Burst allowance (default: 200)Audit Section
audit:
backend: file # log (ephemeral, fail-open; the default) | file (durable, hash-chained)
strict: true # fail a request closed (503) when its audit event cannot be recorded (default: false)
file:
path: /var/lib/guardai/audit/audit.log # required when backend=file
allow_ephemeral: false # explicit waiver of the non-dev durable, strict requirementA non-dev (api_key/jwt/oidc) start requires the durable, strict trail (backend: file plus
strict: true); it refuses to boot otherwise. Set audit.allow_ephemeral: true (or
GUARDAI_ALLOW_EPHEMERAL_AUDIT=1) to explicitly accept an ephemeral or non-strict trail. The
--compliance preset always forces file + strict and ignores the waiver.
Idempotency Section
idempotency:
backend: memory # memory (per-replica) | redis (shared; uses session.redis_url)
ttl_seconds: 86400 # how long a completed Idempotency-Key result is replayed (default: 86400)
in_progress_ttl_seconds: 60 # how long an in-flight claim blocks a duplicate before it is reclaimable (default: 60)Mutating, non-streaming routes (transform, batch transform, RAG ingest/delete, revoke, session
delete) accept an Idempotency-Key header so a client can retry without executing the mutation
twice.
Tenants Section
tenants:
acme-corp:
default_policy: gdpr-strict
rate_limit:
requests_per_second: 50
burst_size: 100
startup-inc:
default_policy: default
rate_limit:
requests_per_second: 200
burst_size: 400Environment Variable Overrides
| Environment Variable | Overrides | Example |
|---|---|---|
GUARDAI_SESSION_SECRET | session.secret | 32-byte production secret |
GUARDAI_DETECTOR_URL | detector.advanced_url | http://detector:9090 |
GUARDAI_DETECTOR_TIMEOUT_SECS | detector.timeout_secs | 5 (max 120) |
GUARDAI_AUTH_MODE | auth.mode | api_key |
GUARDAI_API_KEY_1 .. GUARDAI_API_KEY_N | Injects auth.api_keys entries | openssl rand -hex 32 |
GUARDAI_REVOCATION_BACKEND | revocation_backend | redis |
GUARDAI_REPLAY_BACKEND | replay_backend | redis |
GUARDAI_REDIS_URL | session.redis_url (revocation/replay/session Redis) | redis://redis:6379 |
GUARDAI_AUDIT_BACKEND | audit.backend | file |
GUARDAI_AUDIT_FILE_PATH | audit.file.path | /var/lib/guardai/audit/audit.log |
GUARDAI_AUDIT_STRICT | audit.strict | true |
GUARDAI_ALLOW_EPHEMERAL_AUDIT | audit.allow_ephemeral (waive the non-dev durable, strict audit requirement) | 1 |
GUARDAI_DETECTOR_API_KEY | Shared secret for the server-to-detector hop (X-Detector-API-Key); set on both sides | openssl rand -hex 32 |
GUARDAI_IDEMPOTENCY_BACKEND | idempotency.backend | redis |
GUARDAI_IDEMPOTENCY_TTL_SECONDS | idempotency.ttl_seconds | 86400 |
GUARDAI_IDEMPOTENCY_IN_PROGRESS_TTL_SECONDS | idempotency.in_progress_ttl_seconds | 60 |
GUARDAI_HOST | server.host | 0.0.0.0 |
GUARDAI_PORT | server.port | 3000 |
RUST_LOG | Log level filter | guardai_server=info,tower_http=info |
The provided compose stack relies on
GUARDAI_AUTH_MODE,GUARDAI_API_KEY_1,GUARDAI_REVOCATION_BACKEND,GUARDAI_REDIS_URL, and theGUARDAI_AUDIT_*trio above.deploy/systemd/oguardai.env.exampleis the env-file starting point for systemd deployments; add theGUARDAI_AUDIT_*variables to it, since a non-dev start requires the durable, strict audit trail.
Session Backend Selection
| Backend | Use Case | Status |
|---|---|---|
| Sealed (default) | Stateless deployments, horizontal scaling, edge deployments | Available |
| Memory | Development, testing, single-instance demos | Available |
| Redis | Multi-instance production, shared state, server-side session management (encrypted at rest) | Shipped |
Sealed Backend
The sealed backend is the default session backend. Session state is encrypted (AES-GCM) and returned to the client as an opaque blob. The client must carry this blob between transform and rehydrate calls.
Advantages:
- No server-side state; horizontal scaling on the session path with no session store or session affinity. A multi-replica deployment still shares one dependency, the revocation store (
revocation_backend=redis), so an erased value stays erased across replicas. - You want the client to control session lifecycle.
Considerations:
- Blob size grows with entity count.
- Client must carry blob between requests.
- Your entity counts per request should be moderate (< 100 entities).
The
memorybackend is available for dev/test (in-process, lost on restart, single-instance). Theredissession backend ships for multi-instance shared session state: each session is encrypted at rest (AES-256-GCM) so Redis holds only ciphertext. The Redis revocation and replay backends also ship for multi-instance shared state.
Key Rotation Procedure
OGuardAI seals through a key ring indexed by key ID (kid). New blobs seal under
current_kid; blobs sealed under any other key still in the ring keep decrypting until that
key is removed. This supports both a simple single-key change and a zero-downtime multi-key
rotation.
Single-key change (brief disruption)
With no session.keyring block the runtime holds one key derived from session.secret.
Replacing that secret invalidates every existing blob, so do it in a low-traffic window.
# Generate a cryptographically random 32-byte secret
openssl rand -base64 32session:
secret: "new-production-secret-32-bytes!!"After deploy, blobs sealed under the old secret fail authentication and return
GUARDAI_SESSION_EXPIRED; clients re-transform the original text to get a new blob.
Zero-downtime rotation (multi-key keyring)
Configure a session.keyring and roll the key in stages so in-flight sessions never break:
- Add the new key (kid 1) but keep
current_kidon the old key, then deploy to every replica so all of them know kid 1 before any pod seals under it:
session:
keyring:
current_kid: 0
keys:
- kid: 0
secret: "${OLD_SECRET}"
- kid: 1
secret: "${NEW_SECRET}"- Switch
current_kidto 1 (keep both keys): new sessions seal under kid 1 while kid 0 blobs still decrypt. - After one session TTL, remove kid 0 from the ring. Blobs sealed under it then fail to decrypt.
See the key rotation runbook for the full procedure and ordering constraints.
Key Rotation Timeline
Single-key change:
Old key active | Deploy new secret | New key only
(all sessions) | (old blobs become | (clients re-transform
| undecryptable) | to get new blobs)
Zero-downtime keyring:
Seal under kid 0 | Add kid 1, seal kid 0 | Switch current_kid 1 | Remove kid 0
| | (kid 0 decrypts | (kid 0 blobs
| | until TTL) | now invalid)Monitoring and Alerting
Health Check
The /v1/health endpoint returns the overall system status (requires auth when enabled; use /livez for unauthenticated probes):
curl http://localhost:3000/livez # Unauthenticated liveness probe
curl -H "X-API-Key: $GUARDAI_API_KEY" http://localhost:3000/v1/health # Full health checkResponse:
{
"status": "healthy",
"version": "0.1.0",
"uptime_seconds": 3600.5,
"components": {
"detector": {"status": "healthy", "message": "builtin_and_ner (full entity detection)"},
"session": {"status": "healthy", "message": "sealed"},
"policies": {"status": "healthy", "message": "3 policies loaded"}
}
}Status values: healthy, degraded (partial functionality), unhealthy (service failure).
What to Monitor
| Metric | Source | Alert Threshold |
|---|---|---|
| HTTP response codes | Access logs / reverse proxy | 5xx rate > 1% |
| Request latency (p95) | Structured logs (latency_ms field) | > 500ms for transform, > 200ms for rehydrate |
| Health check status | /v1/health endpoint | Status != "healthy" |
| Entity detection rate | Structured logs (entity_count field) | Sudden drop may indicate detector failure |
| Session seal/unseal errors | Structured logs (error events) | Any GUARDAI_SESSION_EXPIRED spike |
| Rate limit rejections | Structured logs | Sustained 429 responses |
| Container restarts | Kubernetes / Docker | > 0 in 5-minute window |
| Memory usage | Container metrics | > 80% of limit |
| CPU usage | Container metrics | Sustained > 70% |
| Detector sidecar (NER) | /readyz + metrics | /readyz returns 503, or guardai_ner_degraded_total increases |
Recommended Alert Rules
# Example Prometheus alerting rules
groups:
- name: guardai
rules:
- alert: OGuardAIUnhealthy
expr: probe_success{job="guardai-health"} == 0
for: 2m
labels:
severity: critical
annotations:
summary: "OGuardAI health check failing"
- alert: OGuardAIHighLatency
expr: histogram_quantile(0.95, rate(guardai_transform_duration_seconds_bucket{job="guardai"}[5m])) > 0.5
for: 5m
labels:
severity: warning
annotations:
summary: "OGuardAI p95 latency above 500ms"
- alert: OGuardAIHighErrorRate
expr: rate(guardai_errors_total{job="guardai"}[5m]) / (rate(guardai_transforms_total{job="guardai"}[5m]) + rate(guardai_rehydrates_total{job="guardai"}[5m]) + 1) > 0.01
for: 5m
labels:
severity: critical
annotations:
summary: "OGuardAI 5xx error rate above 1%"Log Aggregation
OGuardAI emits structured JSON logs compatible with all major SIEM and log aggregation systems:
{
"timestamp": "2026-04-15T10:30:00Z",
"level": "INFO",
"target": "guardai_server::pipeline",
"message": "transform completed",
"request_id": "req-abc-123",
"session_id": "sess-def-456",
"tenant_id": "acme-corp",
"entity_count": 3,
"latency_ms": 12.5,
"policy": "gdpr-strict"
}Forward logs to your SIEM using standard log collectors (Fluent Bit, Fluentd, Vector, Filebeat).
Troubleshooting
Server does not start
Symptom: oguardai-server exits immediately.
Causes and fixes:
| Cause | Fix |
|---|---|
| Port already in use | Change server.port in config or stop conflicting process |
| Invalid YAML config | Run oguardai config validate or check YAML syntax |
| Missing policies directory | Create policies/ directory or update policy.directory |
"session expired" errors on rehydrate
Symptom: Rehydrate returns {"error": "...", "code": "GUARDAI_SESSION_EXPIRED"}.
Causes and fixes:
| Cause | Fix |
|---|---|
| Session TTL exceeded | Increase session.ttl_seconds or call rehydrate sooner |
| Different session secret between transform and rehydrate | Ensure all server instances use the same GUARDAI_SESSION_SECRET |
| Key rotation mid-flight | With a multi-key session.keyring, rotation is zero-downtime: keep the prior key in the ring so in-flight blobs unseal under their kid until removed or TTL-expired, while new sessions seal under current_kid. A single-key secret change is disruptive; rotate it in a maintenance window. Removing a key invalidates blobs sealed under it |
Detector sidecar not connecting
Symptom: /readyz returns 503, or guardai_ner_degraded_total / GUARDAI_DETECTION_FAILED increase. Note that /v1/health always reports the detector healthy even when a configured NER sidecar is unreachable, so use /readyz (which probes the sidecar) and ner_active on /v1/capabilities to detect a downed sidecar.
Causes and fixes:
| Cause | Fix |
|---|---|
| Sidecar not started | Check docker compose logs detector |
| Wrong URL | Verify detector.advanced_url matches sidecar address |
| Sidecar still loading models | Wait for sidecar health check to pass (may take 20-30s on first start) |
| Network isolation | Ensure server and detector are on the same Docker/Kubernetes network |
High latency on first request
Symptom: First request takes significantly longer than subsequent requests.
Cause: Regex patterns and detection models are compiled on first use.
Fix: This is expected behavior. Subsequent requests reuse compiled patterns. For consistent latency, send a warmup request after deployment.
Entity not detected
Symptom: Known PII is not being replaced.
Causes and fixes:
| Cause | Fix |
|---|---|
| Entity type not in policy | Check policy YAML includes the entity type |
| Below confidence threshold | Lower threshold in detect request or policy |
| Builtin-only mode | Names and companies require the Python NER sidecar |
| Non-standard format | Builtin patterns cover common formats; unusual formats may need custom patterns |
Session seal/unseal failures
Symptom: GUARDAI_SESSION_EXPIRED or seal/unseal errors on rehydrate. The /v1/health
session component always reports healthy and only echoes the configured backend name; it does
not surface per-request seal failures, so watch the error codes and logs rather than the health
status.
Causes and fixes:
| Cause | Fix |
|---|---|
| Wrong or missing session secret | Verify GUARDAI_SESSION_SECRET is set and matches across all instances |
| Key rotation mid-flight | With a multi-key session.keyring, rotation is zero-downtime: keep the prior key in the ring so in-flight blobs unseal under their kid until removed or TTL-expired, while new sessions seal under current_kid. A single-key secret change is disruptive; rotate it in a maintenance window. Removing a key invalidates blobs sealed under it |
| Tampered or corrupted blob | Client must pass the exact session_state blob received from transform |
| Session TTL exceeded | Increase session.ttl_seconds or call rehydrate sooner |
Rate Limiting in Multi-Instance Deployments
OGuardAI's rate limiter operates per-instance. Each server process maintains independent rate limit buckets.
Implications for Kubernetes / multi-instance:
| Deployment | Effective Rate Limit | Example |
|---|---|---|
| Single instance, limit=100/s | 100/s total | As configured |
| 3 instances, limit=100/s each | Up to 300/s total | Clients hitting different instances get 100/s each |
| 3 instances behind load balancer | ~100/s per client (sticky) or ~300/s (round-robin) | Depends on LB strategy |
Recommended configurations:
For strict per-client limits: Use sticky sessions (session affinity) in your load balancer so each client always hits the same instance.
For global limits: Divide the desired global limit by the number of instances:
# 3 instances, want 300 req/s global -> 100/s per instance
rate_limit:
enabled: true
requests_per_second: 100
burst_size: 200For shared rate limiting (advanced): Deploy a Redis-backed rate limiter in front of OGuardAI (e.g., Kong, Envoy, or API gateway rate limiting). OGuardAI's built-in rate limiter then serves as a secondary defense.
Metrics in Multi-Instance Deployments
Each OGuardAI instance exposes its own /metrics endpoint. Prometheus scrapes all instances independently.
Prometheus scrape config:
# prometheus.yml
scrape_configs:
- job_name: 'guardai'
# /metrics requires the `admin` scope whenever auth is enabled. Present an admin
# credential: an Authorization: Bearer token in jwt/oidc mode, or route the scrape
# through a proxy that injects the X-API-Key header in api_key mode.
authorization:
type: Bearer
credentials: '<admin-scoped-jwt>'
kubernetes_sd_configs:
- role: pod
selectors:
- role: pod
label: app=oguardai-server
relabel_configs:
- source_labels: [__meta_kubernetes_pod_ip]
target_label: __address__
replacement: '$1:3000'Aggregation in PromQL:
# Total transforms across all instances
sum(guardai_transforms_total)
# Per-instance error rate
rate(guardai_errors_total[5m])
# p95 latency across all instances
histogram_quantile(0.95, sum(rate(guardai_transform_duration_seconds_bucket[5m])) by (le))Key point:
Per-instance metrics is the standard pattern for Prometheus-based observability. No shared metrics backend needed.
Detector Mode Selection
Quick Decision Guide
| Your Use Case | Recommended Mode | Why |
|---|---|---|
| Low-latency API protection | builtin | Sub-millisecond, no dependencies |
| Full PII coverage (names, companies) | both | Builtin + NER gives best coverage |
| Maximum accuracy, latency acceptable | advanced | All detection via trained models |
| NER sidecar sometimes unavailable | both | Graceful fallback to builtin |
How to Check Current Mode
# Health endpoint shows detector status (requires auth when enabled)
curl -H "X-API-Key: $GUARDAI_API_KEY" http://localhost:3000/v1/health | jq '.components.detector'
# Capabilities shows what entity types are available
curl -H "X-API-Key: $GUARDAI_API_KEY" http://localhost:3000/v1/capabilities | jq '.entity_types[].name'
# Diagnostics shows full config (requires admin scope)
curl -H "X-API-Key: $GUARDAI_API_KEY" http://localhost:3000/v1/diagnostics | jq '.detector_mode'NER Sidecar Availability
When detector.mode: both is configured:
| NER Status | Health Shows | Capabilities Shows | Behavior |
|---|---|---|---|
| Running + healthy | builtin_and_ner (full entity detection) | All types incl. person/company/location (exact list depends on loaded policies) | Full detection |
| Not running | Same (optimistic) | Same (optimistic) | One NER-timeout wait per request (detector.timeout_secs), then fallback to builtin |
| Not configured | builtin_only (person/company/location detection unavailable) | Builtin types only (no person/company/location) | Builtin only, no timeout |
Latency Troubleshooting
If transform latency is high (>1 second p50):
- Check detector mode:
curl -H "X-API-Key: $GUARDAI_API_KEY" http://localhost:3000/v1/diagnostics | jq .detector_mode - If "Both": Check if NER sidecar is running at the configured URL
- If NER is down: Either start the sidecar or switch to
detector.mode: builtin - Verify:
curl -H "X-API-Key: $GUARDAI_API_KEY" http://localhost:3000/metrics | grep transform_duration_seconds
This is the canonical deployment guide. A stub exists at docs/deployment-guide.md for repository navigation.