Key Rotation Runbook

Session-encryption keys protect sealed session blobs. The audit root protects the durable audit chain. They have different continuity requirements and must be rotated as separate operations.

Session-Key Rotation

When to Rotate

• Session secret was exposed (logs, source control, breach).
• Scheduled rotation per compliance policy (e.g., every 90 days).
• Personnel with access to the secret has left the team.

Background

Sealed sessions are encrypted blobs sent to the client. On rehydrate, the server decrypts with the configured key. Wrong key returns GUARDAI_SESSION_EXPIRED. Clients recover by calling /v1/transform again to get a fresh session.

Option A: Simple Rotation (Brief Disruption)

Use when a short spike of session expiration errors is acceptable.

1. Generate new secret:

openssl rand -base64 32

2. Update the secret:

# Kubernetes
kubectl create secret generic oguardai-session-secret \
  --from-literal=session-secret=<NEW_SECRET> --dry-run=client -o yaml | kubectl apply -f -

# systemd
sudo sed -i 's/^GUARDAI_SESSION_SECRET=.*/GUARDAI_SESSION_SECRET=<NEW_SECRET>/' /etc/guardai/oguardai.env

# Docker Compose
export GUARDAI_SESSION_SECRET=<NEW_SECRET>

3. Rolling restart:

kubectl rollout restart deployment/guardai && kubectl rollout status deployment/guardai --timeout=5m
sudo systemctl restart oguardai-server                           # systemd
docker compose -f deploy/docker/docker-compose.yml up -d --no-deps server  # Docker

4. Monitor, expect a brief spike in SessionExpired errors:

kubectl logs -l app.kubernetes.io/name=oguardai -c server --tail=100 | grep 'SessionExpired'

Spike should subside within minutes as clients retry with fresh transforms.

Option B: Zero-Downtime Rotation (KeyRing)

Use when no client-visible errors are acceptable. New sessions seal under current_kid, and every listed key can still decrypt, so old and new sessions both work during the overlap.

Sequence the rollout in two phases: every pod must be able to DECRYPT the new key before any pod starts SEALING under it. Bumping current_kid in the same restart that introduces the key would let a new pod seal a kid 1 blob that an old pod (still without kid 1) rejects with SessionExpired.

1. Generate new secret: openssl rand -base64 32

2. Add the new key but keep current_kid on the old key:

session:
  backend: sealed
  keyring:
    current_kid: 0
    keys:
      - kid: 0
        secret: <OLD_SECRET>
      - kid: 1
        secret: <NEW_SECRET>

3. Rolling restart. Every pod now seals under key 0 but can decrypt key 1:

kubectl rollout restart deployment/guardai && kubectl rollout status deployment/guardai --timeout=5m

4. Activate the new key by switching current_kid to 1 (keep both keys):

session:
  backend: sealed
  keyring:
    current_kid: 1
    keys:
      - kid: 0
        secret: <OLD_SECRET>
      - kid: 1
        secret: <NEW_SECRET>

5. Rolling restart. New sessions seal under key 1, which every pod already decrypts, so no SessionExpired errors occur during the rollout.

6. Wait one full session TTL (default 3600s) for key 0 sessions to drain:

grep ttl /etc/guardai/oguardai.yaml   # check configured TTL

7. Remove the old key from config (keep current_kid on the surviving key):

session:
  backend: sealed
  keyring:
    current_kid: 1
    keys:
      - kid: 1
        secret: <NEW_SECRET>

8. Final rolling restart: kubectl rollout restart deployment/guardai

Impact Summary

Post-Rotation Verification

curl -sf http://guardai.internal:3000/livez
curl -s -X POST http://guardai.internal:3000/v1/transform \
  -H "Content-Type: application/json" \
  -H "X-API-Key: $GUARDAI_API_KEY" \
  -d '{"input": "test@example.com"}' | jq .session_state

Confirm session expired error rate returns to baseline within 10 min (simple) or stays flat (KeyRing).

Audit-Root Rotation

Use a dedicated audit.secret so routine session-keyring rotation does not change audit-chain identity. If it is unset, session.secret is the fallback audit root, so a base session-secret change also requires this procedure.

1. Stop all writers to the live audit file. In a replicated deployment, drain and stop every replica that shares the file.

2. Move the live file to a stable archive path. Keep it a regular file owned by the verification user at mode 0400 or 0600. Do not follow or create symlinks.

3. Configure the new audit root and retain the previous root for archive verification only. When adopting a dedicated audit secret from the session fallback, the previous root is the old session secret.

   audit:
     secret: "${NEW_AUDIT_SECRET}"
     retired_secrets:
       - "${PREVIOUS_AUDIT_ROOT_SECRET}"

4. Restart the runtime. It creates a new exact-0600 live file in the stable, trusted parent.

5. Verify the archive before retiring any copy of its writing secret:

   oguardai-server audit verify --config /etc/guardai/oguardai.yaml \
     --path /var/lib/guardai/audit/audit-previous.jsonl

The source header key id selects the current or retired writing root. For a legacy header, the verifier tries configured candidates against the same opened source. One descriptor supplies key selection and whole-file verification, so a pathname replacement cannot switch the evidence mid-command. The complete custody and export procedure is in Compliance Controls.

Proxy Ingress Key Rotation

The transparent proxy authenticates callers with GUARDAI_PROXY_API_KEY on the X-GuardAI-API-Key header. Rotate it with an overlap window so no caller is rejected mid-rotation. Set a second accepted key with GUARDAI_PROXY_API_KEY_NEXT: both keys authenticate, share the tenant, rate bucket, and audit attribution, and hold the same roles. The next key must differ from the primary and from any metrics key.

1. Generate the new key.

   NEW_KEY=$(openssl rand -hex 32)

2. Set GUARDAI_PROXY_API_KEY_NEXT to the new key on every replica and restart or reload each one. The proxy now accepts the old key and the new key. Confirm the startup posture line reports ingress_keys=2.

3. Roll clients onto the new key: update each client's X-GuardAI-API-Key to NEW_KEY. The old key keeps working throughout.

4. Watch the retiring key drain. While the next key is set, GET /metrics carries a proxy-local counter per key slot:

   curl -s http://proxy.internal:8081/metrics \
     -H "X-GuardAI-API-Key: $NEW_KEY" | grep guardai_proxy_ingress_auth_total
   # guardai_proxy_ingress_auth_total{key_slot="current"} <old key traffic>
   # guardai_proxy_ingress_auth_total{key_slot="next"} <new key traffic>

   Wait until key_slot="current" stops advancing across scrapes: all live traffic is now on the new key.

5. Promote the new key. Set GUARDAI_PROXY_API_KEY to NEW_KEY and unset GUARDAI_PROXY_API_KEY_NEXT on every replica, then restart or reload. The old key is now rejected, and the per-slot counter drops off /metrics (a single-key deployment renders no slot lines).

6. Verify: a request with the old key returns 401, and a request with the new key succeeds.

Proxy Metrics Credential Rotation

GUARDAI_PROXY_METRICS_API_KEY is a single slot, so rotating it is a brief scrape gap, not a caller outage: change the value on every replica, restart or reload, and update the scraper's X-GuardAI-API-Key. The dashboard misses at most one scrape interval. The data plane is untouched, so no client sees an error. The new metrics key must differ from every ingress key.