Key Rotation Runbook

Rotate the AES-256-GCM session encryption key for OGuardAI's sealed session backend.

When to Rotate

• Session secret was exposed (logs, source control, breach).
• Scheduled rotation per compliance policy (e.g., every 90 days).
• Personnel with access to the secret has left the team.

Background

Sealed sessions are encrypted blobs sent to the client. On rehydrate, the server decrypts with the configured key. Wrong key returns GUARDAI_SESSION_EXPIRED. Clients recover by calling /v1/transform again to get a fresh session.

Option A: Simple Rotation (Brief Disruption)

Use when a short spike of session expiration errors is acceptable.

1. Generate new secret:

openssl rand -base64 32

2. Update the secret:

# Kubernetes
kubectl create secret generic oguardai-session-secret \
  --from-literal=session-secret=<NEW_SECRET> --dry-run=client -o yaml | kubectl apply -f -

# systemd
sudo sed -i 's/^GUARDAI_SESSION_SECRET=.*/GUARDAI_SESSION_SECRET=<NEW_SECRET>/' /etc/guardai/oguardai.env

# Docker Compose
export GUARDAI_SESSION_SECRET=<NEW_SECRET>

3. Rolling restart:

kubectl rollout restart deployment/guardai && kubectl rollout status deployment/guardai --timeout=5m
sudo systemctl restart oguardai-server                           # systemd
docker compose -f deploy/docker/docker-compose.yml up -d --no-deps server  # Docker

4. Monitor, expect a brief spike in SessionExpired errors:

kubectl logs -l app.kubernetes.io/name=oguardai -c server --tail=100 | grep 'SessionExpired'

Spike should subside within minutes as clients retry with fresh transforms.

Option B: Zero-Downtime Rotation (KeyRing)

Use when no client-visible errors are acceptable. New sessions seal under current_kid, and every listed key can still decrypt, so old and new sessions both work during the overlap.

Sequence the rollout in two phases: every pod must be able to DECRYPT the new key before any pod starts SEALING under it. Bumping current_kid in the same restart that introduces the key would let a new pod seal a kid 1 blob that an old pod (still without kid 1) rejects with SessionExpired.

1. Generate new secret: openssl rand -base64 32

2. Add the new key but keep current_kid on the old key:

session:
  backend: sealed
  keyring:
    current_kid: 0
    keys:
      - kid: 0
        secret: <OLD_SECRET>
      - kid: 1
        secret: <NEW_SECRET>

3. Rolling restart. Every pod now seals under key 0 but can decrypt key 1:

kubectl rollout restart deployment/guardai && kubectl rollout status deployment/guardai --timeout=5m

4. Activate the new key by switching current_kid to 1 (keep both keys):

session:
  backend: sealed
  keyring:
    current_kid: 1
    keys:
      - kid: 0
        secret: <OLD_SECRET>
      - kid: 1
        secret: <NEW_SECRET>

5. Rolling restart. New sessions seal under key 1, which every pod already decrypts, so no SessionExpired errors occur during the rollout.

6. Wait one full session TTL (default 3600s) for key 0 sessions to drain:

grep ttl /etc/guardai/oguardai.yaml   # check configured TTL

7. Remove the old key from config (keep current_kid on the surviving key):

session:
  backend: sealed
  keyring:
    current_kid: 1
    keys:
      - kid: 1
        secret: <NEW_SECRET>

8. Final rolling restart: kubectl rollout restart deployment/guardai

Impact Summary

Post-Rotation Verification

curl -sf http://guardai.internal:3000/livez
curl -s -X POST http://guardai.internal:3000/v1/transform \
  -H "Content-Type: application/json" \
  -H "X-API-Key: $GUARDAI_API_KEY" \
  -d '{"input": "test@example.com"}' | jq .session_state

Confirm session expired error rate returns to baseline within 10 min (simple) or stays flat (KeyRing).