Incident Triage Playbook

On-call playbook for OGuardAI alerts. Identify the alert, follow the matching section.

Severity Definitions

OGuardAIHighErrorRate

Trigger: rate(guardai_errors_total[5m]) > 0.1 for 2 min.

1. Check logs for failing endpoint:

   kubectl logs -l app.kubernetes.io/name=oguardai -c server --tail=200 | grep ERROR
   journalctl -u oguardai-server --since "10 min ago" --priority=err   # systemd

2. Check detector sidecar health:

   kubectl exec deploy/guardai -- curl -sf http://localhost:9090/health

3. If detector is down, restart: kubectl rollout restart deployment/guardai
4. If errors are on /v1/rehydrate, check for expired session keys. See key-rotation runbook.

guardai_errors_total is a single unlabelled counter with no per-code breakdown, so read the code off the structured log: each failed request logs one request failed event carrying error_code, status, and request_id. The 503 codes mean a dependency is down rather than that the caller sent bad content. Most are retriable once the dependency is back; strict audit needs the extra commit-state check below.

Treat GUARDAI_AUDIT_UNAVAILABLE as outcome-unknown unless the operation's state proves otherwise. Capacity, queue, writer, and acknowledgement failures before the audit claim leave the operation uncommitted or roll it back. A failure after the audit claim can leave the operation committed, and a commit-indeterminate revocation deliberately retains the erasure. For a mutating route that accepts Idempotency-Key, retry only with the same key. Without that protection, reconcile the domain state and audit chain before retrying so an already committed operation is not executed twice.

The offline oguardai-server audit verify and audit export commands reject a symlink, non-regular, broadly readable, or wrongly owned source without modifying it. If an export with --out fails after creating its destination, it can leave a zero-length mode 0000 residual that reserves the name. Confirm the residual is the expected regular zero-byte leaf in the trusted directory, remove that exact leaf, then rerun. If any metadata differs, treat the directory as unsafe and use a new trusted destination. See Compliance Controls.

A 422 GUARDAI_OUTPUT_BLOCKED is the opposite case and is not an outage: the content was scanned and refused, so retrying the same content will be refused again.

OGuardAIHighTransformLatency

Trigger: histogram_quantile(0.95, rate(guardai_transform_duration_seconds_bucket[5m])) > 1.0 for 5 min.

1. Check detector sidecar logs: kubectl logs -l app.kubernetes.io/name=oguardai -c detector --tail=100
2. Check CPU pressure: kubectl top pods -l app.kubernetes.io/name=oguardai
3. If detector CPU saturated, scale: kubectl scale deployment/guardai --replicas=4 (only if revocation is Redis-backed; see the scaling prerequisite below)
4. If latency persists, check if NER detector is the bottleneck:

   kubectl logs -l app.kubernetes.io/name=oguardai -c detector --tail=50 | grep -i timeout

OGuardAIPromptSecuritySpike

Trigger: rate(guardai_prompt_security_triggers_total[5m]) > 1.0 for 1 min. CRITICAL.

1. Review blocked inputs:

   kubectl logs -l app.kubernetes.io/name=oguardai -c server --tail=300 | grep 'prompt_security'

2. Find the source tenant:

   kubectl logs -l app.kubernetes.io/name=oguardai -c server --tail=500 \
     | grep 'prompt_security' | grep -oP 'tenant=\S+' | sort | uniq -c | sort -rn

3. If active attack: block source at ingress/WAF level.
4. If false positive from new integration: adjust prompt security thresholds.
5. Escalate to L3 if pattern is novel or sustained.

OGuardAIOutputGuardSpike

Trigger: rate(guardai_output_guard_triggers_total[5m]) > 0.5 for 2 min.

1. Check which entity types are being caught:

   kubectl logs -l app.kubernetes.io/name=oguardai -c server --tail=200 | grep 'output_guard'

2. Review active policy: kubectl exec deploy/guardai -- cat /app/policies/default/policy.yaml
3. If LLM is hallucinating PII, check for recent model updates at the provider.
4. If new entity type not covered, update policy. See policy-rollback runbook for safe steps.

OGuardAIRateLimitSaturation

Trigger: rate(guardai_rate_limit_rejections_total[5m]) > 10 for 1 min.

1. Check which tenants: kubectl logs -l app.kubernetes.io/name=oguardai -c server --tail=300 | grep 'rate_limit'
2. If one tenant dominates: contact tenant or raise their limit.
3. If all tenants hit limits, scale:

   kubectl scale deployment/guardai --replicas=6
   kubectl patch hpa guardai -p '{"spec":{"maxReplicas":12}}'

Scaling prerequisite: scale beyond one replica only when the shared Redis backends are configured (revocation.backend=redis and idempotency.backend=redis). A release installed at one replica with the default per-pod memory backends is not scale-safe: scaling it lets a value revoked on one pod stay restorable through another, and lets a retried mutation double-execute on another pod. The Helm chart refuses to render a multi-replica release without the Redis backends, but a raw kubectl scale of a single-replica release bypasses that render-time check, so reconfigure first (helm upgrade ... --set revocation.backend=redis --set idempotency.backend=redis --set redis.external.url=...) and let the pods roll before raising the replica count.

OGuardAINoTraffic

Trigger: rate(guardai_transforms_total[5m]) == 0 for 5 min.

1. Health check: curl -sf http://guardai.internal:3000/livez
2. Check pod status: kubectl get pods -l app.kubernetes.io/name=oguardai
3. Check routing: kubectl describe ingress guardai && kubectl get endpoints guardai
4. If healthy but no traffic, verify upstream services have not changed routing.

OGuardAIEntityDetectionSpike

Trigger: rate(guardai_entities_detected_total[5m]) > 100 for 2 min.

1. Check if this is legitimate high-volume traffic or a scanning attack.
2. If suspicious, escalate to L2.

Escalation Path

Escalate if: issue persists after L1, alert is critical, or root cause unclear after 15 min.