Policy Rollback Runbook

Roll back a bad policy deployment. Policies are loaded at server startup from the configured policy directory.

Symptoms of a Bad Policy

• Entities that should be masked are passing through unprotected.
• Entities that should pass through are being blocked or masked.
• Customer complaints about broken responses or missing data.
• Unexpected spike in guardai_errors_total or guardai_output_guard_triggers_total.

Step 1: Confirm the Problem

# View active policy
kubectl exec deploy/guardai -- cat /app/policies/default/policy.yaml      # Kubernetes
cat /etc/guardai/policies/default/policy.yaml                              # systemd

# Test a known input
curl -s -X POST http://guardai.internal:3000/v1/detect \
  -H "Content-Type: application/json" \
  -d '{"input": "My SSN is 123-45-6789 and email is test@example.com"}' | jq .

Compare output against expected behavior for the active policy.

Step 2: Identify What Changed

git log --oneline -10 -- policies/
git diff HEAD~1 -- policies/
diff previous-policy.yaml current-policy.yaml

Step 3: Revert the Policy

Kubernetes (Helm):

helm rollback guardai
helm status guardai

If policy is in a standalone ConfigMap:

kubectl create configmap guardai-policy --from-file=policy.yaml=known-good-policy.yaml \
  --dry-run=client -o yaml | kubectl apply -f -
kubectl rollout restart deployment/guardai
kubectl rollout status deployment/guardai --timeout=5m

Docker Compose (policies bind-mounted from policies/ directory):

cp policies/default/policy.yaml.bak policies/default/policy.yaml
docker compose -f deploy/docker/docker-compose.yml restart server

systemd:

cp /etc/guardai/policies/default/policy.yaml.bak /etc/guardai/policies/default/policy.yaml
sudo systemctl restart oguardai-server

Step 4: Verify the Rollback

curl -sf http://guardai.internal:3000/v1/health

curl -s -X POST http://guardai.internal:3000/v1/detect \
  -H "Content-Type: application/json" \
  -d '{"input": "My SSN is 123-45-6789 and email is test@example.com"}' | jq .

Confirm entity actions (mask, block, passthrough) match expected policy. Monitor guardai_errors_total for 10 min.

Step 5: Investigate Root Cause

After rollback is stable:

1. Review bad policy for: missing entity types, wrong actions (passthrough vs mask), incorrect rule ordering (first-match wins), typos in entity type names.
2. Dry-run the bad policy:

   curl -s -X POST http://guardai.internal:3000/v1/evaluate-policy \
     -H "Content-Type: application/json" \
     -d '{"entities": [{"type": "ssn", "value": "123-45-6789"}], "policy": "strict-pii"}' | jq .

Prevention

1. Back up current policy before every change: cp policy.yaml policy.yaml.bak
2. Test new policies in staging first.
3. Use the evaluate-policy endpoint as a dry run before production deployment.
4. Version control all policy files. Tag releases for easy rollback targets.
5. Always review the diff before deploying: diff old-policy.yaml new-policy.yaml