Version Upgrade Runbook

Upgrade OGuardAI from version N to N+1.

Availability: The registry, image and package installs below are private today and provided under a written agreement with Oronts. Request access at /request.

Pre-Upgrade Checklist

1. Read the changelog. Note breaking changes.
2. Back up current config:

   helm get values guardai -o yaml > guardai-values-backup.yaml          # Helm
   cp /etc/guardai/oguardai.yaml /etc/guardai/oguardai.yaml.bak            # systemd

3. Confirm current deployment is healthy: curl -sf http://guardai.internal:3000/livez
4. Session compatibility: check the changelog for a sealed envelope version bump. Without one, sealed sessions carry across the upgrade under the same key. With one, see the section below; the upgrade is breaking for in-flight sessions and you must drain first.
5. Policy compatibility: minor versions only add fields with defaults. Existing policies work as-is.

Breaking: Sealed Envelope Version Bump

A release that bumps the sealed envelope version invalidates every session sealed by the previous release. The runtime accepts only the exact version it emits, in both directions, so an old node cannot read a new blob and a new node cannot read an old one. This is deliberate: the sealed payload carries the authorization decision that governs restoration, and a node reading a blob it does not fully understand would silently fall back to the live policy and bypass the protections sealed into that session.

Consequences, all of them expected:

• Every in-flight session_state blob stops unsealing. Clients receive 410 GONE with GUARDAI_SESSION_EXPIRED and must re-run transform to obtain a fresh session.
• Sessions in the Redis session backend read as absent, so callers receive a not-found rather than a restore under stale assumptions.
• Persisted RAG chunk sessions stored alongside vector embeddings become unusable. Those corpora need re-ingest. Plan this before the upgrade window, not after; it is the long pole.
• Rolling deploys and rollbacks are both affected. Mixed-version nodes sharing a session key reject each other's blobs rather than misread them.
• Cached idempotency results survive the bump, on purpose. The Redis idempotency namespace (guardai:idem:) is the same across every release, because that one key holds the claim as well as the cached result: versioning it would split the claim space, and during a rolling deploy an old and a new node would each see a key as unclaimed and both execute the same mutation. At-most-once wins over freshness here. The visible effect is that a retry with a pre-upgrade Idempotency-Key replays a response whose session_state the new nodes cannot unseal, exactly like every other pre-upgrade blob after the bump: the client re-runs transform, as it must anyway. Cached results expire on their own within idempotency.ttl_seconds (default 86400). Do not flush the namespace during a rolling deploy; flushing removes live claims and reopens the duplicate-execution window.

Procedure. Choose one:

Drain (preferred). Stop admitting new work, wait one full session TTL (session.ttl_seconds, default 3600) so every in-flight session ages out on its own, then upgrade. No client sees an error.

Accept the invalidation. Upgrade directly and let active sessions fail. Only do this when clients retry transform on 410, and when no RAG corpus depends on persisted chunk sessions. Notify API consumers beforehand and schedule the RAG re-ingest.

Verify after the upgrade that a fresh transform to rehydrate round-trip succeeds (see Post-Upgrade Verification below), then watch for a GUARDAI_SESSION_EXPIRED spike that outlasts one session TTL, which would mean clients are still replaying pre-upgrade blobs. Query the structured application log or the audit stream for it, not the metrics endpoint: guardai_errors_total is a single unlabeled counter with no per-code breakdown, so it cannot separate session expiry from any other failure. Each failed request logs one request failed event carrying error_code, status, and request_id, so filter on error_code = GUARDAI_SESSION_EXPIRED and trend the count. Use guardai_errors_total only as the coarse "is anything failing at all" signal that tells you to go read the log.

Helm Upgrade

helm diff upgrade oguardai oci://ghcr.io/oronts/charts/oguardai --version 0.1.1 --set server.image.tag=0.1.1 -f guardai-values-backup.yaml
helm upgrade oguardai oci://ghcr.io/oronts/charts/oguardai --version 0.1.1 --set server.image.tag=0.1.1 -f guardai-values-backup.yaml --wait --timeout 5m
kubectl rollout status deployment/oguardai --timeout=5m

The image tag is the chart appVersion semver without a leading v (0.1.1, not v0.1.1); the publish pipeline tags images that way, so a v-prefixed tag would not resolve.

The chart's render-time guards run on upgrade too, so a values backup taken from a release installed under an older chart can fail the upgrade rather than the install. A non-dev auth.mode now requires an answer for the revocation table as well as the audit trail: add revocation.backend=redis with redis.external.url for a durable shared table, or revocation.allowEphemeral=true to keep the pod-local one, to guardai-values-backup.yaml before running the upgrade. helm diff upgrade above surfaces this first, without touching the release.

For a fleet that must resolve to the exact published bytes across a rollout and a rollback, pin by digest instead of by tag. Read the digests for the version from the signed release ledger (see "Pin and roll back by digest" in the deployment guide for the cosign verify-blob plus jq lookup), then either set server.image.digest and detector.image.digest directly or fill them into the overlay. The shipped values-digest-pinned.yaml carries all-zero placeholder digests that resolve to no real image, so replace them with the ledger digests before you apply it:

# Replace the placeholder digests in values-digest-pinned.yaml with the ledger digests first.
helm upgrade oguardai oci://ghcr.io/oronts/charts/oguardai --version 0.1.1 \
  -f guardai-values-backup.yaml -f deploy/helm/oguardai/values-digest-pinned.yaml --wait --timeout 5m

A set digest wins over the tag.

Rollback:

helm history oguardai
helm rollback oguardai          # previous revision
helm rollback oguardai 3        # specific revision

When the images were digest-pinned, roll the previous version's digest back in the same way: read its image digests from that version's signed release ledger and re-run helm upgrade with the previous server.image.digest and detector.image.digest (or a values-digest-pinned.yaml filled from that ledger), so the rollback lands on the exact bytes that version published rather than whatever a tag currently resolves to.

Docker Compose Upgrade

The compose files reference the published image by GUARDAI_VERSION (default 0.1.0), so an upgrade pulls the new tag rather than rebuilding:

cd deploy/docker
export GUARDAI_VERSION=0.1.1
docker compose pull server detector
docker compose up -d --no-deps server
docker compose up -d --no-deps detector   # if detector also upgraded
docker compose ps
curl -sf http://localhost:3000/livez

For an exact, digest-pinned rollout instead of a tag, see "Pin and Roll Back by Digest" in the deployment guide.

Rollback:

export GUARDAI_VERSION=0.1.0
docker compose pull server
docker compose up -d --no-deps server

systemd Upgrade

The release-download URL below is illustrative. Published release artifacts are not yet available, so substitute your own build output or internal artifact URL until releases are published.

# Illustrative URL: replace with a published release asset or your own build
curl -L -o /tmp/oguardai-server \
  https://github.com/oronts/OGuardAI/releases/download/v0.1.1/oguardai-server-linux-amd64
chmod +x /tmp/oguardai-server
sudo systemctl stop oguardai-server
cp /usr/local/bin/oguardai-server /usr/local/bin/oguardai-server.bak
mv /tmp/oguardai-server /usr/local/bin/oguardai-server
sudo systemctl start oguardai-server
systemctl status oguardai-server
curl -sf http://localhost:3000/livez

Stop before you replace, as the sequence above does. PDF extraction re-execs the server binary in a bounded sandbox subprocess, so the running image has to stay resolvable, and identical, for the life of the process. On Linux the sandbox re-execs through the /proc/self/exe magic link, which keeps resolving to the live image after its inode is unlinked, so an in-place replacement no longer breaks extraction there. Two cases still depend on the stop-first order: a Linux container that does not mount procfs, and any non-Linux Unix host. Both fall back to the binary's pathname. If the file is gone, every PDF then fails closed with 503 GUARDAI_DOCUMENT_EXTRACTION_UNAVAILABLE until the service restarts; if a new file now sits at that pathname, the child would run different code than the parent. Text, JSON, and image paths are unaffected either way.

Rollback:

sudo systemctl stop oguardai-server
mv /usr/local/bin/oguardai-server.bak /usr/local/bin/oguardai-server
sudo systemctl start oguardai-server

Post-Upgrade Verification

Run these regardless of deployment method:

# Liveness (unauthenticated)
curl -sf http://guardai.internal:3000/livez

# Test transform (with auth)
curl -s -X POST http://guardai.internal:3000/v1/transform \
  -H "Content-Type: application/json" \
  -H "X-API-Key: $GUARDAI_API_KEY" \
  -d '{"input": "My email is test@example.com"}' | jq .

# Test rehydrate (use session_state from transform response)
curl -s -X POST http://guardai.internal:3000/v1/rehydrate \
  -H "Content-Type: application/json" \
  -H "X-API-Key: $GUARDAI_API_KEY" \
  -d '{"output": "Your email is `{{email:xxxx:29ccba2d9234}}`", "session_state": "<blob>"}' | jq .

Monitor guardai_errors_total and guardai_transform_duration_seconds for 10 min after upgrade.

Compatibility Notes

• Sessions: The same key decrypts across minor versions as long as the sealed envelope version is unchanged. Do not rotate keys during upgrade. When the changelog reports an envelope version bump, the key is irrelevant: drain first, per the section above.
• Policies: Additive changes only in minor versions. Existing files work without modification.
• API: From 1.0, /v1/* endpoints follow the stability contract within a major version (new response fields may appear; existing fields are not removed). Pre-1.0 (0.x), the surface may still change.