Observability

This page covers the three signals an operator uses to run OGuardAI: structured logs, traces, and health probes. For Prometheus metrics, the Grafana dashboard, and alerting rules, see Monitoring. This page is the reference for everything else.

The overriding rule across all of these signals: raw detected values never reach any log, audit event, or exported span. Logs, audit events, and exported spans carry entity types, counts, timings, and one-way fingerprints, never the PII that tokenization protects. That guarantee is enforced in code, not by convention (see The no-PII rule below).

Structured logging

The Rust server logs through the tracing crate with a tracing-subscriber registry. There is no println! anywhere in the request path. Every log line is a structured event with typed key/value fields, so it parses cleanly whether you keep the human-readable text format or switch to JSON.

Three kinds of events matter to an operator:

Request and lifecycle events

The pipeline emits an event when an operation finishes, carrying the trace correlation ID as a fingerprint:

transform_completed  trace_id=3f1c9a0b8e5d7c42
rehydrate_completed  trace_id=3f1c9a0b8e5d7c42

HTTP request spans come from tower_http's trace layer, which is wired into the router. These sit on the tower_http target and are enabled at info by the default filter.

Error events

Every error response is logged once, at error level, with a server-generated request_id that is also returned to the client in the JSON error body. The log line and the client response share that ID, so a caller who reports a request_id points you straight at the matching log line:

request failed  request_id=<uuid-v4>  error_code=GUARDAI_POLICY_DENIED  status=403  error=<internal detail>

The error field on the log line may contain the full internal error detail. The response body returned to the client does not: internal and detection failures are mapped to generic client messages (internal error, detection failed) so no internal host, path, or dependency detail leaks to callers. This split is asserted by tests in the error module.

Audit events

Audit events are the operational record of what the pipeline did. They are emitted as structured tracing events on the dedicated guardai::audit target, which lets you route them to a separate sink, a SIEM, or an analytics pipeline by filtering on that target. Each event carries only safe metadata:

The default audit sink (audit.backend=log) emits these to tracing and never fails. The durable, HMAC-chained file sink (audit.backend=file) requires a stable, trusted parent directory that does not traverse symlinks. Its live file must be regular, owned by the runtime user, single-link, and exactly 0600; the runtime can safely narrow a strict permission superset to 0600, but rejects incompatible files. Readiness accounts for the sink (see Readiness). The complete source, verification, and export contract is in Compliance Controls.

On macOS, custody also verifies that the file has no extended ACL entries. Linux rejects audit files and pinned directories that carry POSIX ACL extended attributes and strips them from files it creates, so a parent-directory default ACL is refused at open rather than inherited; exact mode 0600 is still required.

The opened filesystem is part of custody, and the property it certifies is durability. The default accepts any non-volatile filesystem: the certified set (ext4, XFS, Btrfs, APFS) is accepted silently, while remote families (NFS and EFS, SMB and CIFS including Azure Files, 9p, Ceph) and unverified families (FUSE, overlayfs, and any unknown filesystem) are accepted with a startup warning naming the fsync and locking caveat. Memory-backed filesystems (tmpfs, ramfs, hugetlbfs) are always refused for the active log and exports. Set audit.require_certified_filesystem: true (or GUARDAI_PROXY_AUDIT_REQUIRE_CERTIFIED_FS on the proxy) to restore the strict allowlist. Active logs and export destinations must be writable; archive sources may be read-only. A container image layer now works with a warning, but mount a dedicated volume into containers because container-layer contents do not outlive the container.

Correlation: trace_id vs request_id

OGuardAI uses two distinct IDs. Do not confuse them:

• trace_id is the request-correlation ID for the protection lifecycle. The caller passes it on the transform request (if omitted, the server generates a UUID v4), then passes the same value on the matching rehydrate request. It ties together transform -> LLM -> rehydrate for one logical operation. Because it is caller-supplied free text, it is only ever logged and audited as a fingerprint, never raw.
• request_id is generated by the server per error response. It appears in the error log line and in the JSON error body. Its job is to link one client-visible failure to one server log line.

There is no incoming request-ID header that the server reads or propagates. Correlation is driven by the trace_id you pass in the request body.

The no-PII rule

The trust boundary is that raw PII stays in the trusted zone (the runtime and, when deployed, the Python NER sidecar). Logging honors it without exception:

• Session IDs are logged only as a keyed HMAC-SHA-256 fingerprint (16 lowercase hex characters). In memory-session mode the session ID is a live bearer credential, so it must never appear verbatim in any sink. The fingerprint still correlates lines for the same session.
• Client-supplied trace_id values are fingerprinted the same way before logging or auditing, because they are free text that could otherwise carry PII.
• Tenant IDs use the same keyed projection on every sink outside the runtime. A tenant is whatever the credential claims, and a claim like julia.schneider can identify a person. The durable file sink stays raw on purpose: it is the compliance record and its per-tenant hash chain binds the tenant.
• Audit events record entity types and counts, never values. On logs, webhooks, and evidence exports, custom entity labels are reduced to a placeholder. Policy name, policy version, purpose, and data category are each replaced by a deterministic keyed fingerprint in a distinct cryptographic field domain.
• The exact policy_content_hash and closed GDPR legal basis remain readable. They identify policy content and lawful-basis categories without forwarding a free-form label.
• Client-facing error bodies are scrubbed to generic messages for internal and detection failures, so a database host or a missing-file path never reaches the caller.
• The Python NER sidecar logs the request text_length and language, never the text itself.

When you configure downstream log shipping, treat these fields as already safe. Do not add a middleware that re-logs raw request bodies: that would defeat the boundary.

Configuring log level and format

Both controls are environment variables read at startup.

Level

The subscriber uses an EnvFilter sourced from RUST_LOG. When it is unset the server falls back to warn,oguardai_server=info,guardai_server=info,guardai::audit=info,tower_http=info.

Setting RUST_LOG replaces that filter outright, so keep three things in whatever you set. The bare warn is a floor for every crate: without it a component that can fail a request stays silent even at ERROR, including the output guard's block reason. oguardai_server is the binary target, distinct from the guardai_server library, so it carries the startup report; a filter naming only guardai_server mutes it. And guardai::audit=info is what lets the audit trail past the floor, as below.

# Quietest filter that still keeps the audit trail and the server's own block reasons
RUST_LOG="warn,oguardai_server=info,guardai::audit=info" oguardai run

# Turn up the server crate for debugging, keep the floor, startup report, audit trail and request spans
RUST_LOG="warn,oguardai_server=info,guardai_server=debug,guardai::audit=info,tower_http=info" oguardai run

Because audit events ride the guardai::audit target, you can hold them at a different level than the rest of the server, which is the mechanism for routing them separately. They are emitted at INFO, below the bare warn floor, which is why the built-in default names that target explicitly: out of the box a deployment on the log audit backend does get its audit trail on stdout. An override that drops the directive silences it. The durable file backend writes them regardless of the log filter.

Format

Set GUARDAI_LOG_FORMAT=json to emit newline-delimited JSON instead of the default human-readable text. Use JSON in any environment where a log collector parses the stream:

GUARDAI_LOG_FORMAT=json \
  RUST_LOG="warn,oguardai_server=info,guardai_server=info,guardai::audit=info,tower_http=info" \
  oguardai run

An audit event in JSON form carries the fields from the table above, for example:

{
  "level": "INFO",
  "target": "guardai::audit",
  "fields": {
    "message": "audit_event",
    "event_type": "Transform",
    "tenant_id": "822b33ad87c148a0",
    "session_id": "9c1f4e2a7b0d6538",
    "policy": "4d1d5f676d7930f1",
    "policy_content_hash": "<sha256-of-resolved-policy>",
    "entity_types": "[\"email\", \"<non-identifier>\"]",
    "entity_count": 2,
    "entities_tokenized": 2,
    "entities_blocked": 0,
    "duration_ms": 4.1,
    "detector_mode": "both",
    "trace_id": "3f1c9a0b8e5d7c42"
  }
}

Several fields in that sample are projections, not the values the runtime held. tenant_id is the keyed fingerprint of the tenant acme, and policy is the separately keyed fingerprint of the policy name. entity_types keeps one built-in type and replaces one custom label with <non-identifier>. Correlate on the deterministic fingerprints and use policy_content_hash to identify the resolved policy content. Use the protected durable file sink when an investigation requires the original tenant, policy, version, purpose, data category, or custom entity labels.

The JSON toggle applies to the runtime server only. The transparent proxy uses the text format, and its own default filter is warn,oguardai_proxy=info,guardai::audit=info,tower_http=info (the binary's target is oguardai_proxy; there is no guardai_proxy target). The CLI honors RUST_LOG with no built-in fallback.

Ordinary policy and startup diagnostics omit free-form policy names. Policy warnings use the resolved policy_content_hash when available and reduce custom entity labels to <non-identifier>. Proxy startup logs only the provider and upstream scheme, host, and port, without the target path, query, or policy name.

Transparent proxy audit

The proxy defaults to GUARDAI_PROXY_AUDIT_BACKEND=log. Set the backend to file and provide GUARDAI_PROXY_AUDIT_FILE to use the access-controlled, HMAC-chained file writer on qualified storage. Shutdown is clean only when accepted events are finalized and exclusive file custody is released within the shared deadline.

Both services create a dedicated audit-finalizer thread during startup and refuse startup if it cannot be created. Shutdown closes admission first, waits for admitted strict writes, and then finalizes file custody. Secondary cleanup starts after the admission phase and shares the same absolute deadline. Audit phase failures and notification worker timeout, panic, or cancellation make shutdown non-clean. Individual webhook delivery failures remain best effort and stay visible in the notification counters. A commit-indeterminate event keeps the sink unhealthy until that exact event is resolved; another event completing cannot clear it.

Proxy recording remains buffered and fail-open on the request path. A full queue or stopped writer does not change an upstream result, so the file can omit an otherwise successful forwarded request. It is a verifiable chain for records accepted by the writer, not a per-request durability guarantee. Server audit.strict and the server --compliance preset do not make proxy audit strict.

OpenTelemetry tracing

The runtime ships an OTLP span exporter. It is built in but off by default. What the operator adds is the collector.

What is built in

When GUARDAI_OTEL_TRACES_ENDPOINT is set, the server attaches a tracing-opentelemetry layer backed by an OTLP-over-HTTP span exporter to the same subscriber that handles logs. Spans then flow to your collector alongside the existing logs and metrics. When the variable is unset or blank, no trace layer is attached and the server runs exactly as before, with logs and metrics only.

# Export spans to an OTLP/HTTP collector (Jaeger, Tempo, the OTel Collector, ...)
GUARDAI_OTEL_TRACES_ENDPOINT="http://collector:4318/v1/traces" \
GUARDAI_OTEL_SERVICE_NAME="oguardai-server" \
  oguardai run

Implementation details that matter operationally:

• Transport is OTLP over HTTP. There is no gRPC dependency to provision.
• Export runs on a batch processor with a blocking HTTP client on its own thread, so it does not compete with the async request runtime.
• Shutdown flushes: the exporter force-flushes and shuts down cleanly on server exit, so in-flight spans are not lost on a normal stop.
• If the exporter fails to initialize (for example a malformed endpoint), the server logs a warning and continues without trace export rather than refusing to start. Trace export is a best-effort side channel, not a request-path dependency.

What the operator adds

You provide the collector at the endpoint: an OpenTelemetry Collector, Jaeger, Grafana Tempo, or any OTLP/HTTP-compatible backend. The server does not run one for you. A minimal setup is an OTel Collector container with an otlp HTTP receiver on :4318, exporting to your tracing backend, with GUARDAI_OTEL_TRACES_ENDPOINT pointed at it.

Span content and trust level

Exported spans inherit the same discipline as the logs. The request span records only the HTTP method, the matched route template (/v1/sessions/{id}, never the caller's raw target), and a non-reversible fingerprint of that target. Log and audit events carry counts, timings, field-separated policy fingerprints, entity types reduced to the built-in set, and fingerprinted tenant, session, and trace identifiers, never raw detected values. Export only to a collector you control, and verify that pipeline before relying on it.

Only the runtime server exports spans. The proxy and CLI do not.

Health and readiness

OGuardAI separates liveness (is the process running) from readiness (can it actually serve). Both the Rust runtime and the Python NER sidecar expose the same probe shape, so an orchestrator can wire them independently.

Runtime server

Two unauthenticated probes sit ahead of the auth middleware, so a probe never needs credentials:

curl http://localhost:3000/livez    # liveness
curl http://localhost:3000/readyz   # readiness

GET /livez is a cheap always-200 process check. If the handler runs at all, the process is alive:

{"status": "ok"}

GET /readyz returns 200 with {"status": "ready"} only when the runtime can actually serve, otherwise 503 with {"status": "not_ready"}. It reports not-ready when any of the following holds:

• No policy is loaded. An empty policy engine protects nothing, so the pod is kept out of rotation.
• The detector is configured for NER (detector.mode is both or advanced) but the NER sidecar is not wired, or its own /readyz does not answer success within a 2 second probe. This keeps traffic off a pod whose detection backend is down.
• The audit sink is unhealthy. A durable file sink whose writer thread has died can no longer record the audit trail, so the pod reports not-ready. The default log sink is always healthy.

Wire /livez to the liveness probe and /readyz to the readiness probe. Use /readyz, not /v1/health, as the readiness gate, because only /readyz performs the live sidecar reachability check.

GET /v1/health is the richer status endpoint. It requires the configured auth when auth is enabled, so use /livez and /readyz for orchestration and /v1/health for human or dashboard inspection:

curl -H "X-API-Key: $GUARDAI_API_KEY" http://localhost:3000/v1/health

{
  "status": "healthy",
  "version": "0.1.0",
  "uptime_seconds": 1.2,
  "components": {
    "detector": {"status": "healthy", "message": "builtin_and_ner (full entity detection)"},
    "session": {"status": "healthy", "message": "sealed"},
    "policies": {"status": "healthy", "message": "3 policies loaded"}
  },
  "license": {
    "status": "absent",
    "edition": "developer",
    "release_date": "2026-07-29"
  }
}

The detector component reports the configured mode: builtin_only, builtin_and_ner, or ner_required. Note the deliberate design choice: /v1/health always reports the detector as healthy even when an optional NER sidecar is temporarily unreachable, because it must not flip a pod unhealthy on an optional dependency. Two things give you the fuller picture:

• To check live NER sidecar reachability, use /readyz, which probes the sidecar.
• To check whether NER is configured at all, use GET /v1/capabilities, which includes an ner_active field.

The licence block

/v1/health also reports the deployment's licence. Nothing is gated by it. No capability is withheld and no behaviour changes based on what it says, so read this block as a statement of what the deployment is entitled to, not as a switch.

The server loads a signed licence file when GUARDAI_LICENSE_FILE points at one and resolves it once at startup. Resolution checks an Ed25519 signature against a compiled-in key, offline: OGuardAI never contacts Oronts. The block never carries the signature, key material, or entitlement flags.

Verification is not operative in this release. No verifying key is compiled into the binary, so a licence file cannot be checked at all: a correctly signed licence and a forged one are indistinguishable to it. Any licence the server can read reports malformed with a reason naming the missing key, and the deployment runs as the developer edition. That is a defect in the release rather than in the file you were issued, so it is logged as an error and never refuses a start. valid and version_cutoff are unreachable until a release ships the key.

Enforcement is on the version axis, never a wall clock: the binary's compiled-in release date is compared against updates_until. A running process therefore never changes behaviour because a date passed. A release newer than the licence covers reports version_cutoff and keeps running as the developer edition. That grading sits behind verification, so it is unreachable in this release.

Four operator-relevant outcomes:

• No licence configured. Resolves to the developer edition and the server starts normally. This is a supported deployment.
• A licence file this build cannot check, which is every licence it can read while no verifying key is compiled in. Reports malformed with a reason naming the missing key, logs an error, runs as the developer edition, and starts. There is no operator remedy, so it must not stop a deployment.
• A licence that does not cover this release. Reports version_cutoff, logs a warning, and starts. Unreachable until a release ships the key.
• A licence file that cannot be read or is oversized. Startup is refused outside dev auth mode: a deployment handed a licence it cannot open must not silently run as if it had none. Replace the file, unset GUARDAI_LICENSE_FILE, or use auth.mode=dev. Once a key ships, every present-but-unusable licence refuses outside dev.

NER sidecar

The Python detector service exposes the same three-endpoint shape:

curl http://localhost:9090/livez    # {"status": "alive"}
curl http://localhost:9090/readyz   # ready or not_ready (+ backend)
curl http://localhost:9090/health   # full status

GET /livez is always 200 while the process runs. GET /readyz returns 200 {"status": "ready", "backend": "..."} only when a NER model is loaded and not degraded, otherwise 503 {"status": "not_ready", "backend": "..."}. That is the readiness signal the runtime's own /readyz consults when NER is required.

GET /health is always 200 and reports the degraded flag directly:

{
  "status": "healthy",
  "service": "guardai-detector-py",
  "wire_contract": 1,
  "detector_loaded": true,
  "backend": "spacy",
  "ner_backend": "spacy",
  "degraded": false,
  "loaded_models": ["en", "de"],
  "languages": {
    "enrichment": ["en", "de"],
    "models": ["en", "de"],
    "benchmarked": []
  }
}

status is degraded exactly when degraded is true, and degraded is true only when no detector is loaded. loaded_models is a backward-compatible union; prefer the three distinct sets under languages (enrichment packs, loaded-model coverage, and benchmark-proven languages).

By default the sidecar fails closed: NER_BACKEND_STRICT=true (the default) aborts startup when the chosen backend cannot load, rather than silently degrading. The gliner -> spacy -> none fallback and the degraded-serving state only occur when you set NER_BACKEND_STRICT=false (see the plugin development guide and apps/detector-py/README.md). In that opt-in mode, degraded is true when the preferred NER backend was unavailable and the service fell back to no NLP-based recognition; /detect still returns 200 but with degraded: true and a warning, and the runtime's readiness gate treats the sidecar as not-ready. Watch the backend and degraded fields to catch a fallback from GLiNER to spaCy, or from spaCy to none.

Where to look for a given symptom