Observability

This page covers the three signals an operator uses to run OGuardAI: structured logs, traces, and health probes. For Prometheus metrics, the Grafana dashboard, and alerting rules, see Monitoring. This page is the reference for everything else.

The overriding rule across all of these signals: raw detected values never leave the runtime. Logs, audit events, and exported spans carry entity types, counts, timings, and one-way fingerprints, never the PII that tokenization protects. That guarantee is enforced in code, not by convention (see The no-PII rule below).

Structured logging

The Rust server logs through the tracing crate with a tracing-subscriber registry. There is no println! anywhere in the request path. Every log line is a structured event with typed key/value fields, so it parses cleanly whether you keep the human-readable text format or switch to JSON.

Three kinds of events matter to an operator:

Request and lifecycle events

The pipeline emits an event when an operation finishes, carrying the trace correlation ID as a fingerprint:

transform_completed  trace_id=3f1c9a0b8e5d7c42
rehydrate_completed  trace_id=3f1c9a0b8e5d7c42

HTTP request spans come from tower_http's trace layer, which is wired into the router. These sit on the tower_http target and are enabled at info by the default filter.

Error events

Every error response is logged once, at error level, with a server-generated request_id that is also returned to the client in the JSON error body. The log line and the client response share that ID, so a caller who reports a request_id points you straight at the matching log line:

request failed  request_id=<uuid-v4>  error_code=GUARDAI_POLICY_DENIED  status=403  error=<internal detail>

The error field on the log line may contain the full internal error detail. The response body returned to the client does not: internal and detection failures are mapped to generic client messages (internal error, detection failed) so no internal host, path, or dependency detail leaks to callers. This split is asserted by tests in the error module.

Audit events

Audit events are the operational record of what the pipeline did. They are emitted as structured tracing events on the dedicated guardai::audit target, which lets you route them to a separate sink, a SIEM, or an analytics pipeline by filtering on that target. Each event carries only safe metadata:

The default audit sink (audit.backend=log) emits these to tracing and never fails. The durable, HMAC-chained file sink (audit.backend=file) is a separate backend covered in the deployment and runbook docs; readiness accounts for it (see Readiness).

Correlation: trace_id vs request_id

OGuardAI uses two distinct IDs. Do not confuse them:

• trace_id is the request-correlation ID for the protection lifecycle. The caller passes it on the transform request (if omitted, the server generates a UUID v4), then passes the same value on the matching rehydrate request. It ties together transform -> LLM -> rehydrate for one logical operation. Because it is caller-supplied free text, it is only ever logged and audited as a fingerprint, never raw.
• request_id is generated by the server per error response. It appears in the error log line and in the JSON error body. Its job is to link one client-visible failure to one server log line.

There is no incoming request-ID header that the server reads or propagates. Correlation is driven by the trace_id you pass in the request body.

The no-PII rule

The trust boundary is that raw PII exists only inside the runtime. Logging honors it without exception:

• Session IDs are logged only as a truncated SHA-256 fingerprint (16 hex characters). In memory-session mode the session ID is a live bearer credential, so it must never appear verbatim in any sink. The fingerprint still correlates lines for the same session.
• Client-supplied trace_id values are fingerprinted the same way before logging or auditing, because they are free text that could otherwise carry PII.
• Audit events record entity types and counts, never values. This is stated in the audit module and holds for both the log sink and the file sink.
• Client-facing error bodies are scrubbed to generic messages for internal and detection failures, so a database host or a missing-file path never reaches the caller.
• The Python NER sidecar logs the request text_length and language, never the text itself.

When you configure downstream log shipping, treat these fields as already safe. Do not add a middleware that re-logs raw request bodies: that would defeat the boundary.

Configuring log level and format

Both controls are environment variables read at startup.

Level

The subscriber uses an EnvFilter sourced from RUST_LOG. When it is unset the server falls back to guardai_server=info,tower_http=info.

# Quieten everything except warnings, but keep audit events at info
RUST_LOG="guardai_server=warn,guardai::audit=info" oguardai run

# Turn up the server crate for debugging, keep request spans at info
RUST_LOG="guardai_server=debug,tower_http=info" oguardai run

Because audit events ride the guardai::audit target, you can hold them at a different level than the rest of the server, which is the mechanism for routing them separately.

Format

Set GUARDAI_LOG_FORMAT=json to emit newline-delimited JSON instead of the default human-readable text. Use JSON in any environment where a log collector parses the stream:

GUARDAI_LOG_FORMAT=json RUST_LOG="guardai_server=info" oguardai run

An audit event in JSON form carries the fields from the table above, for example:

{
  "level": "INFO",
  "target": "guardai::audit",
  "fields": {
    "message": "audit_event",
    "event_type": "Transform",
    "tenant_id": "acme",
    "session_id": "9c1f4e2a7b0d6538",
    "policy": "german-support",
    "entity_types": "[\"email\", \"person\"]",
    "entity_count": 2,
    "entities_tokenized": 2,
    "entities_blocked": 0,
    "duration_ms": 4.1,
    "detector_mode": "both",
    "trace_id": "3f1c9a0b8e5d7c42"
  }
}

The JSON toggle applies to the runtime server. The transparent proxy and the CLI use the text format and honor RUST_LOG (their default filters are guardai_proxy=info,tower_http=info and the ambient RUST_LOG respectively).

OpenTelemetry tracing

The runtime ships an OTLP span exporter. It is built in but off by default. What the operator adds is the collector.

What is built in

When GUARDAI_OTEL_TRACES_ENDPOINT is set, the server attaches a tracing-opentelemetry layer backed by an OTLP-over-HTTP span exporter to the same subscriber that handles logs. Spans then flow to your collector alongside the existing logs and metrics. When the variable is unset or blank, no trace layer is attached and the server runs exactly as before, with logs and metrics only.

# Export spans to an OTLP/HTTP collector (Jaeger, Tempo, the OTel Collector, ...)
GUARDAI_OTEL_TRACES_ENDPOINT="http://collector:4318/v1/traces" \
GUARDAI_OTEL_SERVICE_NAME="oguardai-server" \
  oguardai run

Implementation details that matter operationally:

• Transport is OTLP over HTTP. There is no gRPC dependency to provision.
• Export runs on a batch processor with a blocking HTTP client on its own thread, so it does not compete with the async request runtime.
• Shutdown flushes: the exporter force-flushes and shuts down cleanly on server exit, so in-flight spans are not lost on a normal stop.
• If the exporter fails to initialize (for example a malformed endpoint), the server logs a warning and continues without trace export rather than refusing to start. Trace export is a best-effort side channel, not a request-path dependency.

What the operator adds

You provide the collector at the endpoint: an OpenTelemetry Collector, Jaeger, Grafana Tempo, or any OTLP/HTTP-compatible backend. The server does not run one for you. A minimal setup is an OTel Collector container with an otlp HTTP receiver on :4318, exporting to your tracing backend, with GUARDAI_OTEL_TRACES_ENDPOINT pointed at it.

Span content and trust level

Exported spans inherit the same discipline as the logs: they carry request identifiers (tenant and policy names) and safe metadata (entity types and counts) plus the fingerprinted trace ID, never the raw detected values. Because the tenant and trace identifiers are caller-controlled, treat exported spans at the same trust level as your logs and export only to a collector you control. Verify the pipeline against your own collector before relying on it.

Only the runtime server exports spans. The proxy and CLI do not.

Health and readiness

OGuardAI separates liveness (is the process running) from readiness (can it actually serve). Both the Rust runtime and the Python NER sidecar expose the same probe shape, so an orchestrator can wire them independently.

Runtime server

Two unauthenticated probes sit ahead of the auth middleware, so a probe never needs credentials:

curl http://localhost:3000/livez    # liveness
curl http://localhost:3000/readyz   # readiness

GET /livez is a cheap always-200 process check. If the handler runs at all, the process is alive:

{"status": "ok"}

GET /readyz returns 200 with {"status": "ready"} only when the runtime can actually serve, otherwise 503 with {"status": "not_ready"}. It reports not-ready when any of the following holds:

• No policy is loaded. An empty policy engine protects nothing, so the pod is kept out of rotation.
• The detector is configured for NER (detector.mode is both or advanced) but the NER sidecar is not wired, or its own /readyz does not answer success within a 2 second probe. This keeps traffic off a pod whose detection backend is down.
• The audit sink is unhealthy. A durable file sink whose writer thread has died can no longer record the audit trail, so the pod reports not-ready. The default log sink is always healthy.

Wire /livez to the liveness probe and /readyz to the readiness probe. Use /readyz, not /v1/health, as the readiness gate, because only /readyz performs the live sidecar reachability check.

GET /v1/health is the richer status endpoint. It requires the configured auth when auth is enabled, so use /livez and /readyz for orchestration and /v1/health for human or dashboard inspection:

curl -H "X-API-Key: $GUARDAI_API_KEY" http://localhost:3000/v1/health

{
  "status": "healthy",
  "version": "0.1.0",
  "uptime_seconds": 1.2,
  "components": {
    "detector": {"status": "healthy", "message": "builtin_and_ner (full entity detection)"},
    "session": {"status": "healthy", "message": "sealed"},
    "policies": {"status": "healthy", "message": "3 policies loaded"}
  }
}

The detector component reports the configured mode: builtin_only, builtin_and_ner, or ner_required. Note the deliberate design choice: /v1/health always reports the detector as healthy even when an optional NER sidecar is temporarily unreachable, because it must not flip a pod unhealthy on an optional dependency. Two things give you the fuller picture:

• To check live NER sidecar reachability, use /readyz, which probes the sidecar.
• To check whether NER is configured at all, use GET /v1/capabilities, which includes an ner_active field.

NER sidecar

The Python detector service exposes the same three-endpoint shape:

curl http://localhost:9090/livez    # {"status": "alive"}
curl http://localhost:9090/readyz   # ready or not_ready (+ backend)
curl http://localhost:9090/health   # full status

GET /livez is always 200 while the process runs. GET /readyz returns 200 {"status": "ready", "backend": "..."} only when a NER model is loaded and not degraded, otherwise 503 {"status": "not_ready", "backend": "..."}. That is the readiness signal the runtime's own /readyz consults when NER is required.

GET /health is always 200 and reports the degraded flag directly:

{
  "status": "degraded",
  "service": "guardai-detector-py",
  "detector_loaded": true,
  "backend": "spacy",
  "ner_backend": "spacy",
  "degraded": false,
  "loaded_models": ["en", "de"]
}

degraded is true when no detector is loaded, or when the preferred NER backend was unavailable and the service fell back to no NLP-based recognition. In degraded mode /detect still returns 200, but with degraded: true and a warning, and the runtime's readiness gate treats the sidecar as not-ready. Watch the backend and degraded fields to catch a silent fallback from GLiNER to spaCy, or from spaCy to none.

Where to look for a given symptom