The Prometheus metrics OGuardAI exposes, the Grafana dashboard and alerting rules it ships, how to wire them up, and the current state of distributed tracing.

OGuardAI exposes operational metrics in Prometheus exposition format and ships a ready-to-import Grafana dashboard plus a starter set of alerting rules. This page is the metric reference and the wiring guide.

The /metrics endpoint

Each server instance exposes GET /metrics in Prometheus text format. The endpoint requires the admin scope, so Prometheus must scrape it with admin credentials, unlike the unauthenticated /livez and /readyz probes. In API-key auth mode that means an admin X-API-Key; in JWT or OIDC mode it means a token that carries the admin scope. In a multi-instance deployment every instance exposes its own /metrics, and Prometheus scrapes each independently. See Deployment for the scrape config and PromQL aggregation across instances.

The transparent proxy exposes its own GET /metrics on the same header auth it uses for the data plane. By default the ingress key (X-GuardAI-API-Key, GUARDAI_PROXY_API_KEY) scrapes it. Set GUARDAI_PROXY_METRICS_API_KEY to a distinct value to move the scrape onto a dedicated credential that is admitted only to /metrics: the ingress key then serves only the data plane, and the metrics key cannot call the upstream, reach /v1/capabilities, or read restored output. It is a least-privilege split for the monitoring system, not an exfiltration control. See the proxy guide for the full credential and role model.

While a second ingress key is configured for a rotation overlap (GUARDAI_PROXY_API_KEY_NEXT), the proxy's /metrics body carries a proxy-local counter guardai_proxy_ingress_auth_total{key_slot="current"} and guardai_proxy_ingress_auth_total{key_slot="next"}, the count of ingress authentications served by each key slot. Watch key_slot="current" fall to zero after rolling clients onto the new key, then promote the new key and unset the next slot. The counter is proxy-local and does not appear on the runtime server's /metrics, and with no next key the proxy's /metrics body is unchanged. See the key rotation runbook.

Metric catalog

Operation counters

Metric	Type	Meaning
`guardai_transforms_total`	counter	Transform operations processed
`guardai_rehydrates_total`	counter	Rehydrate operations processed
`guardai_detections_total`	counter	Detect operations processed
`guardai_errors_total`	counter	Errors returned
`guardai_prompt_security_triggers_total`	counter	Prompt-security engine triggers
`guardai_output_guard_triggers_total`	counter	Output-guard triggers (a model response with newly generated PII, or a raw session value echoed instead of its token)
`guardai_sessions_created_total`	counter	Sealed sessions created
`guardai_rate_limit_rejections_total`	counter	Requests rejected by the rate limiter
`guardai_rate_limit_backend_unavailable_total`	counter	Requests refused because the shared rate-limit backend was unavailable (`rate_limit.backend: redis`), distinct from `guardai_rate_limit_rejections_total` which counts requests that hit their quota. Zero when `rate_limit.backend` is `local`
`guardai_rag_operations_total`	counter	RAG pipeline operations
`guardai_ner_degraded_total`	counter	Operations whose detection produced regex-only coverage although NER was configured: the `both`-mode sidecar-outage fallback, plus a request that explicitly selected only `builtin_regex`. It records the detection outcome, not the request outcome, and is counted per operation (once per batch item, once per RAG stage). An `advanced`-mode outage fails inside detection and never reaches it, but on several paths the increment precedes later fail-closed steps, so it is not disjoint from `GUARDAI_DETECTION_UNAVAILABLE` per request. See Reading the NER degradation counter for the per-path positions

Entity counters (labeled)

Metric	Type	Labels	Meaning
`guardai_entities_detected_total`	counter	`type`	Entities detected, by entity type
`guardai_entities_blocked_total`	counter	`type`	Entities blocked, by entity type

The type label is cardinality-capped at 64 distinct values per counter. Operator-defined custom entity types (policy custom_patterns) beyond that cap are counted under type="other", so a policy cannot grow the Prometheus series set without bound. Label values are escaped for the exposition format. A custom type exports under its own type only when the deployment declares it (a config custom detector, a vocab extra pattern or pattern spec, the external detector webhook's declared types, or a loaded policy's custom pattern, document column, or NER-backed type); an inline custom type a request supplies but the deployment never declared is counted under a single type="custom:other" series, distinct from the series-cap overflow bucket type="other".

Latency histograms

Metric	Type	Meaning
`guardai_transform_duration_seconds`	histogram	Transform latency in seconds
`guardai_rehydrate_duration_seconds`	histogram	Rehydrate latency in seconds

Each histogram exposes _bucket{le="..."}, _sum, and _count. Buckets run from 1 ms to 10 s. Compute quantiles with histogram_quantile, for example:

histogram_quantile(0.95, sum(rate(guardai_transform_duration_seconds_bucket[5m])) by (le))

Grafana dashboard

A ready-to-import dashboard ships at deploy/grafana/oguardai-dashboard.json. In Grafana, open Dashboards, New, Import, then upload the JSON file and select your Prometheus data source. It visualizes request and error rates, transform and rehydrate p95 latency, entity detection and blocking by type, session creation, RAG operations, and the prompt-security and output-guard triggers. Every panel is built on the metrics in the catalog above.

Alerting rules

A starter alert set ships at deploy/prometheus/alerting-rules.yml. Load it with Prometheus --rule-files. It covers:

high error rate (guardai_errors_total)
transform p95 latency above 1 second
prompt-security trigger spikes (critical, possible token-extraction attempts)
output-guard trigger spikes (the model may be emitting PII that was not in the input)
rate-limit saturation
no transform traffic for 5 minutes (possible outage upstream)
entity-detection spikes (possible scanning)

Treat the thresholds as starting points. Tune them to your own traffic and SLOs (see SLO).

Distributed tracing

OGuardAI ships the Prometheus metrics, Grafana dashboard, alerting rules, and optional OpenTelemetry trace export. Set GUARDAI_OTEL_TRACES_ENDPOINT to your collector's OTLP over HTTP endpoint and GUARDAI_OTEL_SERVICE_NAME to override the default oguardai-server service name. When the endpoint is absent, trace export is off. Spans and attached audit events carry safe metadata, keyed identifier and policy fingerprints, and reduced entity labels, never raw detected values. Verify the pipeline against your own collector and export only to one you control.

Monitoring