Prometheus Metrics

The engine daemon exposes Prometheus metrics on GET /metrics on the same --api-addr as the REST API. The full definition catalogue under --all-features (which is how the prebuilt release archives and the GHCR Docker image are built) is 45 metric names across nine concerns: 40 are always registered, and the OTLP (3) and TLS (2) families are feature-gated on daemon-otlp and daemon-tls respectively. The runtime exposes the ones that have ever fired in a given process. The three field-observer surfaces always render their # HELP/# TYPE lines (and stay at zero unless --observe-fields was passed); the others follow the lazy-registration pattern documented per section below.

The exact source of truth is the daemon/metrics module.

Engine core (18 metrics)

These always show up. They cover ingest, matches, queue depth, back-pressure, reloads, and resource usage.

Per-rule labels (2 metrics)

These counters carry labels that identify which rule fired. They surface on /metrics only after the first match for that kind.

rule_title is not guaranteed to be unique in a rule set. If two rules share a title, their counters add together. For collision-free per-rule analytics, scrape rsigma_detection_matches_total and join against your detection NDJSON stream by rule_id outside Prometheus.

These two families feed the silence and noisy signals of rule hygiene (and the production-volume column of rule scorecard): pass a saved scrape or a live endpoint as --metrics and they join per-rule by rule_title.

Dynamic pipeline sources (5 metrics)

Exposed when one or more pipelines declare dynamic sources. The labelled counters surface after the first resolve attempt for the relevant source; source_cache_hits_total and source_resolve_seconds are global (no source_id label).

The error_kind values come from rsigma_runtime::sources::SourceErrorKind. Fetch covers HTTP / file / command / NATS connect-or-read failures (per-protocol details land in the error_message log field, not the label). ResourceLimit covers the 10 MiB body cap, 30 s command exec cap, and similar.

Enrichment (6 metrics)

Exposed when the daemon is built with daemon and --enrichers is passed. Every (enricher_id, kind, status) triple and every HTTP-cache enricher_id row is pre-registered at startup, so all six families render with their # HELP / # TYPE lines and zeroed counters on the first scrape, even before any event has fired. Filtered (kind- or scope-mismatched) enricher calls do not increment any counter, so cardinality stays bounded by the number of configured enrichers.

The kind label is carried even though enricher_id typically already encodes it (asset_lookup_det vs asset_lookup_corr), so dashboards can compute sum by (kind) without depending on a naming convention.

Alert pipeline (13 metrics)

Exposed when the daemon is built with daemon. The fixed label sets on rsigma_dedup_results_total, rsigma_incidents_emitted_total, and rsigma_incident_overmerge_total, plus the rsigma_dedup_store_entries, rsigma_incidents_open, rsigma_silences_active, and rsigma_inhibit_sources_active gauges, are pre-registered at startup, so they render with their # HELP / # TYPE lines and zeroed series on the first scrape, even before --alert-pipeline is passed or any event fires. The rsigma_inhibited_total{rule} series appears once a rule first inhibits. See the Alert Pipeline guide.

Risk-based alerting (9 metrics)

Exposed when the daemon is built with daemon. The fixed label set on rsigma_risk_annotations_total and rsigma_risk_incidents_emitted_total, plus the rsigma_risk_entities_open and rsigma_risk_state_entries gauges, are pre-registered at startup, so they render with their # HELP / # TYPE lines and zeroed series on the first scrape, even before --risk is passed or any event fires. See the Risk-Based Alerting guide.

OTLP (3 metrics)

Exposed when the daemon is built with daemon-otlp and an OTLP receiver is active. The labelled counters surface after the first request of that kind.

TLS (2 metrics)

Exposed when the daemon is built with daemon-tls. Both metrics render with their # HELP and # TYPE lines as soon as TLS is configured, even before the first handshake.

Field observability (3 metrics)

Exposed unconditionally; values stay at zero unless the daemon was started with --observe-fields. All three refresh on every /metrics scrape and after every successful /api/v1/fields/* call. See HTTP API: Field observability for the matching endpoints.

Schema observability (4 metrics)

Exposed unconditionally; values stay at zero unless the daemon was started with --observe-schemas (or --discover-schemas, which implies it). All refresh on every /metrics scrape and on every GET /api/v1/schemas call. See HTTP API: Schema observability for the matching endpoint.

Logsource-aware evaluation (4 metrics)

Exposed unconditionally; values stay at zero unless the daemon was started with --logsource-routing. All refresh on every /metrics scrape. See Logsource-Aware Evaluation.

Live event tap (4 metrics)

Exposed unconditionally; values stay at zero unless the tap is enabled (daemon.tap.enabled: true) and an operator opens a session. See HTTP API: Live event tap and rsigma engine tap.

Live detection tail (2 metrics)

Exposed unconditionally; values stay at zero unless the tail is enabled (daemon.tail.enabled: true) and an operator opens a session. See HTTP API: Live detection tail and rsigma engine tail.

Triage feedback loop (4 metrics)

Exposed when the triage feedback loop is enabled (daemon.dispositions.enabled: true or --enable-dispositions). The ingest counters pre-register their fixed label sets so they render with zeroed series on the first scrape; rsigma_rule_false_positive_ratio is absent for a rule until it reaches daemon.dispositions.min_sample. See the Triage Feedback Loop guide and HTTP API: Dispositions.

Scrape configuration

Minimum Prometheus scrape config:

scrape_configs:
  - job_name: rsigma
    scrape_interval: 15s
    static_configs:
      - targets: ['rsigma.internal:9090']

15-30 s intervals are reasonable. The histograms use the default prometheus bucket boundaries; alert on the _bucket{le="..."} quantiles you care about rather than the average, which becomes meaningless under bimodal latency.

Useful alerts

groups:
  - name: rsigma
    rules:
      # Engine cannot keep up.
      - alert: RsigmaBackPressure
        expr: rate(rsigma_back_pressure_events_total[5m]) > 0
        for: 10m
        labels: {severity: warning}

      # Correlation state above 80% of the default 100000 cap.
      - alert: RsigmaCorrelationStatePressure
        expr: rsigma_correlation_state_entries > 80000
        for: 10m
        labels: {severity: warning}

      # DLQ taking traffic.
      - alert: RsigmaDlqVolume
        expr: rate(rsigma_dlq_events_total[5m]) > 1
        for: 15m
        labels: {severity: warning}

      # Reloads failing means rules are broken on disk.
      - alert: RsigmaReloadsFailing
        expr: rate(rsigma_reloads_failed_total[5m]) > 0
        for: 10m
        labels: {severity: critical}

      # Sustained API auth failures: a misconfigured client or a probe.
      - alert: RsigmaApiAuthFailures
        expr: sum(rate(rsigma_api_auth_failures_total[5m])) > 1
        for: 10m
        labels: {severity: warning}

      # Dynamic source went stale (no successful resolve in 10 minutes).
      - alert: RsigmaSourceStale
        expr: time() - rsigma_source_last_resolved_timestamp > 600
        for: 5m
        labels: {severity: warning}

      # Enricher consistently failing (timeouts or fetch errors).
      - alert: RsigmaEnrichmentFailing
        expr: |
          sum by (enricher_id) (
            rate(rsigma_enrichment_total{status=~"error|timeout"}[5m])
          ) > 1
        for: 10m
        labels: {severity: warning}

      # TLS certificate expires within 14 days.
      - alert: RsigmaTlsCertExpiring
        expr: rsigma_tls_certificate_expiry_seconds < 14 * 86400
        for: 5m
        labels: {severity: warning}

      # TLS certificate has already expired.
      - alert: RsigmaTlsCertExpired
        expr: rsigma_tls_certificate_expiry_seconds < 0
        for: 1m
        labels: {severity: critical}

Histograms: bucket guidance

event_processing_seconds p99 above 5 ms is usually a sign of misuse (regex-heavy rules without --cross-rule-ac, or many |all modifiers).

See also

• Observability for the broader observability story, including the tracing event targets that complement these metrics.
• Performance Tuning for which metric to watch when sizing --buffer-size, --batch-size, or correlation max_state_entries.
• Streaming Detection for how the /metrics endpoint fits into the broader daemon API.
• daemon/metrics source for the registry implementation.