rsigma-eval

rsigma-eval

The detection and correlation engine. Compiles parsed Sigma rules into a matcher tree (via the rsigma-ir HIR), evaluates events against them, runs correlation windows, and applies processing pipelines.

• docs.rs/rsigma-eval
• README
• crates.io/crates/rsigma-eval

When to use

• Run rules against events in an in-process embedding (no daemon, no I/O).
• Build a custom front-end on top of the engine (different input format, different sink shape).
• Reuse the matcher optimizer, pipeline machinery, or correlation engine in another tool.

For streaming I/O (stdin / HTTP / NATS / OTLP), source resolution, and hot-reload, layer rsigma-runtime on top.

Install

[dependencies]
rsigma-parser = "0.24.0"
rsigma-eval = "0.24.0"
serde_json = "1"   # only if you use the JsonEvent shim

Public surface

The full enum of modifiers, the matcher-optimizer constants, the rsigma.* custom-attribute table, and the bloom/cross-rule prefilters live in the crate README.

Transformation::SetState.value is a serde_json::Value, preserving numeric and boolean pipeline state for typed processing_state comparisons. Code that previously constructed it with a String should use serde_json::Value::String. Added in v0.24.0

Minimum example: detection only

use rsigma_eval::{Engine, JsonEvent};
use rsigma_parser::parse_sigma_yaml;
use serde_json::json;

let yaml = r#"
title: Whoami
id: 8b1d8c97-5b3a-4d77-9b48-7c5f7c8b1a2a
logsource: { product: windows, category: process_creation }
detection:
    selection:
        CommandLine|contains: 'whoami'
    condition: selection
level: medium
"#;

let collection = parse_sigma_yaml(yaml)?;
let mut engine = Engine::new();
engine.add_collection(&collection)?;

let event = json!({ "CommandLine": "cmd /c whoami" });
let matches = engine.evaluate(&JsonEvent::borrow(&event));

assert_eq!(matches.len(), 1);
assert_eq!(matches[0].header.rule_title, "Whoami");

With a pipeline

Pipeline applies before compilation. The CLI’s -p flag wires this up; in code:

use rsigma_eval::Engine;
use rsigma_eval::pipeline::parse_pipeline;
use rsigma_parser::parse_sigma_yaml;

let pipeline = parse_pipeline(r#"
name: ecs_windows
priority: 20
transformations:
  - id: ecs_fields
    type: field_name_mapping
    mapping:
      CommandLine: process.command_line
    rule_conditions:
      - type: logsource
        product: windows
"#)?;

let collection = parse_sigma_yaml(rule_yaml)?;

let mut engine = Engine::new();
engine.add_pipeline(pipeline);   // priority sorted; multiple allowed
engine.add_collection(&collection)?;

After this, the rule sees ECS field names; an event with process.command_line matches.

Reading the rule a pipeline produced

add_collection discards the rewritten Sigma AST and keeps only the compiled rules, so a caller that needs to act on what a pipeline decided asks for the rewrite explicitly. The common case is a collector that subscribes to log channels: rather than hardcoding a second copy of the pipeline’s category-to-channel mapping, read the logsource the pipeline rewrote.

use rsigma_eval::{Engine, transform_collection};
use rsigma_eval::pipeline::parse_pipeline;
use rsigma_parser::parse_sigma_yaml;

let pipelines = vec![parse_pipeline(&pipeline_yaml)?];
let collection = parse_sigma_yaml(&rule_yaml)?;

let mut channels = std::collections::BTreeSet::new();
for transformed in transform_collection(&pipelines, &collection)? {
    if let Some(service) = &transformed.rule.logsource.service {
        channels.insert(service.clone());
    }
    // `transformed.applied_items` lists the ids of the transformations that
    // fired, so you can tell "nothing matched" from "matched and rewrote".
}

// The originals are unchanged, so load them normally.
let mut engine = Engine::new();
for pipeline in pipelines {
    engine.add_pipeline(pipeline);
}
engine.add_collection(&collection)?;

TransformedRule also carries state, the merged PipelineState, which is where set_state and query_expression_placeholders values land.

Both functions clone and re-run the pipelines, the same work add_collection does, so they belong on the load or reload path and not on the per-event path. If only the logsource matters, Engine::rules() after loading is cheaper: each CompiledRule retains its post-pipeline logsource.

They also apply pipelines in slice order rather than by priority, matching apply_pipelines. Pass the slice through merge_pipelines when chaining several, or use Engine::transform_collection, which uses the engine’s already-sorted pipelines.

Rule tuning

tune_rule(rule, false_positives, true_positives, config) accepts one parsed, optionally pipeline-transformed SigmaRule plus JSON event values. It verifies that every label fires before filtering, profiles reusable value forms from the FP set, rejects candidates that match a TP, emits a standard filter rule, and verifies the final artifact through Engine::add_collection.

TuneConfig bounds minimum and maximum field count, OR-list cardinality, inferred token length, cluster support, and cluster count. TuneReport contains the paste-ready YAML, ranked field rationale, emitted selections, FP coverage, warnings, and closed before/after counts. TuneError::NoCleanSeparator is returned instead of an unsafe filter when the corpora cannot be separated.

Correlation

For stateful detections, use CorrelationEngine instead of the bare Engine. It owns both the rule set and the sliding-window state:

use rsigma_eval::{CorrelationConfig, CorrelationEngine, JsonEvent, ProcessResultExt};
use rsigma_parser::parse_sigma_yaml;

let collection = parse_sigma_yaml(yaml)?;

let mut correlator = CorrelationEngine::new(CorrelationConfig::default());
correlator.add_collection(&collection)?;

for raw in events {
    let evt = JsonEvent::borrow(&raw);
    let result = correlator.process_event(&evt);
    for m in result.detections() { /* detection match */ }
    for c in result.correlations() { /* correlation firing */ }
}

process_with_detections(event, Vec<EvaluationResult>, timestamp_secs) is the lower-overhead variant for hot loops (pre-compute detections in parallel, feed sequentially to correlation). CorrelationConfig enforces max_state_entries (default 100,000) and the 10-deep correlation-chain limit; see Security Hardening. A correlation whose rules: list other correlations (for example a temporal of two event_count rules) emits the parent result when the chain condition is met, the same way a temporal of detections does. The referenced child correlations update the parent’s state but are output only when a referencing correlation has generate: true or emit_detections is on. Added in v0.24.0

An EvaluationResult carries the rule id but not its name, so process_with_detections and correlate_detections match a detection from a rule without an id to its rule by title. When two such rules share a title, neither feeds the correlations that reference it by name. Give those rules unique titles, or use process_event, process_event_at, or process_batch, which evaluate detections themselves and keep each compiled rule’s id and name. Added in v0.24.0

rule_draft::correlation::draft_correlation accepts positive and negative GroupedExemplar collections, an optional flat baseline, and CorrelationDraftConfig. Each TimedEvent carries exactly one RFC3339 timestamp or Sigma offset plus a JSON event. The result contains paste-ready YAML, inferred grouping/order/window evidence, per-slot support, gap distributions, warnings, and the isolated verification matrix. Caller-supplied ids keep the library deterministic; command-line callers generate UUIDs.

Custom attributes

Pipeline transformations can write rsigma.* attributes that the engine consumes (include_event, correlation_event_mode, max_correlation_events, …). Full table in Custom Attributes.

Performance knobs

Schema classification and routing live alongside the engine: SchemaClassifier::classify and classify_with_ambiguity recognize an event’s schema from declarative SchemaSignature predicates, explain reports why, and validate_schema_config statically checks a config. SchemaRouter builds one engine per pipeline-set, routes each event to its schema’s engine (deriving the event’s logsource from the schema for pruning), feeds one shared correlation store, and reports a per-schema pruning summary via schema_pruning_summary. See the Schema Signatures reference and the Schema Routing guide.

Engine::add_rule and add_compiled_rule are amortized O(1) per call (v0.12.0+), so a control-plane that ingests rules one at a time no longer pays an O(N) cost on every push. The bulk loaders (add_rules, extend_compiled_rules, add_collection) rebuild indexes exactly once per batch. If you enable set_cross_rule_ac(true), prefer the bulk loaders since the daachorse automaton has no incremental update.

Decision matrix in Performance Tuning. Verified Criterion numbers in Benchmarks.

Error handling

EvalError from thiserror. Variants include Parser (re-exports the parser errors), InvalidRegex, InvalidCidr, IncompatibleValue (a value the modifier cannot use, such as a cidr with host bits set), InvalidModifiers, UnknownRuleRef (correlation references a rule that wasn’t added), CorrelationCycle, and Base64. Each carries enough context to point operators at the offending rule.

See also

• Architecture for how Engine and CorrelationEngine fit into the broader runtime.
• Performance Tuning for the bloom and cross-rule AC knobs.
• rsigma-runtime for the streaming-runtime layer that wraps this crate.
• Custom Attributes for the rsigma.* namespace this crate consumes.
• rsigma-eval README for the full API surface.
• docs.rs/rsigma-eval.