Performance Tuning

How fast RSigma evaluates depends on how much of your corpus the candidate index can prune. The index works from rule witnesses: necessary conditions, at least one of which must hold on any event the rule can match. Exact field values, substring needles, keywords, mandatory regex literals, and bare field presence all yield witnesses, so most of a real corpus is prunable and an event is only evaluated against the rules it could plausibly match. Rules whose conditions admit no witness, in practice the ones whose only required branch is negated, are evaluated against every event and cost time linear in their count. For each surviving rule, condition evaluation first computes only the boolean verdict, allowing any and threshold selectors to stop once decided; matched-selection details are collected only when the rule matches.

On the full SigmaHQ ruleset (~3,100 rules) expect roughly 19-66k events/s per core for offline engine eval, rising to 30-106k with --logsource-routing, and 88-287k end-to-end through the daemon (114-401k with routing) on a 12-core machine. Where a workload lands in those ranges depends on event shape: wide structured events are the slow end, narrowly tagged syslog the fast end. The SigmaHQ corpus baseline in Benchmarks documents the full matrix with before-and-after numbers. Small or exact-match-heavy rulesets are considerably faster still.

This page covers the cases where the defaults stop being optimal: very large rule sets, substring-heavy threat-intel feeds, high-throughput daemon ingestion, and memory-constrained deployments. For SigmaHQ-scale corpora the single biggest lever is --logsource-routing, which is also a correctness filter because it stops cross-product keyword false positives. --cross-rule-ac is no longer worth enabling alongside it: the candidate index already does that substring work over a smaller rule population, and layering the cross-rule automaton on top of routing costs 20-35% on every measured lane, offline and end-to-end. The daemon now batches up to 128 events by default, which gives Engine::evaluate_batch enough work to fan detection across rayon without requiring manual tuning. The bloom pre-filter (--bloom-prefilter) is off by default for a reason and should be benchmarked before flipping it on.

Always-on: the matcher optimizer

Three rewrites run at rule-compile time, transparently. There is no flag to disable or configure them.

Threshold choices come from a Criterion sweep documented in the Benchmarks page (8 patterns is where the sequential str::contains path with memchr/Two-Way SIMD acceleration loses to Aho-Corasick on typical haystacks). The compiler invariant is that these are pure rewrites: the optimized tree returns the same bool for the same event as the unoptimized tree.

Because the optimizer is part of compilation, a rule reload picks up any new pattern groupings automatically.

Rule loading at scale

Loading a large rule corpus is no longer the bottleneck it was in v0.11.x. As of v0.12.0, Engine::add_rule and Engine::add_compiled_rule are amortized O(1) per call, and the bulk loaders (Engine::add_rules, Engine::extend_compiled_rules, Engine::add_collection) rebuild the candidate index and the per-field bloom filter exactly once per batch instead of once per rule.

Concrete numbers from the rule_load Criterion group on an Apple M4 Pro (release build):

Reproduce with cargo bench -p rsigma-eval --bench eval -- rule_load. Loading the pinned SigmaHQ corpus (~3,132 rules) costs about 0.3 s in the representative harness.

How the bloom rebuild is amortized. The per-field bloom uses a doubling watermark with a 64-rule floor. A full bloom rebuild only fires when the rule count has at least doubled past the last rebuild, capping false-positive-rate drift while keeping the amortized per-rule cost flat. Rules that introduce a brand-new indexed field get a fresh bloom on the fly. The differential test append_rule_matches_build_verdicts pins the property that incremental and batched indexes accept the same haystacks (with the documented MaybeMatch tolerance between rebuilds).

Caveat: the cross-rule Aho-Corasick index falls back to a full rebuild on add_rule. The daachorse automaton has no incremental update story, so if you enable --cross-rule-ac and then call add_rule in a loop, each call rebuilds the cross-rule AC. The batched loaders (add_collection, add_rules, extend_compiled_rules) keep the single-rebuild-at-end fast path. For very large rule sets with cross-rule AC on, always batch.

Bloom pre-filter for substring-heavy rule sets

The bloom pre-filter is the right knob when:

• Most events do NOT match any rule.
• Your rules are dominated by positive substring needles (|contains, |startswith, |endswith, IOC lists).
• Per-event latency matters more than rule-load latency.

When enabled, the engine builds a per-field bloom filter at rule-load time over every positive substring needle. At eval time, Engine::evaluate short-circuits any positive substring detection item whose field value cannot possibly contain a needle trigram, skipping the matcher entirely.

rsigma engine eval -r rules/ --bloom-prefilter -e @events.ndjson
rsigma engine daemon -r rules/ --bloom-prefilter --bloom-max-bytes 2097152

The trigram probe costs roughly 1 µs on a typical CommandLine field. On rule sets where most events overlap with at least one needle, that 1 µs is pure overhead and the bloom hurts throughput.

Always benchmark first. The eval_bloom_rejection Criterion group in crates/rsigma-eval/benches/eval.rs reports throughput with the bloom on and off on synthetic data; clone the corpus shape onto your own events before deciding.

Cross-rule Aho-Corasick pre-filter

The cross-rule AC index is a feature-gated opt-in for very large rule sets (~5000+) dominated by shared substring patterns. Threat-intel feeds and IOC packs are the canonical case.

When enabled, the engine builds one per-field DoubleArrayAhoCorasick automaton (daachorse) over every rule’s positive substring needles. At eval time, the engine scans each indexed field once with that automaton and drops the AC-prunable rules from the candidate set when zero needles hit the event.

A rule is AC-prunable when:

1. It has at least one positive substring detection item.
2. Every detection consists exclusively of positive substring matchers (Contains, StartsWith, EndsWith, AhoCorasickSet, possibly nested under AnyOf/AllOf/CaseInsensitiveGroup).
3. The condition expression contains no not.

Rules with Exact, Regex, Numeric, or Cidr matchers, or with not selectors in their conditions, are kept in the candidate set unfiltered.

Building with the feature

The flag is feature-gated. The default cargo install rsigma does NOT include it. Build or install with daachorse-index:

cargo install --locked rsigma --features daachorse-index

The released archives (x86_64-unknown-linux-gnu.tar.gz and friends) and the GHCR Docker image are built with --all-features, so they already include the flag. Run rsigma engine eval --help | grep cross-rule-ac to confirm.

rsigma engine eval -r rules/ --cross-rule-ac -e @events.ndjson
rsigma engine daemon -r rules/ --cross-rule-ac

When it pays off

The published benchmark (eval_cross_rule_ac group, 200 non-matching events against pure-substring rules) shows the best-case win:

Those numbers predate witness-based candidate indexing, which closed most of the gap the AC pass existed to close. On the measured SigmaHQ lanes, --cross-rule-ac now loses to --logsource-routing alone on every lane, and adding it on top of routing costs 27-35%. Treat it as a knob to benchmark on pure-substring corpora at 5,000+ rules rather than one to enable by default, and measure it against routing rather than against the bare default.

The pattern-count cap per field is 100,000; rules referencing fields above that cap are kept unfiltered.

Daemon throughput knobs

These knobs only matter for the streaming daemon, not for engine eval.

A typical high-throughput configuration:

RAYON_NUM_THREADS=8 \
RSIGMA_DETECT_INFLIGHT=5 \
RUST_LOG=warn \
rsigma engine daemon \
    --rules rules/ \
    --input http \
    --api-addr 127.0.0.1:19090 \
    --logsource-routing \
    --batch-size 512 \
    --buffer-size 50000 \
    --output 'file:///var/lib/rsigma/detections.ndjson'

The daemon does not wait for a batch to fill: it blocks for the first event, then drains up to --batch-size events already waiting in the queue. A larger value therefore increases the maximum processing quantum rather than adding a batch-fill timer. Lower it when short bursts need the smallest tail latency, or raise it when sustained load and a large corpus justify more parallel work per lock acquisition. The effective value never exceeds --buffer-size; the published SigmaHQ baseline found 512 best on sustained load.

Formatted input parsing fans across the same rayon pool before engine evaluation. When there are no correlation rules, several batches may be in flight so one batch’s serial merge can overlap another’s rayon work; set RSIGMA_DETECT_INFLIGHT=1 to force the older single-batch path. Correlation engines stay one batch at a time. Raising RAYON_NUM_THREADS beyond the physical performance-core count should not be expected to scale linearly. Use rsigma_batch_phase_duration_seconds (parse, decode_merge, observe, evaluate, result_merge, dispatch) to rank where batch time goes on your workload.

--include-event clones the complete input event into every detection result. Its cost scales with matches rather than inputs: the raw Windows baseline produces about 3 matches per event without routing and about 0.02 with routing, so measure this option on the actual match volume. The checked-in daemon matrix includes both forms, a match-heavy lane, and a handcrafted lane covering event-count, value-count, value-sum, and ordered-temporal correlations; the pinned SigmaHQ tree itself contains no correlation rules.

Walkthrough: maximize sustained detection throughput

This recipe targets a detection-only daemon under sustained load. Correlation rules force one batch in flight because their state updates are ordered, while enrichment, serialization, and durable sinks can become the bottleneck on match-heavy workloads.

1. Use a release artifact. Native glibc builds and the released static musl image both scale correctly; the musl artifact selects jemalloc automatically. Debug builds are not representative.
2. Size rayon to the performance cores. Start with one rayon worker per physical performance core, such as RAYON_NUM_THREADS=8 on an eight-core machine. More workers can help on SMT-only servers, but benchmark them rather than assuming logical CPU count is optimal.
3. Enable logsource routing. --logsource-routing is both a performance optimization and a correctness filter when events carry product, service, or category. Configure --event-logsource or --logsource-field-map when those hints do not use the default event fields.
4. Keep enough work ready. Use --batch-size 512 for sustained SigmaHQ-shaped traffic and a larger bounded queue such as --buffer-size 50000. The default batch size of 128 remains the latency-conscious general default.
5. Keep five detection batches in flight at eight or more workers. Five is already the default at that size; setting RSIGMA_DETECT_INFLIGHT=5 makes a benchmark reproducible. Do not raise it for correlation rules, and retune before carrying five to a substantially different worker count or workload.
6. Avoid optional work you do not need. Keep --include-event, --bloom-prefilter, and --cross-rule-ac off unless workload-specific measurements justify them. In particular, adding --cross-rule-ac to routing reduced measured SigmaHQ throughput by 27-35%.
7. Drive the daemon hard enough to fill batches. A checkout can reproduce the sustained HTTP lane with:

LANE=target/perf-fixtures/events/raw_windows.ndjson \
URL=http://127.0.0.1:19090/api/v1/events \
BATCH=500 \
VUS=16 \
DURATION=30s \
k6 run scripts/perf/daemon-load.js

For an engine-only ceiling, temporarily replace the production sink with --output 'file:///dev/null'. Do not use /dev/null to estimate end-to-end capacity for a durable NATS, OTLP, or file sink.

Watch rsigma_input_queue_depth, rsigma_back_pressure_events_total, process CPU, and the batch phase durations while testing. Full queues plus low CPU indicate an upstream or scheduling limit; high CPU plus rising backpressure means the engine has reached its current capacity; low engine pressure plus a growing sink queue points to delivery rather than detection.

What the tuning work improved

The improvements below are separate same-build comparisons on the pinned SigmaHQ raw Windows workload, not numbers that should be multiplied together:

• Witness-based candidate indexing raised the four representative default lanes by 13.0-22.8x over full residual evaluation and kept p95 candidate sets at 0.29-3.86% of loaded rules.
• Raising the daemon batch size from 1 to 128 moved sustained throughput from 28.6k to 157.4k events/s, a 5.5x gain.
• Allowing four correlation-free batches in flight moved the eight-worker routed lane from about 520k to 654k events/s, a 26% gain over one batch in flight while preserving sink and acknowledgment order.
• Retuning four in-flight batches to five moved the same-host median from about 665k to 708k events/s, a 6.5% gain. Native Linux artifact validation measured depth-five/depth-four ratios of 1.0201 on amd64 glibc, 0.9985 on amd64 musl, 1.0166 on arm64 glibc, and 1.0095 on arm64 musl, with lower backpressure in every row.
• The final routed operating point measured 152.8k events/s at one worker and 754.8k at eight workers, or 61.7% multicore efficiency.

Memory pressure and correlation state

Correlation state lives in memory unless --state-db writes periodic snapshots to SQLite. The hard cap is max_state_entries, default 100,000 (correlation, group-key) entries across all correlation rules, settable with --max-state-entries (or daemon.correlation.max_state_entries in the config file). When the cap is hit, the engine evicts the stalest 10% and emits a warning.

The cap bounds the number of groups, not the bytes within one. A single group’s window state grows with timespan x event rate: 8 bytes per in-window event for event_count, 16 for the numeric aggregations (value_sum/value_avg/value_percentile/value_median), and roughly 32 bytes plus the value string for value_count. --max-group-entries (or the per-rule rsigma.max_group_entries custom attribute) caps that within-window growth; when a group exceeds it, the oldest entries are dropped, which can only under-count. Session windows always keep their oldest entry as the span anchor so truncation cannot silently extend the timespan cap. Unset means unbounded, the historical behavior. The measured shape (see Benchmarks): 1M unique session keys against the default cap peaked at 39.8 MiB, and a fully chatty event_count workload (100 groups sustaining 1 event/s through a 2h session cap) held 6.3 MiB. A live but quiet session group costs ~256 bytes, dominated by the group-key strings.

Window modes (sliding/tumbling/session) have identical per-event cost; they differ only in how long entries are retained. tumbling resets per-group state at each bucket boundary and is the cheapest under sustained load; session retains everything between the first event and the timespan cap, so it is the mode to watch on chatty groups.

Two workload shapes deserve attention:

• value_count with high-cardinality values. Every (timestamp, value) pair is retained for the window, and the distinct count is recomputed per event over the whole window — O(window size) per event. At 1,800 distinct values per window, measured throughput drops from ~1.1M to 63K events/s. CPU collapses before memory does. Prefer event_count where distinctness is not actually required, shorten the window, or set --max-group-entries to bound the retained pairs.
• Group-key cardinality floods. Under cap pressure the stalest groups are evicted first, so a burst of unique keys can push a slow-burning session out of state before it completes. The eviction warning in the log is the tripwire; raise --max-state-entries if the warning fires during legitimate traffic.

Watch:

• rsigma_correlation_state_entries (Prometheus gauge) — current size.
• tracing warnings tagged rsigma_eval::correlation_engine (the correlation memory pressure span) — eviction events.

Tune by:

• Lowering the per-correlation window (timespan: 5m instead of 1h) so older state expires faster.
• Setting --max-group-entries (or rsigma.max_group_entries per-rule) to cap within-window growth on chatty groups, e.g. value_count rules over high-rate telemetry.
• Lowering --max-state-entries on memory-constrained deployments, or raising it when the eviction warning fires during legitimate high-cardinality traffic.
• Setting --max-correlation-events 5 (or via rsigma.max_correlation_events per-rule) to cap the per-window event list.
• Setting --correlation-event-mode refs to store lightweight references instead of full event bodies. refs mode keeps timestamps and event IDs only; full retains the deflate-compressed event JSON. none (the default) keeps no events.

See Evaluating Rules for the matching engine eval flags and Streaming Detection for the daemon’s SQLite snapshot path.

Benchmarking your own corpus

Criterion benchmarks live in crates/rsigma-eval/benches/eval.rs and crates/rsigma-runtime/benches/. The full numbers ship in the Benchmarks page.

Quick runs on a checkout:

# Pure eval throughput, single event
cargo bench -p rsigma-eval --bench eval -- eval_single_event

# Bloom on/off comparison
cargo bench -p rsigma-eval --bench eval -- eval_bloom_rejection

# Cross-rule AC on/off comparison
cargo bench -p rsigma-eval --features daachorse-index --bench eval -- eval_cross_rule_ac

# Streaming pipeline throughput
cargo bench -p rsigma-runtime --bench runtime_throughput

# Dynamic pipeline resolve cost (HTTP/file/command source + extract)
cargo bench -p rsigma-runtime --bench dynamic_pipelines

Replace the synthetic Criterion inputs with rules and events that mirror your own corpus. Both the bloom and cross-rule AC wins are workload-shaped: the published numbers above are the upper bound, not what you should expect on mixed data.

For an end-to-end measurement on a real corpus rather than synthetic inputs, scripts/perf/fetch-fixtures.sh pins the SigmaHQ ruleset and generates deterministic event lanes, and scripts/perf/baseline-eval.sh and scripts/perf/daemon-matrix.sh walk the flag matrix over them offline and through the daemon. Point them at your own rules and events to get the same table for your workload.

Quick decision matrix

See also

• Observability for the Prometheus metrics that surface every knob above (rsigma_input_queue_depth, rsigma_back_pressure_events_total, rsigma_correlation_state_entries, rsigma_event_processing_seconds).
• Streaming Detection for daemon-level configuration around hot-reload, state, and back-pressure.
• Logsource-Aware Evaluation for --logsource-routing as both a performance and correctness filter.
• Evaluating Rules for the corresponding engine eval flags.
• Feature Flags reference for daachorse-index, evtx, logfmt, cef, and daemon-* features.
• Benchmarks for the full Criterion results across parser, evaluator, correlation engine, runtime, and dynamic pipelines.
• rsigma-eval/README.md for the matcher optimizer, bloom, and cross-rule AC implementation notes.