Testing

The workspace runs several test tiers. Most are gated in CI on every PR; coverage is advisory, and representative performance is path-selective on PRs with a fuller weekly matrix.

At a glance

Unit tests

Located inside the crate modules they test. Conventional Rust:

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn parses_minimal_rule() {
        let rule = parse_sigma_yaml(MINIMAL_YAML).unwrap();
        assert_eq!(rule.rules.len(), 1);
    }
}

Bias toward unit tests for pure-functional logic (parsers, matchers, formatters). Bias toward integration tests for end-to-end shapes (CLI invocations, daemon HTTP round-trips, dynamic source resolution).

Integration tests (in-process)

These tests link directly against the crate as a library and exercise multi-component flows without spawning the compiled binary.

Helpers (test rule fixtures, common test pipelines) live in crates/<crate>/tests/helpers/mod.rs or crates/<crate>/tests/common/mod.rs. Reuse them; do not duplicate.

Do not duplicate unit-level assertions in integration tests. Integration tests own the boundaries, the multi-component chains, and the error paths.

End-to-end tests

E2E tests cross the binary boundary or stand up real external services through containers. They are the highest-confidence layer and the longest to run.

CLI E2E (crates/rsigma-cli/tests/cli_*.rs)

There are 51 cli_*.rs files under crates/rsigma-cli/tests/. They invoke the freshly built rsigma binary via assert_cmd and exercise stdin, stdout, stderr, exit codes, and (for the daemon tests) the HTTP, NATS, and OTLP wire surface.

Categories for orientation (not a contract on per-file counts):

• Core CLI: config, convert, eval, lint, validate, parse, fields, output format.
• Daemon: core daemon plus auth, TLS, tap, tail, schemas/schema routing, risk, alert pipeline, webhook, UDS, HTTP, NATS, OTLP, enrichment, fields observer, delivery, status, correlations, audit trail, sink format, incident bundle, logsource routing.
• Detection quality / authoring: scorecard, backtest, coverage, tune, hygiene, visibility, doc, draft, discover, classify, explain, disposition recipes.
• Pipeline / sources: pipeline diff, migrate-sources, sources deprecation, dump correlation state, logsource/schema routing.

List discovered tests with cargo test -p rsigma --tests -- --list. Prefer that over memorizing counts.

The shared harness in crates/rsigma-cli/tests/common/mod.rs is the canonical reference for spawning a long-running daemon under test: it drains stdout in a background thread to prevent pipe stalls, forwards stderr lines via mpsc, probes the actual TCP socket with TcpStream::connect_timeout before returning a handle, and wraps the Child in a ChildGuard RAII type that kills it on drop. PR #115 hardened this against macOS-under-load flakes by replacing every std::thread::sleep wait with a poll_until retry loop that polls the actual observable condition (HTTP status, metric counter) every 50 ms up to a 5 s deadline. Use it for any new daemon-level test; do not roll your own.

Container E2E (NATS and Postgres via testcontainers)

Four files spin up real services in Docker containers via testcontainers. Together they cover 29 tests, all guarded by a can_run_linux_containers() probe that shells out to docker info and checks that the daemon reports a Linux OS type. If Docker is missing or only provides Windows containers, the tests print “Skipping” and return successfully.

The skip_without_docker!() macro pattern is identical in all four:

macro_rules! skip_without_docker {
    () => {
        if !can_run_linux_containers() {
            eprintln!("Skipping: Docker with Linux container support is not available");
            return;
        }
    };
}

Use the same skip_without_docker!() pattern for any new test that requires an external service via testcontainers. CI runs these on the Linux matrix entry; macOS and Windows entries skip them.

What “e2e” means here

• Goal: cross every internal boundary the binary has, so a regression in the dispatch / IO / metric / exit-code surface fails CI rather than escaping to a user.
• Scope: the compiled binary; the HTTP API; NATS JetStream wiring (via testcontainers, 21 tests across three files); the OTLP HTTP and gRPC handlers; and the PostgreSQL backend’s generated SQL (via testcontainers, 8 tests).
• Out of scope (today): LynxDB, Splunk, Elastic, and KQL backends only have golden-text coverage, not live-query e2e. Kubernetes deployment has no e2e coverage in this repo.

Golden tests

The dynamic-pipelines suite under tests/fixtures/dynamic-pipelines/ is the canonical golden-file harness:

tests/fixtures/dynamic-pipelines/
├── pipelines/                  # inputs (one *.yml per scenario)
├── source-files/               # per-scenario `--source-file` YAML (top-level sources:)
├── sources/                    # mock source bodies (HTTP, file, command output)
└── golden/                     # expected `rsigma pipeline resolve --pretty` output

The CI loop in the sigma-corpus job iterates pipelines/*.yml, runs rsigma pipeline resolve --pipeline … --source-file … --pretty, and diffs against golden/${name}.json. To run the same check locally:

cargo build --release --all-features --locked -p rsigma
for pipeline in tests/fixtures/dynamic-pipelines/pipelines/*.yml; do
  name=$(basename "$pipeline" .yml)
  golden="tests/fixtures/dynamic-pipelines/golden/${name}.json"
  source_file="tests/fixtures/dynamic-pipelines/source-files/${name}.yml"
  diff -u "$golden" <(./target/release/rsigma pipeline resolve --pipeline "$pipeline" --source-file "$source_file" --pretty) \
    || echo "FAIL: $name"
done

To regenerate a golden after an intentional behavior change:

./target/release/rsigma pipeline resolve \
    --pipeline tests/fixtures/dynamic-pipelines/pipelines/<name>.yml \
    --source-file tests/fixtures/dynamic-pipelines/source-files/<name>.yml \
    --pretty \
    > tests/fixtures/dynamic-pipelines/golden/<name>.json

Then git diff the resulting golden file; if the diff matches your intent, commit it along with the code change. Otherwise revert and investigate.

SigmaHQ corpus regression

CI clones SigmaHQ/sigma at the pinned SHA 994da16651194500b607a3007186c29779e1f961 (SIGMA_CORPUS_SHA in .github/workflows/ci.yml) and runs three checks (job sigma-corpus):

# 1. Every rule must parse and compile.
./target/release/rsigma rule validate /tmp/sigma/rules/ --verbose

# 2. The dynamic-pipelines fixtures must still resolve cleanly against
#    the live corpus, validating that the field-mapping and include
#    expansion stay compatible with rules in the wild.
./target/release/rsigma rule validate /tmp/sigma/rules/ \
    --pipeline tests/fixtures/dynamic-pipelines/pipelines/field_mapping.yml \
    --pipeline tests/fixtures/dynamic-pipelines/pipelines/allowlist.yml \
    --pipeline tests/fixtures/dynamic-pipelines/pipelines/multi_format.yml \
    --pipeline tests/fixtures/dynamic-pipelines/pipelines/extract_languages.yml \
    --pipeline tests/fixtures/dynamic-pipelines/pipelines/include_expansion.yml \
    --source tests/fixtures/dynamic-pipelines/source-files/ \
    --resolve-sources --verbose

# 3. The dynamic-pipelines goldens must match (the diff loop shown above).

A regression in any of those steps fails the PR. Locally:

cargo build --release --all-features --locked -p rsigma
mkdir -p /tmp/sigma && cd /tmp/sigma
git init -q
git remote add origin https://github.com/SigmaHQ/sigma.git
git fetch --depth 1 origin 994da16651194500b607a3007186c29779e1f961
git checkout -q FETCH_HEAD
cd - >/dev/null
./target/release/rsigma rule validate /tmp/sigma/rules/ --verbose

The Performance workflow also uses this pinned corpus for representative throughput and candidate-rate checks. Keep both corpus consumers pinned to the same SHA.

Coverage

The coverage job runs cargo llvm-cov --workspace --all-features --locked --no-report, then cargo llvm-cov report --lcov --output-path lcov.info, on Linux and uploads lcov.info. It is advisory, not gating; there are no per-crate thresholds enforced today. Drops of more than a couple of percentage points warrant a comment on the PR.

Locally a simpler one-shot form is fine:

cargo llvm-cov --workspace --all-features --locked --lcov --output-path lcov.info

Prefer --locked so the resolve matches CI.

Performance regressions

Criterion benchmarks live under crates/<crate>/benches/. Run them manually:

cargo bench -p rsigma-eval -- eval
cargo bench -p rsigma-parser -- parse
cargo bench -p rsigma-runtime -- runtime_throughput

One bench target is not a Criterion suite: correlation_memory installs a counting global allocator and reports peak/settled heap for correlation window-state stress scenarios (high-cardinality group keys, long-lived chatty sessions), which Criterion cannot measure. It prints an aligned table and finishes in about half a minute:

cargo bench -p rsigma-eval --bench correlation_memory

Criterion microbenchmarks are not gated in CI. The numbers in Benchmarks come from a manual run on the development workstation; if a PR makes a hot-path change, attach a before/after Criterion summary in the PR description.

The Criterion suites use synthetic, mostly exact-match-indexable rules, so they measure hot paths, not representative corpus throughput. For a representative before/after, materialize the pinned SigmaHQ workload with scripts/perf/fetch-fixtures.sh and run scripts/perf/baseline-eval.sh (offline eval matrix, single core, net of rule load) and scripts/perf/daemon-matrix.sh (daemon HTTP end to end across lanes and flag variants, wrapping scripts/perf/baseline-daemon.sh). Both take an RSIGMA override, so a pre-change build can be measured through the same harness for an honest before-and-after. The daemon matrix includes --include-event variants, including the match-heavy lane where event cloning matters most, and a handcrafted event-count/value-count/value-sum/temporal-ordered correlation lane because the pinned SigmaHQ tree contains no correlation rules.

The Performance workflow builds the PR and its base revision on the same GitHub-hosted runner, runs the load-corrected median-of-three offline matrix through one checked-in harness, verifies match counts, and rejects only a head/base EPS ratio below 0.5. That intentionally coarse floor catches order-of-magnitude regressions without pretending shared-runner noise can support fine-grained gating. A weekly/manual job runs five samples of the full offline matrix, reports a deterministic bootstrap 95% confidence interval, runs the daemon matrix, and retains the raw artifacts for 90 days.

Weekly/manual runs also build native glibc and static musl image binaries on dedicated eight-core amd64 and arm64 runners. scripts/perf/inflight-compare.sh alternates five samples each at detection depths 4 and 5, then rejects a depth-5 throughput ratio below 0.98 or a backpressure-rate increase above 0.008. This gate covers the allocator and architecture behavior that the native PR comparison cannot observe.

The production candidate index is measured directly, rather than inferred from the witness-audit simulation:

RSIGMA_DIFF_RULES="$PWD/target/perf-fixtures/sigma/rules" \
RSIGMA_DIFF_EVENTS="$PWD/target/perf-fixtures/events" \
    cargo test -p rsigma-eval --all-features \
      corpus_candidate_rate -- --ignored --nocapture

That ignored corpus test reports each lane’s p95 candidate count/rate and fails at 10% of loaded rules. See the SigmaHQ corpus baseline for the recorded matrix.

Tips

• Run only the failing test first. cargo test -p rsigma-runtime nats_e2e::test_replay_from_offset -- --nocapture is much faster than --workspace.
• Run feature-gated tests once with the feature off. A #[cfg(feature = "nats")] fn test_x() is silently skipped if you forget; CI catches that. Locally, cargo test --no-default-features -p rsigma-runtime is a useful smoke test.
• In-process NATS and OTLP servers are spawned by the integration tests in crates/rsigma-runtime/tests/nats_integration.rs and crates/rsigma-cli/tests/cli_daemon_otlp.rs; they do not need external infrastructure.
• Container-backed NATS e2e in crates/rsigma-runtime/tests/nats_e2e.rs needs Docker. On a Mac, colima start or Docker Desktop is the easiest local setup.
• CLI tests use assert_cmd. They invoke the compiled rsigma binary, so the first run is slow because it triggers a full build. Subsequent runs reuse the cache.

See also: Fuzzing, Benchmarks, Contributing.