User Guide
Operate
CI/CD

CI/CD

RSigma is designed to drop into a detection-as-code workflow. The CLI surfaces that matter for CI are rule lint, rule validate, rule backtest, rule coverage, engine eval, and backend convert. Each exits with a structured code that lets CI runners distinguish “no findings, clean exit” from “the tool ran but reported findings” from “the tool could not run because of a configuration or rule error.”

This page covers the exit-code model, the fixture harness (rule backtest), the failure-controlling flags (--fail-on-detection, --fail-level), and copy-paste pipelines for GitHub Actions, GitLab CI, pre-commit, and a generic shell runner.

Exit codes

Every rsigma command uses the same four-code scheme. The split is the conventional CI one shared by tools like pylint and zizmor: distinguish a tool failure from a finding, so the lint/validate steps in a pipeline can be required while the detection-match step can be advisory.

Code	Meaning	Triggered by
`0`	Success. The tool ran cleanly. With `--fail-on-detection`, no detection or correlation fired.	Default happy path. Also returned when `rule lint` produces findings below the `--fail-level` threshold.
`1`	Findings. The tool ran cleanly but found something noteworthy.	`eval --fail-on-detection` with at least one match; `rule lint --fail-level <X>` with at least one finding at or above `X`.
`2`	Rule error. The input rules could not be parsed, compiled, or converted.	`rule validate` with parse or compile errors; `backend convert` when conversion fails or every rule fails; `engine eval` and `rule lint` when the rules path itself cannot be read.
`3`	Configuration error. A pipeline file could not be loaded, a CLI argument was invalid, or the tool was misconfigured.	Bad `-p` path, unknown `-t backend`, malformed `--suppress` duration, unreadable schema file.

The exact source of truth is the exit_code module.

A few non-obvious behaviours worth pinning down:

engine eval exits 0 when a rule file contains a Sigma parse error (it logs a warning to stderr and continues with the rules that did load). Use rule validate if you want a parse error to fail the build.
engine eval exits 0 by default even when matches fire. Pass --fail-on-detection if you want detections to fail the build.
rule lint exits 0 for findings below --fail-level. The default threshold is error, so a clean lint with only info/warning findings still returns 0.
rule backtest exits 1 on a failed expectation or, under --unexpected fail, on a rule firing with no covering expectation. A bad expectations file or a missing corpus path is exit 3, and unreadable rules are exit 2.

`rule backtest` for fixture suites

rule backtest is the recommended fixture harness. It replays a corpus (a file or a directory walked recursively), tallies how many times each rule fired, and diffs those counts against an expectations file. Unlike engine eval --fail-on-detection, which is corpus-global and passes when any rule fires, backtest asserts per rule, so a broken rule cannot be masked by an unrelated rule matching the same fixture.

Declare what each rule must do in an expectations file:

# ci/expectations.yml
defaults:
  unexpected_detections: warn   # fail | warn | ignore
expectations:
  - rule: 5f0d7d3c-3aab-43fa-952f-8f7b2d966ee5   # positive fixture: must fire
    at_least: 1
  - rule: Suspicious Whoami Execution             # negative fixture: must not fire
    exactly: 0

Then run the whole suite in one command:

rsigma rule backtest -r rules/ --corpus ci/corpus/ --expectations ci/expectations.yml

Exit 1 means an expectation failed (or, under --unexpected fail, that a rule fired with no covering expectation, the false-positive signal on a benign corpus). Emit a JUnit report for CI annotation with --junit, and the full JSON report with --report:

rsigma rule backtest -r rules/ --corpus ci/corpus/ \
    --expectations ci/expectations.yml --junit backtest.xml --unexpected fail

Correlation rules are asserted the same way, by id or title. See rule backtest for the full flag table, expectations schema, and report shape.

`rule coverage` for ATT&CK gaps

rule backtest proves your rules fire correctly; rule coverage proves you have rules for the techniques you care about. It exports an ATT&CK Navigator layer and, with --fail-on-gaps, fails the build when a target technique list loses its last covering rule:

rsigma rule coverage -r rules/ --targets ci/threat-model.txt --fail-on-gaps

It can also surface techniques that have an Atomic Red Team test but no rule (--atomics) or that the SigmaHQ baseline covers but you do not (--baseline). See ATT&CK Coverage and the rule coverage reference.

`rule scorecard` for the metrics gate

rule backtest and rule coverage each emit a JSON report; rule scorecard fuses them (optionally enriched with a Prometheus production-volume snapshot and a triage feed) into per-rule keep/tune/retire verdicts and a gate. It is the metrics surface atop the triad: run it last, feeding the two reports the earlier steps wrote, and fail the build when the portfolio accrues retire-grade rules.

rsigma rule backtest -r rules/ --corpus ci/corpus/ --expectations ci/expectations.yml --report backtest.json
rsigma rule coverage -r rules/ --output-format json > coverage.json
rsigma rule scorecard --backtest backtest.json --coverage coverage.json --fail-on retire --report scorecard.md

A retire candidate that is the sole rule covering an ATT&CK technique is downgraded to tune, so the gate never asks you to drop coverage. See Detection Scorecard and the rule scorecard reference.

`rule hygiene` for the retirement cadence

rule scorecard needs the backtest and coverage reports; rule hygiene runs on the rules alone and flags retirement and clean-up candidates: untagged, unowned, undocumented (incomplete ADS), and deprecated/stale rules with no extra inputs, plus silence and noise when you add a Prometheus scrape and broken field coverage when you add a field-observability snapshot. Gate on the conditions your program treats as blocking:

rsigma rule hygiene -r rules/ --metrics metrics.txt \
    --silent-threshold 365d --fail-on silent --fail-on no-owner

It exits 1 when a selected --fail-on condition matches, the same gating model as the rest of the triad. See Rule Hygiene and the rule hygiene reference.

`--fail-on-detection` for `engine eval`

engine eval --fail-on-detection is the zero-setup fallback when you do not want an expectations file: a single fixture and a single rule, where exit 1 means “something fired.”

rsigma engine eval -r rules/ --fail-on-detection -e @ci/should-not-match.ndjson

In a “should match” fixture, you actually want exit 1:

if rsigma engine eval -r rules/ --fail-on-detection -e @ci/should-match.ndjson; then
    echo "ERROR: rule did not fire on the positive fixture"
    exit 1
fi

Because this check is corpus-global, prefer rule backtest once a fixture exercises more than one rule. Pair --fail-on-detection with --no-detections if you only care whether correlations fire:

rsigma engine eval -r rules/ --fail-on-detection --no-detections \
    --correlation-event-mode none < events.ndjson

`--fail-level` for `rule lint`

rule lint uses a tier system. The default threshold is error, meaning info, warning, and hint findings never fail the build:

rsigma rule lint rules/                        # exit 1 only on error findings
rsigma rule lint rules/ --fail-level warning   # exit 1 on warning or error
rsigma rule lint rules/ --fail-level info      # exit 1 on info, warning, or error

For shared repositories, --fail-level warning strikes the right balance: spec violations break the build, missing-author or missing-description findings don’t. For SigmaHQ-style strict contributions, --fail-level info is reasonable.

`rule doc --fail-on-missing` for the ADS gate

To require every status: stable rule to carry a full detection-strategy document, add an ads: block to .rsigma-lint.yml and run rule doc as its own gate:

rsigma rule doc rules/ --fail-on-missing

It exits 1 when any enforced rule is missing a required ADS section, and 0 otherwise. The same sections also surface through rule lint (as ads_missing_* findings) if you prefer a single lint step; rule doc is the standalone view, and --missing-only narrows the report to just the rules below the bar.

The two gates agree on absent sections. They differ only on a section whose key is present but blank: rule doc --fail-on-missing treats it as undocumented and fails, while rule lint reports the lower-severity ads_empty_section (info) and so passes at the default --fail-level error. Run rule lint --fail-level info if you want the lint step to fail on blank sections too.

`rule validate` in CI

rule validate is the cheapest gate: it just parses and compiles every rule, no events involved. Wire it as the first step of any detection-as-code pipeline:

rsigma rule validate rules/

Exit 2 means a parse or compile error somewhere; the stdout summary shows the counts:

Parsed 0 documents from rules/
  Detection rules:   0
  Correlation rules: 0
  Filter rules:      0
  Parse errors:      1
  Compiled OK:       0
  Compile errors:    0

Add -p <pipeline.yml> to validate that your processing pipelines apply cleanly too:

rsigma rule validate rules/ -p pipelines/ecs.yml

For dynamic pipelines, add --resolve-sources so CI also exercises the HTTP/file/command sources at validation time. The job fails with exit 3 if any source is unreachable:

rsigma rule validate rules/ -p pipelines/dynamic.yml --resolve-sources

See Linting Rules and Processing Pipelines for the deeper context on each gate.

GitHub Actions

The fastest path on GitHub is the timescale/rsigma-action composite action, which runs the whole gate in one step. The manual multi-job workflow further down is the equivalent without a third-party action, and the pattern to copy for the other CI systems on this page.

Use the rsigma-action

name: detections

on:
  pull_request:

permissions:
  contents: read
  pull-requests: write # the sticky summary comment

jobs:
  detection-as-code:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
        with:
          persist-credentials: false
          fetch-depth: 0 # required for the merge-base fields-drift diff
      - uses: timescale/rsigma-action@v1
        with:
          version: v0.18.0
          rules: rules/
          corpus: ci/corpus/
          expectations: ci/expectations.yml
          coverage: "true"
          coverage-targets: ci/threat-model.txt

The action installs a checksum- and SLSA-attestation-verified rsigma release (cached per version and target, no insecure fallback), then runs rule lint (PR diff annotations generated from the stable --output-format json envelope, not text-scraping problem matchers), rule validate --resolve-sources, a merge-base fields-drift diff, rule backtest, and rule coverage, and keeps a single sticky summary comment up to date. It needs pull-requests: write for the comment and fetch-depth: 0 on checkout for the fields-drift diff. Pin a concrete version: so a silent rsigma upgrade cannot change CI behaviour between runs; the minimum supported version is the release where rule backtest and rule coverage shipped. See the action repository for the full input and output reference; hardened consumers can pin the action by commit SHA instead of the @v1 major tag.

Manual workflow

A four-job workflow that mirrors a typical detection-engineering loop: lint, validate, backtest, then convert and publish.

name: detections

on:
  push:
    branches: [main]
  pull_request:

# Least-privilege default; jobs that need more (e.g. uploading artifacts) opt in.
permissions: {}

env:
  RSIGMA_VERSION: "0.18.0"

jobs:
  lint:
    runs-on: ubuntu-latest
    permissions:
      contents: read
    steps:
      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
        with:
          persist-credentials: false
      - run: cargo install --locked rsigma --version "${RSIGMA_VERSION}"
      - run: rsigma rule lint rules/ --fail-level warning

  validate:
    runs-on: ubuntu-latest
    permissions:
      contents: read
    steps:
      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
        with:
          persist-credentials: false
      - run: cargo install --locked rsigma --version "${RSIGMA_VERSION}"
      - run: rsigma rule validate rules/ -p pipelines/ecs.yml

  backtest:
    runs-on: ubuntu-latest
    needs: [lint, validate]
    permissions:
      contents: read
    steps:
      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
        with:
          persist-credentials: false
      - run: cargo install --locked rsigma --version "${RSIGMA_VERSION}"
      - name: Backtest fixtures against expectations
        run: |
          rsigma rule backtest -r rules/ \
            --corpus ci/corpus/ \
            --expectations ci/expectations.yml \
            --unexpected fail \
            --junit backtest.xml
      - if: always()
        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
        with:
          name: backtest-junit
          path: backtest.xml

  convert:
    runs-on: ubuntu-latest
    needs: [validate]
    if: github.ref == 'refs/heads/main'
    permissions:
      contents: read
    steps:
      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
        with:
          persist-credentials: false
      - run: cargo install --locked rsigma --version "${RSIGMA_VERSION}"
      - run: rsigma backend convert rules/ -t postgres -f view -p pipelines/ecs.yml -o views.sql
      - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
        with:
          name: postgres-views
          path: views.sql

For a faster CI loop, install from the precompiled archives instead of cargo install. The release page publishes one archive per supported Rust target (see Installation for the full list):

      - name: Install rsigma
        run: |
          curl -fsSL -o rsigma.tar.gz \
            "https://github.com/timescale/rsigma/releases/download/v${RSIGMA_VERSION}/rsigma-x86_64-unknown-linux-gnu.tar.gz"
          tar -xzf rsigma.tar.gz
          sudo install -m 0755 rsigma /usr/local/bin/rsigma
          rsigma --version

RSIGMA_VERSION is taken from the workflow-level env: shown above; pin it to a released tag so a silent rsigma upgrade cannot change CI behaviour between runs.

Audit your detection workflow with zizmor

Detection-as-code repositories should hold themselves to the same supply-chain hygiene they expect from the rest of the org. Run zizmor against .github/workflows/ to catch missing permissions:, unpinned actions, script-injection-prone GitHub-context interpolations, and other GHA pitfalls. RSigma’s own workflows pass zizmor --pedantic with zero findings; the zizmor.yml workflow is a small reference to copy.

GitLab CI

stages: [check, eval, build]

variables:
  RSIGMA_VERSION: "0.18.0"

default:
  image: debian:bookworm-slim

.install-rsigma: &install-rsigma
  before_script:
    - apt-get update && apt-get install -y --no-install-recommends curl ca-certificates
    - curl -fsSL -o rsigma.tar.gz "https://github.com/timescale/rsigma/releases/download/v${RSIGMA_VERSION}/rsigma-x86_64-unknown-linux-gnu.tar.gz"
    - tar -xzf rsigma.tar.gz
    - install -m 0755 rsigma /usr/local/bin/rsigma

lint:
  stage: check
  <<: *install-rsigma
  script:
    - rsigma rule lint rules/ --fail-level warning

validate:
  stage: check
  <<: *install-rsigma
  script:
    - rsigma rule validate rules/ -p pipelines/ecs.yml

backtest:
  stage: eval
  <<: *install-rsigma
  script:
    - rsigma rule backtest -r rules/ --corpus ci/corpus/ --expectations ci/expectations.yml --unexpected fail --junit backtest.xml
  artifacts:
    when: always
    reports:
      junit: backtest.xml

convert-postgres:
  stage: build
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  <<: *install-rsigma
  script:
    - rsigma backend convert rules/ -t postgres -f view -p pipelines/ecs.yml -o views.sql
  artifacts:
    paths: [views.sql]

The only: [main] keyword is deprecated in modern GitLab CI; the rules: form above is the supported replacement and works on both gitlab.com and self-managed instances at 15.x+.

Pre-commit hook

A .pre-commit-config.yaml that runs the linter and validator on staged .yml rules:

repos:
  - repo: local
    hooks:
      - id: rsigma-lint
        name: rsigma rule lint
        entry: rsigma rule lint rules/
        language: system
        pass_filenames: false
        files: '^rules/.*\.ya?ml$'

      - id: rsigma-validate
        name: rsigma rule validate
        entry: rsigma rule validate rules/ -p pipelines/ecs.yml
        language: system
        pass_filenames: false
        files: '^(rules|pipelines)/.*\.ya?ml$'

rsigma rule lint and rsigma rule validate take a single <PATH> argument (a file or a directory), so pass_filenames: false is required on both hooks. The files: glob still scopes pre-commit’s trigger to the rule and pipeline directories, while the hook itself lints or validates the whole tree.

For auto-fix on commit, run --fix then check for diffs:

rsigma rule lint rules/ --fix
git diff --exit-code rules/

Exit code 1 from git diff --exit-code means an auto-fix changed a file that has not been committed yet; the commit hook should fail and ask the developer to re-stage.

Generic shell pipeline

For environments without a managed CI system (cron jobs, scheduled detection regression checks, Concourse, Drone, Argo Workflows):

#!/usr/bin/env bash
set -euo pipefail

RSIGMA_BIN="${RSIGMA_BIN:-rsigma}"
RULES_DIR="${RULES_DIR:-rules/}"
PIPELINE="${PIPELINE:-pipelines/ecs.yml}"

$RSIGMA_BIN rule lint    "$RULES_DIR" --fail-level warning
$RSIGMA_BIN rule validate "$RULES_DIR" -p "$PIPELINE"

$RSIGMA_BIN rule backtest -r "$RULES_DIR" -p "$PIPELINE" \
    --corpus ci/corpus/ \
    --expectations ci/expectations.yml \
    --unexpected fail

set -e plus set -o pipefail makes any non-zero exit fail the script, so the structured exit codes work without explicit if branches: a failed expectation or an unexpected fire returns exit 1 and stops the run.

Tips and gotchas

Cache the rsigma binary between CI runs. The cargo install form compiles rsigma from source and typically takes several minutes on a GitHub-hosted runner; the precompiled archive download completes in under 5 seconds. The release page ships archives for x86_64-unknown-linux-gnu, aarch64-unknown-linux-gnu, x86_64-apple-darwin, aarch64-apple-darwin, x86_64-pc-windows-msvc, and aarch64-pc-windows-msvc.
Pin the rsigma version in CI. Detection-as-code repos test specific behaviour; a silent rsigma upgrade can flip a previously-fixed bug. Use cargo install --locked rsigma --version 0.18.0 or pin the precompiled archive URL.
Separate lint and validate jobs. They fail for different reasons. A combined job hides which check broke.
Avoid set +e around rsigma. Structured exit codes are the API. Wrapping commands in || true or set +e defeats the whole model.
JSON output for diagnostic logs. Pass --log-format json so CI log aggregators (Datadog CI Visibility, Buildkite test analytics) can parse run metadata without regex. Stdout/stderr are unchanged; only structured diagnostic logs flip to JSON. See Observability.

CI/CD

Exit codes

`rule backtest` for fixture suites

`rule coverage` for ATT&CK gaps

`rule scorecard` for the metrics gate

`rule hygiene` for the retirement cadence

`--fail-on-detection` for `engine eval`

`--fail-level` for `rule lint`

`rule doc --fail-on-missing` for the ADS gate

`rule validate` in CI

GitHub Actions

Use the rsigma-action

Manual workflow

GitLab CI

Pre-commit hook

Generic shell pipeline

Tips and gotchas

See also

On This Page

CI/CD

Exit codes

rule backtest for fixture suites

rule coverage for ATT&CK gaps

rule scorecard for the metrics gate

rule hygiene for the retirement cadence

--fail-on-detection for engine eval

--fail-level for rule lint

rule doc --fail-on-missing for the ADS gate

rule validate in CI

GitHub Actions

Use the rsigma-action

Manual workflow

GitLab CI

Pre-commit hook

Generic shell pipeline

Tips and gotchas

See also

On This Page

`rule backtest` for fixture suites

`rule coverage` for ATT&CK gaps

`rule scorecard` for the metrics gate

`rule hygiene` for the retirement cadence

`--fail-on-detection` for `engine eval`

`--fail-level` for `rule lint`

`rule doc --fail-on-missing` for the ADS gate

`rule validate` in CI