Static Quality Assessment of Sigma Detection Rules: Framework and Empirical Evaluation

Abstract

The Sigma rule format has emerged as a widely used open interchange format for SIEMportable detection content, with the public SigmaHQ corpus exceeding 3,000 actively-curated rules. Yet rule quality-whether a rule is correct, maintainable, operationally useful, and free of common false-positive patterns-is assessed primarily by manual review. We present a multi-dimensional framework for static detection-rule quality assessment, implemented in the open-source tool sigmalint, and evaluate it against the full public SigmaHQ corpus (3,132 rules at commit 994da16). The framework separates validity (a binary specconformance gate) from quality (a weighted score across six dimensions: ATT&CK alignment, taxonomy correctness, false-positive risk, metadata completeness, redundancy, and style). Scoring is size-invariant across dimensions of different rule-counts and deterministic given pinned reference data. Our study finds: every SigmaHQ rule passes the validity gate; corpus mean static-quality score is 99.18 / 100; two rule IDs (META004 and FP003) account for 89.8% of all findings, with a third (FP004) pushing the top-3 total to 93.2%; the false-positive-risk dimension provides the strongest discrimination within this curated corpus. A seeded-defect benchmark with nine mutation operators provides controlled recall evidence for the implemented checks across nine injected defect families (0.993 mean target-rule recall across 450 in-scope mutations; low collateral firing). We also disclose a measurement-bias correction we discovered during the empirical pass-a 99.3% false-positive rate in one of our own rules (STY003)-as a methodology check. 

The framework's contribution is structural: the validity/quality split, consumer-parameterized profiles, and dimension-rule-count-invariant scoring. We release the specific dimensions, weights, and heuristics as a starting point for community calibration.

Keywords: Sigma Rules, Detection Engineering, Detection-as-code, SIEM, Security Operations, Static Analysis, Rule Quality, Cyber Threat Detection, False-positive Risk, MITRE ATT&CK, Pysigma, Cybersecurity Measurement, Reproducible Evaluation, Seeded-defect Benchmark, Security Analytics