1. LandOffer
  2. Blog
  3. All articles
  4. Detection Engineering Portfolio: Test a Sigma Rule Against Positive and Negative Events

Job search

Detection Engineering Portfolio: Test a Sigma Rule Against Positive and Negative Events

Publish a reproducible Sigma development sample: labeled events, an explicit pipeline, actual query results and a tuning review that retains false-negative evidence.

9 min readLandOffer team

A detection engineering portfolio should show what a Sigma rule matches, what it should ignore, and which failures remain after tuning. Publish the labeled events, rule versions, processing pipeline, generated query and actual results together. A YAML file and an ATT&CK tag identify intended logic and context; actual event-level results establish what happened in the tested sample.

This guide demonstrates a real local test over original synthetic application-audit events. On October 23, 2026, we converted Sigma YAML with pySigma and its SQLite backend, then executed the generated SQL in an in-memory database. No real system was monitored, no attack was performed, and no production telemetry or private identifiers were used. LandOffer publishes this guide.

An engineer compares marked synthetic event slips with a rule notebook and a separate review stack

Use recent roles on LandOffer.ai to identify the detection responsibilities you want to demonstrate, then make the evidence behind your work inspectable.

Define the behavior and the event contract first

The fictional application, demo_app, records administrative role changes. Our narrow hypothesis is a successful grant of the admin role that lacks the exercise’s explicitly authorized provisioning context. A failed attempt, role removal or viewer-role grant falls outside that hypothesis. These labels express the authored scenario; they are not independently established threat intelligence.

The event contract contains an opaque event ID, source stream, action, target role, result, actor label, workflow and change-ticket label. The expected source is demo.audit.v1. All actor and ticket values are invented. An event describing a successful grant does not prove that a real permission changed, and a matching event does not identify an attacker.

Five fields are required before scoring: event ID, source stream, action, target role and result. The local runner quarantines a record missing one of those values. It does not quietly label incomplete telemetry benign. Actor and context fields can be absent; their interpretation must remain visible in the query and review.

Write the expected decisions before interpreting the rule’s output. Otherwise, “positive” can become shorthand for “the rule happened to match,” which makes evaluation circular. In this project, each event carries a separate label reason based on the fictional change record, rather than borrowing the query result as its label.

Keep the Sigma rule separate from its backend mapping

Sigma’s rule documentation distinguishes the detection, log source and metadata. It also explicitly says falsepositives helps analysts interpret alerts; converters do not use that field to suppress events. An exclusion belongs in executable logic, not just an explanatory note.

Our baseline detection excerpt is:

detection:
  selection:
    Action: role_granted
    TargetRole: admin
    Result: success
  condition: selection

The full local rule is marked experimental and declares the custom application log source. This is an authored demonstration schema, not a claim that a commercial product emits those fields or that Sigma includes a ready-made integration for it.

Sigma’s processing-pipeline guide explains how transformations map fields and constrain the target data. Our pipeline maps six rule names to the fixture’s lowercase fields, including TargetRole to target_role and ChangeTicket to change_ticket. It adds source_stream = demo.audit.v1 to the generated query. Merely writing a logsource description is not our substitute for that explicit stream constraint.

The pipeline is deliberately specific to this fixture. It is not a Sysmon pipeline, a cloud connector or a deployed ingestion configuration. If your actual telemetry uses different names or semantics, an apparently successful conversion still needs verification against that schema.

Use a real converter and preserve its output

We used the official pySigma SQLite backend, whose documented API converts a Sigma collection into SQL. The tested environment was Python 3.12.14, pySigma 2.0.1, backend 2.1.1 and SQLite 3.53.4. Dependencies were installed in this article’s isolated environment, not added to another portfolio project.

The runner reads each rule through SigmaCollection, applies ProcessingPipeline, invokes sqliteBackend and executes that backend’s output. It does not replace Sigma with a hand-written Python predicate and call the result a Sigma test. Generated SQL, package versions and source hashes are retained with the event files.

Two actual checks show why the conversion boundary matters. Without our field-mapping pipeline, the generated query failed against the fixture table with “no such column: TargetRole.” Changing only the falsepositives explanation, while preserving rule order, produced the same baseline SQL. The note changed the analyst explanation while the generated baseline query remained identical.

The SQLite table declares text columns with ASCII case-insensitive collation. That is part of the tested contract, not evidence that every SIEM handles case, nulls, escaping or nested fields identically. The backend documents limitations for those areas. This experiment does not test regular expressions, correlations, Unicode matching or production performance.

Completed asset one: positive, negative and incomplete events

The primary file contains eight authored rows: two positive, five negative and one incomplete record excluded from scoring. These are finite regression examples, not a representative sample of an organization’s activity.

Event Authored meaning Baseline result and assessment
P01 Successful unapproved human admin grant Matched; true positive under the scenario
P02 Provisioning actor grants admin through an unapproved manual workflow Matched; true positive
N01 Approved provisioning actor, workflow and change record Matched; false positive needing review
N02 Failed admin-grant attempt Not matched; true negative for this hypothesis
N03 Role removal Not matched; true negative
N04 Successful viewer-role grant Not matched; true negative
N05 Similar grant from another source stream Not matched; explicit pipeline scope works on this row
Q01 Required target role absent Quarantined before query execution; not a true negative

The actual baseline result was P01, P02 and N01. Among seven scored rows, that is two true positives, one false positive, zero false negatives and four true negatives. The eighth row remains a separately counted data-quality gap.

Precision is true positives divided by all matched scored events: 2 / 3, approximately 66.7%. Recall is true positives divided by all labeled positives: 2 / 2, or 100%. Neither percentage describes production coverage. We deliberately authored the events and their labels; a different sample could produce very different results.

The complete asset is more than this table. It includes the JSONL rows and label reasons, baseline YAML, pipeline YAML, generated SQL, actual matched IDs, quarantine list and confusion counts. Give the reviewer the matched IDs and labeled rows so they can check the claim without relying on a test-banner screenshot.

This exercise also exposes a useful limit: missing target-role data cannot support the intended decision. Quarantining Q01 does not fix the upstream event contract or prove the rule would detect that activity once corrected. Keep that gap in the report instead of raising the apparent score by counting it as harmless.

Two engineers compare a broad exclusion with a narrower context filter while retaining wrongly excluded event slips

Completed asset two: reject the overbroad tuning change

The independent review file contains six newly authored events, T01–T06. It is a separate regression set, not a blind holdout or real alert history. T01 is an authorized provisioning change; T05 is a failed request. T02, T03, T04 and T06 are labeled positives under the stated fictional policy.

An initial tuning proposal excludes every event whose actor is svc_provision. It removes the primary file’s false positive, but also hides P02, the same actor operating outside its approved workflow. On the primary file, precision rises to 100% while recall falls to 50%. Report precision and recall together so removing a false positive does not hide the newly missed positive.

The completed review makes that failure concrete:

Review event Relevant context Broad actor exclusion Context-limited revision
T01 Provisioner, approved workflow and approved change Excluded correctly Excluded correctly
T02 Provisioner uses manual workflow Missed positive Matched
T03 Provisioner uses expected workflow without approved change Missed positive Matched
T04 Unapproved human admin grant Matched Matched
T05 Failed approved-workflow request Correctly outside selection Correctly outside selection
T06 Different actor has matching workflow and ticket labels Matched Matched; cannot inherit provisioner exception

The narrower revision excludes only the conjunction of actor svc_provision, workflow approved_provisioning and ticket DEMO-APPROVED. It retains the original successful-admin-grant selection. The exception requires that exact actor, workflow and ticket combination. T02 and T03 test two ways the provisioning actor can fall outside it.

These values are a fixture-specific teaching policy, not a suggested production allowlist. The project does not establish that a workflow or change-ticket field is trustworthy, that a ticket is approved in a real change system, or that those fields cannot be forged. Real deployment would need an authorized source of that context and an owner for maintaining the exception.

Actual review results were TP=2, FP=0, FN=2, TN=2 for the broad actor exclusion and TP=4, FP=0, FN=0, TN=2 for the context-limited revision. The baseline on these six rows was TP=4, FP=1, FN=0, TN=1. The corrected rule also re-ran against the primary set, retaining P01 and P02 while excluding N01.

The review record keeps the three versions as archived experiments, not three simultaneously active production rules. It identifies the same rule’s progression, the excluded context and the actual positive IDs that must survive a change. Before accepting another exception, a reviewer can add a case that resembles the approved activity but changes one relevant field. That is an explicit new regression requirement, not evidence that the current suite exhausts possible variations.

The review decision is “reject the actor-wide exception; retain the narrower revision for this synthetic package.” It is not “deploy to production.” The saved rule diff, queries and both result sets show the change, its reason and the remaining assumptions.

Package enough evidence for another person to rerun it

The local bundle contains run_lab.py, the two event files, three archived rule versions, the pipeline and SQL output. From the report directory, use the recorded isolated environment to run:

current-venv/bin/python \
  run_lab.py

The script resolves the project root from its saved location, so that command uses the included project layout. If you move the bundle into a standalone repository, update and test the path handling deliberately. Preserve the pinned package versions when reproducing the experiment.

Two fresh-process local executions produced identical saved result hashes. That checks reproducibility in the recorded environment, not every operating system or future dependency release. The results also record the expected missing-mapping error and metadata-only query comparison. No SIEM account, production alert rule or response integration was tested.

For a production-oriented follow-up, identify the authorized log owner, actual schema, label provenance, representative negative activity, ingestion gaps and response handoff. Record those items as deployment prerequisites still unperformed, not production capabilities supplied by the successful local test. ATT&CK mapping can explain a behavioral relationship; it cannot supply an unmeasured detection rate, recall denominator or verified response capability.

Describe the portfolio contribution accurately

A defensible summary is: “Converted original experimental Sigma rules through an explicit pipeline to SQLite; tested two synthetic event sets, preserved a missing-field quarantine, and rejected an exclusion that improved sample precision while reducing sample recall.” Link the reproducible artifacts and state their limitations alongside the result.

The executed contribution is conversion, finite sample evaluation and regression review; production deployment remains outside the evidence. It does not demonstrate production rule ownership, a real compromise, alert-volume reduction or employer acceptance. Keep the question “what did I actually execute?” separate from “what would I need before deploying?”

Compare that evidence with a current role’s requirements when you browse recent roles on LandOffer.ai. Choose a work sample you can explain from input labels through query execution, tuning failure and final review decision.

Download the worked example files

Download the curated educational companion. Read PUBLICATION-README.md after extraction for the public file layout, replay commands and requirements. This is an archival source-and-fixture companion with external runtime requirements; its dependency-based execution was not rerun for publication. The earlier local report and commands describe the original evidence environment; use the companion README for the portable layout. No whole evidence directory, credentials or compiler binaries are included.

Sources and Further Reading