alirezaaminzadeh/sigmaforge-rule-generator

SigmaForge Rule Generator

Fine-tuned Sigma detection-rule generator based on Qwen/Qwen2.5-Coder-1.5B-Instruct. The model drafts and repairs Sigma YAML from natural-language detections, sample events, or broken rules, then downstream tools compile the YAML to Splunk SPL, Elastic Lucene, and Microsoft Sentinel KQL.

Published by the Aria AI Security Research Team.

Training

Property	Value
Base model	Qwen/Qwen2.5-Coder-1.5B-Instruct
Method	LoRA (r=16, α=32) then merge
Data	sigmaforge-detection-rules
Samples seen	7,166 (6 ZeroGPU chunks)
Last train loss	0.73
Compute	Hugging Face ZeroGPU (sigmaforge-trainer)

The supervised mix covers three dataset configs:

• description_to_sigma — analyst description → Sigma YAML
• sigma_repair — broken rule + validation errors → corrected rule
• logs_to_sigma — positive/negative events → expected rule

Held-out evaluation (n=60, description_to_sigma test)

Metric	Score
Valid YAML rate	1.00
Sigma schema pass rate	0.92
Splunk SPL compilation rate	0.83
Elastic Lucene compilation rate	0.83
ATT&CK mapping precision	0.60
ATT&CK mapping recall	0.08
ATT&CK mapping F1	0.14

The generator is reliable at emitting schema-valid, compilable Sigma YAML. Technique tagging is weaker: the model often omits ATT&CK IDs even when the rule body is usable. Pair it with the retrieval + pySigma validation loop in the SigmaForge Space rather than treating raw model output as a deployable rule.

Raw numbers: eval_results.json.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "alirezaaminzadeh/sigmaforge-rule-generator"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype="auto", device_map="auto")

messages = [
    {
        "role": "system",
        "content": "You are SigmaForge. Write valid Sigma YAML only.",
    },
    {
        "role": "user",
        "content": (
            "Write a Sigma rule for the following detection.\n\n"
            "Detection: Suspicious access to LSASS memory from a non-system process\n"
            "Log source: product=windows, category=process_access\n"
            "MITRE ATT&CK: T1003.001"
        ),
    },
]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
ids = tok(prompt, return_tensors="pt").to(model.device)
out = model.generate(**ids, max_new_tokens=700, do_sample=False)
print(tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=True))

Validate every generated rule with pySigma before deploying it to a SIEM.

Related

• Dataset: sigmaforge-detection-rules
• Demo: sigmaforge
• Trainer: sigmaforge-trainer
• Collection: SigmaForge

License

MIT for this fine-tune. Upstream SigmaHQ rules remain under the Detection Rule License (DRL 1.1).