README.md

metadata

license: mit
base_model: Qwen/Qwen2.5-Coder-1.5B-Instruct
library_name: transformers
pipeline_tag: text-generation
tags:
  - cybersecurity
  - detection-engineering
  - sigma
  - siem
  - mitre-attack
  - qwen2
  - lora
datasets:
  - alirezaaminzadeh/sigmaforge-detection-rules
language:
  - en
model-index:
  - name: sigmaforge-rule-generator
    results:
      - task:
          type: text-generation
          name: Sigma rule generation
        dataset:
          name: SigmaForge Detection Rules (description_to_sigma, test)
          type: alirezaaminzadeh/sigmaforge-detection-rules
          config: description_to_sigma
          split: test
        metrics:
          - type: valid_yaml_rate
            value: 1
            name: Valid YAML Rate
          - type: schema_pass_rate
            value: 0.9167
            name: Sigma Schema Pass Rate
          - type: splunk_compilation_rate
            value: 0.8333
            name: Splunk SPL Compilation Rate
          - type: elastic_compilation_rate
            value: 0.8333
            name: Elastic Lucene Compilation Rate
          - type: attack_mapping_f1
            value: 0.1412
            name: MITRE Mapping F1

SigmaForge Rule Generator

Fine-tuned Sigma detection-rule generator based on Qwen/Qwen2.5-Coder-1.5B-Instruct. The model drafts and repairs Sigma YAML from natural-language detections, sample events, or broken rules, then downstream tools compile the YAML to Splunk SPL, Elastic Lucene, and Microsoft Sentinel KQL.

Published by the Aria AI Security Research Team.

Training

Property	Value
Base model	Qwen/Qwen2.5-Coder-1.5B-Instruct
Method	LoRA (r=16, α=32) then merge
Data	sigmaforge-detection-rules
Samples seen	7,166 (6 ZeroGPU chunks)
Last train loss	0.73
Compute	Hugging Face ZeroGPU (sigmaforge-trainer)

The supervised mix covers three dataset configs:

• description_to_sigma — analyst description → Sigma YAML
• sigma_repair — broken rule + validation errors → corrected rule
• logs_to_sigma — positive/negative events → expected rule

Held-out evaluation (n=60, description_to_sigma test)

Metric	Score
Valid YAML rate	1.00
Sigma schema pass rate	0.92
Splunk SPL compilation rate	0.83
Elastic Lucene compilation rate	0.83
ATT&CK mapping precision	0.60
ATT&CK mapping recall	0.08
ATT&CK mapping F1	0.14

The generator is reliable at emitting schema-valid, compilable Sigma YAML. Technique tagging is weaker: the model often omits ATT&CK IDs even when the rule body is usable. Pair it with the retrieval + pySigma validation loop in the SigmaForge Space rather than treating raw model output as a deployable rule.

Raw numbers: eval_results.json.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "alirezaaminzadeh/sigmaforge-rule-generator"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype="auto", device_map="auto")

messages = [
    {
        "role": "system",
        "content": "You are SigmaForge. Write valid Sigma YAML only.",
    },
    {
        "role": "user",
        "content": (
            "Write a Sigma rule for the following detection.\n\n"
            "Detection: Suspicious access to LSASS memory from a non-system process\n"
            "Log source: product=windows, category=process_access\n"
            "MITRE ATT&CK: T1003.001"
        ),
    },
]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
ids = tok(prompt, return_tensors="pt").to(model.device)
out = model.generate(**ids, max_new_tokens=700, do_sample=False)
print(tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=True))

Validate every generated rule with pySigma before deploying it to a SIEM.

Related

• Dataset: sigmaforge-detection-rules
• Demo: sigmaforge
• Trainer: sigmaforge-trainer
• Collection: SigmaForge

License

MIT for this fine-tune. Upstream SigmaHQ rules remain under the Detection Rule License (DRL 1.1).