---
license: mit
base_model: Qwen/Qwen2.5-Coder-1.5B-Instruct
library_name: transformers
pipeline_tag: text-generation
tags:
- cybersecurity
- detection-engineering
- sigma
- siem
- mitre-attack
- qwen2
- lora
datasets:
- alirezaaminzadeh/sigmaforge-detection-rules
language:
- en
model-index:
- name: sigmaforge-rule-generator
  results:
  - task:
      type: text-generation
      name: Sigma rule generation
    dataset:
      name: SigmaForge Detection Rules (description_to_sigma, test)
      type: alirezaaminzadeh/sigmaforge-detection-rules
      config: description_to_sigma
      split: test
    metrics:
    - type: valid_yaml_rate
      value: 1.0
      name: Valid YAML Rate
    - type: schema_pass_rate
      value: 0.9167
      name: Sigma Schema Pass Rate
    - type: splunk_compilation_rate
      value: 0.8333
      name: Splunk SPL Compilation Rate
    - type: elastic_compilation_rate
      value: 0.8333
      name: Elastic Lucene Compilation Rate
    - type: attack_mapping_f1
      value: 0.1412
      name: MITRE Mapping F1
---

# SigmaForge Rule Generator

Fine-tuned Sigma detection-rule generator based on
`Qwen/Qwen2.5-Coder-1.5B-Instruct`. The model drafts and repairs
[Sigma](https://sigmahq.io/) YAML from natural-language detections, sample
events, or broken rules, then downstream tools compile the YAML to Splunk SPL,
Elastic Lucene, and Microsoft Sentinel KQL.

Published by the Aria AI Security Research Team.

## Training

| Property | Value |
|---|---|
| Base model | `Qwen/Qwen2.5-Coder-1.5B-Instruct` |
| Method | LoRA (r=16, α=32) then merge |
| Data | [sigmaforge-detection-rules](https://huggingface.co/datasets/alirezaaminzadeh/sigmaforge-detection-rules) |
| Samples seen | 7,166 (6 ZeroGPU chunks) |
| Last train loss | 0.73 |
| Compute | Hugging Face ZeroGPU (`sigmaforge-trainer`) |

The supervised mix covers three dataset configs:

- `description_to_sigma` — analyst description → Sigma YAML
- `sigma_repair` — broken rule + validation errors → corrected rule
- `logs_to_sigma` — positive/negative events → expected rule

## Held-out evaluation (n=60, `description_to_sigma` test)

| Metric | Score |
|---|---:|
| Valid YAML rate | **1.00** |
| Sigma schema pass rate | **0.92** |
| Splunk SPL compilation rate | **0.83** |
| Elastic Lucene compilation rate | **0.83** |
| ATT&CK mapping precision | 0.60 |
| ATT&CK mapping recall | 0.08 |
| ATT&CK mapping F1 | 0.14 |

The generator is reliable at emitting schema-valid, compilable Sigma YAML.
Technique tagging is weaker: the model often omits ATT&CK IDs even when the
rule body is usable. Pair it with the retrieval + pySigma validation loop in
the [SigmaForge Space](https://huggingface.co/spaces/alirezaaminzadeh/sigmaforge)
rather than treating raw model output as a deployable rule.

Raw numbers: [`eval_results.json`](./eval_results.json).

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "alirezaaminzadeh/sigmaforge-rule-generator"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype="auto", device_map="auto")

messages = [
    {
        "role": "system",
        "content": "You are SigmaForge. Write valid Sigma YAML only.",
    },
    {
        "role": "user",
        "content": (
            "Write a Sigma rule for the following detection.\n\n"
            "Detection: Suspicious access to LSASS memory from a non-system process\n"
            "Log source: product=windows, category=process_access\n"
            "MITRE ATT&CK: T1003.001"
        ),
    },
]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
ids = tok(prompt, return_tensors="pt").to(model.device)
out = model.generate(**ids, max_new_tokens=700, do_sample=False)
print(tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=True))
```

Validate every generated rule with pySigma before deploying it to a SIEM.

## Related

- Dataset: [sigmaforge-detection-rules](https://huggingface.co/datasets/alirezaaminzadeh/sigmaforge-detection-rules)
- Demo: [sigmaforge](https://huggingface.co/spaces/alirezaaminzadeh/sigmaforge)
- Trainer: [sigmaforge-trainer](https://huggingface.co/spaces/alirezaaminzadeh/sigmaforge-trainer)
- Collection: [SigmaForge](https://huggingface.co/collections/alirezaaminzadeh/sigmaforge-6aa93dde06ccb8ad3403eecc)

## License

MIT for this fine-tune. Upstream SigmaHQ rules remain under the Detection Rule
License (DRL 1.1).