File size: 4,621 Bytes
f5a5d10 3896c8f f5a5d10 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 | ---
license: mit
base_model: Qwen/Qwen2.5-Coder-1.5B-Instruct
library_name: transformers
pipeline_tag: text-generation
tags:
- cybersecurity
- detection-engineering
- sigma
- siem
- mitre-attack
- qwen2
- lora
datasets:
- alirezaaminzadeh/sigmaforge-detection-rules
language:
- en
model-index:
- name: sigmaforge-rule-generator
results:
- task:
type: text-generation
name: Sigma rule generation
dataset:
name: SigmaForge Detection Rules (description_to_sigma, test)
type: alirezaaminzadeh/sigmaforge-detection-rules
config: description_to_sigma
split: test
metrics:
- type: valid_yaml_rate
value: 1.0
name: Valid YAML Rate
- type: schema_pass_rate
value: 0.9167
name: Sigma Schema Pass Rate
- type: splunk_compilation_rate
value: 0.8333
name: Splunk SPL Compilation Rate
- type: elastic_compilation_rate
value: 0.8333
name: Elastic Lucene Compilation Rate
- type: attack_mapping_f1
value: 0.1412
name: MITRE Mapping F1
---
# SigmaForge Rule Generator
Fine-tuned Sigma detection-rule generator based on
`Qwen/Qwen2.5-Coder-1.5B-Instruct`. The model drafts and repairs
[Sigma](https://sigmahq.io/) YAML from natural-language detections, sample
events, or broken rules, then downstream tools compile the YAML to Splunk SPL,
Elastic Lucene, and Microsoft Sentinel KQL.
Published by the Aria AI Security Research Team.
## Training
| Property | Value |
|---|---|
| Base model | `Qwen/Qwen2.5-Coder-1.5B-Instruct` |
| Method | LoRA (r=16, α=32) then merge |
| Data | [sigmaforge-detection-rules](https://huggingface.co/datasets/alirezaaminzadeh/sigmaforge-detection-rules) |
| Samples seen | 7,166 (6 ZeroGPU chunks) |
| Last train loss | 0.73 |
| Compute | Hugging Face ZeroGPU (`sigmaforge-trainer`) |
The supervised mix covers three dataset configs:
- `description_to_sigma` — analyst description → Sigma YAML
- `sigma_repair` — broken rule + validation errors → corrected rule
- `logs_to_sigma` — positive/negative events → expected rule
## Held-out evaluation (n=60, `description_to_sigma` test)
| Metric | Score |
|---|---:|
| Valid YAML rate | **1.00** |
| Sigma schema pass rate | **0.92** |
| Splunk SPL compilation rate | **0.83** |
| Elastic Lucene compilation rate | **0.83** |
| ATT&CK mapping precision | 0.60 |
| ATT&CK mapping recall | 0.08 |
| ATT&CK mapping F1 | 0.14 |
The generator is reliable at emitting schema-valid, compilable Sigma YAML.
Technique tagging is weaker: the model often omits ATT&CK IDs even when the
rule body is usable. Pair it with the retrieval + pySigma validation loop in
the [SigmaForge Space](https://huggingface.co/spaces/alirezaaminzadeh/sigmaforge)
rather than treating raw model output as a deployable rule.
Raw numbers: [`eval_results.json`](./eval_results.json).
## Usage
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "alirezaaminzadeh/sigmaforge-rule-generator"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype="auto", device_map="auto")
messages = [
{
"role": "system",
"content": "You are SigmaForge. Write valid Sigma YAML only.",
},
{
"role": "user",
"content": (
"Write a Sigma rule for the following detection.\n\n"
"Detection: Suspicious access to LSASS memory from a non-system process\n"
"Log source: product=windows, category=process_access\n"
"MITRE ATT&CK: T1003.001"
),
},
]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
ids = tok(prompt, return_tensors="pt").to(model.device)
out = model.generate(**ids, max_new_tokens=700, do_sample=False)
print(tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=True))
```
Validate every generated rule with pySigma before deploying it to a SIEM.
## Related
- Dataset: [sigmaforge-detection-rules](https://huggingface.co/datasets/alirezaaminzadeh/sigmaforge-detection-rules)
- Demo: [sigmaforge](https://huggingface.co/spaces/alirezaaminzadeh/sigmaforge)
- Trainer: [sigmaforge-trainer](https://huggingface.co/spaces/alirezaaminzadeh/sigmaforge-trainer)
- Collection: [SigmaForge](https://huggingface.co/collections/alirezaaminzadeh/sigmaforge-6aa93dde06ccb8ad3403eecc)
## License
MIT for this fine-tune. Upstream SigmaHQ rules remain under the Detection Rule
License (DRL 1.1).
|