Upload folder using huggingface_hub

README.md

@@ -5,210 +5,114 @@ language:
 - en
 tags:
 - cybersecurity
-- detection-engineering
-- threat-detection
 - soc-triage
-- qwen
-- qwen3
+- threat-detection
 - qlora
 - peft
-- security
-- malware-detection
+- qwen3
 pipeline_tag: text-generation
 library_name: peft
 ---
 
 # ThreatQwen-1.7B-Detect
 
-A QLoRA fine-tune of **Qwen3-1.7B-Instruct** for **cybersecurity event triage**.
-
-Given a raw security event (Sysmon, CloudTrail, HTTP log, etc.), the model returns a structured JSON verdict classifying the event as **malicious** or **benign**.
-
----
+A QLoRA fine-tune of **Qwen3-1.7B-Instruct** for cybersecurity event triage. Given a raw security event (Sysmon, CloudTrail, HTTP log, etc.), the model returns a structured JSON verdict.
 
 ## Results
 
-Evaluated on a balanced held-out test set of **882 samples (441 malicious + 441 benign)**,
-same examples presented to all models:
-
-| Model | Size | Accuracy | Mal. Recall | Ben. Recall | Macro F1 |
-|---|---|---|---|---|---|
-| **ThreatQwen-1.7B-Detect (ours)** | **1.7B** | **98.1%** | **96.1%** | **100.0%** | **0.981** |
-| Base Qwen3-1.7B (no fine-tune) | 1.7B | 85.1% | 92.1% | 78.2% | 0.851 |
-| GPT-4o (Azure, zero-shot) | ~200B+ | 51.8% | 85.0% | 18.6% | 0.458 |
-| GPT-4o-mini (Azure, zero-shot) | ~8B | 51.9% | 59.6% | 44.2% | 0.516 |
+Evaluated on a balanced held-out test set of **882 samples (441 malicious + 441 benign)**:
 
-**Fine-tuning improves the base model by +13 percentage points.**
-**ThreatQwen-1.7B outperforms GPT-4o by +46.3 percentage points on this task.**
-
----
+| Model | Accuracy | Mal. Recall | Ben. Recall | Macro F1 |
+|---|---|---|---|---|
+| **ThreatQwen-1.7B-Detect (ours)** | **95.1%** | **96.1%** | **94.2%** | **0.951** |
+| Base Qwen3-1.7B (no fine-tune) | 85.1% | 92.1% | 78.2% | 0.851 |
+| GPT-4o (Azure, zero-shot) | 51.8% | 85.0% | 18.6% | 0.458 |
+| GPT-4o-mini (Azure, zero-shot) | 51.9% | 59.6% | 44.2% | 0.516 |
 
-## What Changed from v1 (ThreatQwen-1.5B-Detect)
-
-- Upgraded base model from Qwen2.5-Coder-1.5B → **Qwen3-1.7B**
-- Benign recall improved from **0% → 100%** via:
-  - 4x benign oversampling during training
-  - Removal of severity field (was causing data leakage)
-  - 1,500 additional GPT-4o-generated benign examples covering Windows,
-    web, email, database, and VPN/remote-access activity
-- Output schema simplified to verdict-only JSON (removed MITRE field —
-  63.9% of training examples had empty MITRE arrays)
-- MAX_SEQ_LEN reduced 1024 → 512 (covers 100% of data, 2x faster training)
-- Full test set evaluation: 882 samples vs 50 in v1
-
----
+Fine-tuning improves the base model by **+10 percentage points** and outperforms GPT-4o by **+43.3 percentage points**.
 
 ## Output Schema
 
 ```json
-{
-  "verdict": "malicious | benign"
-}
+{"verdict": "malicious | benign"}
 ```
 
----
-
 ## Training
 
 | Parameter | Value |
 |---|---|
 | Base model | unsloth/Qwen3-1.7B-unsloth-bnb-4bit |
-| Method | QLoRA (4-bit NF4) |
+| Method | QLoRA 4-bit NF4 |
 | LoRA rank / alpha | 16 / 32 |
-| Target modules | q/k/v/o_proj, gate/up/down_proj |
 | Epochs | 3 |
-| Batch size | 4 (grad accum 4, effective batch 16) |
-| Learning rate | 2e-4 (cosine schedule) |
-| MAX_SEQ_LEN | 512 tokens |
+| Effective batch size | 16 (batch 4 × accum 4) |
+| Learning rate | 2e-4 cosine |
+| MAX_SEQ_LEN | 512 |
 | Hardware | Tesla T4 (Kaggle free tier) |
-| Training time | ~10 minutes |
-
----
+| Training time | ~10 min |
 
 ## Dataset
 
-Training data combines:
-
-| Source | Type | Examples |
-|---|---|---|
-| SigmaHQ detection rules | Real malicious | ~1,968 |
-| Elastic Detection Rules | Real malicious | ~1,358 |
-| Nuclei Templates | Real malicious | ~2,497 |
-| synthetic_benign | Synthetic benign | ~1,255 |
-| synthetic_web_benign | Synthetic benign | ~1,000 |
-| synthetic_ambiguous | Mixed | ~376 |
-| GPT-4o generated (Windows) | Synthetic benign | ~499 |
-| GPT-4o generated (Web) | Synthetic benign | ~500 |
-| GPT-4o generated (Email/DB/VPN) | Synthetic benign | ~989 |
-
-After deduplication and cleaning: **~9,400 unique examples**.
-
----
+Combines SigmaHQ, Elastic Detection Rules, and Nuclei templates (~5,823 real malicious events) with GPT-4o-generated synthetic benign examples (~3,743 benign covering Windows, web, email, database, VPN/remote-access). Full dataset: [minar-svn/ThreatQwen-detection-dataset](https://huggingface.co/datasets/minar-svn/ThreatQwen-detection-dataset)
 
 ## Deployment
 
-- Runs fully **offline** — no internet required (air-gapped SOC capable)
-- **~3.1 GB VRAM** at inference (4-bit quantized)
-- **~18 tokens/sec** on Tesla T4
-- **~5–7 sec/event** end-to-end latency
-- Adapter size: **~434 MB**
-- Minimum GPU: RTX 3060 (8 GB VRAM) or equivalent
-
----
+- Fully offline — no internet required (air-gapped SOC capable)
+- ~3.1 GB VRAM (4-bit quantized)
+- ~18 tokens/sec on Tesla T4
+- ~5–7 sec/event latency
+- Adapter: ~434 MB
+- Min GPU: RTX 3060 (8 GB)
 
 ## Usage
 
 ```python
 from transformers import AutoModelForCausalLM, AutoTokenizer
 from peft import PeftModel
-import torch, json, re
+import torch, json
 
-# Load base + adapter
 base = AutoModelForCausalLM.from_pretrained(
     "unsloth/Qwen3-1.7B-unsloth-bnb-4bit",
-    torch_dtype  = torch.float16,
-    device_map   = "auto",
+    torch_dtype=torch.float16, device_map="auto",
 )
-model = PeftModel.from_pretrained(base, "minar-svn/ThreatQwen-1.7B-Detect")
+model     = PeftModel.from_pretrained(base, "minar-svn/ThreatQwen-1.7B-Detect")
 tokenizer = AutoTokenizer.from_pretrained("minar-svn/ThreatQwen-1.7B-Detect")
 
-SYSTEM_PROMPT = """You are a cybersecurity detection model.
-Respond ONLY with valid JSON — no markdown, no explanations, no extra text.
+SYSTEM = 'You are a cybersecurity detection model. Respond ONLY with valid JSON. Format: {"verdict": "malicious | benign"}'
 
-Format:
-{
-  "verdict": "malicious | benign"
-}
-"""
-
-event = """EventID: 1 (Process Creation)
-Image: C:\\Windows\\System32\\rundll32.exe
-ParentImage: C:\\Windows\\System32\\cmd.exe
-CommandLine: rundll32.exe C:\\Windows\\System32\\comsvcs.dll MiniDump 624 C:\\temp\\lsass.dmp full
-User: admin"""
-
-messages = [
-    {"role": "system", "content": SYSTEM_PROMPT},
-    {"role": "user",   "content": f"Analyze this security event:\n\n{event}"},
-]
-
-prompt = tokenizer.apply_chat_template(
-    messages, tokenize=False, add_generation_prompt=True
-).strip()
+event = "EventID: 1 Image: rundll32.exe CommandLine: comsvcs.dll MiniDump User: admin"
 
+msgs   = [{"role":"system","content":SYSTEM},{"role":"user","content":f"Analyze:\n\n{event}"}]
+prompt = tokenizer.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True).strip()
 inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
+
 with torch.no_grad():
-    out = model.generate(
-        **inputs,
-        max_new_tokens     = 80,
-        do_sample          = False,
-        repetition_penalty = 1.02,
-        pad_token_id       = tokenizer.eos_token_id,
-    )
-
-response = tokenizer.decode(
-    out[0][inputs["input_ids"].shape[1]:],
-    skip_special_tokens=True,
-).strip()
-
-# Clean and parse
-for tok in ["<think>", "</think>", "```json", "```"]:
-    response = response.replace(tok, "")
-response = response.strip()
-blocks   = [b.strip() for b in response.split("\n\n") if b.strip()]
-response = blocks[-1] if blocks else response
+    out = model.generate(**inputs, max_new_tokens=40, do_sample=False,
+                         pad_token_id=tokenizer.eos_token_id)
 
-result = json.loads(response)
-print(result)
-# Output: {"verdict": "malicious"}
+response = tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True).strip()
+blocks   = [b.strip() for b in response.split("\n\n") if b.strip()]
+print(json.loads(blocks[-1]))
+# {"verdict": "malicious"}
 ```
 
----
-
 ## Limitations
 
-- All benign training examples are synthetic — real-world benign generalization is untested
-- Best on structured log formats (Sysmon, CloudTrail, HTTP); degrades on free-form text
+- All benign test examples are synthetic — real-world benign generalization untested
 - English only
-- Should be used as an analyst aid, not as a sole decision authority
-- May over-flag Windows administrative utilities that share patterns with LOLBAS abuse
-
----
+- Analyst aid only — not a sole decision authority
 
 ## Citation
 
 ```bibtex
 @misc{threatqwen2026,
   author    = {Md. Minaruzzaman Shovon},
-  title     = {ThreatQwen-1.7B-Detect: A Small Open-Source LLM for Cybersecurity Event Triage},
+  title     = {ThreatQwen-1.7B-Detect},
   year      = {2026},
   publisher = {Hugging Face},
   url       = {https://huggingface.co/minar-svn/ThreatQwen-1.7B-Detect}
 }
 ```
 
----
-
 ## License
-
-Apache 2.0 — inherits from the Qwen3 base model.
+Apache 2.0