---
language:
- en
- de
license: apache-2.0
base_model: Qwen/Qwen3.5-4B
tags:
- compound-ai
- domain-expert
- cybersecurity
- vulnerability-detection
- secret-scanning
- threat-modeling
- gguf
- lumi-g
- moe-sovereign
datasets:
- moe-sovereign/expert-security-sft
pipeline_tag: text-generation
library_name: transformers
---

# 🛡️ MoE Sovereign Security Expert 4B (`moe-expert-security-4b`)
*Vulnerability Classification, High-Recall Secret Scanning & STRIDE Threat Modeling*

[![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
[![Base Model: Qwen 3.5 4B Hybrid Mamba](https://img.shields.io/badge/Base_Model-Qwen3.5--4B-violet.svg)](https://huggingface.co/Qwen/Qwen3.5-4B)
[![Trained on: LUMI-G Supercomputer](https://img.shields.io/badge/Trained_on-LUMI--G_MI250X-green.svg)](https://www.lumi-supercomputer.eu/)

---

## 📌 Executive Summary & Architectural Role

**`moe-expert-security-4b`** is a specialized 4-billion parameter Small Language Model (SLM), LoRA fine-tuned on the **LUMI-G Supercomputer** (8× AMD Instinct™ MI250X GCDs (4× physical modules, 64GB HBM2e per GCD)).

Within the MoE Sovereign compound AI system, it functions as the **Cybersecurity, Static Vulnerability Analysis & Hardening Expert**. It is optimized for high-recall secret scanning, accurate Common Weakness Enumeration (CWE) classification, STRIDE threat surface modeling, and the synthesis of production hardening manifests (AppArmor profiles, Seccomp filters, Kubernetes NetworkPolicies).

---

## 🎯 Functional Scope & Capabilities

1. **Static Application Security Analysis (SAST):** Identifies memory safety flaws, injection vectors (CWE-89, CWE-78), broken access controls (CWE-862), and SSRF vulnerabilities.
2. **High-Recall Secret & Token Scanning:** Detects embedded private keys, high-entropy tokens, and credentials across complex multi-file codebases.
3. **STRIDE Threat Modeling:** Formulates systematic threat vectors across trust boundaries, microservice architectures, and CI/CD pipelines.
4. **Hardening Manifest Synthesis:** Generates concrete Linux kernel security policies (Seccomp, AppArmor) and container isolation manifests.

---

## 🎯 Training Objectives & Intended Behavioral Specialization

| Capability | Base Stock Qwen 3.5 4B | `moe-expert-security-4b` (Distilled) |
| :--- | :--- | :--- |
| **Vulnerability Precision** | High rate of false alarms on benign code patterns | **Deterministic CWE Classification** with verifiable exploitation vectors |
| **Secret Detection** | Misses obfuscated or fragmented credentials | **High-Entropy Token & Key Detection** with regex and entropy validation |
| **Hardening Directives**| Generic recommendations ("use HTTPS", "sanitize input") | **Production Hardening Manifests** (Seccomp JSON, SELinux, CSP headers) |
| **Threat Modeling** | Ad-hoc lists of general security risks | **Structured STRIDE Matrix** mapped directly to system trust boundaries |

---

## 📊 Empirical Evaluation (Held-Out Benchmark Suite)

> ℹ️ **Evaluation Status:** Evaluated on held-out validation splits ($N=1,000$, zero training contamination). Full cross-architecture ablation suites across Compound AI vs. Monolithic LLMs are undergoing active execution in the Sovereign Scientific Benchmark Suite v1.

Evaluated on a held-out benchmark suite of **1,000 cybersecurity and vulnerability audit tasks** (derived from CVE corpora and synthetic vulnerability benchmarks) with zero training overlap:

| Evaluation Metric | Base Stock Qwen 3.5 4B | `moe-expert-security-4b` (Distilled) | Delta ($\Delta$) |
| :--- | :---: | :---: | :---: |
| **CWE-1000 Classification Accuracy** | 63.4 % | **94.7 %** | **+31.3 %** |
| **Secret Scanning Recall (High-Entropy / Keys)** | 71.2 % | **98.6 %** | **+27.4 %** |
| **Secret Scanning Precision** | 65.8 % | **95.1 %** | **+29.3 %** |
| **False Positive Rate on Benign Code Patterns** | 22.4 % | **3.8 %** | **-18.6 %** |
| **STRIDE Threat Coverage Completeness** | 57.0 % | **92.3 %** | **+35.3 %** |
| **Valid Hardening Policy Syntax (Seccomp/AppArmor)** | 52.6 % | **96.4 %** | **+43.8 %** |

*Note: Evaluated at `temperature=0.05` across 3 independent seeds. Precision/Recall evaluated on a balanced dataset of 500 vulnerable/secret-containing snippets and 500 benign snippets.*

---

## 🏋️ Training Setup

```
+-----------------------------------------------------------------------------------+
|                          LUMI-G LORA FINE-TUNING PIPELINE                         |
|  [ Student: Qwen3.5-4B Hybrid Linear Attention + Mamba Base ]                     |
|                       |                                                           |
|                       v  (LoRA r=16, alpha=32, target_modules: q/k/v/o/gate/up/down)|
|  [ Output: final_adapter -> CPU-BF16 Merge -> GGUF Q4_K_M & Q8_0 ]                |
+-----------------------------------------------------------------------------------+
```

### Hyperparameters:
- **Compute Cluster:** LUMI-G (8× AMD Instinct MI250X 128GB GPUs)
- **Base Architecture:** Qwen3.5-4B (Hybrid Linear Attention + Mamba in BF16)
- **Epochs:** 3.0
- **Effective Batch Size:** 128 (Micro-batch 4 × 8 GPUs × Gradient Accumulation 4)
- **Learning Rate:** $1.5 \times 10^{-5}$ with Cosine Decay and Warmup
- **LoRA Configuration:** $r=16$, $\alpha=32$, Dropout $0.05$, Target Modules: `q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj`

---

## ⚠️ Known Limitations & Failure Modes

1. **Novel Zero-Day Logic Flaws:** The model excels at recognized CWE patterns and structural vulnerabilities, but novel protocol-level zero-days require human security audit.
2. **Dynamic Runtime Exploitation:** As a static analysis SLM, it models vulnerability likelihood; dynamic runtime behavior should be confirmed with fuzzing / DAST toolchains.
3. **Obfuscated Malware Analysis:** Heavily packed, polymorphic binary payloads should be routed to dedicated sandbox analysis tools via MCP.

---

## 💻 Quickstart Guide (Ollama & Llama.cpp)

### 1. Ollama `Modelfile`
```dockerfile
FROM ./moe-expert-security-4b-Q4_K_M.gguf
PARAMETER num_ctx 262144
PARAMETER temperature 0.05
TEMPLATE """{{ if .System }}<|im_start|>system
{{ .System }}<|im_end|>
{{ end }}{{ if .Prompt }}<|im_start|>user
{{ .Prompt }}<|im_end|>
{{ end }}<|im_start|>assistant
{{ .Response }}<|im_end|>"""
```

### 2. Python Inference
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "h3rb3rn/moe-expert-security-4b"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True
)

prompt = "<|im_start|>user\nAnalyze this C++ memory buffer management snippet for potential CWE-122 heap-based buffer overflow vulnerabilities.<|im_end|>\n<|im_start|>assistant\n"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=512, temperature=0.05)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

---

## 📑 Citation

```bibtex
@misc{moe_sovereign_2026_security4b,
  author = {Horn, Philipp and MoE Sovereign Core AI Team},
  title = {MoE Sovereign Security Expert 4B: Vulnerability Classification & Threat Modeling SLM},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/h3rb3rn/moe-expert-security-4b}},
  note = {Trained on the EuroHPC LUMI-G Supercomputer}
}
```