--- # ---- Identity ------------------------------------------------------------- base_model: Qwen/Qwen3.5-9B base_model_relation: adapter library_name: peft pipeline_tag: text-generation language: - en license: apache-2.0 # verified: inherited from Qwen/Qwen3.5-9B, whose Hub # metadata declares license:apache-2.0 and which ships an # Apache-2.0 LICENSE file (copied here verbatim). # ---- Discovery ------------------------------------------------------------ tags: - lora - qlora - sft - trl - peft - text-classification - cve - cwe - vulnerability - security datasets: - exploitintel/cve-cwe-consensus metrics: - f1 - accuracy # ---- Structured evaluation ------------------------------------------------ model-index: - name: qwen3.5-9b-cve-cwe-lora results: - task: type: text-classification name: CVE description to CWE weakness class (117-way, generated as text) dataset: type: exploitintel/cve-cwe-consensus name: exploitintel/cve-cwe-consensus, 300-row natural-distribution sample of the validation split split: validation revision: 606cef101302fc2e0f69fd298f0de28b20a2aacf metrics: - type: f1 name: Macro-F1 over the union of gold and predicted classes (81 classes) value: 0.473288 - type: f1 name: Macro-F1 over gold classes only (68 classes) value: 0.563769 - type: accuracy name: Accuracy (= micro-F1 for single-label output) value: 0.71 --- # Qwen3.5-9B CVE-to-CWE Classifier (LoRA) Given the free-text description of a published CVE, emits the single CWE identifier that best characterises the root-cause weakness, as one line of text (`CWE-79`, `CWE-787`, ...), over the 117-class label space of [exploitintel/cve-cwe-consensus](https://huggingface.co/datasets/exploitintel/cve-cwe-consensus). This is a **LoRA adapter for** [Qwen/Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B), trained with **QLoRA (4-bit NF4 base, bf16 compute)** via [TRL](https://github.com/huggingface/trl) SFT. > **The headline number is one draw from a noisy measurement.** This checkpoint > scored macro-F1 0.473 on 300 held-out rows, but the identical configuration > re-run later in the same search scored 0.395. Read > [How these values were chosen](#how-these-values-were-chosen) and > [Evaluation](#evaluation) before quoting anything, and do not report 0.473 as > the model's expected performance. ## Model details | | | |---|---| | Developed by | SASVA AI Model Cognition Labs (MCL) Team | | Base model | [`Qwen/Qwen3.5-9B`](https://huggingface.co/Qwen/Qwen3.5-9B) | | Base revision | `c202236235762e1c871ad0ccb60c8ee5ba337b9a` | | Base parameters | 9.65B (dense; 9,653,104,368 in the safetensors index, ~19.3 GB bf16) | | Architecture family | `qwen3_5` (`Qwen3_5ForConditionalGeneration`, text tower `qwen3_5_text`) | | Adaptation | LoRA (`r=128`, `alpha=256`, `dropout=0.15`, rsLoRA off, DoRA off) | | Trainable modules | `in_proj_qkv`, `in_proj_z`, `out_proj` (Gated DeltaNet layers); `q_proj`, `k_proj`, `v_proj`, `o_proj` (attention layers); `gate_proj`, `up_proj`, `down_proj` (every layer) | | Excluded modules | none (the base is text-only in practice; no adapter tensor touches the vision tower) | | Training method | `qlora` (`--load-in-4bit`, run 3 / trial 23) | | Refinement | none | | Precision | 4-bit NF4 base with double quantisation, bf16 compute; adapter stored in float32 | | Language | English | | License | Apache-2.0 (inherited from the base model) | Trainable parameters: **320,864,256** across 200 modules, 3.2975% of the 9,730,678,000 parameters PEFT counted with the adapter attached. The adapter file is 1,283,518,408 bytes (400 tensors, `lora_A` + `lora_B` per module, all float32). Every tensor sits under `base_model.model.model.language_model`. **One Qwen 3.5 structural fact shapes the module list.** Confirmed against the base model's `config.json`: the 32 layers follow a 3-linear / 1-full pattern (`full_attention_interval: 4`), so 24 layers are Gated DeltaNet linear attention and expose `in_proj_qkv` / `in_proj_z` / `out_proj`, while the 8 full-attention layers (3, 7, 11, ..., 31) are grouped-query attention with 16 heads over 4 KV heads and expose `q_proj` / `k_proj` / `v_proj` / `o_proj`. The adapter's tensor counts match exactly: 48 per linear-attention projection (24 layers x A/B), 16 per attention projection (8 layers x A/B), 64 per MLP projection (32 layers x A/B). There are no MoE experts in this base. ## Intended use **Direct use.** Map a CVE description to one CWE id, for triage and labelling pipelines that already consume CWE ids. The 117-class label space is the dataset's; ids outside it were never seen in training. The model was trained on a specific prompt shape and that shape is part of the contract: - System prompt (verbatim): *"You are a CVE-to-CWE classifier. Given a CVE vulnerability description, identify the single root-cause CWE weakness class that best characterizes the flaw. Output exactly one CWE identifier (e.g., CWE-79) on a single line with no explanation or additional text."* - User turn: the instruction *"Classify the following CVE description into exactly one CWE weakness class. Reply with the CWE ID only, for example CWE-79."*, a blank line, then the CVE description inside a bare ```` ``` ```` fence. This is the exact string the evaluator rendered (`instruction + "\n\n```\n" + description + "\n```"`). - Applied through the tokenizer's chat template (`chat_template.jinja`, shipped in this repo) with `add_generation_prompt=True` and `enable_thinking=False`. Do not concatenate strings by hand. - The output is one line, `CWE-`. In evaluation all 300 generations were a single well-formed id; the first line of the output is the prediction. - Decode greedily (`do_sample=False`) with a small budget; 64 new tokens is what the metric was scored with. ## How to get started ```python import torch from peft import PeftModel from transformers import AutoModelForImageTextToText, AutoTokenizer BASE = "Qwen/Qwen3.5-9B" ADAPTER = "SASVAAI/Qwen-3.5-9B-CVE-to-CWE" # NOTE: AutoModelForImageTextToText, not AutoModelForCausalLM. Qwen 3.5's # CausalLM mapping raises AttributeError: 'Qwen3_5Config' object has no # attribute 'vocab_size' on the transformers build used here; the # image-text-to-text mapping resolves to Qwen3_5ForConditionalGeneration and # loads the text model correctly. Training used the same class. tokenizer = AutoTokenizer.from_pretrained(ADAPTER, trust_remote_code=True) model = AutoModelForImageTextToText.from_pretrained( BASE, dtype=torch.bfloat16, device_map="auto", trust_remote_code=True ) model = PeftModel.from_pretrained(model, ADAPTER) model.eval() # Verbatim from the training invocation. Do not paraphrase. SYSTEM = ( "You are a CVE-to-CWE classifier. Given a CVE vulnerability description, " "identify the single root-cause CWE weakness class that best characterizes " "the flaw. Output exactly one CWE identifier (e.g., CWE-79) on a single line " "with no explanation or additional text." ) INSTRUCTION = ( "Classify the following CVE description into exactly one CWE weakness class. " "Reply with the CWE ID only, for example CWE-79." ) description = ( "A stored cross-site scripting flaw in the FAQ page lets an attacker inject " "script that runs in other users' browsers." ) messages = [ {"role": "system", "content": SYSTEM}, {"role": "user", "content": f"{INSTRUCTION}\n\n```\n{description}\n```"}, ] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, enable_thinking=False, return_tensors="pt", return_dict=True, ).to(model.device) out = model.generate(**inputs, max_new_tokens=64, do_sample=False) print(tokenizer.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True).strip()) # -> CWE-79 ``` The base model is ~19 GB in bfloat16; one 24 GB-class GPU is enough for inference with the adapter. Loading the base in 4-bit with `bitsandbytes` (`load_in_4bit=True`, `bnb_4bit_quant_type="nf4"`, `bnb_4bit_use_double_quant=True`, `bnb_4bit_compute_dtype=torch.bfloat16`) reproduces the training-time numerics and needs about 7 GB. > Decoding matters. The metric was scored greedily with `max_new_tokens=64` > through the chat template with thinking disabled. No sampling setting was > validated, and enabling thinking changes the prompt the model sees. ## Training details **Data.** 700 training rows and 300 validation rows built from [exploitintel/cve-cwe-consensus](https://huggingface.co/datasets/exploitintel/cve-cwe-consensus) at revision `606cef101302fc2e0f69fd298f0de28b20a2aacf` by a deterministic script (`autocatalyst.datagen.cve_cwe_consensus`, seed 0). No model generated any training content. Row selection, as recorded in the builder's manifest: - Source rows are `{"messages": [system, user, assistant]}`; the user turn is the CVE description, the assistant turn the label. - Rows whose label is not exactly one `CWE-` id were dropped (about 9% of the dataset are comma-separated multi-label lists). Rows whose id is outside the dataset's 117-class set, or whose description is empty, were dropped. Duplicate descriptions within a split were collapsed to the first. - **Train**: from the dataset's `train` split (50,074 raw rows, 44,337 kept after dropping 4,390 multi-label or malformed and 1,347 duplicate rows), 700 rows were sampled with one row guaranteed per class and the remainder weighted by the square root of class frequency, so the head classes do not crowd out the tail that macro-F1 scores. All 117 classes are present. - **Validation**: from the dataset's own `validation` split (11,052 raw rows, 8,878 kept), 300 rows sampled at the natural class distribution. 68 classes appear; every one of them is in the training set; no description text is shared with the 700 training rows (0 overlaps measured). | | | |---|---| | Train samples | 700 rows / 117 classes | | Validation samples | 300 rows / 68 classes | | Text overlap | 0 rows | | Prompt format | chat template + system prompt + instruction/fenced-description user turn (see Intended use) | | Loss masking | answer tokens only; prompt tokens set to `-100` | | Truncation | prompt left-truncated to fit `max_seq_len` 2048; answer never truncated | The dataset's head is heavy: in the 300 validation rows CWE-79 appears 49 times, CWE-862 22, CWE-89 17, CWE-200 and CWE-22 13 each, and 68 classes share the rest. An LLM (Claude Opus 4.6, `claude-opus-4-6`, via an internal inference gateway) proposed the hyperparameters the search tried. It generated no training content and computed no metric. ### Method | | | |---|---| | SFT method | `qlora` | | Base quantisation during training | 4-bit NF4, double quantisation, bf16 compute (`bitsandbytes`) | | Refinement stage | none | | Attention implementation | `flash_attention_2` for the 8 GQA layers; the 24 Gated DeltaNet layers ran the PyTorch fallback because `flash-linear-attention` was not installed | | Auto class | `AutoModelForImageTextToText` (see How to get started) | | Hardware | 8x NVIDIA H100 80GB HBM3, `torchrun --nproc_per_node=8` | The project allowed two methods for this run (`bf16_lora`, `qlora`) and the search tried both; see the trial table below. No refinement stage ran; the published weights are the SFT adapter. ### Final hyperparameters | Hyperparameter | Value | Source | |---|---|---| | `learning_rate` | 0.0002 | `[TRAIN]` cmdline | | `lr_scheduler_type` | cosine | `[TRAIN]` cmdline | | `num_train_epochs` | 5 | `[TRAIN]` cmdline, `trainer_state.json` | | `per_device_train_batch_size` | 2 | `[TRAIN]` cmdline, `trainer_state.json` | | `gradient_accumulation_steps` | 4 | `[TRAIN]` cmdline | | `max_seq_length` | 2048 | `[TRAIN]` cmdline | | `warmup_ratio` | 0.05 | `[TRAIN]` cmdline | | `weight_decay` | 0.05 | `[TRAIN]` cmdline | | `loraplus_lr_ratio` | 2.0 (LoRA `B` matrices at 2x the learning rate) | `[TRAIN]` cmdline | | `lora_r` / `lora_alpha` / `lora_dropout` | 128 / 256 / 0.15 | `adapter_config.json` | | `use_rslora` / `use_dora` | `false` / `false` | `adapter_config.json` | | `target_modules` | the 10 listed in Model details | `adapter_config.json` | | `load_in_4bit` | `true` (NF4, double quant, bf16 compute) | `[TRAIN]` cmdline | **Effective batch size: 64** (`2 x 4 x 8`). Optimizer steps: 55 (11 per epoch). `neftune_noise_alpha` (0.0), `use_liger_kernel`, `use_sample_packing` and `lora_init` (`default`) were left at their no-op defaults. KD parameters are omitted deliberately: this is a `qlora` run, not a distillation run. > Provenance note: every value above was recovered from the platform database > (`runs`, `experiments`, `events` tables for run 3) and cross-checked against > the literal `[TRAIN]` command line recorded in the run log and against the > shipped `adapter_config.json`. ### How these values were chosen > These hyperparameters were selected by an automated search > (`autocatalyst.cli.run_autoresearch`): an agent proposes one change at a > time, runs train then eval, and keeps or discards on `f1_macro` (higher is > better). Run 3 ran **46 trials in 7 h 17 m** (2026-09-08 20:29 to 2026-09-09 03:46); 35 scored and 11 errored. This checkpoint is **trial 23**, the run's best. **Every trial is comparable.** The search's five "data strategies" all re-adapted the same fixed 700-row training file (the `train_v1` ... `train_v5` files below hold the same 700 rows) and every trial was scored on the same 300 validation rows, so the whole table is one comparison. | # | Method | rsLoRA | r | Dropout | LR | Epochs | Grad accum | Warmup | Weight decay | LoRA+ ratio | Data | f1_macro | Kept | |---|---|---|---|---|---|---|---|---|---|---|---|---|---| | 1 | bf16_lora | no | 16 | 0.05 | 2e-4 | 2 | 4 | 0.05 | 0.01 | 1.0 | train_v1 | 0.271028 | yes | | 2 | qlora | no | 16 | 0.05 | 2e-4 | 2 | 4 | 0.05 | 0.01 | 1.0 | train_v1 | 0.323589 | yes | | 3 | bf16_lora | no | 16 | 0.05 | 2e-4 | 2 | 4 | 0.05 | 0.01 | 1.0 | train_v1 | 0.271161 | no | | 4 | qlora | no | 16 | 0.05 | 1e-4 | 2 | 4 | 0.05 | 0.01 | 1.0 | train_v1 | 0.224824 | no | | 5 | qlora | no | 16 | 0.05 | 2e-4 | 5 | 4 | 0.05 | 0.01 | 1.0 | train_v1 | 0.328347 | yes | | 6 | qlora | no | 16 | 0.05 | 2e-4 | 8 | 4 | 0.05 | 0.01 | 1.0 | train_v1 | 0.315655 | no | | 7 | qlora | no | 32 | 0.05 | 2e-4 | 5 | 4 | 0.05 | 0.01 | 1.0 | train_v1 | 0.339796 | yes | | 8 | qlora | no | 32 | 0.15 | 2e-4 | 5 | 4 | 0.05 | 0.01 | 1.0 | train_v1 | 0.358649 | yes | | 9 | qlora | no | 32 | 0.15 | 2e-4 | 5 | 4 | 0.05 | 0.01 | 2.0 | train_v1 | 0.379137 | yes | | 10 | qlora | no | 64 | 0.15 | 2e-4 | 5 | 4 | 0.05 | 0.01 | 2.0 | train_v1 | 0.411651 | yes | | 11 | qlora | yes | 64 | 0.15 | 2e-4 | 5 | 4 | 0.05 | 0.01 | 1.0 | train_v1 | 0.385129 | no | | 12 | qlora | no | 64 | 0.05 | 2e-4 | 5 | 4 | 0.05 | 0.01 | 2.0 | train_v1 | 0.405714 | no | | 13 | qlora | no | 64 | 0.15 | 2e-4 | 5 | 8 | 0.05 | 0.01 | 2.0 | train_v1 | 0.377237 | no | | 14 | qlora | no | 128 | 0.15 | 2e-4 | 5 | 4 | 0.05 | 0.01 | 2.0 | train_v1 | 0.440878 | yes | | 15 | qlora | no | 128 | 0.15 | 2e-4 | 8 | 4 | 0.05 | 0.01 | 2.0 | train_v1 | 0.377763 | no | | 16 | qlora | yes | 128 | 0.15 | 2e-4 | 5 | 4 | 0.05 | 0.01 | 1.0 | train_v1 | error | no | | 17 | qlora | no | 128 | 0.15 | 1.5e-4 | 5 | 4 | 0.05 | 0.01 | 2.0 | train_v1 | 0.386976 | no | | 18 | qlora | yes | 128 | 0.15 | 2e-4 | 5 | 4 | 0.05 | 0.01 | 1.0 | train_v1 | error | no | | 19 | qlora | no | 128 | 0.15 | 2e-4 | 5 | 4 | 0.05 | 0.01 | 2.0 | train_v1 | 0.396817 | no | | 20 | qlora | no | 128 | 0.15 | 2e-4 | 8 | 4 | 0.05 | 0.01 | 2.0 | train_v2 | 0.407741 | no | | 21 | bf16_lora | no | 128 | 0.15 | 2e-4 | 5 | 4 | 0.05 | 0.01 | 2.0 | train_v2 | 0.403981 | no | | 22 | qlora | yes | 128 | 0.15 | 2e-4 | 5 | 4 | 0.05 | 0.01 | 1.0 | train_v2 | error | no | | **23** | **qlora** | **no** | **128** | **0.15** | **2e-4** | **5** | **4** | **0.05** | **0.05** | **2.0** | **train_v2** | **0.473288** | **yes** | | 24 | qlora | no | 128 | 0.10 | 2e-4 | 5 | 4 | 0.05 | 0.05 | 2.0 | train_v2 | 0.444999 | no | | 25 | qlora | yes | 128 | 0.15 | 2e-4 | 5 | 4 | 0.05 | 0.05 | 1.0 | train_v2 | error | no | | 26 | qlora | no | 128 | 0.15 | 2.5e-4 | 5 | 4 | 0.05 | 0.05 | 2.0 | train_v2 | 0.427152 | no | | 27 | qlora | yes | 128 | 0.15 | 2e-4 | 5 | 4 | 0.05 | 0.05 | 1.0 | train_v2 | error | no | | 28 | qlora | yes | 128 | 0.15 | 2e-4 | 5 | 4 | 0.05 | 0.05 | 1.0 | train_v2 | error | no | | 29 | qlora | yes | 128 | 0.15 | 2e-4 | 5 | 4 | 0.05 | 0.05 | 1.0 | train_v3 | error | no | | 30 | qlora | no | 128 | 0.15 | 2e-4 | 8 | 4 | 0.05 | 0.05 | 2.0 | train_v3 | 0.384342 | no | | 31 | bf16_lora | no | 128 | 0.15 | 2e-4 | 5 | 4 | 0.05 | 0.05 | 2.0 | train_v3 | 0.403688 | no | | 32 | qlora | no | 128 | 0.05 | 2e-4 | 5 | 4 | 0.05 | 0.05 | 2.0 | train_v3 | 0.415561 | no | | 33 | qlora | no | 128 | 0.15 | 1.75e-4 | 5 | 4 | 0.05 | 0.05 | 2.0 | train_v3 | 0.435022 | no | | 34 | qlora | no | 128 | 0.15 | 2e-4 | 5 | 4 | 0.03 | 0.05 | 2.0 | train_v3 | 0.416758 | no | | 35 | qlora | no | 128 | 0.15 | 2e-4 | 8 | 4 | 0.05 | 0.05 | 2.0 | train_v4 | 0.406259 | no | | 36 | bf16_lora | no | 128 | 0.15 | 2e-4 | 5 | 4 | 0.05 | 0.05 | 2.0 | train_v4 | 0.408250 | no | | 37 | qlora | yes | 128 | 0.15 | 2e-4 | 5 | 4 | 0.05 | 0.05 | 1.0 | train_v4 | error | no | | 38 | qlora | no | 128 | 0.15 | 2e-4 | 5 | 4 | 0.05 | 0.05 | 2.0 | train_v4 | 0.395110 | no | | 39 | qlora | no | 128 | 0.15 | 2e-4 | 5 | 4 | 0.05 | 0.01 | 2.0 | train_v4 | 0.437202 | no | | 40 | qlora | yes | 128 | 0.15 | 2e-4 | 5 | 4 | 0.05 | 0.05 | 1.0 | train_v4 | error | no | | 41 | qlora | no | 128 | 0.15 | 2e-4 | 8 | 4 | 0.05 | 0.05 | 2.0 | train_v5 | 0.391961 | no | | 42 | bf16_lora | no | 128 | 0.15 | 2e-4 | 5 | 4 | 0.05 | 0.05 | 2.0 | train_v5 | 0.428970 | no | | 43 | qlora | no | 128 | 0.05 | 2e-4 | 5 | 4 | 0.05 | 0.05 | 2.0 | train_v5 | 0.423615 | no | | 44 | qlora | yes | 128 | 0.15 | 2e-4 | 5 | 4 | 0.05 | 0.05 | 1.0 | train_v5 | error | no | | 45 | qlora | yes | 128 | 0.15 | 2e-4 | 5 | 4 | 0.05 | 0.05 | 1.0 | train_v5 | error | no | | 46 | qlora | no | 128 | 0.15 | 2e-4 | 5 | 8 | 0.05 | 0.05 | 2.0 | train_v5 | 0.402578 | no | All trials: `lora_alpha = 2 x r`, cosine schedule, batch 2 per device, `max_seq_len` 2048. **What the search actually established.** - **Rank dominates, up to a point.** Holding QLoRA, 5 epochs, dropout 0.15, LR 2e-4, weight decay 0.01, LoRA+ 2.0: `r` 32 → 64 → 128 moved macro-F1 0.379137 → 0.411651 → 0.440878 (#9, #10, #14). - **LoRA+ helped.** The one clean A/B at `r=32` (#8 → #9) moved 0.358649 → 0.379137 by putting the `B` matrices at 2x the learning rate. - **More epochs hurt.** 8 epochs lost to 5 every time it was tried at `r=128` (#15, #20, #30, #35, #41), by 0.01 to 0.06. - **4-bit versus bf16 base: no measurable difference.** At `r=16` the 4-bit base beat bf16 (#2 at 0.323589 against #1 and #3 at 0.271); at `r=128` with weight decay 0.05 the three bf16 runs (#31, #36, #42: 0.404 to 0.429) sit inside the two 4-bit runs of the same configuration (#38 at 0.395, #23 at 0.473). Quantising the base cost nothing this search could detect. - **rsLoRA at `r=128` diverges.** All 11 rsLoRA trials at `r=128` (#16, #18, #22, #25, #27, #28, #29, #37, #40, #44, #45) were aborted by the training watchdog at the third logging step with gradient norm above 10 (the configured limit; observed 12.3 in the first). The likely cause is the scaling rule: rsLoRA scales updates by `alpha / sqrt(r)` instead of `alpha / r`, which with `alpha = 2r` is `2 sqrt(r)`, about 22.6 at `r=128` against 2.0 without it, an 11x larger effective update. The one rsLoRA trial that finished (#11, `r=64`) scored below its non-rsLoRA sibling (#10). The 11 aborted trials are why the search shows 46 trials but 35 scores. **What it did not establish: the winning margin.** The search's decision "weight decay 0.01 → 0.05 improved 0.440878 → 0.473288" (#14 → #23) does not survive the re-runs. Identical configurations were run more than once because each "data strategy" restarted the inner loop on the same data: | Configuration (all QLoRA, r=128, dropout 0.15, LR 2e-4, 5 epochs, grad accum 4, warmup 0.05, LoRA+ 2.0) | Trials | f1_macro | Spread | |---|---|---|---| | weight decay 0.05 (**this checkpoint's config**) | #23, #38 | 0.473288, 0.395110 | 0.078 | | weight decay 0.01 | #14, #19, #39 | 0.440878, 0.396817, 0.437202 | 0.044 | | weight decay 0.05, bf16 base instead of 4-bit | #31, #36, #42 | 0.403688, 0.408250, 0.428970 | 0.025 | | weight decay 0.05, 8 epochs | #30, #35, #41 | 0.384342, 0.406259, 0.391961 | 0.022 | | weight decay 0.05, dropout 0.05 | #32, #43 | 0.415561, 0.423615 | 0.008 | Seeds were not pinned, so these differ only in initialisation and data order. The 0.078 gap between #23 and #38 is larger than the 0.032 "improvement" the search kept, and larger than most differences in the table. Read 0.473 as the high draw of a configuration whose expected score is somewhere in the low-to-mid 0.4s, and treat any two trials within about 0.05 of each other as tied. **Search space.** Six knobs were varied (`LORA_R`, `LORA_DROPOUT`, `LEARNING_RATE`, `EPOCHS`, `WEIGHT_DECAY`, `LORAPLUS_LR_RATIO`) plus one `GRAD_ACCUM` probe (#13, #46), one warmup probe (#34), the bf16-vs-4-bit method switch, and the rsLoRA attempts. `LR_SCHEDULER`, `MAX_SEQ_LEN`, `BATCH_SIZE`, `NEFTUNE_NOISE_ALPHA`, `USE_DORA`, `LORA_INIT`, `USE_LIGER_KERNEL` and `USE_SAMPLE_PACKING` were never moved. **Observed training metrics** (this checkpoint). | | | |---|---| | Final train loss (mean over the run) | 0.6052066683769226 | | Final eval loss (teacher-forced, answer tokens) | 0.9717274904251099 | | Eval mean token accuracy | 0.8136 | | Train runtime | 387.7625 s | | Total FLOPs | 4.311845712166912e+16 | | Throughput | 9.026 samples/s, 0.142 steps/s | 55 optimizer steps ran. The logged train loss fell from 1.2962 at step 10 (grad norm 1.06) to the run mean of 0.6052; the reported train loss is the mean over the run, not a converged value. ## Evaluation **Protocol.** All 300 validation rows, greedy decoding, `max_new_tokens=64`, prompts rendered through the chat template with `enable_thinking=False`. The project's `classification` evaluator takes the first line of the generation as the predicted label and compares it to the gold id. Generation ran through Hugging Face `generate` in batches of 4 (`max_input_len` 4096) after vLLM 0.19.1 refused the LoRA on this architecture and the evaluator fell back; the whole pass took 63 s on 8 GPUs. Every one of the 300 outputs was a single well-formed `CWE-` id, so no prediction was lost to formatting, and the 64-token budget cut nothing (the longest output is one id). | Metric | Value | |---|---| | Macro-F1, union of gold and predicted classes (**the search metric**) | 0.473288 | | Macro-F1, gold classes only | 0.563769 | | Accuracy (= micro-F1) | 0.71 (213 / 300) | | Classes in gold / predicted / union | 68 / 72 / 81 | **Two macro-F1 numbers, one convention.** The evaluator averages F1 over the *union* of gold and predicted classes, so the 13 classes the model predicted that never occur in the 300 gold rows each contribute an F1 of 0 and pull the macro average down from 0.5638 to 0.4733. Both are reported; the union convention is the one the search optimised and the one the published baselines below appear to use, but check before comparing. **Published comparison points.** The dataset authors report, on their own evaluation of the same dataset (a much larger split than these 300 rows): | Model | Method | Micro-F1 | Macro-F1 | |---|---|---|---| | Qwen3-32B ([exploitintel/cve-cwe-qwen3-32b](https://huggingface.co/exploitintel/cve-cwe-qwen3-32b)) | QLoRA r=16 | 0.729 | 0.595 | | Qwen3-8B (same author, same recipe) | QLoRA r=16 | 0.702 | 0.511 | | **This adapter** (Qwen3.5-9B, QLoRA r=128, 700 training rows) | | **0.71** | **0.473** | Those models trained on the full ~44K-row training split; this adapter saw 700 rows. The micro-F1 is in the same range; the macro-F1 is 0.04 to 0.12 lower, which is the long tail this adapter had one to twelve examples of per class to learn from. The evaluation sets also differ in size and composition, so read this as context, not a controlled comparison. **Sibling run on the same 700 / 300 split.** A Gemma 4 E4B adapter trained by the same platform on this split reached accuracy 0.697 and union macro-F1 0.4587, essentially tied with this checkpoint given the spread above. **Baseline for comparison. Not measured.** The untuned `Qwen/Qwen3.5-9B` was never scored on these 300 rows, so nothing here quantifies how much of the score the fine-tuning is responsible for. This is the most important gap in this card. **This is a validation split the search selected against.** 35 trials were scored on these same 300 rows and the best was kept, so expect optimistic bias on top of the re-run spread already described. The rows are drawn from the dataset authors' own validation split, so they are unseen CVEs from the same period as training, not future CVEs. **The evaluation set is reproducible.** `predictions.jsonl` in this repo holds every one of the 300 rows: instruction, description, prediction and gold. The builder's manifest (dataset revision, seed, drop counts, per-class counts) is summarised under Training details. ## Limitations and bias **One number, wide error bars.** The same configuration scored 0.473 and 0.395 in two runs. Anyone deploying this should re-evaluate on their own data rather than trust either figure. **No baseline, so no established gain.** See Evaluation. **Head classes dominate what accuracy measures.** CWE-79 alone is 16% of the validation rows. A model that got only the top ten classes right would post a respectable accuracy and a poor macro-F1; the two numbers here disagree by 0.24 for that reason. **Tail classes were barely trained.** 117 classes over 700 rows means many classes had a single training example. Expect the model to fall back to a frequent neighbour (CWE-20 for input validation issues, CWE-200 for disclosure) when the description is ambiguous, and to emit ids it saw rarely with low reliability. It also produced 13 ids in evaluation that never occur in the 300 gold rows; some may be reasonable alternative labels, some are wrong. **Only one label.** Real CVEs often carry two CWE ids (about 9% of the source dataset). The training data dropped those rows, so the model always commits to one. **Prompt shape is the contract.** Change the system prompt, the instruction sentence, the code fence, or enable thinking, and you are evaluating a model nobody measured. **Domain narrowness.** English CVE descriptions in NVD / CNA style. Advisories in other formats, other languages, source-code inputs and exploit write-ups are unmeasured. **Inherits all biases and limitations of the base model.** This adapter changes 3.3% of the parameters and was not evaluated for safety or fairness. The base model's own card governs those properties. ## Environmental impact | | | |---|---| | Hardware | 8x NVIDIA H100 80GB HBM3 | | Training time | 6.46 minutes (387.7625 s) | | Cloud provider / region | on-premise | Covers this trial only. The full 46-trial search that selected it took 7 h 17 m on the same hardware. ## Framework versions - PEFT 0.18.1 - TRL: 1.0.0 - Transformers: 5.7.0.dev0 (git main) - Pytorch: 2.5.1+cu121 - bitsandbytes: 0.49.2 - flash-attn: 2.8.3 - Python: 3.12 PEFT's version is the one recorded in `adapter_config.json` at save time; the rest are the pinned versions of the training environment. `transformers` is a git-main build: the `qwen3_5` architecture is not in the stable PyPI release. ## Citation ```bibtex @misc{qwen35_cve_cwe_lora_2026, title = {Qwen3.5-9B CVE-to-CWE Classifier LoRA}, author = {Banerjee, Aaron and Anbuselvan, Pooja and Jodhpurkar, Om}, year = {2026}, url = {https://huggingface.co/SASVAAI/Qwen-3.5-9B-CVE-to-CWE} } ```