README.md

---
# ---- Identity -------------------------------------------------------------
base_model: google/gemma-4-12B
base_model_relation: adapter
library_name: peft
pipeline_tag: text-generation
language:
  - en
# verified: the base model's card declares `license: apache-2.0` with this
# link, and the repo is ungated. The standard Apache-2.0 text is shipped here
# as LICENSE (Sec. 4(a)); the base repo itself ships no LICENSE file.
license: apache-2.0
license_link: https://ai.google.dev/gemma/docs/gemma_4_license

# ---- Discovery ------------------------------------------------------------
tags:
  - lora
  - sft
  - trl
  - peft
  - bf16_lora
  - defect-detection
  - code

datasets:
  - google/code_x_glue_cc_defect_detection

metrics:
  - f1
  - accuracy

# ---- Structured evaluation ------------------------------------------------
model-index:
  - name: gemma4-12b-defect-detection
    results:
      - task:
          type: text-generation
          name: Binary security-defect classification of C functions
        dataset:
          type: google/code_x_glue_cc_defect_detection
          name: CodeXGLUE CC Defect Detection (246-row random holdout, see Evaluation)
          split: validation
        metrics:
          - type: f1
            name: F1 Macro
            value: 0.5429997560563697
            args:
              average: macro
          - type: f1
            name: F1 Micro
            value: 0.5975609756097561
            args:
              average: micro
          - type: accuracy
            name: Accuracy
            value: 0.5975609756097561
---

# gemma-4-12B Defect Detection (LoRA)

Classifies the source of a single C function as `defective` or `clean`, for
triaging C code for security defects.

This is a **LoRA adapter for**
[google/gemma-4-12B](https://huggingface.co/google/gemma-4-12B), trained with
**bf16 LoRA (no quantisation)** via [TRL](https://github.com/huggingface/trl) SFT.

## Model details

| | |
|---|---|
| Developed by | SASVA AI Model Cognition Labs(MCL) Team |
| Base model | [`google/gemma-4-12B`](https://huggingface.co/google/gemma-4-12B) |
| Base parameters | 11,959,730,224 (11.96B) |
| Architecture family | `gemma4_unified` |
| Adaptation | LoRA (`r=16`, `alpha=32`, `dropout=0.05`) |
| Trainable modules | `q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj` |
| Training method | `bf16_lora` |
| Refinement | none |
| Precision | bf16 base (no quantisation), bf16 compute |
| Language | English (C source and identifiers) |
| License | Apache-2.0 (inherited from the base model) |

Trainable parameters: **65,568,768** across 328 modules — 0.548% of the base.
The adapter file is 262,373,216 bytes (656 tensors: `lora_A` + `lora_B` per
module). All 656 sit under `model.language_model`; the base model's vision and
audio towers are loaded but untouched. `exclude_modules` is `null` in
`adapter_config.json` and none was needed — unlike the 26B/31B Gemma 4 variants,
this unified checkpoint has no `vision_tower` submodule for the seven names to
match against.

Of the 48 text layers, **40 carry a `v_proj` adapter and 8 do not**. This is not
a gap in coverage: layers 5, 11, 17, 23, 29, 35, 41 and 47 are the
`full_attention` layers (`config.json → text_config.layer_types`), and the base
model sets `attention_k_eq_v: true`, so those layers ship no separate `v_proj`
weight at all. Every `v_proj` that exists in the base — 40 of 40 — is adapted.

## Intended use

**Direct use.** Binary classification of one C function at a time. The model was
trained on a specific prompt shape and **that shape is part of the contract**:

- `google/gemma-4-12B` ships **no chat template** (`tokenizer.chat_template` is
  `null` on both the base and this adapter's tokenizer). Training and evaluation
  therefore used a plain-text prompt, assembled by hand — see
  [How to get started](#how-to-get-started) for the exact string. Do **not**
  call `apply_chat_template`; there is nothing for it to apply.
- The instruction is fixed and verbatim: *"Classify whether the following C
  function contains a security defect. Answer with exactly one word: defective
  or clean."*
- The function source is wrapped in a triple-backtick fence.
- The label is the **first line** of the generation, lowercased; discard
  anything after it.

**Out of scope.**
- **Not a security gate.** A `clean` verdict is weak evidence. Use the model to
  prioritise review, not to sign off on code.
- Localising, explaining, or repairing a defect. It emits one word and no
  rationale, and was never trained to produce an explanation.
- Any specific CWE or vulnerability class. The label is an undifferentiated
  binary inherited from the CodeXGLUE corpus.
- Languages other than C, and multi-function or whole-file inputs. Every
  training and eval example is a single C function.
- Not a general-purpose assistant. It emits a bare label, never prose, and will
  degrade on open-ended chat.

## How to get started

```python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

BASE = "google/gemma-4-12B"
ADAPTER = "SASVAAI/Gemma4-12b-defect-detection"

tokenizer = AutoTokenizer.from_pretrained(ADAPTER)
model = AutoModelForCausalLM.from_pretrained(BASE, dtype="bfloat16", device_map="auto")
model = PeftModel.from_pretrained(model, ADAPTER)
model.eval()

func = """static void unix_process_msgfd(CharDriverState *chr, struct msghdr *msg)
{
    TCPCharDriver *s = chr->opaque;
    struct cmsghdr *cmsg;
    /* ... */
}"""

INSTRUCTION = (
    "Classify whether the following C function contains a security defect. "
    "Answer with exactly one word: defective or clean."
)
SYSTEM = (
    "You are a security auditor for C/C++ code. Given a function, determine "
    "whether it contains a security vulnerability (such as missing bounds "
    "checks, NULL pointer dereferences, buffer overflows, use-after-free, "
    "integer overflows, or other subtle defects). Respond with exactly one "
    "word: `defective` if the function contains a vulnerability, or `clean` "
    "if it does not."
)

# This model has NO chat template. The prompt is this literal string — the same
# one the training and eval harness built. Reproduce it exactly.
prompt = f"<|system|>\n{SYSTEM}\n<|user|>\n{INSTRUCTION}\n\n```\n{func}\n```\n<|assistant|>"

inputs = tokenizer(
    prompt, return_tensors="pt", truncation=True, max_length=4096,
).to(model.device)

out = model.generate(**inputs, max_new_tokens=16, do_sample=False)
text = tokenizer.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True)
print(text.strip().splitlines()[0].strip().lower())
# -> clean
```

> **Decoding matters.** This model was evaluated with greedy decoding
> (`do_sample=False`, `max_new_tokens=16`) inside a **4096-token input
> window**. A prompt that overflows that window is not truncated at either
> end — both ends are load bearing, the instruction at the head and the
> `<|assistant|>` cue at the tail — so the harness elides the *middle of the
> function source* and substitutes the literal marker
> `/* ... source elided ... */`, keeping the surrounding scaffolding intact.
> One of the 246 holdout rows needed this. Sampling will not reproduce the
> reported numbers.

The base model is ~24 GB in bfloat16 and fits on one 80 GB-class GPU.

## Training details

**Data.** [CodeXGLUE CC Defect Detection](https://huggingface.co/datasets/google/code_x_glue_cc_defect_detection)
(Devign — C functions from QEMU and FFmpeg), rendered to flat
`{instruction, input, output}` JSONL: `instruction` is the fixed directive
above, `input` is the raw `func` field, `output` is the binary `target` rendered
as the word `defective` or `clean`.

**The upstream train and validation splits were concatenated and re-split.** The
harness pooled them into 24,586 rows and drew a 1% random holdout
(`val_split_ratio = 0.01`), giving 24,340 train / 246 validation.

| | |
|---|---|
| Train rows in the split file | 24,340 |
| Validation rows | 246 |
| Prompt format | plain text, no chat template (see Intended use) |
| Truncation | right (tokenizer default), at 2048 tokens; rows whose rendered prompt overran the cap were dropped rather than truncated |
| Loss | full-sequence SFT (not answer-only masking) |

The loss is computed over the whole sequence, including the C source — the model
was trained to predict the function text as well as the label. That is part of
why the prompt shape is load-bearing.

### Method

| | |
|---|---|
| SFT method | `bf16_lora` |
| Base quantisation during training | none — bf16 base, bf16 compute |
| Base quantisation during evaluation | none |
| Refinement stage | none |
| Hardware | 4x NVIDIA H200 (141 GB), `torchrun --nproc_per_node=4` |

`bf16_lora` trains *and* evaluates without quantisation, so the score below is a
clean bf16 measurement with no quantise/dequantise mismatch between training and
inference.

No refinement stage ran; the published weights are the SFT adapter.

### Final hyperparameters

| Hyperparameter | Value | Source |
|---|---|---|
| `learning_rate` | 0.0002 | run record |
| `lr_scheduler_type` | cosine | run record |
| `num_train_epochs` | 2 | `trainer_state.json` |
| `per_device_train_batch_size` | 1 | `trainer_state.json` |
| `gradient_accumulation_steps` | 4 | run record |
| `max_seq_length` | 2048 | run record |
| `warmup_ratio` | 0.05 | run record |
| `weight_decay` | 0.01 | run record |
| `lora_r` / `lora_alpha` / `lora_dropout` | 16 / 32 / 0.05 | `adapter_config.json` |
| `target_modules` | the 7 listed in Model details | `adapter_config.json` |

**Effective batch size: 16** (`1 x 4 x 4`). Optimizer steps: 2,918.

Every remaining knob sat at its default and is omitted rather than printed:
`use_dora`, `use_rslora`, `use_liger_kernel` and `use_sample_packing` were all
false, `lora_init` was `default`, `loraplus_lr_ratio` 1.0 and
`neftune_noise_alpha` 0.0. KD parameters are omitted deliberately — this is a
`bf16_lora` run, not a KD run, so `kd_alpha`/`kd_beta`/`kd_temperature` carry
inert defaults that would imply distillation that did not happen.

**Observed training metrics.**

| | |
|---|---|
| Final train loss | 0.40944306608583114 |
| Final eval loss | 0.378094345331192 |
| Eval mean token accuracy | 0.8924608574075213 |
| Train runtime | 8768.7437s |
| Total FLOPs | 1.9065004466718638e+18 |
| Throughput | 5.322 samples/s, 0.333 steps/s |

Loss falls from 1.2440 at step 10 to 0.5481 by step 50, then improves slowly:
0.4544 at the half-epoch mark, 0.4099 at the end of epoch 1, and 0.3561 by step
2,910. Most of the task is learned in the first 2% of epoch 1; the second epoch
buys roughly 0.05 of training loss.

## Evaluation

**Protocol.** All 246 holdout rows, no sampling. Predictions generated greedily
(`do_sample=False`, `max_new_tokens=16`) with the base loaded in bf16 and the
adapter applied — the adapter is **not** merged before evaluation. The predicted
label is the first line of the generation, lowercased and matched against the
two valid labels; anything matching neither counts as unparsed and scores as
wrong. F1-macro is computed over exactly the two labels present in the
references, so absent classes cannot inflate it. Prompts were built inside a
4096-token input window with middle-elision on overflow (1 of the 246 rows),
and generated in left-padded batches over length-sorted prompts.

| Metric | Value |
|---|---|
| F1 Macro | 0.5429997560563697 |
| F1 Micro | 0.5975609756097561 |
| Accuracy | 0.5975609756097561 |
| Unparsed rate | 0.0 |
| Samples | 246 |

**This is a validation split, not a held-out test set.** Model selection used
this split, so expect some optimistic bias. It is also small (246 examples) —
expect sampling noise on the order of +/-0.03 F1-macro. And because the upstream
train and validation sets were pooled before the holdout was drawn, these
numbers are not comparable to published CodeXGLUE results.

## Limitations and bias

**Domain narrowness.** C functions from the Devign corpus (QEMU and FFmpeg).
Expect a substantial drop on other codebases, other C styles, and any other
language — none of which were measured.

**Inherits all biases and limitations of the base model.** This adapter changes
0.548% of the parameters and was not evaluated for social bias, safety, or
fairness across demographic groups. It is a code classifier and was not assessed
for adversarial inputs — a function crafted to look benign is untested.

## Environmental impact

| | |
|---|---|
| Hardware | 4x NVIDIA H200 (141 GB) |
| Training time | 146.1 minutes (8768.7437s) |
| Cloud provider / region | on-premise |

Covers this run only.

## Framework versions

- PEFT 0.18.1
- TRL: 1.0.0
- Transformers: 5.16.0.dev0
- Pytorch: 2.5.1+cu121
- Datasets: 4.8.4
- Tokenizers: 0.23.1

`transformers` is a git-main build: `gemma4_unified` is not in the stable PyPI
release.

## Citation

```bibtex
@misc{gemma4_12b_defect_detection_2026,
  title  = {gemma-4-12B Defect Detection (LoRA)},
  author = {{SASVA AI Model Cognition Labs (MCL) Team}},
  year   = {2026},
  url    = {https://huggingface.co/SASVAAI/Gemma4-12b-defect-detection}
}
```