File size: 2,178 Bytes
bb759b7 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 | ---
license: apache-2.0
base_model: mlx-community/Qwen3.5-4B-4bit
tags:
- roblox
- luau
- lora
- mlx
- archived
- experimental
---
# Vertigo-Qwen3.5-4B-v0.5 (Archived Experiment)
> **⚠️ ARCHIVED — This model is an early experiment and should NOT be used in production.**
> It is preserved here for research reproducibility only.
## What This Is
An early LoRA fine-tune of Qwen3.5-4B for Roblox Luau code generation, trained as part of the [Vertigo](https://github.com/adpena/vertigo) game engine project.
## Why It's Archived
Rigorous execution-backed evaluation revealed that this adapter provides **zero improvement** in functional code generation over the base model:
| Metric | Base Qwen3.5-4B | This Adapter |
|---|---|---|
| Pattern-match score | 43.8% | 48.0% (+4.2pp) |
| **Execution pass@1** | **9.5%** | **9.7%** (not significant) |
The pattern-match improvement (+4.2pp) measures keyword presence, not working code. When evaluated on whether generated code actually **compiles and passes test assertions**, the adapter shows no improvement.
## Key Findings
1. **LoRA SFT on pattern-match-scored data does not improve execution pass rate** — the training signal was disconnected from functional correctness
2. **The Qwen3.5-4B hybrid architecture** (24 GatedDeltaNet + 8 standard attention layers) requires targeting both `self_attn` and `linear_attn` LoRA keys
3. **An agentic repair loop on the larger 35B-A3B model** achieves 43.3% execution pass rate on calibrated tasks — without any training
Full technical report: [vertigo-ml/docs/technical-report-lora-v1.md](https://github.com/adpena/vertigo-ml/blob/main/lora/docs/technical-report-lora-v1.md)
## Training Details
- Base: `mlx-community/Qwen3.5-4B-4bit`
- Method: LoRA rank-32, scale 20.0 (later found to be 10x too high)
- Data: ~2200 examples, pattern-match quality filtered
- Hardware: 128GB Apple Silicon, MLX
## Don't Use This Model
Use the base `Qwen3.5-4B` model directly, or better yet, use a larger model (35B-A3B) with an agentic repair loop. See the [vertigo-ml](https://github.com/adpena/vertigo-ml) repository for the benchmark and evaluation infrastructure.
|