---
library_name: transformers
license: apache-2.0
language:
  - en
pipeline_tag: text-generation
tags:
  - moe
  - code
  - qwen3
  - deepseek
  - speculative-decoding
  - 4-bit
  - consumer-gpu
base_model:
  - Qwen/Qwen3-4B
---

# Mini Whale 1 12B

A 12B-parameter Mixture-of-Experts model that fuses **Qwen3-4B** (host) with **260 coding experts** extracted from **DeepSeek-V4-Flash**, fine-tuned with QLoRA and optimized for local inference on consumer GPUs (RTX 3060 12GB).

---

## Table of Contents

1. [What is this model?](#what-is-this-model)
2. [Architecture (A to Z)](#architecture-a-to-z)
3. [Training](#training)
4. [Local Inference Guide](#local-inference-guide)
5. [DSpark Speculative Decoding](#dspark-speculative-decoding)
6. [Performance](#performance)
7. [File Structure](#file-structure)
8. [Citation](#citation)

---

## What is this model?

Mini Whale 1 12B is a **fusion architecture** that combines:

- **Host model**: Qwen3-4B (36 layers, hidden dim 2560, BF16) — handles language understanding, reasoning, and general text generation
- **Coding experts**: 260 SwiGLU experts from DeepSeek-V4-Flash (each ~25M params, hidden dim 4096) — specialized for code generation
- **Bridge layers**: Low-rank adapters (rank 7) that translate between Qwen3's 2560-dim representation space and DeepSeek's 4096-dim expert space
- **Router**: sqrt-softplus scoring with top-2 expert selection per token

The result is a model that **thinks like Qwen3** but **codes like DeepSeek** — all while fitting in 12GB VRAM with 4-bit quantization.

---

## Architecture (A to Z)

### Overview

```
Input Tokens
    │
    ▼
┌─────────────────────────────────────────┐
│  Qwen3-4B Host (36 layers, hidden=2560) │
│                                         │
│  For each layer:                        │
│  1. Self-Attention (SDPA, 32 heads)     │
│  2. Host MLP (SwiGLU, 9728 intermediate)│
│  3. [Augmented layers only]:            │
│     a. Bridge In (2560 → 4096)          │
│     b. Router (sqrt-softplus, top-2)    │
│     c. Expert SwiGLU (2 of N experts)   │
│     d. Bridge Out (4096 → 2560)         │
│     e. RMSNorm + Sigmoid Gate           │
│     f. Residual Repair (rank-7)         │
│     g. Residual Addition                │
│  4. RMSNorm                             │
└─────────────────────────────────────────┘
    │
    ▼
LM Head (tied embeddings, 151936 vocab)
    │
    ▼
Output Logits → Token
```

### Layer Structure (36 layers)

All 36 layers are **augmented** (have coding experts), but the number of experts per layer varies:

| Layers | Expert Count | Description |
|--------|-------------|-------------|
| 0-8    | 1-8         | Early layers: fewer experts, basic coding patterns |
| 9      | 1           | Minimal augmentation (dense-like) |
| 10-28  | 2-13        | Middle layers: moderate expert density |
| 29-35  | 8-25        | Deep layers: heavy expert density for complex code |

**Total experts across all layers: 303 expert instances** (260 unique experts, some shared across layers).

### Expert Computation

Each expert is a standard SwiGLU MLP:

```python
expert_output = down_proj(
    clamp(silu(gate_proj(x)) * up_proj(x), -10.0, 10.0)
)
```

The `clamp(-10, 10)` is critical — it matches DeepSeek-V4-Flash's native `swiglu_limit` and prevents activation explosion across layers.

### Bridge Layers

The bridge is the key innovation. It translates between two different representation spaces:

- **bridge_in**: `Linear(2560, 4096)` — maps Qwen3 hidden states to DeepSeek expert space
- **bridge_out**: `Linear(4096, 2560)` — maps expert outputs back to Qwen3 space

Both are low-rank (rank 7) to keep parameter count low and prevent the experts from overwhelming the host signal.

### Router

```python
logits = router.gate(hidden_states)                    # (tokens, num_experts)
scores = sqrt(clamp(softplus(logits), min=1e-6))      # sqrt-softplus scoring
topk_weights, topk_indices = scores.topk(2, dim=-1)   # top-2 selection
topk_weights = topk_weights / topk_weights.sum(dim=-1) # normalize
```

The sqrt-softplus scoring is smoother than softmax and prevents expert collapse (one expert dominating all tokens).

### Residual Safety

The coding delta is gated and clamped to prevent it from disrupting the host's representation:

```python
gate = sigmoid(coding_gate)          # learnable scalar, init -2.0 → ~12% gate
coding_delta = bridge_out(expert_output)
coding_delta = coding_norm(coding_delta)
coding_delta = coding_delta * gate

# Clamp to max_delta_ratio of host signal norm
scale = min(1.0, (host_norm * max_delta_ratio) / delta_norm)
coding_delta = coding_delta * scale

# Repair (rank-7 residual correction)
repair_delta = repair_up(repair_down(host_hidden))

# Final residual
output = host_hidden + coding_delta + repair_delta
```

This ensures the coding experts **contribute** to the output without **dominating** it. The host model's reasoning ability is preserved.

### Key Config

| Parameter | Value |
|-----------|-------|
| Host hidden size | 2560 |
| Host intermediate size | 9728 |
| Host layers | 36 |
| Host attention heads | 32 (Q) / 8 (KV) |
| Head dim | 128 |
| Expert hidden size | 4096 |
| Expert intermediate size | 2048 |
| Top-k experts | 2 |
| Bridge rank | 7 |
| Vocab size | 151,936 |
| Max position | 40,960 |
| RoPE theta | 1,000,000 |
| Total params | ~12B |
| Quantized size (4-bit) | 23.4 GB (BF16) / 8.9 GB (4-bit) |

---

## Training

### Stage 1: Bridge + Router Training (QLoRA)

- **Method**: QLoRA (4-bit NF4 host + trainable LoRA adapters on attention + bridge layers)
- **Trainable params**: 19.5M (bridge_in, bridge_out, router.gate, repair, q/k/v/o_proj LoRA)
- **Dataset**: Coding instruction dataset (Python, JavaScript, TypeScript)
- **Steps**: 500
- **Loss**: 0.62 → 0.19
- **Optimizer**: AdamW, lr=2e-4, cosine schedule
- **Hardware**: RTX 3060 12GB (local)

### Stage 2: Merge

LoRA adapters were merged into the base model using streaming tensor-by-tensor merge:

- 216 LoRA modules merged (B = A @ B for each adapter)
- 144 checkpoint overrides applied
- 1285 tensors copied unchanged
- Output: 23.4 GB BF16 model (5 shards)

### Critical Runtime Fixes

Two fixes are required at inference time (applied automatically by the model code):

1. **SwiGLU clamping** (`limit=10.0`): DeepSeek-V4-Flash uses `swiglu_limit=10.0` to clamp expert intermediate activations. Without this, outlier values grow exponentially across layers.

2. **Router stability**: `softplus` is clamped to `min=1e-6` before `sqrt()` to prevent NaN gradients.

---

## Local Inference Guide

### Requirements

| Component | Minimum | Recommended |
|-----------|---------|-------------|
| GPU | RTX 3060 12GB | RTX 4070 16GB+ |
| RAM | 16GB | 32GB |
| Python | 3.10+ | 3.11+ |
| CUDA | 12.1+ | 12.6+ |
| VRAM (4-bit) | 8.9 GB | — |
| VRAM (4-bit + DSpark) | 10.9 GB | — |

### Installation

```bash
# Create virtual environment
python -m venv fuse2-venv
fuse2-venv\Scripts\activate

# Install dependencies
pip install torch --index-url https://download.pytorch.org/whl/cu126
pip install transformers==5.14.1 bitsandbytes==0.49 peft==0.20 accelerate
pip install safetensors
```

### Download the Model

```bash
pip install huggingface_hub
huggingface-cli download Akahsizrr/Mini-Whale-1-12B --local-dir ./mini-whale-1-12b
```

### Quick Start: One-Liner 4-bit Loading (~5 tok/s)

The model has all runtime fixes **baked into the model code** (`fuse2_model.py`):
- SwiGLU clamping (limit=10.0) — built into `SwiGLUExpert.forward`
- Router stability (softplus clamp) — built into `Fuse2Router.forward`
- Meta param auto-init + norm dtype fix — handled by `from_pretrained` override

So you can load and generate with zero post-load patching:

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

MODEL = "Akahsizrr/Mini-Whale-1-12B"

# 4-bit NF4 quantization (matches config.json quantization_config)
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)

# Load model — runtime fixes applied automatically by from_pretrained override
model = AutoModelForCausalLM.from_pretrained(
    MODEL,
    trust_remote_code=True,
    quantization_config=bnb_config,
    device_map="auto",
    attn_implementation="sdpa",
)
model.eval()

# Tokenizer
tok = AutoTokenizer.from_pretrained(MODEL, trust_remote_code=True)

# Generate
messages = [{"role": "user", "content": "Write a Python fizzbuzz."}]
text = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
input_ids = tok(text, return_tensors="pt").input_ids.to(model.device)

with torch.inference_mode():
    out = model.generate(input_ids, max_new_tokens=512, do_sample=False,
                         repetition_penalty=1.3)
print(tok.decode(out[0], skip_special_tokens=True))
```

That's it — no manual SwiGLU patching, no meta device fixes, no norm dtype casting. The model code handles all of it.

### Quick Start: Manual 4-bit Loading (~5 tok/s)

For more control over the loading process (e.g., selective quantization of attention vs MLP layers):

```python
import torch, json, os, sys, gc
from transformers import AutoConfig, AutoTokenizer
from accelerate import init_empty_weights
from safetensors import safe_open
import bitsandbytes as bnb

MODEL_PATH = "./mini-whale-1-12b"
DEVICE = "cuda:0"
sys.path.insert(0, MODEL_PATH)
import fuse2_model_local

# Load config with SDPA attention
config = AutoConfig.from_pretrained(MODEL_PATH, trust_remote_code=True)
config._attn_implementation = "sdpa"

# Create model on meta device (saves RAM)
with init_empty_weights():
    model = fuse2_model_local.Fuse2ForCausalLM(config)

# Load weights with 4-bit quantization
quant_suffixes = ("q_proj.weight", "k_proj.weight", "v_proj.weight", "o_proj.weight",
                  "gate_proj.weight", "up_proj.weight", "down_proj.weight")

with open(f"{MODEL_PATH}/model.safetensors.index.json") as f:
    index = json.load(f)
weight_map = index["weight_map"]

def nav(model, key):
    parts = key.split(".")
    obj = model
    for p in parts[:-1]:
        obj = obj[int(p)] if p.isdigit() else getattr(obj, p)
    return obj, parts[-1]

def find_linear(model, key):
    parts = key.split(".")
    obj = model
    for p in parts[:-2]:
        obj = obj[int(p)] if p.isdigit() else getattr(obj, p)
    return obj, parts[-2]

param_names = set(dict(model.named_parameters()).keys())
replaced = {}

for shard_name in sorted(set(weight_map.values())):
    with safe_open(os.path.join(MODEL_PATH, shard_name), framework="pt", device="cpu") as f:
        for key in [k for k, v in weight_map.items() if v == shard_name]:
            if key not in param_names:
                continue
            tensor = f.get_tensor(key)
            if any(key.endswith(s) for s in quant_suffixes):
                owner, attr = find_linear(model, key)
                old = getattr(owner, attr)
                new = bnb.nn.Linear4bit(old.in_features, old.out_features,
                                        bias=False, quant_type="nf4",
                                        compute_dtype=torch.bfloat16, device=DEVICE)
                new.weight = bnb.nn.Params4bit(tensor.to(torch.bfloat16),
                                               requires_grad=False, quant_type="nf4").cuda(0)
                setattr(owner, attr, new)
            else:
                parent, pname = nav(model, key)
                parent._parameters[pname] = torch.nn.Parameter(
                    tensor.to(torch.bfloat16).to(DEVICE), requires_grad=False)
            del tensor
    gc.collect(); torch.cuda.empty_cache()

# Fix tied embeddings
if model.lm_head.weight.device.type == 'meta':
    model.lm_head.weight = torch.nn.Parameter(
        model.model.embed_tokens.weight.data.clone(), requires_grad=False)

# Apply runtime fixes (SwiGLU clamp + router stability)
from fuse2_model_local import Fuse2AugmentedLayer
import torch.nn.functional as F

for layer in model.model.layers:
    if not isinstance(layer, Fuse2AugmentedLayer):
        continue
    if hasattr(layer, 'coding_gate') and layer.coding_gate.device.type == 'meta':
        layer.coding_gate = torch.nn.Parameter(torch.tensor(-2.0, device=DEVICE))
    if hasattr(layer, 'coding_norm') and layer.coding_norm.weight.device.type == 'meta':
        layer.coding_norm = torch.nn.RMSNorm(
            layer.coding_norm.weight.shape[0], eps=1e-6).to(DEVICE)
    experts = getattr(layer, "experts", None)
    if experts:
        for expert in experts:
            gp, up, dp = expert.gate_proj, expert.up_proj, expert.down_proj
            def make_fwd(g, u, d, lim=10.0):
                def forward(x):
                    return d(torch.clamp(F.silu(g(x)) * u(x), -lim, lim))
                return forward
            expert.forward = make_fwd(gp, up, dp)
    router = getattr(layer, "router", None)
    if router:
        gate, top_k = router.gate, router.top_k
        def make_router(g, tk):
            def forward(h):
                logits = g(h)
                scores = torch.clamp(F.softplus(logits), min=1e-6).sqrt()
                w, idx = scores.topk(tk, dim=-1)
                return w / (w.sum(dim=-1, keepdim=True) + 1e-8), idx, logits
            return forward
        router.forward = make_router(gate, top_k)

model.set_coding_enabled(True)
model.to(DEVICE)
model.eval()

# Fix norm dtypes (float32 → bfloat16 for fused kernels)
for module in model.modules():
    if hasattr(module, 'weight') and hasattr(module, 'eps'):
        if module.weight.dtype == torch.float32:
            module.weight.data = module.weight.data.to(torch.bfloat16)

# Tokenizer
tok = AutoTokenizer.from_pretrained(MODEL_PATH, trust_remote_code=True)

# Generate
messages = [{"role": "user", "content": "Write a Python fizzbuzz."}]
text = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
input_ids = tok(text, return_tensors="pt").input_ids.to(DEVICE)

with torch.inference_mode():
    out = model.generate(input_ids, max_new_tokens=512, do_sample=False,
                         repetition_penalty=1.3)
print(tok.decode(out[0], skip_special_tokens=True))
```

### DSpark Speculative Decoding (~10 tok/s)

For 2x speedup, use DSpark speculative decoding. This requires the drafter model:

```bash
# The drafter is included in the model directory
# It's a 5-layer Qwen3 model that predicts 7 tokens at once
```

See the `fuse2_dspark_fast.py` script (included in this repo) for the full implementation.

### VRAM Breakdown

| Configuration | VRAM (allocated) | VRAM (reserved) | Speed |
|--------------|------------------|-----------------|-------|
| 4-bit target only | 8.9 GB | 9.4 GB | ~5 tok/s |
| 4-bit target + BF16 drafter | 11.7 GB | 12.2 GB | **OVERFLOW** |
| 4-bit target + 4-bit drafter | 10.3 GB | 10.8 GB | ~8 tok/s |
| BF16-attn target + 4-bit drafter (shared embed) | 10.9 GB | 11.4 GB | **~10 tok/s** |

**Critical**: On RTX 3060 12GB, Windows reserves ~0.5GB for display. If reserved VRAM exceeds ~11.5GB, CUDA silently spills to system RAM via PCIe, causing a **50x slowdown**. Always check `torch.cuda.memory_reserved(0)`.

---

## DSpark Speculative Decoding

This model supports **DSpark** (Draft Speculative) decoding for 2x speedup:

### How it works

1. **Drafter** (5-layer Qwen3, 2560 hidden): Predicts 7 tokens in a single forward pass using masked attention + target hidden states
2. **Target** (12B Fuse-2): Verifies all 7 tokens in ONE forward pass
3. **Greedy verification**: Accept the longest prefix where drafter matches target's argmax
4. **Bonus token**: Target always produces 1 bonus token (even if all 7 are rejected)

### Key optimizations

- **Forward hooks** on 5 target layers (not `output_hidden_states=True` on all 36)
- **Single target forward** per block (no sequential fallback)
- **4-bit drafter** to save VRAM
- **Shared embedding** between target and drafter (saves 0.8 GB)
- **Sliding window KV cache** (512 tokens) for constant speed on long sequences

### Acceptance rate

- Short prompts: 3.0-3.4/7 (43-49%)
- Long prompts: 2.3-2.5/7 (33-36%)
- Code patterns: up to 4.5/7 (64%)

---

## Performance

### Speed (RTX 3060 12GB, 4-bit NF4)

| Mode | Speed (short) | Speed (long, 512+ tok) | VRAM |
|------|--------------|------------------------|------|
| Basic generation | 5.5 tok/s | 5.5 tok/s | 8.9 GB |
| DSpark speculative | 10.0 tok/s | 8.5 tok/s | 10.9 GB |

### Quality

The model produces:
- **Reasoning**: Qwen3-quality chain-of-thought (the host handles this)
- **Code**: DeepSeek-quality code generation (the experts handle this)
- **Mixed**: Seamless switching between reasoning and code

### Example output

Prompt: "Write a Python function to check if a number is prime."

Output (excerpt):
```
Okay, I need to write a Python function to check if a number is prime.
Let me think about how to approach this.

First, a prime number is a number greater than 1 that has no divisors
other than 1 and itself...

1. Check if the number is less than 2 → return False.
2. Check if the number is 2 → return True
3. Check if the number is even → if yes, return False
4. Iterate from 3 to sqrt(n), stepping by 2
5. For each i, check if n is divisible by i
6. If none divide n, return True
```

---

## File Structure

```
mini-whale-1-12b/
├── config.json                      # Model configuration
├── generation_config.json           # Generation defaults
├── model-00001-of-00005.safetensors # Shard 1 (5.1 GB)
├── model-00002-of-00005.safetensors # Shard 2 (5.1 GB)
├── model-00003-of-00005.safetensors # Shard 3 (5.1 GB)
├── model-00004-of-00005.safetensors # Shard 4 (5.1 GB)
├── model-00005-of-00005.safetensors # Shard 5 (3.5 GB)
├── model.safetensors.index.json     # Shard index
├── fuse2_model.py                   # Model code (HF auto_map)
├── fuse2_model_local.py             # Model code (local import)
├── tokenizer.json                   # Tokenizer
├── tokenizer_config.json            # Tokenizer config
├── vocab.json                       # Vocabulary
├── merges.txt                       # BPE merges
└── README.md                        # This file
```

---

## Citation

```bibtex
@misc{miniwhale1,
  title={Mini Whale 1 12B: Fusing Qwen3-4B with DeepSeek Coding Experts},
  author={Vasko Djack},
  year={2026},
  url={https://huggingface.co/Akahsizrr/Mini-Whale-1-12B}
}
```

---

## License

Apache 2.0 — see LICENSE file for details.

## Acknowledgments

- **Qwen3-4B** by Alibaba/Qwen Team — host model
- **DeepSeek-V4-Flash** by DeepSeek AI — coding experts
- **DSpark** speculative decoding framework
- **bitsandbytes** for 4-bit NF4 quantization