File size: 3,038 Bytes
3292bf9 6d1e3e3 1d2f97a | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 | ---
license: apache-2.0
base_model: Qwen/Qwen3.5-4B
library_name: transformers
pipeline_tag: image-text-to-text
tags:
- qwen3_5_moe
- moe
- upcycled
- research
---
# Qwen3.5-4B-A3B-Upcycled
Experimental sparse-MoE initialization derived from `Qwen/Qwen3.5-4B`.
> **Research checkpoint, not a trained release.** The dense FFNs were compressed
> and the experts have not undergone continued pretraining or distillation.
> Expect quality loss relative to the base model. Do not treat benchmark results
> from the base model as results for this checkpoint.
## Architecture
| Property | Value |
|---|---:|
| Total parameters | 4,036,686,336 |
| Active parameters including vision | 2,998,596,096 |
| Experts per layer | 8 |
| Experts selected per token | 2 |
| Shared expert width | 1536 |
| Routed expert width | 704 |
The original 9,216-wide dense FFN is reduced to a 1,536-wide shared expert and
eight 704-wide routed experts with top-2 routing. Neurons are ranked per layer
by the product of their gate, up, and down projection norms. The strongest
shared and routed slices are retained. Routed experts start identically so the
untrained router does not make the initial function nondeterministic.
## Intended use
This checkpoint is intended as an initialization for router warm-up, knowledge
distillation from `Qwen/Qwen3.5-4B`, and continued pretraining. It is not
recommended for production or user-facing inference before recovery training
and evaluation.
Load with a Transformers release that provides `Qwen3_5MoeForConditionalGeneration`:
```python
from transformers import Qwen3_5MoeForConditionalGeneration, AutoProcessor
model = Qwen3_5MoeForConditionalGeneration.from_pretrained(
"sepsy070716/Qwen3.5-4B-A3B-Upcycled",
device_map="auto",
)
processor = AutoProcessor.from_pretrained("sepsy070716/Qwen3.5-4B-A3B-Upcycled")
```
See `conversion_manifest.json` and `neuron_selection.json` for reproducibility.
## Router warm-up v1
A router-only MPS warm-up artifact is published under
`research/router-warmup-v1/`. It updates 655,360 router parameters and leaves
all attention, expert, embedding, and vision weights untouched.
On 32 held-out FineWeb2 Korean documents (8,109 valid tokens), normalized
routing entropy improved from `0.99599` to `0.99748`, while the maximum/minimum
expert usage ratio improved from `1.528` to `1.390`. This adapter balances
routing but does **not** recover the quality lost by dense-FFN compression;
expert distillation is still required.
## Layerwise distillation pilots
Accepted layer adapters for depths 0, 16, and 31 are published under
`research/layer-distillation-pilots/`. Each was trained for 10 local updates
with the v1 router frozen. On eight held-out Korean documents, dense-FFN
relative MSE improved by 4.07%, 8.24%, and 8.33% respectively, with improvement
on all 24 document/layer comparisons.
These local reconstruction results are proof of direction, not an end-to-end
model benchmark. The root checkpoint has not been modified by the pilot
adapters.
|