NPU-MoE-VLM

A unified Mixture-of-Experts Vision-Language Model designed from scratch for Intel NPU hardware.

Why?

Intel NPUs (Arrow Lake 3720 and similar) have severe constraints:

• 4 MB SRAM (2 MB per NCE tile), software-managed, no cache
• <10 GB/s DMA bandwidth to system RAM
• FP16-only compute (6.5 TOPS), INT8 at 13 TOPS
• Static shapes only — no dynamic dimensions

Standard dense models are memory-bound on decode: a 1B model at INT4 needs 500 MB DMA'd through a 10 GB/s pipe every token → 50ms/token just moving data. Compute is negligible.

MoE solves this. With top-2 routing over 32 experts, only ~125M active params (63 MB INT4) need loading per token → 6.3ms DMA theoretical. Experts are sized to tile cleanly in NPU SRAM.

Architecture

d_model = 512
d_ff = 1024 per expert
n_experts = 32, top-k = 2
n_layers = 24 (all MoE)
Standard GQA attention (8 query heads, 2 KV heads)

Property	Value
Total params	1.25B
Active params per token	125.8M
INT4 deploy size	0.62 GB
Expert INT4 size	786 KB (fits 4 per NCE tile)
Tokenizer	SmolLM2 (49K vocab)

Unified Vision + Language

No separate ViT, no projection layer. One transformer:

Image → Conv2d patch embed → [visual tokens]
Text → Embedding lookup    → [text tokens]
         ↓                     ↓
    [visual_tokens, text_tokens]  ← single sequence
         ↓
    Shared MoE Transformer
         ↓
    Text logits

The patch embedding is a single nn.Conv2d(3, 512, kernel_size=16, stride=16). Cross-modal attention is free — it's just self-attention over the concatenated sequence. Experts naturally specialize between visual and linguistic processing.

Training

Streams everything from HuggingFace — zero local disk required.

pip install torch transformers datasets bitsandbytes pillow huggingface_hub
pip install --pre openvino openvino-genai openvino-tokenizers \
  --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/nightly

export HF_TOKEN=your_token_here
python3 train.py

• Text: HuggingFaceFW/fineweb (10B token sample), streamed
• Images: laion/laion-coco (30% of batches), streamed with on-the-fly download
• Optimizer: Adafactor (near-zero state, fits 8 GB VRAM)
• Precision: FP16 weights, gradient checkpointing
• Auto-resume: Picks up from local or HF Hub checkpoint (whichever is newer)
• Auto-push: Checkpoints to HuggingFace Hub every 1000 steps

NPU Deployment (WIP)

After training, the model exports to OpenVINO IR with INT4 weight compression and runs on Intel NPU via openvino_genai. The expert dimensions are designed to tile within the NPU's 2 MB SRAM per NCE tile for maximum DMA efficiency.

Hardware Reference

Spec	Intel NPU 3720 (Arrow Lake)
MACs	4096 INT8 / 2048 FP16 per cycle
Clock	~1.16 GHz
Peak	13.1 INT8 TOPS / 6.5 FP16 TOPS
SRAM	4 MB (2 MB × 2 NCE tiles)
DMA	<10 GB/s to system RAM
Compute types	FP16, INT8 only (no FP32)

License

Mozilla Public License 2.0 — commercially permissive copyleft.

Author

Two (Nhoodie) — independent researcher, Salem, Oregon.