README.md

---

license: mit
tags:
- amd
- ryzen-ai
- npu
- xdna2
- fastflowlm
- strix-halo
- lora
- fine-tuning
- tools
pipeline_tag: text-generation
---


# NPU-Forge 🔥 — fine-tune a model and put it on your AMD Ryzen AI NPU in ~3 minutes

**Measured, on a Strix Halo (Ryzen AI MAX+ 395), June 2026:** a LoRA fine-tune of
Llama-3.2-1B on 300+ real chat exchanges — trained, merged, behavior-verified,
converted to GGUF, re-quantized to FastFlowLM's Q4NX, NPU-ready — in
**183 seconds of cloud time** (≈ $0.10 on a rented T4):

```

forge tune my-chats.jsonl --name grandma   # proven: coherent + IN-BAND on NPU

  ├─ LoRA fine-tune (cloud GPU)        122 s

  ├─ merge                               3 s

  ├─ voice proof (model speaks first!)   5 s

  ├─ HF -> f16 -> Q4_K_M GGUF           20 s

  └─ GGUF -> Q4NX (NPU format)          24 s

forge register        (one UAC click)

flm run grandma-forge:1b

```

The "voice proof" stage generates a sample from the merged model *inside the

training job*, before any conversion — so you know the tune actually took.
Ours came back with the persona's exact ritual phrases after 2 minutes of
training. That's the bar.

## What's in this repo

- **`forge.js` / `forge.bat`** — the CLI: `tune`, `convert`, `register`,
  `list`, `doctor`, `serve`
- **`modal/tune_npu.py`** — the whole tune→NPU pipeline as one

  [Modal](https://modal.com) job (bring your own Modal account; T4 is plenty)

- **`modal/convert_q4nx.py`** — just the GGUF→Q4NX stage (65 s for a 1B)
- **`bin/assemble.js`** — downloads results and stages the FLM model folder
- **`bin/register.js` + `register-admin.bat`** — the permanent custom-model
  registry that survives FLM updates (see below)
- **`registry.example.json`** — entry template

Chat data format: one JSON per line, `{"messages":[{"role":"user","content":...},{"role":"assistant","content":...}]}`.

## The registry problem (why `forge register` exists)

FLM's `model_list.json` lives in `C:\Program Files\flm\` and **every FLM update

resets it**, silently de-registering all your custom models. Your model files
survive (they're in `Documents\flm\models\`) but they vanish from `flm list`.
Forge keeps its own user-space `registry.json` forever and re-merges with one
click. `forge doctor` tells you when an update has eaten your registrations.


## NEW in v0.3 — a voice-verifier "ear" that runs on the NPU

Train a ~111KB classification head over EmbeddingGemma-300m embeddings
(`modal/train_ear_head.py`, bring your own labeled texts), then run it locally
with `bin/ear.js` against FLM's `/v1/embeddings` (`flm serve <model> --embed 1`).
The embeddings come off the NPU; the head is plain JS. In our tests the
111KB head **matched a fine-tuned 268MB DistilBERT on real-voice accuracy

(95.9%) and beat it on the hard boundary cases**, live on a Strix Halo NPU.

Also measured: llama3.2:1b chat on the NPU = **47.8 tokens/s** including
prefill (FLM, performance pmode).

Snag #8: FLM's embeddings endpoint closes the TCP connection per request —
retry once on ECONNRESET (ear.js does).

`start.bat` gives you a menu: doctor / list / register / serve / tune guide.

## The snag ledger — ten walls we hit so you don't

1. **The Q4NX converter's `convert.py` CLI is broken at HEAD** (uncommented
   debug `sys.argv` override hijacks every invocation). Call the module API:
   `from q4nx import create_converter; create_converter(gguf, "").convert(q4nx_path=out, weights_type="language")`
2. Converter needs `einops` and `tqdm` beyond its README list, and **must run

   with cwd = its repo root** (relative `configs/<arch>.json` loads).
3. **Llama-3.2 tokenizers need `transformers>=4.46`** — the error
   `untagged enum ModelWrapper` is that wall exactly.
4. **`transformers 4.46` needs `accelerate>=1.0`** — the error
   `'AdamW' object has no attribute 'train'` at step 0 is that skew.
5. **T4 + Llama-3.2's 128k vocab OOMs at batch 4** (loss-logits blowup).
   Floor: batch 1 × grad-accum 8 + gradient checkpointing.
6. **NPU driver minimum for current FLM: `32.0.203.304`** (`.311`
   recommended). `flm validate` will tell you; so will `forge doctor`.
7. **EmbeddingGemma needs `transformers>=4.5x` + `sentence-transformers 5.x`**
   and the official weights are license-gated (use the `unsloth/` mirror, or
   accept the Gemma license on your HF account + pass an `HF_TOKEN` secret).
8. **FLM's `/v1/embeddings` closes the TCP connection per request** — retry
   once on `ECONNRESET` (the ear runtime does).
9. **For a FINE-TUNED model, exporting GGUF as `q8_0` produces repetition

   garbage on the NPU** even though the merged model is perfect — the q8_0

   then Q4NX re-quant is a lossy double-quantization. **Use `Q4_K_M`.**
10. **The Q4NX converter's llama path rejects `f16`** (`not enough values to
    unpack` — it expects pre-quantized blocks). So the GGUF must be quantized

    *before* Q4NX, and `Q4_K_M` is the format proven to produce a coherent,

    in-voice NPU model. Pipeline: HF → f16 → `llama-quantize Q4_K_M` → Q4NX.


**Frozen known-good stack** (the whole point — never debug this again):
`torch 2.4.1 · transformers 4.46.3 · trl 0.9.6 · peft 0.12.0 ·
accelerate 1.1.1 · datasets 2.21.0 · gguf · amd-quark · einops · tqdm ·
protobuf` + a compiled `llama-quantize` (the Modal job builds it).

## Proven, measured (Strix Halo, June 2026)

A LoRA fine-tune of Llama-3.2-1B on 300 real chat exchanges, run through the
whole pipeline and served on the NPU:

- **Coherent and in-voice** — the persona's rituals and endearments intact.
- **41.9 tokens/s** on the NPU (FLM, performance pmode).
- **In-band against the source voiceprint** — mean 0.845 vs the original's
  own held-out band of 0.83 ± 0.07 (3 prompts). A separate stylometric scorer
  certified the NPU model speaks like the source it was tuned on.

That is the bar: not "it converts," but "it talks like itself, on the NPU."

## Requirements

- AMD Ryzen AI machine with XDNA2 NPU (Strix, Strix Halo, Kraken…) +
  [FastFlowLM](https://github.com/FastFlowLM/FastFlowLM)
- Node.js (the CLI), Python + a [Modal](https://modal.com) account (the cloud legs)
- NPU driver ≥ 32.0.203.304

Part of an ongoing project to make local NPUs a first-class home for personal
AI — voices you own, on silicon you own.