license: apache-2.0
base_model:
- llm-jp/llm-jp-4.1-32b-a3b-thinking
language:
- ja
- en
pipeline_tag: text-generation
tags:
- openvino
- npu
- intel
- onw
- moe
- agent
library_name: onw
llm-jp-4.1-32b-a3b-thinking for onw (INT8, high precision) — a Japanese MoE running entirely on the Intel NPU
日本語 | English
llm-jp/llm-jp-4.1-32b-a3b-thinking converted for onw, an engine that runs LLMs entirely on the Intel NPU. This repository holds the model only (the engine is a separate download).
A thinking MoE trained from scratch, mainly in Japanese, by LLM-jp at NII: 32B in total, ~3B per token (8 of 128 experts), so more knowledge than llm-jp-4.1-8b at about the same speed.
Thinking and tool calls: the harmony format, like llm-jp-4.1-8b; it always thinks first. The thinking comes back as reasoning_content in thinking mode and counts against max_tokens (use 300 or more). Tools work through the OpenAI-compatible tools.
The lighter INT4 version (16 GB, runs on 16 GB PCs too): ryugyosoft/llm-jp-4.1-32b-a3b-thinking-onw.
| NPU 3720 (Core Ultra 9 285HX, 192 GB) | |
|---|---|
| Input | text |
| Decode | 4.6 tok/s |
| Prompt | 94 tokens in 11 s |
| Download | 30 GB |
| Memory | ~32 GB |
| First start (NPU compile) | ~9 min |
PCs with less memory: for PCs with 32 GB or more (on 16 GB use the INT4 version).
How to use
- Install onw (the engine) 1.2.0 or newer (Install in the onw README).
- In the onw window's Models tab pick llm-jp-4.1-32B-A3B-thinking (INT8) and press Download (30 GB).
- Load it on the Server tab, then use the Chat or the OpenAI-compatible API (
http://localhost:8000/v1).
Advanced: hf download ryugyosoft/llm-jp-4.1-32b-a3b-thinking-int8-onw and run the folder with onw serve FOLDER.
How it works
- Static graphs with one segment per layer: [the previous layer's experts -> this layer's attention -> router]; channel-wise INT8 experts and attention; the INT8 LM head is a segment of its own.
- The experts are not baked into the graphs: the 8 picked ones are handed to the NPU each step (streamed from the SSD on PCs short of memory).
- See onw's technical notes.
Quality
Teacher-forced against the original model (bf16, CPU): 292/300 tokens of a 300-token Japanese answer, top-5 logits within 0.26 on average (INT4 version: 270/300, 1.6). Asked for Japan's capital it answers 日本の首都は東京です。
License
Apache 2.0, as the original model. The weights are requantized and repacked from the original model; no training.