README_en.md

---
license: apache-2.0
base_model:
  - llm-jp/llm-jp-4.1-32b-a3b-thinking
language:
  - ja
  - en
pipeline_tag: text-generation
tags:
  - openvino
  - npu
  - intel
  - onw
  - moe
  - agent
library_name: onw
---

# llm-jp-4.1-32b-a3b-thinking for onw (INT8, high precision) — a Japanese MoE running entirely on the Intel NPU

[日本語](README.md) | **English**

[llm-jp/llm-jp-4.1-32b-a3b-thinking](https://huggingface.co/llm-jp/llm-jp-4.1-32b-a3b-thinking) converted for **[onw](https://huggingface.co/ryugyosoft/onw)**, an engine that runs LLMs entirely on the Intel NPU. This repository holds the model only (the engine is a separate download).

A thinking MoE trained from scratch, mainly in Japanese, by LLM-jp at NII: 32B in total, ~3B per token (8 of 128 experts), so more knowledge than llm-jp-4.1-8b at about the same speed.

**Thinking and tool calls**: the harmony format, like llm-jp-4.1-8b; it always thinks first. The thinking comes back as `reasoning_content` in thinking mode and counts against `max_tokens` (use 300 or more). Tools work through the OpenAI-compatible `tools`.

The lighter INT4 version (16 GB, runs on 16 GB PCs too): [ryugyosoft/llm-jp-4.1-32b-a3b-thinking-onw](https://huggingface.co/ryugyosoft/llm-jp-4.1-32b-a3b-thinking-onw).

| NPU 3720 (Core Ultra 9 285HX, 192 GB) | |
|---|---|
| Input | text |
| Decode | **4.6 tok/s** |
| Prompt | 94 tokens in 11 s |
| Download | **30 GB** |
| Memory | ~32 GB |
| First start (NPU compile) | ~9 min |

**PCs with less memory**: for PCs with 32 GB or more (on 16 GB use the INT4 version).

## How to use

1. **Install onw (the engine) 1.2.0 or newer** ([Install in the onw README](https://huggingface.co/ryugyosoft/onw)).
2. In the onw window's Models tab pick **llm-jp-4.1-32B-A3B-thinking (INT8)** and press Download (30 GB).
3. Load it on the Server tab, then use the Chat or the **OpenAI-compatible API** (`http://localhost:8000/v1`).

Advanced: `hf download ryugyosoft/llm-jp-4.1-32b-a3b-thinking-int8-onw` and run the folder with `onw serve FOLDER`.

## How it works

- Static graphs with one segment per layer: [the previous layer's experts -> this layer's attention -> router]; channel-wise INT8 experts and attention; the INT8 LM head is a segment of its own.
- The experts are not baked into the graphs: the 8 picked ones are handed to the NPU each step (streamed from the SSD on PCs short of memory).
- See [onw's technical notes](https://huggingface.co/ryugyosoft/onw/blob/main/TECHNICAL_en.md).

## Quality

Teacher-forced against the original model (bf16, CPU): 292/300 tokens of a 300-token Japanese answer, top-5 logits within 0.26 on average (INT4 version: 270/300, 1.6). Asked for Japan's capital it answers 日本の首都は東京です。

## License

Apache 2.0, as the original model. The weights are requantized and repacked from the original model; no training.