---
base_model: Qwen/Qwen3-32B-FP8
license: apache-2.0
license_link: https://huggingface.co/Qwen/Qwen3-32B-FP8/blob/main/LICENSE
pipeline_tag: text-generation
library_name: furiosa-llm
tags:
  - furiosa-ai
  - qwen3
---
# Qwen3-32B-FP8

This repository contains [Qwen/Qwen3-32B-FP8](https://huggingface.co/Qwen/Qwen3-32B-FP8)
together with a Furiosa Executable Bundle (FXB) for running it on
[FuriosaAI RNGD](https://furiosa.ai) with [Furiosa-LLM](https://developer.furiosa.ai/latest/en/furiosa_llm/intro.html).
The same model also runs on other frameworks (such as vLLM, SGLang, and
Transformers); for usage with those, see the upstream
[Qwen/Qwen3-32B-FP8](https://huggingface.co/Qwen/Qwen3-32B-FP8) model card.

## Overview

Qwen3-32B is the 32.8B-parameter **dense** model of the Qwen3 series, a causal
transformer with grouped-query attention. Its hallmark is seamless switching
between a thinking mode — emitting a chain of thought before the final answer
for complex reasoning, math, and coding — and a non-thinking mode for efficient
general dialogue, within a single model. It also offers strong tool-calling and
agent capabilities and multilingual support. Its intended use is the same as
the upstream [Qwen/Qwen3-32B-FP8](https://huggingface.co/Qwen/Qwen3-32B-FP8),
and it is released under the [Apache 2.0 License](https://huggingface.co/Qwen/Qwen3-32B-FP8/blob/main/LICENSE).

- **Architecture:** Qwen3 (dense)
- **Input / Output:** Text / Text
- **Supported Inference Engine:** Furiosa LLM
- **Supported Hardware:** FuriosaAI RNGD

### Quantization

The model weights are quantized to **FP8** (static), using the same fine-grained
FP8 quantization (block size 128) the upstream model ships in, and activations
use **dynamic FP8 quantization** at runtime (per-token / per-block). The KV cache
stays in 16-bit precision.

### Features

- **Reasoning.** Qwen3-32B is a hybrid reasoning model: thinking mode is on by default and can be toggled per request via `enable_thinking`. Launch the server with `--reasoning-parser qwen3` to have the reasoning content returned in a separate field (see [Basic Usage](#basic-usage) below).
- **Tool calling.** The model supports tool (function) calling through the `hermes` tool-call parser.

### Parallelism Strategy

On RNGD, Qwen3-32B-FP8 runs with a **tensor-parallel size of 32 PEs**, which
maps to **four RNGD cards** (8 PEs per card).

## Usage

To run this model with Furiosa-LLM, follow the example commands below after
[installing Furiosa-LLM and its prerequisites](https://developer.furiosa.ai/latest/en/get_started/furiosa_llm.html#installing-furiosa-llm).

### Launch the server

Serve the model with the `qwen3` reasoning parser so the chain of thought is
returned in a separate field:

```sh
furiosa-llm serve furiosa-ai/Qwen3-32B-FP8 \
  --reasoning-parser qwen3
```

To also enable tool (function) calling, add the `hermes` tool-call parser (the
parser used by the Qwen3 series):

```sh
furiosa-llm serve furiosa-ai/Qwen3-32B-FP8 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser hermes
```

When the server is ready, you will see:

```sh
INFO:     Started server process [27507]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
```

### Basic Usage

The server exposes an OpenAI-compatible API. You can send a request with `curl`:

```sh
curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
    "model": "furiosa-ai/Qwen3-32B-FP8",
    "messages": [{"role": "user", "content": "What is the capital of France?"}]
    }' \
    | python -m json.tool
```

With `--reasoning-parser qwen3`, the thinking content is returned separately from
the final answer:

* `response.choices[].message.reasoning` (non-streaming)
* `response.choices[].delta.reasoning` (streaming)

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

response = client.chat.completions.create(
    model="furiosa-ai/Qwen3-32B-FP8",
    messages=[{"role": "user", "content": "How many r's are in 'strawberry'?"}],
)

print("Reasoning:", response.choices[0].message.reasoning)
print("Answer:", response.choices[0].message.content)
```

> **Note:** The `reasoning` field is not part of the OpenAI API specification but
> is a widely followed convention (the OpenAI Agents SDK, vLLM, and others). It
> appears only in responses that contain reasoning content; accessing it otherwise
> raises an `AttributeError`.

### Advanced Usage

**Toggling thinking.** Qwen3-32B is a hybrid model: it reasons by default and can
switch thinking on and off. To turn thinking off for a single request, pass
`enable_thinking` through `chat_template_kwargs`; the response then carries no
reasoning content, so read only `message.content`:

```python
# Disable thinking for a single request
response = client.chat.completions.create(
    model="furiosa-ai/Qwen3-32B-FP8",
    messages=[{"role": "user", "content": "What is the capital of France?"}],
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)
print(response.choices[0].message.content)
```

To default every request to non-thinking, launch the server with
`--default-chat-template-kwargs` (a request can still re-enable thinking with its
own `chat_template_kwargs`):

```sh
furiosa-llm serve furiosa-ai/Qwen3-32B-FP8 \
  --reasoning-parser qwen3 \
  --default-chat-template-kwargs '{"enable_thinking": false}'
```

**Tool calling.** With the server launched using
`--enable-auto-tool-choice --tool-call-parser hermes` (see
[Launch the server](#launch-the-server)), pass `tools` in the request and let the
model decide when to call them. See the
[Tool Calling guide](https://developer.furiosa.ai/latest/en/furiosa_llm/toolcalling.html)
for a complete client example and details on tool-choice options.

## Learn more

* [Tool Calling](https://developer.furiosa.ai/latest/en/furiosa_llm/toolcalling.html) — parsers, tool-choice options, and more examples
* [Furiosa-LLM Server (`furiosa-llm serve`)](https://developer.furiosa.ai/latest/en/furiosa_llm/furiosa-llm-serve.html) — full OpenAI-compatible API reference and serving options
* [Qwen/Qwen3-32B-FP8](https://huggingface.co/Qwen/Qwen3-32B-FP8) — upstream model card