Instructions to use AEON-7/Qwen3.6-27B-AEON-Ultimate-Uncensored-Text-NVFP4-MTP with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use AEON-7/Qwen3.6-27B-AEON-Ultimate-Uncensored-Text-NVFP4-MTP with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="AEON-7/Qwen3.6-27B-AEON-Ultimate-Uncensored-Text-NVFP4-MTP")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("AEON-7/Qwen3.6-27B-AEON-Ultimate-Uncensored-Text-NVFP4-MTP")
model = AutoModelForMultimodalLM.from_pretrained("AEON-7/Qwen3.6-27B-AEON-Ultimate-Uncensored-Text-NVFP4-MTP", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use AEON-7/Qwen3.6-27B-AEON-Ultimate-Uncensored-Text-NVFP4-MTP with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "AEON-7/Qwen3.6-27B-AEON-Ultimate-Uncensored-Text-NVFP4-MTP"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AEON-7/Qwen3.6-27B-AEON-Ultimate-Uncensored-Text-NVFP4-MTP",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/AEON-7/Qwen3.6-27B-AEON-Ultimate-Uncensored-Text-NVFP4-MTP

SGLang

How to use AEON-7/Qwen3.6-27B-AEON-Ultimate-Uncensored-Text-NVFP4-MTP with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "AEON-7/Qwen3.6-27B-AEON-Ultimate-Uncensored-Text-NVFP4-MTP" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AEON-7/Qwen3.6-27B-AEON-Ultimate-Uncensored-Text-NVFP4-MTP",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "AEON-7/Qwen3.6-27B-AEON-Ultimate-Uncensored-Text-NVFP4-MTP" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AEON-7/Qwen3.6-27B-AEON-Ultimate-Uncensored-Text-NVFP4-MTP",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use AEON-7/Qwen3.6-27B-AEON-Ultimate-Uncensored-Text-NVFP4-MTP with Docker Model Runner:
```
docker model run hf.co/AEON-7/Qwen3.6-27B-AEON-Ultimate-Uncensored-Text-NVFP4-MTP
```

AEON-7 Claude Opus 4.8 (1M context) commited on Jun 18

Commit

9d7f810

verified ·

1 Parent(s): 57681b5

docs: vLLM compatibility status on aeon-vllm-ultimate:latest

Browse files

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Files changed (2) hide show

README.md +46 -1
assets/perf/qwen27b_variant_family.svg +36 -0

README.md CHANGED Viewed

@@ -41,12 +41,57 @@ tags:
 > ## 🆕 AEON vLLM Ultimate container
 >
-> [`ghcr.io/aeon-7/aeon-vllm-ultimate:latest`](https://github.com/AEON-7/vllm-ultimate-dgx-spark) (= tag `:2026-06-11-pr41703`; rollback `:2026-06-04-pr44389`) — the unified vLLM image for **all** Qwen3.6-27B AEON repos: NVFP4 KV cache + DFlash + TurboQuant K8V4 + AEON sm_121a patches, plus PR #40898 (sliding-window attention for the DFlash drafter) and PR #41703 (prefix-caching corruption-immunity with DFlash). This variant uses the same modelopt NVFP4 format, the same `qwen3_5_mtp` native head, and the same hybrid GDN+attention stack — it serves identically with `--quantization modelopt` and `--speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}'` (native MTP, the recommended drafter on dedicated-VRAM Blackwell — see Usage below).
 >
 > The image ENTRYPOINT is `/bin/bash`, so a raw `docker run` must pass `--entrypoint vllm` and then `serve ...` (do **not** write `IMAGE vllm serve`, which runs `bash vllm serve` and fails).
 >
 > **Why the long-context fix matters (DFlash path on DGX Spark):** if you instead drive this body with a DFlash drafter on a Spark/GB10, note that the z-lab Qwen3.6-27B DFlash drafter is a sliding-window model — 4 of its 5 layers use sliding-window attention (window 2048). PR #40898 (in `aeon-vllm-ultimate:latest`) runs those layers as proper SWA; earlier images ran them as full attention, so drafting collapsed once context grew past ~2048 tokens. PR #41703 additionally makes `--enable-prefix-caching` corruption-immune with DFlash. Net: long-context drafting holds up; short-context (<2048, one window) is unchanged. (This MTP repo's native `qwen3_5_mtp` head is the recommended drafter on dedicated VRAM — see the [`-Multimodal-NVFP4-MTP-XS`](https://huggingface.co/AEON-7/Qwen3.6-27B-AEON-Ultimate-Uncensored-Multimodal-NVFP4-MTP-XS) sibling for the Spark-side DFlash benchmarks.) Full setup + bench comparison: [container README](https://github.com/AEON-7/vllm-ultimate-dgx-spark).
 ## Variants
 | Format | Size | Use case |

 > ## 🆕 AEON vLLM Ultimate container
 >
+> [`ghcr.io/aeon-7/aeon-vllm-ultimate:latest`](https://github.com/AEON-7/vllm-ultimate-dgx-spark) (= tag `:2026-06-18-v0.23.0-dflashfix`; rollback `:2026-06-11-pr41703`) — the unified **vLLM v0.23.0** image (built from source for sm_121a) for **all** Qwen3.6-27B AEON repos: NVFP4 KV cache + DFlash + TurboQuant K8V4 + AEON sm_121a patches, plus PR #40898 (sliding-window attention for the DFlash drafter) and the DFlash high-concurrency block-table fix. On **dedicated-VRAM Blackwell** this variant serves with `--quantization modelopt` and `--speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}'` (native MTP — see Usage below). **On the DGX Spark / GB10 this text export does not load** — see the vLLM-compatibility note immediately below.
 >
 > The image ENTRYPOINT is `/bin/bash`, so a raw `docker run` must pass `--entrypoint vllm` and then `serve ...` (do **not** write `IMAGE vllm serve`, which runs `bash vllm serve` and fails).
 >
 > **Why the long-context fix matters (DFlash path on DGX Spark):** if you instead drive this body with a DFlash drafter on a Spark/GB10, note that the z-lab Qwen3.6-27B DFlash drafter is a sliding-window model — 4 of its 5 layers use sliding-window attention (window 2048). PR #40898 (in `aeon-vllm-ultimate:latest`) runs those layers as proper SWA; earlier images ran them as full attention, so drafting collapsed once context grew past ~2048 tokens. PR #41703 additionally makes `--enable-prefix-caching` corruption-immune with DFlash. Net: long-context drafting holds up; short-context (<2048, one window) is unchanged. (This MTP repo's native `qwen3_5_mtp` head is the recommended drafter on dedicated VRAM — see the [`-Multimodal-NVFP4-MTP-XS`](https://huggingface.co/AEON-7/Qwen3.6-27B-AEON-Ultimate-Uncensored-Multimodal-NVFP4-MTP-XS) sibling for the Spark-side DFlash benchmarks.) Full setup + bench comparison: [container README](https://github.com/AEON-7/vllm-ultimate-dgx-spark).
+## vLLM compatibility (DGX Spark / aeon-vllm-ultimate:latest)
+**Status: this text export does not load on the unified container as shipped.** Validated 2026-06-18 on [`ghcr.io/aeon-7/aeon-vllm-ultimate:latest`](https://github.com/AEON-7/vllm-ultimate-dgx-spark) (vLLM v0.23.0, sm_121a).
+The reason is purely a packaging mismatch, not a problem with the weights:
+- `config.json` declares the **multimodal architecture** `Qwen3_5ForConditionalGeneration` (`model_type: qwen3_5`), so vLLM routes it through the **multimodal loader**.
+- The multimodal loader expects an **image processor** in the repo, but this is a text-only export (the vision tower was stripped during quantization), so no processor files ship here.
+- vLLM therefore aborts during init with **"cannot load image processor"** before serving can start.
+**This is fixable two ways**, either of which would let this exact NVFP4+MTP body serve on the Spark:
+1. **Re-export with a text-only architecture** (e.g. a `*ForCausalLM` head class instead of `Qwen3_5ForConditionalGeneration`), so vLLM uses the text loader and never looks for an image processor; or
+2. **Add the processor files** (`preprocessor_config.json` / image-processor config) so the multimodal loader can initialize even though no image input is used.
+Until then, **for vLLM serving on `aeon-vllm-ultimate:latest` (DGX Spark / GB10), use the [`-Multimodal-NVFP4-MTP-XS`](https://huggingface.co/AEON-7/Qwen3.6-27B-AEON-Ultimate-Uncensored-Multimodal-NVFP4-MTP-XS) sibling instead** — it is the smallest NVFP4 export, loads cleanly, and is the **fastest single-stream** option in this family (~42 tok/s at c=1 with DFlash). This card still serves fine on **dedicated-VRAM Blackwell** (RTX PRO 6000 / B100 / B200) via the text path with native MTP — see Usage below.
+### Where this variant sits in the family
+<p align="center"><img src="assets/perf/qwen27b_variant_family.svg" width="100%" alt="Qwen3.6-27B-AEON-Ultimate variant comparison: Multimodal-MTP-XS is fastest single-stream and the recommended Spark choice; BF16 baseline is ~2× the memory"></p>
+### Recommended Spark alternative — measured c=1 throughput (Multimodal-NVFP4-MTP-XS)
+These are the per-category single-stream numbers for the working sibling on `aeon-vllm-ultimate:latest` (DGX Spark / GB10, DFlash speculative decoding):
+| Category | Decode tok/s | TTFT (ms) | TPOT (ms) | Prefill (tok/s) | DFlash accept % |
+|---|---:|---:|---:|---:|---:|
+| Coding | 42.6 | 141 | 23.5 | 318 | 34.5 |
+| Math | 55.9 | 248 | 17.9 | 246 | 48.0 |
+| Reasoning | 49.3 | 232 | 20.3 | 211 | 41.7 |
+| Prose | 31.2 | 229 | 32.1 | 166 | 23.2 |
+| Natural language | 34.8 | 228 | 28.7 | 175 | 26.6 |
+| Extraction / JSON | 57.4 | 234 | 17.4 | 231 | 49.3 |
+Long-context (≈16k–32k) DFlash acceptance holds at ~45%. Aggregate throughput scales to **c=64** on the unified container (Reasoning peaks ~340 tok/s). Full per-concurrency data lives on the [XS sibling card](https://huggingface.co/AEON-7/Qwen3.6-27B-AEON-Ultimate-Uncensored-Multimodal-NVFP4-MTP-XS).
+## What we fixed for the DGX Spark
+All AEON models run on one unified container — **`ghcr.io/aeon-7/aeon-vllm-ultimate:latest`** (= `:2026-06-18-v0.23.0-dflashfix`; rollback `:2026-06-11-pr41703`). It is **vLLM v0.23.0 built from source for sm_121a (GB10 / Blackwell)** and merged with the AEON speculative-decoding stack.
+- **DFlash high-concurrency fix** *(new)* — slices the speculative drafter's KV block-table to the unpadded batch (`block_table[:num_reqs]`). The drafter previously **crashed at ≥32 concurrent requests** (padded-vs-unpadded block-table shape mismatch in FlashAttention); it now scales cleanly to **c=64**. A port of upstream PR #43982, which fixed this for MTP but never for DFlash.
+- **Unified vLLM 0.23.0 image** — NVFP4 KV cache (PR #44389, the only 4-bit KV path on sm_121a) + DFlash sliding-window attention (PR #40898, so long-context draft acceptance holds) + sm_121a-native CUTLASS NVFP4/FP8 kernels + boot/CUDA-graph patches, all in a single tag.
+> **Stock baseline pending fresh vanilla re-bench:** no apples-to-apples stock (vanilla vLLM, no DFlash, no sm_121a opts) baseline exists for this family yet. A fully-vanilla benchmark on the current version is pending; the optimized figures above are measured on `aeon-vllm-ultimate:latest` (vLLM 0.23.0).
 ## Variants
 | Format | Size | Use case |

assets/perf/qwen27b_variant_family.svg ADDED Viewed