Instructions to use hoborific/Pantheon-Reasoning-31B-1.1-W8A16-FP8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use hoborific/Pantheon-Reasoning-31B-1.1-W8A16-FP8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="hoborific/Pantheon-Reasoning-31B-1.1-W8A16-FP8")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("hoborific/Pantheon-Reasoning-31B-1.1-W8A16-FP8")
model = AutoModelForMultimodalLM.from_pretrained("hoborific/Pantheon-Reasoning-31B-1.1-W8A16-FP8", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use hoborific/Pantheon-Reasoning-31B-1.1-W8A16-FP8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "hoborific/Pantheon-Reasoning-31B-1.1-W8A16-FP8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "hoborific/Pantheon-Reasoning-31B-1.1-W8A16-FP8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/hoborific/Pantheon-Reasoning-31B-1.1-W8A16-FP8

SGLang

How to use hoborific/Pantheon-Reasoning-31B-1.1-W8A16-FP8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "hoborific/Pantheon-Reasoning-31B-1.1-W8A16-FP8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "hoborific/Pantheon-Reasoning-31B-1.1-W8A16-FP8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "hoborific/Pantheon-Reasoning-31B-1.1-W8A16-FP8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "hoborific/Pantheon-Reasoning-31B-1.1-W8A16-FP8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use hoborific/Pantheon-Reasoning-31B-1.1-W8A16-FP8 with Docker Model Runner:
```
docker model run hf.co/hoborific/Pantheon-Reasoning-31B-1.1-W8A16-FP8
```

hoborific commited on 26 days ago

Commit

cbc4242

verified ·

1 Parent(s): 1b009ed

Upload README.md with huggingface_hub

Browse files

Files changed (1) hide show

README.md +30 -88

README.md CHANGED Viewed

@@ -1,100 +1,42 @@
 ---
-license: apache-2.0
-language:
-- en
-base_model:
-- Gryphe/Gemma-4-31B-StyleTune
 tags:
-- gemma4
-- conversational
-- instruct
-- finetune
-- roleplay
-- reasoning
-- creative-writing
-pipeline_tag: text-generation
 ---
-# Pantheon-Reasoning-31B-1.1
-[![image/jpg](Pantheon-Reasoning-31B.jpg)](Pantheon-Reasoning-31B.jpg)
-Considering I derived the Gemma 4 31B Style Tune from this same type of data it felt appropriate to give the layers the same treatment. Fun fact - Compared to the base model, this resulted in a 67.6% reduction in cliches, which actually reveals how little the layers are in control of this bit compared to that singular lm_head tensor. Average response lengths have shortened a fair amount, but in a way that makes the writing punchier. I included the full analysis further down if you're interesting!
-The same theory from my previous release applies: take the data that Pantheon is built on, pair it with full thinking traces, and let the model reason its way through character work — weighing tone, planning narrative beats, considering how a character would actually respond before committing to a line. Whether that meaningfully improves roleplay quality over a non-reasoning model is a question you'll hopefully be able to help me answer.
-**New in 1.1:** I really tightened down on the reasoning traces this time around, with each going through multiple QA stages to ensure they're as perfect as can be. I also drastically altered the recipe to allow only the highest quality to make it through, which meant dropping the WorldSim and Tiamat datasets, with my cobbled together WorldSim data and Tiamat's highly specific style simply not meeting my personal standards.
-## Model details
-Base model is [Gryphe/Gemma-4-31B-StyleTune](https://huggingface.co/Gryphe/Gemma-4-31B-StyleTune), obviously!
-All training sources include full reasoning traces, with thinking active across every assistant turn:
-- **Pantheon data** (~26%) - the core Pantheon roleplay corpus with reasoning traces back-generated using the method described below
-- **General roleplay data** (~36%) - a broad collection of highly varied roleplay transcripts with reasoning back-generated, helping the model generalise well to arbitrary character setups
-- **Text adventure data** (~38%) - high stakes interactive fiction and text adventure content with reasoning back-generated, lending the model a more grounded, prose-forward writing style
-A special training template was used to ensure Gemma saw all reasoning included with each turn since unlike Qwen they do not use preserve_thinking, which would have meant only the last turn of each sample had its reasoning trained on.
-Note that I deliberately skipped including an instruct set this time around, being far more interested in pushing writing style first and foremost.
-### Reasoning back-generation
-For the Pantheon, text adventure and general roleplay data, thinking traces were generated using DeepSeek 3.2 after the fact rather than being native to the source material. I tried V4 Flash as well but it proved to be terrible at this specific task. The approach prompts the model to think **as a writer planning their next response** — before writing — rather than annotating a response that already exists. This distinction matters: the goal is genuine forward planning (considering character psychology, tone, and narrative direction), not post-hoc explanation.
-Each generated trace was validated by a judge model before being kept. Traces that slipped into character voice, produced pure restatement, or read as analysis rather than planning were rejected and retried. The result is thinking that reflects real craft decisions rather than a summary of what the response contains. For 1.1 I introduced a master judge that oversaw each trace generation project and further discarded and regenerated traces that had slipped through in the form of a self-iterating pipeline. Self-iterating pipelines are neat.
-The theory is that this reasoning ties semi-seamlessly into Gemma 4's native training and therefore enhances, rather than blatantly overwrites. The reasoning traces aren't as condensed as I expected them to be, but they've definitely condensed down into something far more bearable. Gemma continues to be a stubborn architecture to deal with.
-## What is Pantheon?
-Pantheon is my ongoing series of roleplay-focused finetunes built around a collection of diverse personas — characters with distinct personalities, voices, accents and mannerisms. Though I made sure to mention exactly which personas these were in the past in reality I'm generally the only one bothering to actually use them (lol) so I'm not going to bother with a huge list this time around. Since the original dataset had duplicate subjects I distilled this down to a smaller, more meaningful core, emphasizing variety first and foremost.
-TLDR: Ten personas put through hundreds of scenarios, from good to bad and anything in-between.
-## Inference
-These settings have been working well for me:
-```
-"temperature": 1.0,
-"repetition_penalty": 1.0,
-"min_p": 0.05
-```
-Reasoning models seem to work better without a repetition penalty — likely because it also affects the thinking traces, even though those aren't visible in the output.
-I obviously recommend leaving thinking enabled. Having said that, I'm also very curious about non-reasoning performance!
-## Prompt Format
-The model was trained using Gemma 4's native chat template, which should be applied automatically.
-Since reasoning doesn't tend to play nice with character name prefixes enabled I'm inclined to recommend against using them.
-## Bonus: Writing Quality Metrics
-| Metric | Base | Style Tune | Pantheon Reasoning |
-|--------|------|-----------|------|
-| Avg sentence length | 15.1w | 15.0w | 12.5w |
-| Type-token ratio | 0.705 | 0.735 | 0.774 |
-| Avg paragraph length | 54.5w | 57.3w | 55.6w |
-| Jaccard top-500 ngrams | — | 21.7% overlap w/ base | 14.0% overlap w/ base |
-| Model | Clichés/resp | Clichés/100w | Reduction | Avg reasoning | Avg output |
-|-------|-------------|--------------|-----------|---------------|------------|
-| Base instruct | 2.38 | 1.231 | — | 310.5w | 193.4w |
-| Style Tune | 0.92 | 0.523 | −57.5% | 294.0w | 175.0w |
-| Pantheon Reasoning | 0.49 | 0.399 | −67.6% | 214.4w | 122.8w |
-## Notes
-This is, like most of my releases nowadays, a research release and hasn't gone through extensive quality testing beyond basic sanity checks. The core question — does reasoning actually help roleplay, or does it just add latency? — is one I'm genuinely curious about, and your feedback will be far more informative than my own bias here. Let me know what you find!
-## Credits
-- Everyone from [Anthracite](https://huggingface.co/anthracite-org)! Hi, guys!
-- [Latitude](https://huggingface.co/LatitudeGames), for which I am still producing finetunes on a regular basis, helping me keep my skills sharp and up-to-date!
-- All the folks I chat with on a daily basis on Discord! You know who you are.
-- Anyone I forgot to mention, just in case!

 ---
+base_model: Gryphe/Pantheon-Reasoning-31B-1.1
+library_name: transformers
 tags:
+- quantized
+- w8a16
+- fp8
 ---
+# Pantheon-Reasoning-31B-1.1-W8A16-FP8
+Quantized version of [Gryphe/Pantheon-Reasoning-31B-1.1](https://huggingface.co/Gryphe/Pantheon-Reasoning-31B-1.1).
+## Format
+Offline-quantized **W8A16 FP8** in the
+[compressed-tensors](https://github.com/neuralmagic/compressed-tensors)
+`float-quantized` format: weights in `float8_e4m3fn` with per-output-channel
+symmetric scales, activations kept in bf16/fp16.
+## How it was quantized
+For each linear layer, every output row gets its own scale starting from
+`amax / 448`, refined by an MSE clip search over ~9 clip fractions
+(0.8–1.0× amax) picking the lowest-error scale per row. Weights are then
+quantized `q = e4m3(w / scale)` with round-to-nearest and saturation. This
+per-channel + clipping scheme gives better SNR than vLLM's online per-tensor
+`--quantization fp8` path.
+Only 2D linear projection weights are quantized (attention q/k/v/o, MLP
+gate/up/down). Embeddings, norms, lm_head, routers/experts, and the vision
+tower stay in bf16 and are listed in the checkpoint's `ignore` list, so vLLM
+leaves them untouched.
+## Supported vLLM platforms
+- **Intel XPU** — `XPUW8A16FP8LinearKernel` (the intended target).
+- **NVIDIA CUDA** (SM75+, i.e. Turing and newer) —
+  `HummingFP8ScaledMMLinearKernel` when the `humming` package is installed,
+  otherwise `MarlinFP8ScaledMMLinearKernel`.
+- **Not supported**: ROCm, CPU, TPU — vLLM has no W8A16-FP8 kernel for these
+  backends yet, so loading will fail with a "no kernel" error.