Add files using upload-large-folder tool

README.md

@@ -0,0 +1,162 @@
+---
+license: other
+license_name: mistral-ai-research-license
+license_link: https://mistral.ai/licenses/MNPL-0.1.md
+language:
+- en
+base_model:
+- tacodevs/Behemoth-T1-123B
+- tacodevs/Behemoth-X-R1-123B
+- TheDrummer/Behemoth-X-123B-v2
+- TheDrummer/Behemoth-R1-123B-v2
+tags:
+- mistral
+- mistral-large
+- 123b
+- roleplay
+- creative-writing
+- thinking
+- reasoning
+- fp8
+- w8a8
+- quantized
+- llm-compressor
+pipeline_tag: text-generation
+library_name: transformers
+---
+
+<div align="center">
+
+<img src="https://huggingface.co/tacodevs/Behemoth-T1-123B/resolve/main/images/01_header_main.png" alt="Behemoth-T1" width="100%" />
+
+<h1>🌴 Behemoth-T1-123B-FP8 🌴</h1>
+
+<p><i>The party where literary craft meets unhinged creative writing — production sweet spot.</i></p>
+
+<p>
+  <a href="https://huggingface.co/tacodevs/Behemoth-T1-123B"><img src="https://img.shields.io/badge/BF16-tacodevs%2FBehemoth--T1--123B-ff7eb6?style=for-the-badge" alt="BF16"/></a>
+  <a href="https://huggingface.co/tacodevs/Behemoth-T1-123B-FP8"><img src="https://img.shields.io/badge/FP8-Behemoth--T1--123B--FP8-7eddff?style=for-the-badge" alt="FP8"/></a>
+  <a href="https://huggingface.co/tacodevs/Behemoth-T1-123B-GPTQ"><img src="https://img.shields.io/badge/GPTQ--W4A16-Behemoth--T1--123B--GPTQ-bdff7e?style=for-the-badge" alt="GPTQ"/></a>
+</p>
+
+</div>
+
+## ☀️ The pitch
+
+This is the **FP8 W8A8 dynamic** quantized version of [tacodevs/Behemoth-T1-123B](https://huggingface.co/tacodevs/Behemoth-T1-123B) — a 123B Mistral Large roleplay model that **thinks like a literary author before it writes like a storyteller**.
+
+FP8 is the **production sweet spot**: half the VRAM of BF16, ~99% of the quality (essentially lossless), runs cleanly on Hopper GPUs (H100, H200) with native FP8 acceleration. This is the variant most users should pick.
+
+For the full pitch, training details, and the philosophy behind T1, see the [BF16 model card](https://huggingface.co/tacodevs/Behemoth-T1-123B).
+
+## ⚡ This variant
+
+| | Value |
+|---|---|
+| **Base** | [tacodevs/Behemoth-T1-123B](https://huggingface.co/tacodevs/Behemoth-T1-123B) (BF16) |
+| **Quantization** | FP8 W8A8 dynamic (8-bit weights, 8-bit activations) |
+| **Calibration** | Data-free (per-tensor scale computed analytically) |
+| **Quantizer** | [llm-compressor](https://github.com/vllm-project/llm-compressor) QuantizationModifier |
+| **Size on disk** | ~115 GB (2× smaller than BF16) |
+| **VRAM (8k ctx)** | **~125 GB → fits on 2× 80 GB or 1× 144 GB GPU** |
+| **Quality vs BF16** | ~99% (essentially lossless) |
+| **Speed** | Faster than BF16 on H100/H200 (native FP8 tensor cores) |
+
+## 🎤 How to use
+
+T1 expects a **prefilled `<think>` block** to enter literary thinking mode. Use the same 7 prefill phrases as the BF16 model:
+
+```python
+import openai
+
+client = openai.OpenAI(base_url="http://localhost:8000/v1", api_key="-")
+
+PREFILLS = {
+    "analytical": "Ok i need to think about how to respond — what does the character feel right now, what from their experience is relevant, what do they value, and what are they trying to achieve, so",
+    "creative":   "Ok i need to think as a creative writer — what twist would surprise here? Let me find an engaging new direction nobody saw coming, so",
+    "unhinged":   "Ok i need to think as an unhinged author — raw, explicit, intense, fully in character with no holding back, so",
+}
+
+response = client.chat.completions.create(
+    model="tacodevs/Behemoth-T1-123B-FP8",
+    messages=[
+        {"role": "system", "content": CHARACTER_CARD},
+        *conversation_history,
+        {"role": "user", "content": user_message},
+        {"role": "assistant", "content": f"<think>\n{PREFILLS['creative']}\n"},
+    ],
+    extra_body={
+        "continue_final_message": True,
+        "add_generation_prompt": False,
+    },
+    temperature=0.6,
+    max_tokens=2048,
+    stop=["[INST]", "</s>"],
+)
+```
+
+## 🚀 Serving with vLLM
+
+```bash
+vllm serve tacodevs/Behemoth-T1-123B-FP8 \
+    --tokenizer-mode auto \
+    --tensor-parallel-size 2 \
+    --max-model-len 8192 \
+    --kv-cache-dtype fp8
+```
+
+**Important**: use `--tokenizer-mode auto`, **not** `mistral` — `mistral_common` mode silently mis-templates merged-LoRA checkpoints.
+
+Recommended hardware:
+- **2× H100 80GB** — comfortable, ~62GB per GPU
+- **2× H200 144GB** — luxury, plenty of headroom for 32k+ context
+- **1× H100 NVL 188GB** — single-card option
+
+FP8 W8A8 uses **native FP8 tensor cores on Hopper GPUs** for both weights and activations, so this variant is also **faster** than BF16 in practice, not just smaller.
+
+## ✅ Quality notes
+
+FP8 W8A8 dynamic quantization is essentially lossless for 100B+ models:
+
+- ✅ **Stream-of-consciousness thinking shape** — preserved
+- ✅ **Detail surfacing from character cards** — preserved
+- ✅ **Word-for-word output similarity to BF16** — typically 95%+ token overlap on greedy sampling
+- ✅ **Beats base R1 in side-by-side** — preserved
+- ✅ **Production reliability** — same recipe Mistral officially uses for [Mistral Large 3 FP8](https://docs.vllm.ai/projects/recipes/en/latest/Mistral/Mistral-Large-3.html)
+
+If you need the absolute reference quality and have 4× 80 GB GPUs, use the [BF16 reference](https://huggingface.co/tacodevs/Behemoth-T1-123B). For most production use cases, **FP8 is the right pick**.
+
+## 🛠️ Training details (from base T1)
+
+T1 is a LoRA distillation of Claude Opus 4.5 literary thinking onto
+[`tacodevs/Behemoth-X-R1-123B`](https://huggingface.co/tacodevs/Behemoth-X-R1-123B)
+(itself an SCE merge of Behemoth-X creative writing + Behemoth-R1 reasoning).
+
+| | |
+|---|---|
+| **LoRA rank** | 32 (alpha 64, dropout 0.05, all 7 projection modules) |
+| **Trainable params** | 559M / 123B (0.45%) |
+| **Dataset** | 1000 Claude Opus 4.5 thinking traces on real RP conversations |
+| **Loss masking** | Think-only (only the post-prefill thinking continuation gets loss) |
+| **Sequence length** | 4096 |
+| **Epochs** | 2 |
+| **Final eval loss** | 0.9898 |
+
+The LoRA only learns the *shape* of literary thinking. The base model's RP prose engine receives **zero gradient updates** — the underlying creative writing voice is structurally preserved.
+
+## 📜 Citation
+
+```bibtex
+@misc{behemoth-t1-2026,
+  title  = {Behemoth-T1-123B: Literary Thinking Distillation for RP},
+  author = {tacodevs},
+  year   = {2026},
+  url    = {https://huggingface.co/tacodevs/Behemoth-T1-123B},
+}
+```
+
+<div align="center">
+
+<i>The party doesn't end. We just go to bed.</i>
+
+</div>

chat_template.jinja

@@ -0,0 +1 @@
+{{ bos_token }}{% for message in messages %}{% if message['role'] == 'user' %}{{ '[INST] ' + message['content'] + '[/INST]' }}{% elif message['role'] == 'system' %}{{ '[SYSTEM_PROMPT] ' + message['content'] + '[/SYSTEM_PROMPT]' }}{% elif message['role'] == 'assistant' %}{{ ' ' + message['content'] + eos_token }}{% else %}{{ raise_exception('Only user, system and assistant roles are supported!') }}{% endif %}{% endfor %}

config.json

@@ -0,0 +1,76 @@
+{
+  "architectures": [
+    "MistralForCausalLM"
+  ],
+  "attention_dropout": 0.0,
+  "bos_token_id": 1,
+  "dtype": "bfloat16",
+  "eos_token_id": 2,
+  "head_dim": 128,
+  "hidden_act": "silu",
+  "hidden_size": 12288,
+  "initializer_range": 0.02,
+  "intermediate_size": 28672,
+  "max_position_embeddings": 131072,
+  "model_type": "mistral",
+  "num_attention_heads": 96,
+  "num_hidden_layers": 88,
+  "num_key_value_heads": 8,
+  "quantization_config": {
+    "config_groups": {
+      "group_0": {
+        "format": "float-quantized",
+        "input_activations": {
+          "actorder": null,
+          "block_structure": null,
+          "dynamic": true,
+          "group_size": null,
+          "num_bits": 8,
+          "observer": null,
+          "observer_kwargs": {},
+          "scale_dtype": null,
+          "strategy": "token",
+          "symmetric": true,
+          "type": "float",
+          "zp_dtype": null
+        },
+        "output_activations": null,
+        "targets": [
+          "Linear"
+        ],
+        "weights": {
+          "actorder": null,
+          "block_structure": null,
+          "dynamic": false,
+          "group_size": null,
+          "num_bits": 8,
+          "observer": "memoryless_minmax",
+          "observer_kwargs": {},
+          "scale_dtype": null,
+          "strategy": "channel",
+          "symmetric": true,
+          "type": "float",
+          "zp_dtype": null
+        }
+      }
+    },
+    "format": "float-quantized",
+    "global_compression_ratio": null,
+    "ignore": [
+      "lm_head"
+    ],
+    "kv_cache_scheme": null,
+    "quant_method": "compressed-tensors",
+    "quantization_status": "compressed",
+    "sparsity_config": {},
+    "transform_config": {},
+    "version": "0.14.0.1"
+  },
+  "rms_norm_eps": 1e-05,
+  "rope_theta": 1000000.0,
+  "sliding_window": null,
+  "tie_word_embeddings": false,
+  "transformers_version": "4.57.6",
+  "use_cache": true,
+  "vocab_size": 32768
+}

generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "transformers_version": "4.57.6"
+}

model-00001-of-00026.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:73a43d695712653aef06313996578a5084544887c188aa3431ff87439e26b93f
+size 4958398128

model-00002-of-00026.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:be327efcc73492294870c16f13f018b6ff79659be1e4fef10ab35eb70c212115
+size 4832680488

model-00003-of-00026.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7ddc93d770e854a048546372b62eae484e82f63580620bbeb189f684d4afddb0
+size 4857866304

model-00004-of-00026.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:877c2b368648de761380e054976a9aee850a6d393a52fd26dcce8c3dda2bb5de
+size 4832680552

model-00005-of-00026.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b537154cef8ea0e83987820718b9db52e1b09e647770b8b762690ad6159147f0
+size 4857866352

model-00006-of-00026.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:57c30cdf754255fabbee65205386b9568c046e2b0e076c43896a8db980ba6290
+size 4832680552

model-00007-of-00026.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0f04df9d2b98877a3204c75d7061427045fc4e80c6ecdcb72c7c1d7a5b85a1b0
+size 4857866352

model-00008-of-00026.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:27a8648490f1cad1b78d5325887ea70acd11bfd32b2ac1bc0cfee07070fd7717
+size 4832680552

model-00009-of-00026.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b44e8f5820df9203aebe5f10b2fcd9ae1fc807acc674d0af1885f709de24b26f
+size 4857866352

model-00010-of-00026.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3e84af8d4e6c7b9762960eb82dfb85f005ecf2ea6eaba0f75623de1a70786440
+size 4832680552

model-00011-of-00026.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6b3b1317bca753d6deaa36b8bbd5bc5964a5ec9f94c47725cd6baa65284bf0fe
+size 4857866352

model-00012-of-00026.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e9d9bd77fbe403a656a49bb06dd253a4feffc82431f954671649fa68a7b0be7e
+size 4832680552

model-00013-of-00026.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b4d544095c778b10502f518b9d0da1e355a67585bef57d860477a6cdfdf4611d
+size 4857866352

model-00014-of-00026.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9fde53f096768a7ea9cf3a2506ca3e56c2025ae670f09fe8b7ae42fc06e86115
+size 4832680552

model-00015-of-00026.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:26533fe731c6e12826978f6754e36665a43ca06c092016ee449b2f664b4567e4
+size 4857866352

model-00016-of-00026.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:80993bac9a9b1c8b9ca2f00b4f3f267ad0cb2f6bad2ef9dfffc9f3d92e87013c
+size 4832680552

model-00017-of-00026.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:190b8cc158e96390507573e29d3a9b116778fedbba071810f2e5e89d782f5e39
+size 4857866352

model-00018-of-00026.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:83b43b783ef90862bb8e5743a9cf2f48e4932aeb8184e43ee99e630075cdb4c6
+size 4832680552

model-00019-of-00026.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ae5fce9f51cc34abd4536e751449af8d4fe7b2f4b3df22b81d09765ce52c218b
+size 4857866352

model-00020-of-00026.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:958266dfc10e5570cb5c70e969cdaae5eb7f99ae85e98b182d4535783c9ac023
+size 4832680552

model-00021-of-00026.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f9515a89c3dc01f5f12967e71bee23a4402c829c2b47f7258a4d5f334d4fa78b
+size 4857866352

model-00022-of-00026.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9d1ff4b37b8726da95bf1bbaa4f66cd36ec1e92a01b211d518f6a53858a94d03
+size 4832680552

model-00023-of-00026.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f7c1901b24f5b936153c2e3b62bf771adf2d40872dac9de74f245f311564aaa7
+size 4857866352

model-00024-of-00026.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8eef824d2a23280b7a50e51e21c04a2b3d14d223dd4846d514500797e8f3bd62
+size 4832680552

model-00025-of-00026.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a00bf1cba7614844deca4c718f0a59db60d5c2232ea250b1291a37ff07bfe67f
+size 4857866352

model-00026-of-00026.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1a40641032fe24b4b82552eac056d8b1310fd580f720fe4d42ab808c53695f28
+size 2189695048

recipe.yaml

@@ -0,0 +1,7 @@
+default_stage:
+  default_modifiers:
+    QuantizationModifier:
+      targets: [Linear]
+      ignore: [lm_head]
+      scheme: FP8_DYNAMIC
+      bypass_divisibility_checks: false

special_tokens_map.json

@@ -0,0 +1,23 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1b968b8dc352f42192367337c78ccc61e1eaddc6d641a579372d4f20694beb7a
+size 587562