Upload README.md

README.md

@@ -1,17 +1,226 @@
-# Merge Report: TripleTrouble-V3 (v6 Geodesic Refinement)
-
-This model represents an advanced merge of three state-of-the-art checkpoints from the **Qwen 35B-A3B** sparse Mixture-of-Experts architecture (40 layers, 256 routed experts + 1 shared expert, hybrid Gated DeltaNet linear recurrence).
-
-### Merged Checkpoints
-1. **Anchor / Codebase Specialist:** `Kwaipilot/KAT-Coder-V2.5-Dev` - Excels at autonomous repository manipulation, SWE-bench coding workflows, and syntax trees.
-2. **Auxiliary 1 / Agentic Reasoning:** `ornith-ai/Ornith-1.5-35B-A3B` - Excels at multi-turn tool calling, complex reasoning chains, and agentic workflows.
-3. **Auxiliary 2 / World Simulator:** `Qwen/Qwen-AgentWorld-35B-A3B` - Native language world model specialized in next-state environment simulation.
-
-### Mathematical Innovations
-* **Decoupled Normalized Geodesic Consensus (NGC):** Separates directional alignment from magnitude scaling, projecting parameters onto unit hyperspheres before scaling by the expected target Frobenius norm.
-* **Row-Wise Router Manifold Calibration:** Calibrates all 256 expert routing hyperplanes row-by-row on `mlp.gate.weight`, preserving routing logit entropy and activation thresholds.
-* **Functional Depth-Aware Scheduling:**
-  - *Early Layers (0-11):* Weighted toward KAT-Coder (up to 53%) to lock in low-level code syntax and AST parsing.
-  - *Middle Layers (12-27):* AgentWorld peaks (up to 35%) to supply rich environment transition dynamics.
-  - *Deep Layers (28-39):* Ornith dominates (up to 47%) to command tool execution, planning schemas, and API formatting.
-* **DeltaNet Spectral Stability:** Maintains spectral radius and contraction operators across recurrent attention matrices.
+---
+license: apache-2.0
+language:
+- en
+- zh
+pipeline_tag: text-generation
+tags:
+- merge
+- moe
+- qwen
+- qwen-35b-a3b
+- agent
+- coding
+- tool-use
+- world-model
+- gated-deltanet
+base_model:
+- Kwaipilot/KAT-Coder-V2.5-Dev
+- ornith-ai/Ornith-1.5-35B-A3B
+- Qwen/Qwen-AgentWorld-35B-A3B
+model_creator: OliviaRossi
+model_name: TripleTrouble-V3
+---
+
+<div align="center">
+
+# ⚡ TripleTrouble-V3 ⚡
+### *The Triad of Code, Tool Agency, and World Simulation*
+
+[![Model Architecture](https://img.shields.io/badge/Architecture-Qwen--35B--A3B-blueviolet?style=for-the-badge&logo=cpu)](https://huggingface.co/OliviaRossi/TripleTrouble-V3)
+[![Total Params](https://img.shields.io/badge/Total%20Params-34.7B-blue?style=for-the-badge)](https://huggingface.co/OliviaRossi/TripleTrouble-V3)
+[![Active Params](https://img.shields.io/badge/Active%20Params-~3.3B%20%2F%20token-cyan?style=for-the-badge)](https://huggingface.co/OliviaRossi/TripleTrouble-V3)
+[![MoE Topology](https://img.shields.io/badge/MoE-256%20Routed%20%2B%201%20Shared-success?style=for-the-badge)](https://huggingface.co/OliviaRossi/TripleTrouble-V3)
+[![License](https://img.shields.io/badge/License-Apache%202.0-red?style=for-the-badge)](https://www.apache.org/licenses/LICENSE-2.0)
+
+<p align="center">
+  <b>A unified sovereign agent model merging three apex Qwen 35B-A3B checkpoints via Decoupled Normalized Geodesic Consensus (NGC), Row-Wise Router Manifold Calibration, and Functional Depth Scheduling.</b>
+</p>
+
+[Model Details](#-overview) • [The Triad](#-the-triad) • [Merge Engineering](#-merge-mathematics) • [Serving with vLLM](#-fast-inference-with-vllm) • [Transformers](#-quickstart-transformers)
+
+---
+
+</div>
+
+## 🌌 Overview
+
+**TripleTrouble-V3** is a 35B-class sparse Mixture-of-Experts (MoE) foundation model created by fusing three divergent, specialized post-trained models of the **Qwen 35B-A3B** architecture:
+
+1. **The Coder (KAT-Coder-V2.5-Dev):** Autonomous codebase manipulation, SWE-bench refactoring, repository traversal, and concrete AST syntax trees.
+2. **The Agent (Ornith-1.5-35B-A3B):** Multi-turn API calling, complex structured JSON emissions, mathematical deduction, and competitive programmatic reasoning.
+3. **The Simulator (Qwen-AgentWorld-35B-A3B):** Large World Model (LWM) dynamics, environment state transition modeling (MCP, OS, Web, Android), and next-state observation forecasting.
+
+By moving beyond naive weight averaging and avoiding the rank collapse of pseudo-base TIES pruning, **TripleTrouble-V3** preserves the singular value spectrum of all 256 experts and maintains the spectral radius of Qwen's hybrid **Gated DeltaNet** linear recurrence dynamics.
+
+---
+
+## 🧬 The Triad
+
+```mermaid
+flowchart TD
+    subgraph MERGED ["⚡ TripleTrouble-V3"]
+        ROOT["<b>TripleTrouble-V3</b><br/>34.7B MoE • ~3.3B Active per token"]
+    end
+
+    ROOT -->|"40% Weight"| KAT["💻 <b>KAT-Coder-V2.5-Dev</b><br/>• Code & SWE-bench Refactoring<br/>• AST & Syntax Trees<br/>• Terminal Command Logic"]
+    ROOT -->|"35% Weight"| ORN["🦅 <b>Ornith-1.5-35B-A3B</b><br/>• Multi-Turn Tool Calling<br/>• MCP Protocol Execution<br/>• Structured JSON & Math"]
+    ROOT -->|"25% Weight"| AGW["🌐 <b>Qwen-AgentWorld-35B</b><br/>�� Large World Model (LWM)<br/>• Environment Dynamics<br/>• State Observation Loops"]
+
+    classDef default fill:#1e293b,stroke:#475569,stroke-width:1px,color:#f8fafc;
+    classDef highlight fill:#4338ca,stroke:#818cf8,stroke-width:2px,color:#ffffff;
+    class ROOT highlight;
+```
+
+---
+
+## 🧮 Merge Mathematics
+
+Traditional model mergers suffer from two major failure modes on 35B hybrid MoE models:
+* **Naive Linear Soups:** Cause high-dimensional variance collapse ($\mathbb{E}[\|\sum w_i \theta_i\|] \ll \mathbb{E}[\|\theta_i\|]$), dampening activations across 40 layers.
+* **Pseudo-Base TIES/DARE:** Truncate 70% of delta coordinates, destroying the singular value spectrum of small ($2048 \times 1024$) MoE experts and shattering the Householder contraction operator of Gated DeltaNet attention.
+
+**TripleTrouble-V3** was built using a continuous hyperspherical framework designed specifically for this architecture:
+
+### 1. Decoupled Normalized Geodesic Consensus (NGC)
+Directional consensus and magnitude scaling are strictly decoupled. Checkpoints with larger gradient norms cannot skew the consensus angle:
+
+$$\hat{W}_i = \frac{W_i}{\|W_i\|_F}$$
+
+$$D_{\text{consensus}} = \sum_{i=1}^3 w_i(l) \hat{W}_i, \quad \hat{D} = \frac{D_{\text{consensus}}}{\|D_{\text{consensus}}\|_F}$$
+
+$$W_{\text{final}} = \hat{D} \times \left( \sum_{i=1}^3 w_i(l) \|W_i\|_F \right)$$
+
+### 2. Row-Wise Router Manifold Calibration
+In an MoE with 256 routed experts, the router gate $W_{\text{gate}} \in \mathbb{R}^{256 \times d_{\text{model}}}$ consists of 256 distinct hyperplanes $g_e$. Global matrix scaling alters individual expert activation probabilities. We calibrate every expert hyperplane row-by-row:
+
+$$g_{e, \text{final}} = \frac{\sum_i w_i \frac{g_{e, i}}{\|g_{e, i}\|_2}}{\left\| \sum_i w_i \frac{g_{e, i}}{\|g_{e, i}\|_2} \right\|_2} \times \left( \sum_{i=1}^3 w_i \|g_{e, i}\|_2 \right)$$
+
+This guarantees router logit sharpness, prevents entropy collapse, and keeps expert dispatch distributions intact.
+
+### 3. Functional Depth-Aware Dynamic Scheduling
+Layer weights dynamically modulate across the 40 layers to match functional network mechanics:
+
+| Depth Bracket | Target Specialization | Dominant Driver | Allocation Distribution |
+| :--- | :--- | :--- | :--- |
+| **Layers 0 – 11**<br>*(Shallow)* | **Syntax & Token ASTs**<br>Code grammar, indentation, and subword abstractions | **KAT-Coder**<br>`~53%` | `KAT` 🟩🟩🟩🟩🟩<br>`ORN` 🟦🟦<br>`AGW` 🟨🟨 |
+| **Layers 12 – 27**<br>*(Middle)* | **Latent World Simulation**<br>Environment state tracking and entity transition dynamics | **AgentWorld**<br>`~35% (Peak)` | `KAT` 🟩🟩🟩<br>`ORN` 🟦🟦🟦<br>`AGW` 🟨🟨🟨🟨 |
+| **Layers 28 – 39**<br>*(Deep)* | **Action Policy & Tool Execution**<br>Function schemas, JSON formatting, and multi-turn planning | **Ornith**<br>`~47%` | `KAT` 🟩🟩🟩<br>`ORN` 🟦🟦🟦🟦🟦<br>`AGW` 🟨🟨 |
+
+---
+
+## ⚙️ Architectural Specifications
+
+| Parameter | Value |
+| :--- | :--- |
+| **Total Parameters** | **34.7 Billion** |
+| **Active Parameters per Token** | **~3.3 Billion** |
+| **Layers** | **40** |
+| **Routed Experts** | **256** (Top-8 active per token) |
+| **Shared Experts** | **1** (Always active) |
+| **Attention Mechanism** | **Hybrid Gated DeltaNet (Linear Attention) + GQA** |
+| **Attention Heads** | **16 Query Heads / 2 Key-Value Heads** |
+| **Hidden Dimension ($d_{\text{model}}$)** | **2048** |
+| **Intermediate Dimension ($d_{\text{ffn}}$)** | **1024 (per routed expert)** |
+| **Vocabulary Size** | **248,320** |
+| **Context Window** | **131,072 tokens** |
+
+---
+
+## 🚀 Fast Inference with vLLM
+
+Thanks to its sparse MoE architecture, **TripleTrouble-V3** runs at the inference speed of a ~3.3B dense model while delivering 35B-scale reasoning.
+
+### Installation
+```bash
+pip install vllm>=0.6.0
+```
+
+### Launch an OpenAI-Compatible API Server
+```bash
+vllm serve OliviaRossi/TripleTrouble-V3 \
+  --tensor-parallel-size 2 \
+  --max-model-len 32768 \
+  --gpu-memory-utilization 0.90 \
+  --trust-remote-code
+```
+*(For a single 80GB GPU, run using FP8 or AWQ quantization: `--quantization fp8`)*.
+
+---
+
+## 💻 Quickstart: Transformers
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_id = "OliviaRossi/TripleTrouble-V3"
+
+tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
+model = AutoModelForCausalLM.from_pretrained(
+    model_id,
+    torch_dtype=torch.bfloat16,
+    device_map="auto",
+    trust_remote_code=True
+)
+
+messages = [
+    {
+        "role": "system",
+        "content": (
+            "You are TripleTrouble, an expert agent combining deep codebase mastery, "
+            "rigorous multi-turn tool planning, and environment simulation capabilities."
+        )
+    },
+    {
+        "role": "user",
+        "content": "Analyze the concurrency bottlenecks in an asynchronous Python producer-consumer queue and write a resilient implementation using asyncio.Queue."
+    }
+]
+
+prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
+
+outputs = model.generate(
+    **inputs,
+    max_new_tokens=2048,
+    temperature=0.6,
+    top_p=0.9,
+    repetition_penalty=1.05
+)
+
+response = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
+print(response)
+```
+
+---
+
+## 🛠️ Recommended Sampling Parameters
+
+| Workload | Temperature | Top-P | Repetition Penalty | Notes |
+| :--- | :---: | :---: | :---: | :--- |
+| **Code Synthesis & Bug Fixing** | `0.2` | `0.85` | `1.02` | Maximizes syntax precision and strict AST adherence. |
+| **Tool Calling & MCP Agents** | `0.3` | `0.90` | `1.00` | Ensures strict valid JSON schemas and tool-call formatting. |
+| **General Problem Solving** | `0.6` | `0.92` | `1.05` | Balances creative reasoning with grounded problem deduction. |
+| **World Simulation & Planning** | `0.7` | `0.95` | `1.08` | Explores diverse trajectory states and action paths. |
+
+---
+
+## ⚖️ License & Attribution
+
+This model is released under the **Apache 2.0 License**.
+
+### Source Models:
+* **KAT-Coder-V2.5-Dev** by [Kwaipilot](https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev)
+* **Ornith-1.5-35B-A3B** by [ornith-ai](https://huggingface.co/ornith-ai/Ornith-1.5-35B-A3B)
+* **Qwen-AgentWorld-35B-A3B** by [Qwen / Alibaba Cloud](https://huggingface.co/Qwen/Qwen-AgentWorld-35B-A3B)
+
+```bibtex
+@misc{tripletrouble2026,
+  author = {Olivia Rossi},
+  title = {TripleTrouble-V3: A Geodesic Manifold Merge of Code, Tool Agency, and World Simulation},
+  year = {2026},
+  publisher = {Hugging Face},
+  howpublished = {\url{https://huggingface.co/OliviaRossi/TripleTrouble-V3}}
+}
+```