--- base_model: - Jackrong/Qwopus3.6-35B-A3B-Coder - ornith-ai/Ornith-1.5-35B-A3B - Kwaipilot/KAT-Coder-V2.5-Dev - Qwen/Qwen-AgentWorld-35B-A3B base_model_relation: merge library_name: transformers tags: - merge - ties - dare - moe - qwen - qwen3.5 - qwen3.6 - causal-lm - deltanet - agentic - reasoning - code license: apache-2.0 language: - en - zh pipeline_tag: text-generation model_type: qwen3_5_moe ---
![PentaCoder-9B](https://huggingface.co/pragmaticcs/Qwen-35B-A3B-SignOfFour-Coder/resolve/main/assets/banner.png) [![License](https://img.shields.io/badge/License-Apache%202.0-6E56CF?style=for-the-badge)](https://opensource.org/licenses/Apache-2.0) [![Library](https://img.shields.io/badge/Library-transformers-FFD21E?style=for-the-badge&logo=huggingface&logoColor=black)](https://github.com/huggingface/transformers) [![Merge Method](https://img.shields.io/badge/Merge%20Method-DARE--TIES-27AE60?style=for-the-badge)](#merge-methodology) [![Architecture](https://img.shields.io/badge/Architecture-Qwen%2035B--A3B%20MoE-2D9CDB?style=for-the-badge)](#architectural-specifications) [![Experts](https://img.shields.io/badge/Routed%20Experts-256-EB5757?style=for-the-badge)](#architectural-specifications) [![Layers](https://img.shields.io/badge/Decoder%20Layers-40-F2994A?style=for-the-badge)](#architectural-specifications)
A four-way MoE merge of the Qwen 35B-A3B architecture, fusing task vectors from three specialized fine-tunes into a base anchor via DARE-TIES with sinusoidal depth modulation. > [!IMPORTANT] > Designed specifically to consolidate software engineering, code synthesis, and agentic tool execution capabilities. Multimodal vision weights and Multi-Token Prediction (MTP) heads were stripped to reduce VRAM footprint and maximize throughput during coding tasks. --- ### Contents - [Architectural Specifications](#architectural-specifications) - [Composition](#composition) - [Merge Methodology](#merge-methodology) - [Layer-Stratified Policies](#layer-stratified-policies) - [Chat Template](#chat-template) - [Available quants](#available-quants) - [Generation Parameters](#recommended-generation-parameters) - [Lineage](#lineage) - [References](#citation--references) --- ## Architectural Specifications | Spec | Value | | :--- | :---: | | Total parameters | 35B | | Active parameters / token | 3B | | Decoder layers | 40 | | Routed experts | 256 | | Shared experts | 1 | | Attention | Gated DeltaNet hybrid linear attention | | Merge algorithm | DARE-TIES + sine depth scaling | ## Composition `Jackrong/Qwopus3.6-35B-A3B-Coder` serves as the base anchor (W₀); the remaining three models contribute task vectors at the listed weights. | Model | Role | Task Weight (α) | | :--- | :--- | :---: | | [Jackrong/Qwopus3.6-35B-A3B-Coder](https://huggingface.co/Jackrong/Qwopus3.6-35B-A3B-Coder) | Base anchor (W₀) | 1.00 | | [ornith-ai/Ornith-1.5-35B-A3B](https://huggingface.co/ornith-ai/Ornith-1.5-35B-A3B) | Donor (D₁) | 0.30 | | [Kwaipilot/KAT-Coder-V2.5-Dev](https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev) | Donor (D₂) | 0.25 | | [Qwen/Qwen-AgentWorld-35B-A3B](https://huggingface.co/Qwen/Qwen-AgentWorld-35B-A3B) | Donor (D₃) | 0.20 | --- ## Merge Methodology For each floating-point parameter, a task delta is computed per donor \\(k\\): $$ \Delta_k = D_k - W_0 $$ **DARE pruning.** A Bernoulli mask at retention density \\(p\\) zeroes out low-magnitude updates; surviving values are rescaled by \\(p^{-1}\\): $$ \tilde{\Delta}_k = \frac{1}{p} \left(\Delta_k \odot M_k\right), \quad M_k \sim \text{Bernoulli}(p) $$ **TIES sign election.** A consensus sign \\(\Gamma\\) is computed via weighted vote across donors, and any donor update conflicting with it is dropped before averaging: $$ \Gamma = \operatorname{sgn}\left(\sum_{k=1}^K \alpha_k \tilde{\Delta}_k\right) $$ $$ \Delta_{\text{TIES}} = \frac{\sum_{k=1}^K \alpha_k \tilde{\Delta}_k \odot \mathbb{I}\left(\operatorname{sgn}(\tilde{\Delta}_k) = \Gamma\right)}{\sum_{k=1}^K \alpha_k \cdot \mathbb{I}\left(\operatorname{sgn}(\tilde{\Delta}_k) = \Gamma\right) + \epsilon} $$ **Depth-scaled reconstruction.** The merged weight is reconstructed as: $$ W_{\text{final}} = W_0 + \lambda(l) \cdot \Delta_{\text{TIES}} $$ where the layer scaling factor \\(\lambda(l)\\) across decoder layer index \\(l \in [0, 39]\\) is defined as: $$ \lambda(l) = \beta \cdot \left(0.5 + 0.5 \sin\left(\pi \frac{l}{39}\right)\right) $$ This keeps input/output projections closer to the base and applies the strongest task transfer to middle layers \\((l \in [12, 28])\\). --- ## Layer-Stratified Policies | Parameter Group | Match Substring | Policy | Density (p) | Base Scale (β) | | :--- | :--- | :---: | :---: | :---: | | **Embeddings / LM head** | `embed_tokens`, `lm_head` | Linear | — | 1.00 | | **Norms / biases** | `norm`, `bias`, 1D tensors | Linear | — | 1.00 | | **DeltaNet recurrent state** | `a_log`, `dt_bias`, `conv1d` | Linear | — | 1.00 | | **MoE router gate** | `mlp.gate.weight`, `block_sparse_moe.gate` | Linear | — | 1.00 | | **MoE shared expert** | `shared_expert` | DARE‑TIES | 0.70 | 0.60 | | **Attention projections** | `attn`, `rotary`, `in_proj`, `out_proj`, `x_proj` | DARE‑TIES | 0.75 | 0.60 | | **Routed experts (×256)** | `experts`, `mlp` | DARE‑TIES | 0.65 | 0.55 | - **Router protection:** Gate weights use linear interpolation (~57% base, ~43% donors) rather than DARE to avoid destabilizing expert routing. - **DeltaNet stability:** Recurrent state kernels are excluded from DARE to prevent divergence in the linear-attention state space. - **MTP removed:** Multi-token-prediction heads beyond the 40 primary decoder blocks were stripped for standard CausalLM inference. --- ## Chat Template This model uses the [Improved Chat Template for Qwen 3.x by Olivia Rossi](https://huggingface.co/OliviaRossi/Improved-Chat-Template-for-Qwen-3.x) to support multi-tier Chain-of-Thought (CoT) reasoning, dual-format agentic tool execution, automatic error-recovery heuristics, and strict token-waste elimination. --- ## Available quants | Quantization | File | Size | | :--- | :--- | :---: | | **Q4_K_M** | `Qwen-35B-A3B-SignOfFour-Coder-Q4_K_M.gguf` | 21.5 GB | | **APEX-High-Quality** | `Qwen-35B-A3B-SignOfFour-Coder-APEX-High-Quality.gguf` | 25.1 GB | | **Q5_K_M** | `Qwen-35B-A3B-SignOfFour-Coder-Q5_K_M.gguf` | 25.0 GB | | **Q6_K** | `Qwen-35B-A3B-SignOfFour-Coder-Q6_K.gguf` | 28.7 GB | | **Q8_0** | `Qwen-35B-A3B-SignOfFour-Coder-Q8_0.gguf` | 36.9 GB | | **BF16 (Split)** | `Qwen-35B-A3B-SignOfFour-Coder-BF16-*.gguf` | 69.4 GB | > [!TIP] > **Need fast GPU inference?** Running a 35B MoE with CPU offloading or on under-provisioned consumer RAM can result in low token throughput. If you lack a local 24 GB+ GPU and want faster inference, you can spin up an hourly instance on **RunPod** in one click to run `llama-server` with **100% layer offload**. New accounts created through the link below receive **bonus compute credits** ($5–$500 on your first $10 top-up). > > [![Deploy on RunPod](https://img.shields.io/badge/RunPod-Deploy%20Cloud%20Pod-5D29F0?style=for-the-badge)](https://runpod.io?ref=zk13vwvk) > > ***Affiliate Disclosure:** If you register an account and load funds using this link, I receive a small commission and platform credits from RunPod at **no extra cost to you**. These earnings are directly reinvested into renting GPU compute for ongoing open-source model merging, testing, and evaluation. Thank you for your support!* --- ## Recommended Generation Parameters For code generation and agentic task trajectories, avoid high temperatures to maintain routing stability and syntax validity. | Parameter | Coding / Terminal Agent | Creative Reasoning | | :--- | :---: | :---: | | **Temperature** | `0.6` | `1.0` | | **Top-P** | `0.95` | `0.95` | | **Min-P** | `0.0` | `0.01` | | **Repetition Penalty** | `off` | `1.05` | --- ## Lineage ``` Qwen/Qwen3.6-35B-A3B └── pragmaticcs/Qwen-35B-A3B-SignOfFour-Coder ├── base: Jackrong/Qwopus3.6-35B-A3B-Coder ├── donor: ornith-ai/Ornith-1.5-35B-A3B ├── donor: Kwaipilot/KAT-Coder-V2.5-Dev └── donor: Qwen/Qwen-AgentWorld-35B-A3B ``` --- ## Citation & References - [Jackrong/Qwopus3.6-35B-A3B-Coder](https://huggingface.co/Jackrong/Qwopus3.6-35B-A3B-Coder) - [ornith-ai/Ornith-1.5-35B-A3B](https://huggingface.co/ornith-ai/Ornith-1.5-35B-A3B) - [Kwaipilot/KAT-Coder-V2.5-Dev](https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev) - [Qwen/Qwen-AgentWorld-35B-A3B](https://huggingface.co/Qwen/Qwen-AgentWorld-35B-A3B) - [Improved Chat Template for Qwen 3.x](https://huggingface.co/OliviaRossi/Improved-Chat-Template-for-Qwen-3.x) ```bibtex @inproceedings{yu2024dare, title={Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch}, author={Yu, Le and Yu, Bowen and Yu, Haiyang and Huang, Fei and Li, Yongbin}, booktitle={International Conference on Machine Learning (ICML)}, year={2024} } @inproceedings{yadav2023ties, title={Resolving Interference When Merging Models}, author={Yadav, Prateek and Tam, Derek and Choshen, Leshem and Raffel, Colin and Bansal, Mohit}, booktitle={Advances in Neural Information Processing Systems (NeurIPS)}, year={2023} } ```