--- base_model: - Jackrong/Qwopus3.6-35B-A3B-Coder - ornith-ai/Ornith-1.5-35B-A3B - Kwaipilot/KAT-Coder-V2.5-Dev - Qwen/Qwen-AgentWorld-35B-A3B base_model_relation: merge library_name: transformers tags: - merge - ties - dare - moe - qwen - qwen3.5 - qwen3.6 - causal-lm - deltanet - agentic - reasoning - code license: apache-2.0 language: - en - zh pipeline_tag: text-generation model_type: qwen3_5_moe ---
[![License](https://img.shields.io/badge/License-Apache%202.0-6E56CF?style=for-the-badge)](https://opensource.org/licenses/Apache-2.0) [![Library](https://img.shields.io/badge/Library-transformers-FFD21E?style=for-the-badge&logo=huggingface&logoColor=black)](https://github.com/huggingface/transformers) [![Merge Method](https://img.shields.io/badge/Merge%20Method-DARE--TIES-27AE60?style=for-the-badge)](#merge-methodology) [![Architecture](https://img.shields.io/badge/Architecture-Qwen%2035B--A3B%20MoE-2D9CDB?style=for-the-badge)](#architectural-specifications) [![Experts](https://img.shields.io/badge/Routed%20Experts-256-EB5757?style=for-the-badge)](#architectural-specifications) [![Layers](https://img.shields.io/badge/Decoder%20Layers-40-F2994A?style=for-the-badge)](#architectural-specifications)
A four-way MoE merge of the Qwen 35B-A3B architecture, fusing task vectors from three specialized fine-tunes into a base anchor via DARE-TIES with sinusoidal depth modulation. > [!IMPORTANT] > Designed specifically to consolidate software engineering, code synthesis, and agentic tool execution capabilities. Multimodal vision weights and Multi-Token Prediction (MTP) heads were stripped to reduce VRAM footprint and maximize throughput during coding tasks. --- ### Contents - [Architectural Specifications](#architectural-specifications) - [Composition](#composition) - [Merge Methodology](#merge-methodology) - [Layer-Stratified Policies](#layer-stratified-policies) - [Chat Template](#chat-template) - [Generation Parameters](#recommended-generation-parameters) - [How to Use](#how-to-use) - [Lineage](#lineage) - [References](#citation--references) --- ## Architectural Specifications | Spec | Value | | :--- | :---: | | Total parameters | 35B | | Active parameters / token | 3B | | Decoder layers | 40 | | Routed experts | 256 | | Shared experts | 1 | | Attention | Gated DeltaNet hybrid linear attention | | Merge algorithm | DARE-TIES + sine depth scaling | ## Composition `Jackrong/Qwopus3.6-35B-A3B-Coder` serves as the base anchor (W₀); the remaining three models contribute task vectors at the listed weights. | Model | Role | Task Weight (α) | | :--- | :--- | :---: | | [Jackrong/Qwopus3.6-35B-A3B-Coder](https://huggingface.co/Jackrong/Qwopus3.6-35B-A3B-Coder) | Base anchor (W₀) | 1.00 | | [ornith-ai/Ornith-1.5-35B-A3B](https://huggingface.co/ornith-ai/Ornith-1.5-35B-A3B) | Donor (D₁) | 0.30 | | [Kwaipilot/KAT-Coder-V2.5-Dev](https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev) | Donor (D₂) | 0.25 | | [Qwen/Qwen-AgentWorld-35B-A3B](https://huggingface.co/Qwen/Qwen-AgentWorld-35B-A3B) | Donor (D₃) | 0.20 | --- ## Merge Methodology For each floating-point parameter, a task delta is computed per donor \\(k\\): $$ \Delta_k = D_k - W_0 $$ **DARE pruning.** A Bernoulli mask at retention density \\(p\\) zeroes out low-magnitude updates; surviving values are rescaled by \\(p^{-1}\\): $$ \tilde{\Delta}_k = \frac{1}{p} \left(\Delta_k \odot M_k\right), \quad M_k \sim \text{Bernoulli}(p) $$ **TIES sign election.** A consensus sign \\(\Gamma\\) is computed via weighted vote across donors, and any donor update conflicting with it is dropped before averaging: $$ \Gamma = \operatorname{sgn}\left(\sum_{k=1}^K \alpha_k \tilde{\Delta}_k\right) $$ $$ \Delta_{\text{TIES}} = \frac{\sum_{k=1}^K \alpha_k \tilde{\Delta}_k \odot \mathbb{I}\left(\operatorname{sgn}(\tilde{\Delta}_k) = \Gamma\right)}{\sum_{k=1}^K \alpha_k \cdot \mathbb{I}\left(\operatorname{sgn}(\tilde{\Delta}_k) = \Gamma\right) + \epsilon} $$ **Depth-scaled reconstruction.** The merged weight is reconstructed as: $$ W_{\text{final}} = W_0 + \lambda(l) \cdot \Delta_{\text{TIES}} $$ where the layer scaling factor \\(\lambda(l)\\) across decoder layer index \\(l \in [0, 39]\\) is defined as: $$ \lambda(l) = \beta \cdot \left(0.5 + 0.5 \sin\left(\pi \frac{l}{39}\right)\right) $$ This keeps input/output projections closer to the base and applies the strongest task transfer to middle layers \\((l \in [12, 28])\\). --- ## Layer-Stratified Policies | Parameter Group | Match Substring | Policy | Density (p) | Base Scale (β) | | :--- | :--- | :---: | :---: | :---: | | **Embeddings / LM head** | `embed_tokens`, `lm_head` | Linear | — | 1.00 | | **Norms / biases** | `norm`, `bias`, 1D tensors | Linear | — | 1.00 | | **DeltaNet recurrent state** | `a_log`, `dt_bias`, `conv1d` | Linear | — | 1.00 | | **MoE router gate** | `mlp.gate.weight`, `block_sparse_moe.gate` | Linear | — | 1.00 | | **MoE shared expert** | `shared_expert` | DARE‑TIES | 0.70 | 0.60 | | **Attention projections** | `attn`, `rotary`, `in_proj`, `out_proj`, `x_proj` | DARE‑TIES | 0.75 | 0.60 | | **Routed experts (×256)** | `experts`, `mlp` | DARE‑TIES | 0.65 | 0.55 | - **Router protection:** Gate weights use linear interpolation (~57% base, ~43% donors) rather than DARE to avoid destabilizing expert routing. - **DeltaNet stability:** Recurrent state kernels are excluded from DARE to prevent divergence in the linear-attention state space. - **MTP removed:** Multi-token-prediction heads beyond the 40 primary decoder blocks were stripped for standard CausalLM inference. --- ## Chat Template This model uses the [Improved Chat Template for Qwen 3.x by Olivia Rossi](https://huggingface.co/OliviaRossi/Improved-Chat-Template-for-Qwen-3.x) to support multi-tier Chain-of-Thought (CoT) reasoning, dual-format agentic tool execution, automatic error-recovery heuristics, and strict token-waste elimination. --- ## Recommended Generation Parameters For code generation and agentic task trajectories, avoid high temperatures to maintain routing stability and syntax validity. | Parameter | Coding / Terminal Agent | Creative Reasoning | | :--- | :---: | :---: | | **Temperature** | `0.6` | `1.0` | | **Top-P** | `0.95` | `0.95` | | **Min-P** | `0.0` | `0.01` | | **Repetition Penalty** | `off` | `1.05` | --- ## How to Use ### Transformers ```python import torch from transformers import AutoModelForCausalLM, AutoTokenizer model_id = "pragmaticcs/SignOfFour" tokenizer = AutoTokenizer.from_pretrained(model_id) model = AutoModelForCausalLM.from_pretrained( model_id, torch_dtype=torch.bfloat16, device_map="auto", ) messages = [ {"role": "system", "content": "You are a precise agentic software engineer. Solve problems concisely."}, {"role": "user", "content": "Write an asynchronous Python queue consumer with retry backoff."} ] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, return_tensors="pt" ).to(model.device) output = model.generate( inputs, max_new_tokens=1024, temperature=0.6, top_p=0.95, min_p=0.01, do_sample=True, ) print(tokenizer.decode(output[0][inputs.shape[-1]:], skip_special_tokens=True)) ``` --- ## Lineage ``` Qwen/Qwen3.6-35B-A3B └── pragmaticcs/SignOfFour ├── base: Jackrong/Qwopus3.6-35B-A3B-Coder ├── donor: ornith-ai/Ornith-1.5-35B-A3B ├── donor: Kwaipilot/KAT-Coder-V2.5-Dev └── donor: Qwen/Qwen-AgentWorld-35B-A3B ``` --- ## Citation & References - [Jackrong/Qwopus3.6-35B-A3B-Coder](https://huggingface.co/Jackrong/Qwopus3.6-35B-A3B-Coder) - [ornith-ai/Ornith-1.5-35B-A3B](https://huggingface.co/ornith-ai/Ornith-1.5-35B-A3B) - [Kwaipilot/KAT-Coder-V2.5-Dev](https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev) - [Qwen/Qwen-AgentWorld-35B-A3B](https://huggingface.co/Qwen/Qwen-AgentWorld-35B-A3B) - [Improved Chat Template for Qwen 3.x](https://huggingface.co/OliviaRossi/Improved-Chat-Template-for-Qwen-3.x) ```bibtex @inproceedings{yu2024dare, title={Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch}, author={Yu, Le and Yu, Bowen and Yu, Haiyang and Huang, Fei and Li, Yongbin}, booktitle={International Conference on Machine Learning (ICML)}, year={2024} } @inproceedings{yadav2023ties, title={Resolving Interference When Merging Models}, author={Yadav, Prateek and Tam, Derek and Choshen, Leshem and Raffel, Colin and Bansal, Mohit}, booktitle={Advances in Neural Information Processing Systems (NeurIPS)}, year={2023} } ```