Update README.md

README.md

@@ -1,3 +1,217 @@
 ---
+base_model:
+  - Jackrong/Qwopus3.6-35B-A3B-Coder
+  - ornith-ai/Ornith-1.5-35B-A3B
+  - Kwaipilot/KAT-Coder-V2.5-Dev
+  - Qwen/Qwen-AgentWorld-35B-A3B
+base_model_relation: merge
+library_name: transformers
+tags:
+  - merge
+  - ties
+  - dare
+  - moe
+  - qwen
+  - causal-lm
+  - deltanet
+  - agentic
+  - code
 license: apache-2.0
+language:
+  - en
+  - zh
+pipeline_tag: text-generation
+model_type: qwen3_5_moe
 ---
+
+<div align="center">
+
+[![License](https://img.shields.io/badge/License-Apache%202.0-6E56CF?style=for-the-badge)](https://opensource.org/licenses/Apache-2.0)
+[![Library](https://img.shields.io/badge/Library-transformers-FFD21E?style=for-the-badge&logo=huggingface&logoColor=black)](https://github.com/huggingface/transformers)
+[![Merge Method](https://img.shields.io/badge/Merge%20Method-DARE--TIES-27AE60?style=for-the-badge)](#merge-methodology)
+
+[![Architecture](https://img.shields.io/badge/Architecture-Qwen%2035B--A3B%20MoE-2D9CDB?style=for-the-badge)](#architectural-specifications)
+[![Experts](https://img.shields.io/badge/Routed%20Experts-256-EB5757?style=for-the-badge)](#architectural-specifications)
+[![Layers](https://img.shields.io/badge/Decoder%20Layers-40-F2994A?style=for-the-badge)](#architectural-specifications)
+
+</div>
+
+A four-way merge of the Qwen 35B-A3B Mixture-of-Experts architecture, integrating task deltas from three fine-tunes into a base anchor via DARE-TIES with sinusoidal depth modulation.
+
+> [!IMPORTANT]
+> Merged specifically to consolidate software engineering, code synthesis, and agentic tool execution capabilities. Multimodal vision weights and Multi-Token Prediction (MTP) heads were stripped to reduce VRAM footprint and maximize throughput during coding tasks.
+
+---
+
+### Contents
+
+- [Architectural Specifications](#architectural-specifications) 
+- [Composition](#composition) 
+- [Merge Methodology](#merge-methodology) 
+- [Layer-Stratified Policies](#layer-stratified-policies) 
+- [Generation Parameters](#recommended-generation-parameters) 
+- [How to Use](#how-to-use)
+- [Lineage](#lineage) 
+- [References](#citation--references)
+
+---
+
+## Architectural Specifications
+
+| Spec | Value |
+| :--- | :---: |
+| Total parameters | 35B |
+| Active parameters / token | 3B |
+| Decoder layers | 40 |
+| Routed experts | 256 |
+| Shared experts | 1 |
+| Attention | Gated DeltaNet hybrid linear attention |
+| Merge algorithm | DARE-TIES + sine depth scaling |
+
+## Composition
+
+`Jackrong/Qwopus3.6-35B-A3B-Coder` serves as the base anchor (W₀); the remaining three models contribute task vectors at the listed weights.
+
+| Model | Role | Task Weight (α) |
+| :--- | :--- | :---: |
+| [Jackrong/Qwopus3.6-35B-A3B-Coder](https://huggingface.co/Jackrong/Qwopus3.6-35B-A3B-Coder) | Base anchor (W₀) | 1.00 |
+| [ornith-ai/Ornith-1.5-35B-A3B](https://huggingface.co/ornith-ai/Ornith-1.5-35B-A3B) | Donor (D₁) | 0.30 |
+| [Kwaipilot/KAT-Coder-V2.5-Dev](https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev) | Donor (D₂) | 0.25 |
+| [Qwen/Qwen-AgentWorld-35B-A3B](https://huggingface.co/Qwen/Qwen-AgentWorld-35B-A3B) | Donor (D₃) | 0.20 |
+
+---
+
+## Merge Methodology
+
+For each floating-point parameter, a task delta is computed per donor $k$:
+
+$$\Delta_k = D_k - W_0$$
+
+**DARE pruning.** A Bernoulli mask at retention density $p$ zeroes out low-magnitude updates; surviving values are rescaled by $p^{-1}$:
+
+$$\tilde{\Delta}_k = \frac{1}{p} \left(\Delta_k \odot M_k\right), \quad M_k \sim \text{Bernoulli}(p)$$
+
+**TIES sign election.** A consensus sign $\Gamma$ is computed via weighted vote across donors, and any donor update conflicting with it is dropped before averaging:
+
+$$\Gamma = \text{sgn}\left(\sum_{k=1}^K \alpha_k \tilde{\Delta}_k\right)$$
+
+$$\Delta_{\text{TIES}} = \frac{\sum_{k=1}^K \alpha_k \tilde{\Delta}_k \odot \mathbb{I}\left(\text{sgn}(\tilde{\Delta}_k) = \Gamma\right)}{\sum_{k=1}^K \alpha_k \cdot \mathbb{I}\left(\text{sgn}(\tilde{\Delta}_k) = \Gamma\right) + \epsilon}$$
+
+**Depth-scaled reconstruction.** The merged weight is reconstructed as:
+
+$$W_{\text{final}} = W_0 + \lambda(l) \cdot \Delta_{\text{TIES}}$$
+
+where the layer scaling factor $\lambda(l)$ across decoder layer index $l \in [0, 39]$ is defined as:
+
+$$\lambda(l) = \beta \cdot \left(0.5 + 0.5 \sin\left(\pi \frac{l}{39}\right)\right)$$
+
+This keeps input/output projections closer to the base and applies the strongest task transfer to middle layers ($l \in [12, 28]$).
+
+---
+
+## Layer-Stratified Policies
+
+| Parameter Group | Match Substring | Policy | Density ($p$) | Base Scale ($\beta$) |
+| :--- | :--- | :---: | :---: | :---: |
+| **Embeddings / LM head** | `embed_tokens`, `lm_head` | Linear | — | 1.00 |
+| **Norms / biases** | `norm`, `bias`, 1D tensors | Linear | — | 1.00 |
+| **DeltaNet recurrent state** | `a_log`, `dt_bias`, `conv1d` | Linear | — | 1.00 |
+| **MoE router gate** | `mlp.gate.weight`, `block_sparse_moe.gate` | Linear | — | 1.00 |
+| **MoE shared expert** | `shared_expert` | DARE-TIES | 0.70 | 0.60 |
+| **Attention projections** | `attn`, `rotary`, `in_proj`, `out_proj`, `x_proj` | DARE-TIES | 0.75 | 0.60 |
+| **Routed experts (×256)** | `experts`, `mlp` | DARE-TIES | 0.65 | 0.55 |
+
+- **Router protection:** Gate weights use linear interpolation (~57% base, ~43% donors) rather than DARE to avoid destabilizing expert routing.
+- **DeltaNet stability:** Recurrent state kernels are excluded from DARE to prevent divergence in the linear-attention state space.
+- **MTP removed:** Multi-token-prediction heads beyond the 40 primary decoder blocks were stripped for standard CausalLM inference.
+
+---
+
+## Recommended Generation Parameters
+
+For code generation and agentic task trajectories, avoid high temperatures to maintain routing stability and syntax validity.
+
+| Parameter | Coding / Terminal Agent | Creative Reasoning |
+| :--- | :---: | :---: |
+| **Temperature** | `0.6` | `1.0` |
+| **Top-P** | `0.95` | `0.95` |
+| **Min-P** | `0.0` | `0.01` |
+| **Repetition Penalty** | `off` | `1.05` |
+
+---
+
+## How to Use
+
+### Transformers
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_id = "pragmaticcs/SignOfFour"
+
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForCausalLM.from_pretrained(
+    model_id,
+    torch_dtype=torch.bfloat16,
+    device_map="auto",
+)
+
+messages = [
+    {"role": "system", "content": "You are a precise agentic software engineer. Solve problems concisely."},
+    {"role": "user", "content": "Write an asynchronous Python queue consumer with retry backoff."}
+]
+
+inputs = tokenizer.apply_chat_template(
+    messages, add_generation_prompt=True, return_tensors="pt"
+).to(model.device)
+
+output = model.generate(
+    inputs,
+    max_new_tokens=1024,
+    temperature=0.6,
+    top_p=0.95,
+    min_p=0.01,
+    do_sample=True,
+)
+
+print(tokenizer.decode(output[0][inputs.shape[-1]:], skip_special_tokens=True))
+```
+
+---
+
+## Lineage
+
+```
+Qwen/Qwen3.6-35B-A3B
+└── pragmaticcs/SignOfFour
+    ├── base:  Jackrong/Qwopus3.6-35B-A3B-Coder
+    ├── donor: ornith-ai/Ornith-1.5-35B-A3B
+    ├── donor: Kwaipilot/KAT-Coder-V2.5-Dev
+    └── donor: Qwen/Qwen-AgentWorld-35B-A3B
+```
+
+---
+
+## Citation & References
+
+- [Jackrong/Qwopus3.6-35B-A3B-Coder](https://huggingface.co/Jackrong/Qwopus3.6-35B-A3B-Coder)
+- [ornith-ai/Ornith-1.5-35B-A3B](https://huggingface.co/ornith-ai/Ornith-1.5-35B-A3B)
+- [Kwaipilot/KAT-Coder-V2.5-Dev](https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev)
+- [Qwen/Qwen-AgentWorld-35B-A3B](https://huggingface.co/Qwen/Qwen-AgentWorld-35B-A3B)
+
+```bibtex
+@inproceedings{yu2024dare,
+  title={Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch},
+  author={Yu, Le and Yu, Bowen and Yu, Haiyang and Huang, Fei and Li, Yongbin},
+  booktitle={International Conference on Machine Learning (ICML)},
+  year={2024}
+}
+
+@inproceedings{yadav2023ties,
+  title={Resolving Interference When Merging Models},
+  author={Yadav, Prateek and Tam, Derek and Choshen, Leshem and Raffel, Colin and Bansal, Mohit},
+  booktitle={Advances in Neural Information Processing Systems (NeurIPS)},
+  year={2023}
+}
+```