--- license: apache-2.0 tags: - jgos - ourbox - darwin - darwin-platform - evolutionary-merge - ffn-merge - model-breeding - korean - korean-specialized - reasoning - advanced-reasoning - chain-of-thought - thinking - qwen3.6 - qwen - moe - mixture-of-experts - multi-token-prediction - multilingual - gpqa - benchmark - open-source - apache-2.0 - vidraft - eval-results language: - ko - en - zh - ja - de - fr - es - ru - ar - multilingual pipeline_tag: text-generation library_name: transformers model-index: - name: Ourbox-35B-JGOS results: - task: type: question-answering name: Question Answering dataset: name: GPQA Diamond type: Idavidrein/gpqa config: gpqa_diamond metrics: - type: accuracy value: 86.36 name: Accuracy --- # Ourbox-35B-JGOS: Korean-Specialized, Darwin-Evolved 35B-A3B Reasoning MoE β€” 86.36% on GPQA Diamond

GPQA Korean

Sibling FINAL Bench GPQA

> Darwin-Evolved on Qwen3.6-35B-A3B | 35B total / ~3B active | πŸ‡°πŸ‡· Korean-Specialized | Thinking Mode | Hybrid Linear/Full Attention | Multi-Token Prediction | 262K Context | BF16 | Apache 2.0 > **Darwin FFN-level evolutionary merge β†’ Korean specialization β†’ 86.36% on GPQA Diamond (majority-of-8+)** --- ## Abstract **Ourbox-35B-JGOS** is a 35-billion-parameter mixture-of-experts (MoE) reasoning model produced by the **Darwin** evolutionary breeding platform (FINAL-Bench / VIDRAFT_LAB). Rather than retraining from scratch, Darwin recombines the **feed-forward (FFN / MoE expert) tensors** of the **Qwen3.6-35B-A3B** backbone with those of additional specialized donor models, then **evolves** the merged descendant toward a target objective β€” here, **Korean-language specialization**. Because the merge operates at the **expert-FFN level**, Ourbox inherits complementary domain and language competencies from multiple sources while preserving the backbone's hybrid-attention topology and 262K long-context behavior. The result is a Korean-specialized reasoning model that remains highly competitive on English graduate-level science: on **GPQA Diamond** (198 questions across physics, chemistry, biology), Ourbox-35B-JGOS scores **86.36% (171/198)** under a majority-of-8+ protocol β€” surpassing open flagships more than **20Γ— its active size** (e.g. the 744B GLM-5.1) with only **~3B active parameters**. --- ## GPQA Diamond Leaderboard Position of Ourbox-35B-JGOS relative to the reference leaderboard (frontier open models, GPQA Diamond): | Rank | Model | Parameters | GPQA Diamond | |---|---|---|---| | 1 | TNSA/NGen-4-Pro | β€” | 91.1% | | 2 | TNSA/NGen-4 | β€” | 90.1% | | 3 | Qwen/Qwen3.5-397B-A17B | 397B | 88.4% | | 3 | FINAL-Bench/Darwin-36B-Opus | 36B (A3B) | 88.4% | | 5 | moonshotai/Kimi-K2.5 | β€” | 87.6% | | 6 | FINAL-Bench/Darwin-27B-Opus | 27B | 86.9% | | 7 | Qwen/Qwen3.5-122B-A10B | 122B | 86.6% | | **β˜…** | **FINAL-Bench/Ourbox-35B-JGOS** πŸ‡°πŸ‡· | **35B (A3B)** | **86.36%** | | 8 | zai-org/GLM-5.1 | 744B | 86.2% | | 9 | zai-org/GLM-5 | 744B | 86.0% | | 10 | zai-org/GLM-4.7 | β€” | 85.7% | A **35B-parameter MoE (β‰ˆ3B active)**, Korean-specialized, that **surpasses the 744B GLM-5.1 and GLM-5** and closes to within **0.24 points of the 122B-active-10B Qwen3.5-122B** β€” at roughly **1/20th the active compute** of the largest systems it beats. Retaining this level of English scientific reasoning *after* Korean specialization is the headline result. --- ## What Is Darwin? **Darwin** is the evolutionary model-breeding platform developed by FINAL-Bench / VIDRAFT_LAB. Rather than allocating further compute to gradient optimization, Darwin treats trained checkpoints as a **genetic pool** and discovers high-performing descendants through principled recombination of their weight tensors β€” with a particular focus on the **FFN / MoE expert** subspace, where domain and language competence is concentrated. At a high level, the platform performs: 1. **Per-tensor compatibility analysis** across the backbone and donor models to identify which FFN experts and components transfer cleanly and which require weighted recombination. 2. **FFN-level merge & evolution** β€” the descendant's expert tensors are assembled from the pool and iteratively evolved toward a target objective (Korean specialization for Ourbox). 3. **Verification** via a multi-phase scientific benchmark before release. Specific algorithmic details of the Darwin engine are proprietary to FINAL-Bench. All Darwin models are released under the base model's open-source license (Apache 2.0). **JGOS** is the reasoning-model line built with Darwin; **Ourbox** is its Korean-specialized 35B-A3B member. --- ## Evolution Process Ourbox-35B-JGOS is bred, not trained: - **Backbone**: [Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen) β€” the foundation MoE, contributing its hybrid-attention topology (ΒΎ linear + ΒΌ full), 256-expert routing, MTP head, and 262K context. - **FFN donors**: additional specialized models whose **feed-forward / expert tensors** are recombined into the backbone by the Darwin engine, contributing complementary domain and Korean-language competence. - **Evolution objective**: Korean specialization β€” the evolutionary selection biases the merged expert population toward stronger Korean reasoning and generation, while structural and long-context behavior is inherited intact from the backbone. The merge operates **without gradient optimization on the final assembly**; a deployable bfloat16 checkpoint is produced by the Darwin pipeline directly. --- ## πŸ‡°πŸ‡· Korean Specialization Ourbox-35B-JGOS is specialized for **Korean**. The Darwin evolutionary process selects and recombines FFN experts to strengthen Korean-language reasoning, comprehension, and generation β€” targeting natural Korean output, robust handling of Korean scientific/technical text, and reduced character-level corruption on large Korean inputs. Crucially, this specialization does **not** come at the cost of general capability: the model's **86.36% GPQA Diamond** (in English) places it above 744B-parameter open flagships, evidence that the FFN-merge preserved scientific reasoning depth while adding Korean strength. The model remains fully multilingual (Korean-first), with English, Chinese, Japanese, and other languages inherited from the backbone. --- ## Architecture Ourbox-35B-JGOS retains the full Qwen3.6-35B-A3B architecture (`qwen3_5_moe` codebase): | | | |---|---| | Foundation | Qwen3.6-35B-A3B (`Qwen3_5MoeForCausalLM`) | | Breeding platform | Darwin (FFN-level evolutionary merge) | | Total parameters | ~35 B | | Active parameters | ~3 B (top-8 of 256 routed experts per layer) | | Layers | 40 | | Hidden size | 2048 | | Attention | **Hybrid** β€” 30 linear-attention + 10 full-attention layers (`full_attention_interval = 4`) | | Full-attention heads | 16 Q / 2 KV (GQA), head dim 256, partial rotary 0.25 | | Linear attention | Gated-DeltaNet style β€” 16 key heads Γ— 128, 32 value heads Γ— 128, conv kernel 4 | | Experts per layer | 256 routed (top-8) + 1 shared, expert intermediate 512 | | Multi-Token Prediction | 1 MTP layer (`mtp_num_hidden_layers = 1`) | | Context length | 262,144 tokens | | Vocabulary | 248,320 | | RoPE | ΞΈ = 1e7, interleaved mRoPE, sections [11, 11, 10] | | Dtype | bfloat16 | | Checkpoint size | ~69 GB (2 shards) | | License | Apache 2.0 | The **hybrid attention** design (ΒΎ linear + ΒΌ full) gives near-linear KV-cache scaling across the 262K window, and the **Multi-Token Prediction** head provides a built-in draft for speculative decoding. --- ## GPQA Diamond Evaluation ### Methodology Ourbox-35B-JGOS was evaluated on all **198 GPQA Diamond** questions using a two-pass **majority-of-8+** protocol (identical to sibling FINAL-Bench reasoning models, for cross-model comparability): **Pass 1 β€” Greedy baseline** - All 198 questions, deterministic decoding (`do_sample=False`) - Up to 5,120 new tokens per question (full `` trajectories) - Standard multiple-choice prompt format **Pass 2 β€” Stochastic majority vote with tiebreaker** - Each question is answered by **8 independent stochastic generations** (`temperature=0.7`, `max_tokens=5120`); the majority answer is taken - Where the 8-vote margin is inconclusive (e.g. 3:3 / 3:4 / 4:4), an additional **16-vote tiebreaker** round (`temperature=0.5`) resolves the answer The final answer for each question is extracted after the `` delimiter. ### Result | Metric | Value | |---|---| | Correct | **171 / 198** | | **GPQA Diamond accuracy (maj@8+)** | **86.36%** | Evaluated against the [`Idavidrein/gpqa`](https://huggingface.co/datasets/Idavidrein/gpqa) `gpqa_diamond` split. The majority-of-8+ protocol surfaces answers that greedy decoding leaves subdominant β€” a pattern characteristic of well-formed chain-of-thought models β€” carrying Ourbox past the 744B-parameter open flagships on graduate-level science. --- ## Usage ```python from transformers import AutoTokenizer, AutoModelForCausalLM import torch tok = AutoTokenizer.from_pretrained("FINAL-Bench/Ourbox-35B-JGOS", trust_remote_code=True) model = AutoModelForCausalLM.from_pretrained( "FINAL-Bench/Ourbox-35B-JGOS", torch_dtype=torch.bfloat16, device_map="auto", trust_remote_code=True, ) messages = [ {"role": "user", "content": "μƒλŒ€λ‘ μ  μš΄λ™μ—λ„ˆμ§€ 식을 μœ λ„ν•΄μ€˜."} ] text = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) inputs = tok(text, return_tensors="pt").to(model.device) outputs = model.generate(**inputs, max_new_tokens=5120, temperature=0.6, do_sample=True) print(tok.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True)) ``` ### Answer extraction for evaluations This is a **thinking model** β€” responses always begin with a `` reasoning trace. For benchmarks, extract the final answer after ``: ```python response = tok.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True) idx = response.rfind("") answer_part = response[idx + len(""):].strip() if idx >= 0 else response ``` ### Recommended settings - **Temperature**: 0.6–0.7 for reasoning / majority voting; 0.0 for greedy deterministic - **max_new_tokens**: β‰₯5120 to accommodate full `` trajectories - **Chat template**: assistant turn opens with `` when `apply_chat_template(add_generation_prompt=True)` is used --- ## VRAM Requirements | Precision | VRAM | Recommended GPU | |---|---|---| | bf16 (full) | ~72 GB | 1Γ— H100 80GB / 1Γ— B200 | | 8-bit | ~40 GB | 1Γ— A100 40GB+ / 1Γ— L40S | | 4-bit | ~22 GB | 1Γ— RTX 4090 / 1Γ— A10 | --- ## Key Findings 1. **Korean specialization without capability loss.** Darwin's FFN-level merge adds Korean-language strength while retaining **86.36% GPQA Diamond** β€” above 744B open flagships. Specialization and general reasoning are not a zero-sum trade under expert-level recombination. 2. **~3B active parameters beat 744B open flagships.** Ourbox surpasses GLM-5.1 and GLM-5 (86.2% / 86.0%, 744B each) on GPQA Diamond while activating roughly 1/20th of their compute per token. 3. **Breeding beats retraining for specialization.** A deployable, Korean-specialized 35B checkpoint is produced by evolutionary FFN recombination β€” no full-model gradient training on the final assembly β€” demonstrating Darwin as an efficient route to targeted, high-capability models. --- ## References - Rein et al., *GPQA: A Graduate-Level Google-Proof Q&A Benchmark*, 2024. [dataset](https://huggingface.co/datasets/Idavidrein/gpqa) - Qwen Team, *Qwen3.6 Technical Report*, 2026. --- ## Built By **FINAL-Bench / VIDRAFT_LAB** β€” Darwin evolutionary breeding platform, JGOS Korean-specialized reasoning line. Backbone weights by the Qwen Team (Qwen3.6-35B-A3B). Released under Apache 2.0. --- ## Citation ```bibtex @misc{ourbox-35b-jgos, title = {Ourbox-35B-JGOS: Korean-Specialized, Darwin-Evolved 35B-A3B Reasoning MoE}, author = {FINAL-Bench and VIDRAFT_LAB}, year = {2026}, url = {https://huggingface.co/FINAL-Bench/Ourbox-35B-JGOS}, note = {Qwen3.6-35B-A3B backbone, Darwin FFN-level evolutionary merge, Korean-specialized, 86.36% GPQA Diamond (maj@8+)} } ```