Yigit-Karaman

/

Oguz-MoE-354M-Base

Like 1

Hi Yigit,

Executing a complete four-stage progressive scaling pipeline (from-scratch 80M pretrain -> dual depth upscaling to 21 layers -> FFN-to-MoE upcycle to 354M) entirely on a single consumer RTX 4060 (8GB VRAM) with custom PyTorch dispatch and 8-bit AdamW is phenomenal resource engineering. Publishing exact VRAM allocations, step counts, and checkpoint restoration traces sets a great open benchmark.

A few architectural observations on your scaling dynamics and parameter allocations:

1. Expert specialization versus healing horizon in 4-expert MoE:
   Your near-function-preserving initialization (copying the 1536-wide dense FFN into the first dimensions and adding near-silent noise to the rest) is textbook practice to prevent catastrophic forgetting during upcycling.
   However, with a 150M-token healing run on top-2 of 4 routing, each individual expert received gradient updates across roughly 75M effective tokens.
   Because all 4 experts start from near-identical weight coordinates, breaking representational symmetry and driving expert orthogonalization usually requires a substantially longer token runway. At 75M tokens per expert, the router often behaves more like a stochastic ensemble over parallel dense projections rather than cleanly routing distinct syntactic or domain pathways.

2. The 36% vocabulary footprint in the Stage 0 foundation:
   With a 50,257 Turkish GPT-2 vocabulary at hidden dimension 576, the tied embedding table consumes 28.95M parameters.
   While this is ~14% of your final 205.5M active MoE token compute, in your Stage 0 dense base (80M parameters), that single static lookup table locked up over 36% of the entire model.
   Decoupling token lookup through low-rank factorized projections would have reclaimed ~20M parameters early on, allowing you to train the dense base at hidden dimension 768 within the exact same 8GB VRAM budget.

3. Context expansion bottlenecks on 8GB GPUs:
   Your VRAM telemetry shows memory scaling from 4.42 GB (batch 4, ctx 512) to 5.15 GB (batch 2, ctx 1024). Because you avoided gradient checkpointing to preserve the auxiliary-loss autograd graph, standard attention activation tensors scale steeply at 21 layers, effectively capping your context ceiling around 1024 tokens on consumer hardware.

In an open architecture project called Maba v2 (reference release: https://huggingface.co/AndrewThompson1233/maba-v2-architecture), we explore bypassing this exact activation wall on 8GB GPUs:
We route 75% of depth through Decoupled Gated Delta Attention (DGDA) paired with MABA-SA sparse attention and native NoPE.
Because linear recurrence maintains an associative matrix state in constant O(1) memory, sequences can scale to 4,096+ tokens on an 8GB card without quadratic activation blowouts or aggressive microbatch slicing.

If you are planning an instruction tune or an Oğuz v2 run, checking out the parameter allocation layout and hybrid layer formulation in the Maba v2 repo might give you some practical reference points for scaling context on consumer GPUs.

Did you measure the cosine similarity between the weight matrices of the 4 experts after the 150M healing run to track how far they diverged from the original dense FFN?

Best,
Andrew