perkyfever commited on
Commit
d10dcae
·
verified ·
1 Parent(s): bdd1d89

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +4 -0
README.md CHANGED
@@ -50,6 +50,10 @@ Standard attention grows more expensive with context length: the longer the requ
50
 
51
  Large models tend to develop implicit self-stabilization (attention/residual sinks), routing most of the signal through a single token or feature to hold the activation scale — which is poorly controlled and can itself become a source of noise at scale. GatedNorm replaces these implicit anchors with an explicit multiplicative gate after RMSNorm, letting the network rescale the signal across features directly. It is made scale-neutral at init via the `2 · sigmoid` reparametrization (a plain sigmoid starts near 0.5 and would halve the scale; the factor 2 keeps the gate near 1.0), so it barely perturbs the data flow at start and learns where to attenuate.
52
 
 
 
 
 
53
  ### Multi-Token Prediction (MTP)
54
 
55
  GigaChat Ultra 3.0 had a single MTP head; in GigaChat Ultra 3.5 we added two MTP heads. Greedy decoding accelerates the generation speed ~1.5× with one head and up to 2.2× with two.
 
50
 
51
  Large models tend to develop implicit self-stabilization (attention/residual sinks), routing most of the signal through a single token or feature to hold the activation scale — which is poorly controlled and can itself become a source of noise at scale. GatedNorm replaces these implicit anchors with an explicit multiplicative gate after RMSNorm, letting the network rescale the signal across features directly. It is made scale-neutral at init via the `2 · sigmoid` reparametrization (a plain sigmoid starts near 0.5 and would halve the scale; the factor 2 keeps the gate near 1.0), so it barely perturbs the data flow at start and learns where to attenuate.
52
 
53
+ ### Overall architecture
54
+
55
+ ![image](https://cdn-uploads.huggingface.co/production/uploads/67a4f89d46a7612efd49c561/AsNtVz8i9vxHhqcT5LT9Y.png)
56
+
57
  ### Multi-Token Prediction (MTP)
58
 
59
  GigaChat Ultra 3.0 had a single MTP head; in GigaChat Ultra 3.5 we added two MTP heads. Greedy decoding accelerates the generation speed ~1.5× with one head and up to 2.2× with two.