Update README.md
Browse files
README.md
CHANGED
|
@@ -50,6 +50,10 @@ Standard attention grows more expensive with context length: the longer the requ
|
|
| 50 |
|
| 51 |
Large models tend to develop implicit self-stabilization (attention/residual sinks), routing most of the signal through a single token or feature to hold the activation scale — which is poorly controlled and can itself become a source of noise at scale. GatedNorm replaces these implicit anchors with an explicit multiplicative gate after RMSNorm, letting the network rescale the signal across features directly. It is made scale-neutral at init via the `2 · sigmoid` reparametrization (a plain sigmoid starts near 0.5 and would halve the scale; the factor 2 keeps the gate near 1.0), so it barely perturbs the data flow at start and learns where to attenuate.
|
| 52 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 53 |
### Multi-Token Prediction (MTP)
|
| 54 |
|
| 55 |
GigaChat Ultra 3.0 had a single MTP head; in GigaChat Ultra 3.5 we added two MTP heads. Greedy decoding accelerates the generation speed ~1.5× with one head and up to 2.2× with two.
|
|
|
|
| 50 |
|
| 51 |
Large models tend to develop implicit self-stabilization (attention/residual sinks), routing most of the signal through a single token or feature to hold the activation scale — which is poorly controlled and can itself become a source of noise at scale. GatedNorm replaces these implicit anchors with an explicit multiplicative gate after RMSNorm, letting the network rescale the signal across features directly. It is made scale-neutral at init via the `2 · sigmoid` reparametrization (a plain sigmoid starts near 0.5 and would halve the scale; the factor 2 keeps the gate near 1.0), so it barely perturbs the data flow at start and learns where to attenuate.
|
| 52 |
|
| 53 |
+
### Overall architecture
|
| 54 |
+
|
| 55 |
+

|
| 56 |
+
|
| 57 |
### Multi-Token Prediction (MTP)
|
| 58 |
|
| 59 |
GigaChat Ultra 3.0 had a single MTP head; in GigaChat Ultra 3.5 we added two MTP heads. Greedy decoding accelerates the generation speed ~1.5× with one head and up to 2.2× with two.
|