QUANTIZATION_ARMS.md

# Per-Tensor Quantization Arms — Methodology & Principles

> Companion technical document for [Confucius4-R2T2-Q4_K_M-GGUF](https://huggingface.co/Nairod785/Confucius4-R2T2-Q4_K_M-GGUF).
> Details the general per-tensor quantization arms methodology, `--keep-type` semantics, cross-lingual validation protocols, and core architectural rules.

---

## 1. Why Per-Tensor Arms?

Standard model quantization tools apply uniform presets (e.g. standard Q4_K_M or Q5_K_M) across entire layers or architectures. For many speech models, however, uniform quantization either fails catastrophic quality checks or leaves substantial memory optimization on the table.

Building **quantization arms** means creating experimental variants that differ in exactly one block or layer type, isolating the quality, memory, and speed impact of each architectural component:
- Isolating which blocks represent non-negotiable precision floors.
- Finding which blocks can be aggressively quantized without measurable loss.
- Measuring real kernel execution speed across mixed-precision representations.

---

## 2. Converter Overrides & Mechanics

Per-tensor control requires specifying precision at the individual weight level:
- **Base Type:** Sets the baseline default type for all quantizable weights in the graph.
- **Explicit Overrides (`--keep-type <tensor>=<type>`):** Selectively overrides the precision of specific tensors.
- **Source Naming Contract:** Override rules must target the exact source parameter names (e.g. `thinker.model.layers.0.mlp.down_proj.weight`), not transformed runtime names.
- **Asserting Tensor Census:** Always assert a strict tensor census per architectural block before and after conversion to prevent silent fallbacks or unmatched patterns:
  - Tower: 147 tensors
  - Attention (`q`, `k`, `v`, `o_proj`): 112 tensors
  - Gate / Up (`gate_proj`, `up_proj`): 56 tensors
  - Down (`down_proj`): 28 tensors
  - Embeddings: 1 tensor
- **Command-Line Limits:** When passing dozens of tensor overrides on Windows systems, drive conversion scripts from bash or JSON specifications rather than `cmd.exe` to avoid the 8191-character command-line length truncation limit.

---

## 3. Verification Protocol: Verify What Was Built

Never infer a model arm's composition from the conversion arguments alone—conversion flags can fail silently without throwing errors.
Always verify the resulting GGUF by inspecting the tensor header table:
1. Verify that all 28 `down_proj` weights are assigned their target type.
2. Confirm that `embed_tokens` was successfully overridden from the default F16 pin to the desired low-bit type.
3. Confirm that sensitive encoder layers retain their target mixed precision.

---

## 4. Cross-Lingual Validation Protocol

A critical lesson learned during quantization validation:

### English and Chinese Are Not a Screen
Speech models like R2T2 are trained with massive data allocations for English and Chinese, making them the model's most resilient input distributions.
- Multiple quantization configurations matched the Q8_0 reference baseline on English (`jfk`) and Chinese (`zh`), but completely broke down on non-target languages.
- For example, naive Q5_K matched Q8_0 on English and Chinese, but silently flipped to English when fed Russian audio.
- Other experimental configurations passed English and Chinese tests, but emitted English translations when given German audio.

### The Non-Target Probe Principle
Always evaluate quantization quality against languages with smaller training footprints (e.g. German, French, Russian). If a model switches languages or drops tokens on secondary languages, its representation space has degraded.

### Automated Detection
Rather than manually inspecting hours of transcripts, automated validation checks should monitor:
1. **Detected Language Output:** Grep the model runtime's detected language tag (`detected-language:`). A switch from `de` or `fr` to `en` immediately identifies representation collapse.
2. **Three-Tier Word Diffing:**
   - **Raw:** Exact matching including punctuation and casing differences (benign).
   - **Strict:** Lowercase, punctuation-stripped matching.
   - **Loose:** Diacritics folded (NFD normalization). Flags true **content drift** (omitted, substituted, or hallucinated words).
3. **Empty Output Detection:** Low-bit matmuls can collapse abruptly to empty strings. Ensure output files are confirmed non-empty before running text diffs.

---

## 5. Architectural Principles That Generalize

The findings from the R2T2 quantization campaign reveal principles that apply broadly to modern transformer and speech architectures:

1. **Quantization Error is Cumulative, Not Per-Tensor:**
   There is no fixed precision floor for a block in isolation. A block at Q4_K may work perfectly alone, but cause collapse when another block is also reduced. The total error budget is shared network-wide.

2. **Residual Stream Writes Compound Strictly:**
   Blocks that write directly into the residual stream (`down_proj`) compound numerical error through every subsequent layer. Consequently, `down_proj` imposes the strictest precision requirement (Q6_K for R2T2).

3. **Non-Linearities Bound Error Propagation:**
   Projections that feed into bounded activation functions (such as SwiGLU in `gate_proj` and `up_proj`) tolerate lower bitwidths (Q4_K) because the non-linearity bounds error growth.

4. **Lookup Tables Do Not Compound:**
   Token embeddings (`embed_tokens`) are accessed via table lookup. Their quantization error is introduced once per token and does not propagate recurrently. Aggressively quantizing embeddings (even to Q2_K) saves substantial memory with virtually no degradation.

5. **Low-Bit Matmul Cliff:**
   Matmul quantization does not degrade gracefully below Q4_K. Q3_K matmuls in R2T2 produced immediate empty output cliffs.

6. **Speed Does Not Track Bit Width Alone:**
   Kernel optimization quality dominates hardware throughput. Q5_K performed ~20% slower than Q8_0 due to non-vectorized paths, whereas highly optimized Q4_K and Q6_K routines provided significant speedups.