Upload Q2_K/Q2_REPAIR_REPORT.md with huggingface_hub

Q2_K/Q2_REPAIR_REPORT.md

@@ -0,0 +1,341 @@
+# DeepSeek V4.1 Flash Q2_K Repair Report
+
+## Summary
+
+The original seven-shard Q2_K model required two distinct classes of repair:
+
+1. Its Engram metadata was incompatible with the `deepseek41` runtime and was incomplete.
+2. Eighty-four architecture-critical tensors had been quantized to Q2_K even though their roles require substantially higher precision.
+
+The final quality artifact contains:
+
+- Correct and complete `deepseek41.engram.*` metadata.
+- Four official BF16 Engram gate tensors.
+- Eighty official F32 multi-stream Hyper-Connection (mHC) tensors.
+- The original Q2/Q3 backbone and MoE weights everywhere else.
+
+The model was repaired selectively. It was not converted or requantized as a whole.
+
+## 1. Engram metadata repair
+
+The original first shard contained only four Engram metadata values, and they used the wrong architecture namespace:
+
+```text
+deepseek4.engram.head_count
+deepseek4.engram.key_length
+deepseek4.engram.layer_ids
+deepseek4.engram.max_ngram_size
+```
+
+The runtime resolves architecture-dependent GGUF keys as `{arch}.{key}`. Because the model declares `general.architecture = deepseek41`, the loader looks for `deepseek41.engram.*`, not `deepseek4.engram.*`.
+
+The four existing keys were renamed to:
+
+```text
+deepseek41.engram.head_count
+deepseek41.engram.key_length
+deepseek41.engram.layer_ids
+deepseek41.engram.max_ngram_size
+```
+
+Five required values were also missing entirely and were added:
+
+```text
+deepseek41.engram.multipliers
+deepseek41.engram.primes
+deepseek41.engram.offsets
+deepseek41.engram.token_map
+deepseek41.engram.pad_id
+```
+
+The metadata count in shard 1 changed from 63 to 68: four old keys were replaced by four correctly namespaced keys, and five new values were added.
+
+### Why this was necessary
+
+Without this repair, the current loader cannot initialize the Engram subsystem. The namespace mismatch prevents it from finding even the four values present in the file, while the missing arrays prevent reproduction of the checkpoint's token compression and hash calculation.
+
+The restored data covers:
+
+- A 129,280-entry INT32 token map.
+- 8 UINT64 hash multipliers.
+- 48 UINT64 primes.
+- 48 UINT64 offsets.
+- Compressed vocabulary size 99,092.
+- Compressed padding ID 2.
+
+The constants were regenerated using the converter algorithm with the official configuration and tokenizer from checkpoint revision:
+
+```text
+dba1be0a40aa45a94ad051997016db3960a90277
+```
+
+They matched the independent JigSawPT GGUF representation exactly.
+
+## 2. Engram gate tensor restoration
+
+The original Q2 model stored these tensors in Q2_K:
+
+```text
+blk.1.engram_q.weight
+blk.1.engram_k.weight
+blk.14.engram_q.weight
+blk.14.engram_k.weight
+```
+
+Each tensor has GGUF shape:
+
+```text
+[5120, 4]
+```
+
+They were replaced with the original BF16 checkpoint payloads:
+
+```text
+engram_q/k: Q2_K -> BF16
+```
+
+### Why this was necessary
+
+These small tensors control the Engram gate. They behave as sensitive scale-like parameters rather than ordinary large backbone matrices. Q2_K heavily distorts their relative magnitudes and therefore changes how Engram table output is injected into the model state.
+
+The practical effect was visible in full-model testing. The source Q2 with quantized gate tensors produced a weak, factually incorrect continuation. Replacing only these four tensors with their official BF16 values changed the deterministic trajectory and produced the expected `Au` token for a gold-symbol prompt.
+
+The four payloads were extracted with HTTP range requests from the official safetensors shards. Each payload was independently compared with the corresponding JigSawPT tensor and matched byte-for-byte.
+
+## 3. mHC tensor restoration
+
+DeepSeek V4.1 Flash has two multi-stream Hyper-Connection parameter matrices in every one of its 40 layers:
+
+```text
+blk.N.hc_attn_fn.weight
+blk.N.hc_ffn_fn.weight
+```
+
+This gives:
+
+```text
+40 layers x 2 tensors = 80 tensors
+```
+
+Each tensor has GGUF shape:
+
+```text
+[20480, 24]
+```
+
+The community Q2 model stored all 80 tensors in Q2_K. They were replaced with official F32 payloads:
+
+```text
+hc_attn_fn: Q2_K -> F32
+hc_ffn_fn: Q2_K -> F32
+```
+
+### Why this was necessary
+
+The mHC matrices generate coefficients that mix the model's four residual streams around every attention and feed-forward sublayer. Their values are small and numerically sensitive. An error in these parameters changes residual routing twice per layer and can accumulate through all 40 layers.
+
+The official reference, vLLM, and SGLang implementations keep these parameters in FP32. Operation-level validation also demonstrated a substantial precision difference:
+
+```text
+BF16 projection error: up to 1.16e-2
+F32 projection error:  up to 1.43e-6
+```
+
+For this reason, restoring the mHC tensors only to BF16 was not sufficient for close agreement with the reference. All 80 were restored to F32.
+
+The payloads were extracted from the official checkpoint revision listed above. They were distributed across official safetensors shards 3 through 42. The complete extracted payload size was 157,286,400 bytes (150 MiB).
+
+## 4. What was not changed
+
+The complete model was not requantized. The following data was preserved:
+
+- MoE expert weights.
+- Attention and feed-forward backbone weights.
+- Token embeddings.
+- Indexer weights.
+- Compressed-attention weights.
+- Both large Engram tables.
+- Every Q2_K, Q3_K, Q6_K, BF16, and F32 tensor not explicitly listed for replacement.
+- Tokenizer data and chat template.
+- Existing non-Engram metadata.
+
+Only 84 tensors were replaced:
+
+```text
+80 mHC tensors + 4 Engram q/k tensors = 84 tensors
+```
+
+This targeted approach preserved the original community quantization for the large backbone while restoring reference precision only where ordinary low-bit quantization was unsuitable.
+
+## 5. Tensor type changes
+
+The complete seven-shard model contains 1,046 unique tensors. Tensor type counts changed as follows:
+
+| Tensor type | Original Q2 | Repaired Q2 |
+|---|---:|---:|
+| Q2_K | 476 | 392 |
+| F32 | 449 | 529 |
+| BF16 | 40 | 44 |
+| Q3_K | 80 | 80 |
+| Q6_K | 1 | 1 |
+
+The repair occurred in two tensor-replacement steps:
+
+1. Restoring the four Engram gate tensors changed Q2_K from 476 to 472 and BF16 from 40 to 44.
+2. Restoring the 80 mHC tensors changed Q2_K from 472 to 392 and F32 from 449 to 529.
+
+The complete size changed as follows:
+
+```text
+Original Q2: 264,514,761,248 bytes
+Repaired Q2: 264,659,800,416 bytes
+Growth:        145,039,168 bytes (approximately 138.3 MiB)
+```
+
+The total tensor count and every tensor shape remained unchanged.
+
+## 6. Rewritten shards
+
+Critical tensors were present in all seven Q2 shards:
+
+```text
+1, 2, 3, 4, 5, 6, and 7
+```
+
+Their distribution was:
+
+| Shard | mHC tensors | Engram q/k tensors | Total replaced |
+|---:|---:|---:|---:|
+| 1 | 4 | 2 | 6 |
+| 2 | 18 | 0 | 18 |
+| 3 | 6 | 0 | 6 |
+| 4 | 4 | 2 | 6 |
+| 5 | 18 | 0 | 18 |
+| 6 | 18 | 0 | 18 |
+| 7 | 12 | 0 | 12 |
+| **Total** | **80** | **4** | **84** |
+
+Shard 1 also required the complete Engram metadata repair because it contains the model-level GGUF metadata.
+
+During the first quality-repair step, only shards 1 and 4 required tensor rewriting because they contain `engram_q/k`; shard 1 was also rewritten for metadata. Restoring all 80 mHC tensors in the final step required rewriting every shard in the seven-file split set.
+
+Because the replacement payloads are larger than their Q2_K versions, each affected rewrite recalculated:
+
+- GGUF tensor type entries.
+- Tensor payload sizes.
+- Tensor offsets.
+- 32-byte alignment padding.
+- The data-section offset.
+- The final shard size.
+
+## 7. Integrity and post-write validation
+
+The source files were never modified in place. Both repair stages used streaming rewrites into new files and checked:
+
+- Full source-file SHA-256 hashes.
+- Official replacement-payload SHA-256 hashes.
+- Preservation of the tensor count.
+- Preservation of all tensor shapes.
+- Correct output types for replacement tensors.
+- Unchanged types and payloads for all other tensors.
+- Correct offsets, sizes, and GGUF alignment.
+- Complete and correctly typed Engram metadata.
+- Absence of the obsolete `deepseek4.engram.*` namespace.
+- Successful parsing of the complete seven-shard output set.
+- Exactly 1,046 unique tensor names with no duplicates.
+
+The final manifest confirms:
+
+```text
+general.architecture = deepseek41
+GGUF version         = 3
+split.count          = 7
+tensor count         = 1046
+Engram q/k           = 4 x BF16
+mHC tensors          = 80 x F32
+```
+
+Relevant artifacts:
+
+- [`stage1/quality-gguf-manifest.json`](stage1/quality-gguf-manifest.json): metadata and BF16 Engram repair.
+- [`stage14/quality-f32-mhc-gguf-manifest.json`](stage14/quality-f32-mhc-gguf-manifest.json): final complete model manifest.
+- [`stage14/quality-f32-mhc-checksums.sha256`](stage14/quality-f32-mhc-checksums.sha256): final shard hashes.
+- [`tools/rewrite_q2_mhc_shard.py`](tools/rewrite_q2_mhc_shard.py): streaming F32 mHC rewrite tool.
+
+## 8. Runtime validation
+
+The repaired seven-shard model successfully loaded and executed across six CUDA GPUs. The broader bring-up validation completed 27 out of 27 checks, including:
+
+- Discovery and loading of all seven GGUF shards.
+- Loading all 1,046 tensors.
+- Offloading all 41 model layers.
+- Lazy loading of both large Engram tables.
+- Prompt execution at several supported context sizes.
+- Deterministic repeated generation.
+- Server health, completion, and metrics endpoints.
+- Clean shutdown and GPU-memory release.
+- No NaN/Inf logits, allocation loop, or first-token hang.
+
+See [`STAGE_9_REPORT.md`](STAGE_9_REPORT.md) for the complete bring-up validation.
+
+## 9. Paired quality validation
+
+A single greedy continuation is highly sensitive to small numeric changes and was not used as the final quality gate for the F32 mHC repair. Instead, the Engram-repaired model with Q2_K mHC and the final model with F32 mHC were compared on the same 32 Wikitext-2 chunks.
+
+Test parameters:
+
+```text
+Context:             512
+Chunks:              32
+Scored tokens/chunk: 255
+Total scored tokens: 8160
+Batch:               512
+Micro-batch:         128
+```
+
+Results:
+
+| Model set | mHC precision | Perplexity | Uncertainty |
+|---|---|---:|---:|
+| Previous Engram quality overlay | Q2_K | 6.3085 | 0.17503 |
+| Final quality artifact | F32 | **6.2046** | 0.17046 |
+
+The F32-mHC model:
+
+- Improved perplexity by 1.647%.
+- Was better on 25 of the 32 paired chunks.
+- Achieved a mean paired NLL gain of 0.016607.
+- Produced a paired t-statistic of 3.326.
+
+This limited paired test supports keeping the official mHC tensors in F32. It is a selection and regression gate, not a complete published benchmark against the official FP8 implementation.
+
+See [`STAGE_14_REPORT.md`](STAGE_14_REPORT.md) and [`stage14/ppl-pair-c512-chunk32.json`](stage14/ppl-pair-c512-chunk32.json) for details.
+
+## 10. Final artifact
+
+The recommended logical path is:
+
+```text
+/home/sexxxx/_my_deepseek41/artifacts/quality-f32-mhc/DeepSeek-V4.1-Flash-Q2_K-00001-of-00007.gguf
+```
+
+The physical files are stored at:
+
+```text
+/media/sexxxx/WW2T/models/vcruz305/DeepSeek-V4.1-Flash-Q2_K-quality-f32-mhc/
+```
+
+The logical artifact directory contains symbolic links to the seven physical shards.
+
+## Result
+
+The final model keeps the low-bit community quantization for the large backbone and expert weights while using reference precision for architecture-critical parameters:
+
+```text
+Backbone and experts: Q2_K/Q3_K
+Engram q/k:          BF16
+mHC parameters:      F32
+Engram metadata:     complete deepseek41 namespace
+```
+
+This is the recommended Q2 development and validation artifact. It corrects both the loader-blocking metadata defects and the quality-critical tensor quantization defects without requantizing the complete model.