---
license: apache-2.0
library_name: llama.cpp
tags:
- gguf
- moe
- xing4_0
- apex-quant
- mixture-of-experts
- multi-latent-attention
pipeline_tag: text-generation
base_model:
- XingChen-AGI/Xing4.0-29B-A4B
---

# Xing4.0-29B-A4B — Apex Quant GGUF Models

This repository contains 3 quantized GGUF profiles of the **XingChen-AGI/Xing4.0-29B-A4B** model, produced using **Apex-Quant** mixed-precision quantization technology.

> **Note:** The original source model is available at [XingChen-AGI/Xing4.0-29B-A4B](https://huggingface.co/XingChen-AGI/Xing4.0-29B-A4B) under Apache 2.0 license. This repository provides quantized variants only.

## 📦 Quick Overview

| Profile | Size | BPW | Best For |
|---------|------|-----|----------|
| **i-quality** | 19.21 GB | ~5.2 | Highest fidelity, production use, research |
| **i-balanced** | 22 GB | ~5.5 | Balanced quality/performance trade-off |
| **i-compact** | 14 GB | 3.85 | Resource-constrained deployment, fast inference |

> **BPW** = Bits Per Weight. Higher value generally correlates with better generation quality.

## 📁 Repository Contents

```
├── Xing4.0-29B-A4B-i-quality.gguf   # 19.21 GB — highest quality
├── Xing4.0-29B-A4B-i-balanced.gguf  # 22 GB — balanced
├── Xing4.0-29B-A4B-i-compact.gguf   # 14 GB — compact
└── patches/
    └── 0001-xing4_0-llama-cpp-build.patch  # Reproducible build instructions
```

## 🏗️ Source Model Information

| Property | Value |
|----------|-------|
| **Architecture** | `Xing4_0ForCausalLM` |
| **Total Parameters** | 29B (4B active per token) |
| **Layers** | 40 (2 dense lead-in + 38 MoE blocks) |
| **Hidden Size** | 3,584 |
| **Vocabulary** | 131,072 tokens (SentencePiece) |
| **Context Length** | 262,144 tokens (YaRN extended from 4,096) |
| **License** | Apache-2.0 |
| **Source** | China Telecom / XingChen-AGI |

### Key Architecture Features

- **Multi-Latent Attention (MLA):** DeepSeek-V3-style MLA compression with `q_lora_rank=768`, `kv_lora_rank=512`, enabling efficient attention without full projection matrices.
- **Mixture-of-Experts (MoE):** 64 routed experts per layer, top-4 active per token (`noaux_tc` gating method), plus 1 shared expert per layer.
- **Masked Hyper-Connections (mHC):** Sinkhorn-gated routing (`mh_mult=4`, `sinkhorn_iters=20`) for enhanced expert collaboration.
- **Multi-Token Prediction (MTP):** Draft head at layer 40 supporting speculative decoding.
- **RoPE:** YaRN positional embeddings with scaling factor 64, theta 10,000.

## 🔬 Quantization Methodology

All three profiles were generated using the **Apex-Quant** framework with importance-weighted mixed-precision quantization:

### Apex-Quant Configuration

```bash
# Generate per-profile config (EDGE/NEAR/MID layer bands)
bash scripts/generate_config.sh \
  --profile i-quality \
  --layers 40 \
  --dense-layers 2 > config.txt

# Append MTP block patterns (blk.40) for draft-head quantization
echo "blk.40.ffn_gate_exps=iq4_xs" >> config.txt
echo "blk.40.ffn_up_exps=iq4_xs" >> config.txt
echo "blk.40.ffn_down_exps=iq4_xs" >> config.txt
# ... (see config files for full content)

# Quantize with imatrix
llama-quantize \
  --tensor-type-file config.txt \
  --imatrix imatrix-Xing4.0-29B-A4B-820chunks.gguf \
  Xing4.0-29B-A4B-BF16.gguf \
  Xing4.0-29B-A4B-i-quality.gguf \
  Q6_K
```

### Layer Band Allocation

| Region | Layers | i-quality Experts | i-balanced Experts | i-compact Experts |
|--------|--------|-------------------|--------------------|-------------------|
| **EDGE** | L0–4, L35–39 | Q6_K | Q6_K | Q4_K |
| **NEAR** | L5–9, L30–34 | Q5_K | Q5_K | Q3_K |
| **MID** | L10–29 | IQ4_XS | Q5_K | Q3_K |
| **Shared FFN** | All layers | Q8_0 | Q8_0 | Q6_K |
| **Attention** | L0–2, L37–39 | Q6_K | Q6_K | Q4_K |
| **Attention** | L3–36 | Q6_K | Q6_K | Q4_K |

### Important Implementation Notes

1. **MTP Block (blk.40):** The multi-token prediction draft head has no imatrix entries because it is not executed during trunk-only inference. Types requiring imatrix (**IQ1/IQ2 series**) cannot be used for blk.40 tensors — this profile uses IQ4_XS/Q5_K/Q3_K which do not require imatrix.

2. **mHC Gate Tensors:** Automatically excluded from quantization by the llama.cpp fork (`hc_attn_fn.weight`, `hc_ffn_fn.weight`, `hc_head_fn.weight` preserved at F32).

3. **MLA B-Matrices (`attn_k_b`):** Tensors with shape `[128, 512, 32]` have an inner dimension (128) that is not divisible by Q6_K's block size (256). These automatically fall back to **Q8_0** via the built-in fallback mechanism — a safe direction (higher precision).

4. **Router Tensors (`ffn_gate_inp`):** Fall through to the base type (Q6_K for i-quality, Q5_K for i-balanced, Q4_K for i-compact).

5. **Config File Format:** The `--tensor-type-file` parser does NOT support comment lines (`#`). Only pure `pattern=value` lines are valid.

## 💻 Inference Usage

### Recommended Settings (from official repo)

```ini
[main]
ctx-size = 65536
flash-attn = true
cache-type-k = q8_0
cache-type-v = q8_0
n-gpu-layers = 999
```

### llama.cpp CLI Examples

```powershell
# i-compact — fastest, lowest VRAM requirement
.\llama-cli.exe ^
  -m models\Xing4.0-29B-A4B-i-compact.gguf ^
  -ngl 999 -c 65536 --flash-attn ^
  --temp 0.7 --top-k 50 --top-p 0.95 ^
  -p "Türkiye'nin en büyük şehri neresidir?" -n 64

# i-quality — highest accuracy
.\llama-cli.exe ^
  -m models\Xing4.0-29B-A4B-i-quality.gguf ^
  -ngl 999 -c 65536 --flash-attn ^
  --temp 0.7 --seed 42 ^
  -p "Explain quantum computing in simple terms." -n 64

# Server mode
.\llama-server.exe ^
  -m models\Xing4.0-29B-A4B-i-compact.gguf ^
  -ngl 999 -c 65536 --host 0.0.0.0 --port 8080
```

### Python (llama-cpp-python)

```python
from llama_cpp import Llama

llm = Llama(
    model_path="Xing4.0-29B-A4B-i-compact.gguf",
    n_ctx=65536,
    n_gpu_layers=999,
    flash_attn=True,
    seed=42,
)

response = llm.create_chat_completion(
    messages=[{"role": "user", "content": "Hello, tell me about Istanbul."}],
    max_tokens=256,
)
print(response["choices"][0]["message"]["content"])
```

### VRAM Sizing Guide

| VRAM | Profile | Offload Strategy |
|------|---------|-----------------|
| **8 GB** | i-compact | Partial (`-ngl 6–10`), expect slow CPU decode |
| **16 GB** | i-compact | Full or near-full (`-ngl 20–35`); i-quality partial |
| **24 GB** | i-compact (full), i-quality (partial) | `-ngl 35–40` for i-quality |
| **40+ GB** | All profiles | Full offload (`-ngl 999`) |

> **KV Cache:** At 65,536 context with `q8_0/kv` ≈ 2.5 GB. To save VRAM, disable GPU KV offload with `--no-kv-offload` (moves KV cache to system RAM).

## 🛠️ Reproducible Build

To reproduce these quantizations from scratch:

### Step 1: Clone and Build the Xing4.0 Fork

```bash
git clone -b xing4_0-port https://github.com/shuxiaoqiong/llama.cpp.git llama-cpp-xing4
cd llama-cpp-xing4

cmake -G "Visual Studio 17 2022" -A x64 ^
      -DCMAKE_BUILD_TYPE=Release -DLLAMA_CUDA=ON ^
      -B build
cd build
cmake --build . --config Release --parallel
```

Required binaries: `llama-quantize.exe`, `llama-imatrix.exe`, `llama-cli.exe`

### Step 2: Convert Source Model to GGUF

```powershell
python convert_hf_to_gguf.py ^
  <path-to-safetensors-dir> ^
  --outfile Xing4.0-29B-A4B-BF16.gguf ^
  --outtype bf16
```

Expected output: ~59 GB GGUF with 977 tensors.

### Step 3: Generate Apex Configs

```bash
# Quality profile
bash scripts/generate_config.sh --profile i-quality --layers 40 --dense-layers 2 > configs/xing4_quality.txt

# Balanced profile
bash scripts/generate_config.sh --profile i-balanced --layers 40 --dense-layers 2 > configs/xing4_balanced.txt

# Compact profile
bash scripts/generate_config.sh --profile i-compact --layers 40 --dense-layers 2 > configs/xing4_compact.txt
```

Append MTP block (blk.40) patterns to each file:

```text
blk.40.ffn_gate_exps=<EXP_TYPE>
blk.40.ffn_up_exps=<EXP_TYPE>
blk.40.ffn_down_exps=<EXP_TYPE>
blk.40.ffn_gate_shexp=<SHARED_TYPE>
blk.40.ffn_up_shexp=<SHARED_TYPE>
blk.40.ffn_down_shexp=<SHARED_TYPE>
blk.40.attn_q=<ATTN_TYPE>
blk.40.attn_k=<ATTN_TYPE>
blk.40.attn_v=<ATTN_TYPE>
blk.40.attn_output=<ATTN_TYPE>
```

Where `<EXP_TYPE>` = iq4_xs (quality) / Q5_K (balanced) / Q3_K (compact), and `<ATTN_TYPE>` = Q6_K (quality/balanced) / Q4_K (compact).

### Step 4: Run Quantization

See the batch script at `.tmp/apex_quant_xing4.ps1` for the automated 3-profile run, or invoke individually:

```powershell
llama-quantize.exe ^
  --tensor-type-file configs\xing4_quality.txt ^
  --imatrix imatrix-Xing4.0-29B-A4B-820chunks.gguf ^
  Xing4.0-29B-A4B-BF16.gguf ^
  Xing4.0-29B-A4B-i-quality.gguf ^
  Q6_K
```

## 📊 Technical Summary

### Tensor Distribution Statistics

| Component | Parameter Count | Approx. Size (BF16) |
|-----------|----------------|---------------------|
| Token Embedding | 470M | 940 MB |
| Output Projection | 470M | 940 MB |
| Attention Projections (all layers) | ~1.2B | ~2.4 GB |
| Expert FFNs (routed, 64×38 layers) | ~23B | ~46 GB |
| Shared Expert FFNs (1×40 layers) | ~360M | ~720 MB |
| mHC Gates & Norms | ~50M | ~100 MB |
| MTP Draft Head (blk.40) | ~3B | ~6 GB |
| **Total** | **~29B** | **~59 GB** |

### Quantization Impact

| Metric | BF16 Base | i-quality | i-balanced | i-compact |
|--------|-----------|-----------|------------|-----------|
| Total Size | 59 GB | 19.21 GB | 22 GB | 14 GB |
| Compression Ratio | 1× | 3.07× | 2.68× | 4.21× |
| Expert FFN Precision | F16 | Q6/Q5/IQ4 | Q6/Q5 | Q4/Q3 |
| Shared FFN | F16 | Q8_0 | Q8_0 | Q6_K |
| Attention | F16 | Q6_K | Q6_K | Q4_K |

## 🙏 Acknowledgments

- **[localai-org/apex-quant](https://github.com/localai-org/apex-quant)** — Apex-Quant MoE-aware mixed-precision quantization framework
- **[shuxiaoqiong/llama.cpp](https://github.com/shuxiaoqiong/llama.cpp)** — Xing4.0 port fork (branch `xing4_0-port`) with native architecture support
- **[ggerganov/llama.cpp](https://github.com/ggerganov/llama.cpp)** — GGUF format specification and quantization engine
- **[XingChen-AGI](https://huggingface.co/XingChen-AGI)** — Original Xing4.0-29B-A4B model creators (China Telecom / XingChen-AGI)
- **[jmarceno](https://huggingface.co/jmarceno)** — Community imatrix and quantization recipe reference (`imatrix-Xing4.0-29B-A4B-820chunks.gguf`)

## 📝 Known Issues & Limitations

1. **Smoke Test Output:** Initial smoke testing with `llama-cli` returned whitespace-only output on CPU (`-ngl 0`) and partial GPU offload (`-ngl 20`). This may indicate MoE expert routing issues in certain runtime configurations. Further testing with different sampler parameters (`--top-k`, `--min-p`) and full GPU offload is recommended before production use.

2. **MTP Draft Head:** The draft head at blk.40 is loaded but not executed during standard inference. Speculative decoding support depends on the inference engine's MTP implementation.

3. **Imatrix Dependency:** While IQ4_XS, Q5_K, Q4_K, and Q3_K types work without imatrix, any future profiles using IQ1/IQ2 series will fail on blk.40 tensors due to missing imatrix entries for the MTP block.

## 📄 License

The original model **XingChen-AGI/Xing4.0-29B-A4B** is distributed under the **Apache-2.0** license. These quantized derivatives are shared under the same license. See the [original model card](https://huggingface.co/XingChen-AGI/Xing4.0-29B-A4B) for full license terms and usage restrictions.

## 🔗 Related Links

- **Original Model:** https://huggingface.co/XingChen-AGI/Xing4.0-29B-A4B
- **Official GGUF Repo:** https://huggingface.co/XingChen-AGI/Xing4.0-29B-A4B-GGUF
- **Deploy Tutorial:** https://github.com/shuxiaoqiong/xing4_0-llama-pc-deploy
- **Apex-Quant Framework:** https://github.com/localai-org/apex-quant
- **llama.cpp Engine:** https://github.com/ggerganov/llama.cpp
- **GGUF Specification:** https://github.com/ggerganov/ggml/blob/master/docs/gguf.md