0xKitkat/Qwen3.8-Flash-Next-125B-UltraLite-37GiB-GGUF

⚡ Qwen3.8 Flash Next 125B — UltraLite 37 GiB GGUF

125B parameters. 37 GiB. One wild GGUF.

This experimental text-only build compresses Qwen3.8-Flash-Next from a 72.55 GB calibrated source GGUF to 39,721,239,200 bytes:

• 36.993287 GiB
• 39.721239 decimal GB
• 1.80 BPW tensor payload
• 125B total parameters / approximately 6B active parameters
• 278,760,800 bytes below a strict 40,000,000,000-byte ceiling

The release is not a blind whole-model requant. It uses a patched GGUF layout, streaming row repacking, mixed quantization by tensor role, boundary-layer protection, byte-for-byte preservation of sensitive paths, and structural removal of optional components not used by ordinary text generation.

Follow the experiments

If you want more aggressively compressed models, weird inference experiments, and build notes, follow @procrastiness on X.

Important runtime requirement

This model uses a padded physical representation for per_layer_token_embd.weight. It therefore requires the included patches/qwen4exp-under40.patch applied to the pinned Qwen4Exp llama.cpp revision. Stock llama.cpp does not understand this custom physical/logical row layout yet.

The patch is part of the release, and the exact build/runtime revision is pinned in build-manifest.txt.

What was compressed

The source is Unsloth's UD-IQ1_S release at revision 83cadfda58d30be06c110518208d1bb918b33f10. Its three source shards total 72,546,461,344 bytes, with 72,535,436,800 bytes of tensor data.

Tensor group	Source payload	Source format	UltraLite format	Reasoning
320,001,536 × 160 per-layer token table	28,800,138,240 B	IQ4_NL	padded Q1_0	Largest single tensor; custom streaming repack makes the target size possible
48 MoE down-projection banks	22,649,241,600 B	IQ4_NL	Q2_0	Down projections are retained at two-bit precision across every layer
96 MoE gate/up banks	17,196,646,400 B	IQ1_S / IQ2_XXS	Q1_0 interior, Q2_0 boundaries	Maximum savings in redundant expert interiors while protecting both ends of the network
Attention, routers, shared experts, hyper-connections, norms, SSM, indexers, token/output embeddings	3,889,410,560 B	BF16/F32/Q8/Q6/Q5/Q4 mix	copied unchanged	Keeps sensitive dense and routing paths identical to the calibrated source

Boundary-layer protection

Expert gate and up projections use Q2_0 in layers:

0, 1, 2, 44, 45, 46, 47

Layers 3–43 use Q1_0 for those expert banks. All 48 expert down banks remain Q2_0. This spends the final byte budget on the transformer entrance and exit, where error propagation is especially undesirable.

The 160 → 256 padded PLE architecture

The per-layer embedding table contains 320,001,536 logical rows of 160 values. The target ultra-low-bit block layout cannot directly encode that row width. The included llama.cpp patch therefore:

1. Reads the source table in bounded chunks of 32,768 rows.
2. Decodes each 160-value logical row.
3. Appends 96 zeros, producing a 256-value physical row.
4. Quantizes that physical row to Q1_0.
5. Stores the physical width in GGUF tensor metadata.
6. At inference, gathers the padded physical row.
7. Creates a strided view of its first 160 values.
8. Materializes the original logical shape before the normal Qwen4Exp graph consumes it.

The padding does not prune learned PLE values: every original element remains present. Only the 96 appended positions are zeros. Quantization error still exists, as expected from Q1_0.

Structural pruning

• The optional vision projector is not included; this is a text-only release.
• The normal-generation source GGUF does not contain the separate 4B MTP training/speculation companion head.
• No ordinary text-generation transformer layer was removed.
• No attention, router, shared-expert, normalization, SSM, indexer, token embedding, or output tensor was pruned.

Exact result

Artifact	Bytes	SHA-256
Qwen3.8-Flash-Next-125B-UltraLite-37GiB-00001-of-00003.gguf	10,946,624	7d0be62f3335e53f7d83a6f2b903a5619014f5de4f7453f0f2095d6d28fb4dda
Qwen3.8-Flash-Next-125B-UltraLite-37GiB-00002-of-00003.gguf	25,357,596,224	b7870d03fdadb5acf38cfaaf46015a638713d67af96db2ae3268c4e0b8200e53
Qwen3.8-Flash-Next-125B-UltraLite-37GiB-00003-of-00003.gguf	14,352,696,352	58cd80f5225550df9b0de7b8b6c1299c242774cff798256b5f445ce42a053795
Total	39,721,239,200	See SHA256SUMS

All 1,224 tensors converted successfully. The patched runtime loaded the three shards, accepted the padded PLE representation, evaluated a prompt, generated a deterministic first token, and exited with code 0. Smoke-test stderr was empty.

Build method

The reproducible pipeline is included as scripts/remote_build.sh. It performs the following complete sequence:

1. Pins the source model repository and exact source revision.
2. Downloads all three source shards with resumable parallel transfers.
3. Validates every source shard against its exact expected byte count.
4. Checks out Unsloth's Qwen4Exp llama.cpp branch at commit 250b61446efc91e3a179c8677956f2667c8fbda0.
5. Applies patches/qwen4exp-under40.patch.
6. Builds llama-quantize, llama-cli, and llama-perplexity.
7. Executes an exact dry-run allocation with explicit per-tensor overrides.
8. Refuses conversion unless projected payload plus 32 MiB of GGUF overhead is below 40,000,000,000 bytes.
9. Streams and requantizes only the selected tensors; unchanged tensors are copied byte-for-byte.
10. Keeps the source three-shard split.
11. Enforces the hard size limit against final filesystem byte counts.
12. Runs a deterministic non-interactive generation smoke test.
13. Generates SHA-256 checksums and a complete build manifest.

The effective quantization command is:

llama-quantize \
  --allow-requantize \
  --keep-split \
  --tensor-type '^per_layer_token_embd[.]weight$=Q1_0' \
  --tensor-type '^blk[.](0|1|2|44|45|46|47)[.]ffn_(gate|up)_exps[.]weight$=Q2_0' \
  --tensor-type '^blk[.][0-9]+[.]ffn_down_exps[.]weight$=Q2_0' \
  --tensor-type '^blk[.][0-9]+[.]ffn_(gate|up)_exps[.]weight$=Q1_0' \
  SOURCE-00001-of-00003.gguf \
  Qwen3.8-Flash-Next-125B-UltraLite-37GiB \
  COPY \
  "$(nproc)"

COPY is intentionally used as the default. The patch permits explicit tensor overrides on top of COPY, so everything not matched by the four expressions retains its original calibrated encoding.

Build it yourself

Required tools on Linux:

sudo apt-get update
sudo apt-get install -y git git-lfs cmake build-essential curl aria2

Then place the patch beside the script and run:

mkdir -p /root/qwen38-compress
cp scripts/remote_build.sh /root/qwen38-compress/
cp patches/qwen4exp-under40.patch /root/qwen38-compress/
chmod +x /root/qwen38-compress/remote_build.sh
/root/qwen38-compress/remote_build.sh

The process needs roughly 73 GB for the source, 40 GB for the output, build space, and temporary headroom. The custom PLE conversion is chunked so it does not allocate the full decoded table in memory.

Run the model

Clone and patch the pinned runtime:

git clone --branch qwen4exp/qwen3.8-flash-next \
  https://github.com/unslothai/llama.cpp.git
cd llama.cpp
git checkout 250b61446efc91e3a179c8677956f2667c8fbda0
git apply ../patches/qwen4exp-under40.patch

cmake -S . -B build -DGGML_CUDA=OFF -DCMAKE_BUILD_TYPE=Release
cmake --build build -j "$(nproc)" --target llama-cli

Run from the directory containing all three GGUF shards:

./llama.cpp/build/bin/llama-cli \
  -m Qwen3.8-Flash-Next-125B-UltraLite-37GiB-00001-of-00003.gguf \
  -c 4096 \
  -t "$(nproc)"

The build was validated CPU-only. GPU backends can be added to the patched runtime in the normal llama.cpp build configuration when supported by the host.

Perplexity comparison

Use the same corpus, context, batch sizes, and runtime for both source and UltraLite:

LLAMA=./llama.cpp/build/bin/llama-perplexity
SRC=/path/to/Qwen3.8-Flash-Next-UD-IQ1_S-00001-of-00003.gguf
DST=/path/to/Qwen3.8-Flash-Next-125B-UltraLite-37GiB-00001-of-00003.gguf
CORPUS=/path/to/calibration.txt

"$LLAMA" -m "$SRC" -f "$CORPUS" -c 2048 -b 128 -ub 128 -t "$(nproc)"
"$LLAMA" -m "$DST" -f "$CORPUS" -c 2048 -b 128 -ub 128 -t "$(nproc)"

Reproducibility files

• build-manifest.txt — exact sizes, revisions, format choices, smoke result, and checksums
• SHA256SUMS — shard integrity hashes
• patches/qwen4exp-under40.patch — converter and runtime support
• scripts/remote_build.sh — complete resumable build pipeline
• scripts/deploy_to_triologay.ps1 — Tailscale deployment helper used for the original build

Credits

• Qwen for the model family
• Unsloth's Qwen3.8-Flash-Next GGUF release
• llama.cpp
• Qwen4Exp llama.cpp work

---

Built as an aggressive compression experiment. For the next one, follow @procrastiness on X.