Baekpica/K-EXAONE-236B-A23B-Mixed-Quant-GGUF

K-EXAONE-236B-A23B — Mixed-Quant GGUF

A 237-billion-parameter model, structurally intact, resident on one 128 GB DGX Spark — with its full 262 144-token context.

Not a distillation. Not a pruned or expert-dropped variant. Not a layer-truncated one. Every one of the 128 routed experts is present in every one of the 47 MoE layers, alongside the shared expert, the dense layer 0, and the original 1-layer MTP block. The tensor count matches the BF16 source: 781 tensors, 237.10 B parameters. The only thing that changed is the number of bits each tensor is stored in — assigned by what the tensor does, not by a global bit budget.

The 250 B-class weight class normally implies a multi-GPU host. This artifact fits 85.56 GiB of weights and 12.30 GiB of 256K KV cache into a single GB10's unified memory, measured at 103.95 GiB of 121.6 GiB resident and serving over an OpenAI-compatible API. That is the result this repository exists to demonstrate.

Parameters	237.10 B (A23B active)
Routed experts kept	128 / 128, all 47 MoE layers
Tensors	781 — identical to the BF16 source
BF16 size	441.63 GiB
This artifact (v1)	85.56 GiB — 5.16× smaller
Context served on one GB10	262 144 tokens
Resident at 256K, measured	103.95 GiB / 121.6 GiB

Mixed-precision GGUF builds of LGAI-EXAONE/K-EXAONE-236B-A23B, quantized per module role rather than uniformly, keeping the parts that matter most at 8 bit.

ds4-dfm

This model is served by ds4-dfm, a common ds4 release line for DFM (독자 파운데이션 모델, 독파모) model families. It was developed to serve full-scale models on a single NVIDIA DGX Spark with 128 GB of unified memory through explicit model-family C/CUDA paths and device kernels.

Use the pinned Baekpica/ds4 v0.5.6.3-dfm release. One ds4-server command and one HTTP contract cover the integrated DeepSeek, Solar Open2, K-EXAONE, and Motif-3 families; only the GGUF path and its matching weight-owner manifest change. The server provides OpenAI Chat Completions, OpenAI Completions, OpenAI Responses, and Anthropic Messages.

The production GGUF in this repository was loaded through the common release binary on DGX Spark. That is an integration and API-surface result; context length, throughput, and quality claims remain limited to the model-specific evidence below.

Variants

Variant	Size	Routed gate/up	Routed down	Built with imatrix
v1 …-MXQ-IQ2XXS-Q3K-Q4Edge-Q8Dense-MTPQ8-v1	85.56 GiB	IQ2_XXS	Q3_K	yes
pilot …-MXQ-Q2K-Q4Edge-Q8Dense-MTPQ8-pilot-v1	87.84 GiB	Q2_K	Q2_K	no

Each is published as three shards (-00001-of-00003 …) because the Hub caps individual files at 50 GB. Point llama.cpp at the first shard; it loads the rest automatically. No merge step is needed:

llama-server -m K-EXAONE-236B-A23B-MXQ-IQ2XXS-Q3K-Q4Edge-Q8Dense-MTPQ8-v1-00001-of-00003.gguf \
  -ngl 99 -c 8192

The pilot exists because llama.cpp treats IQ2_XXS without an importance matrix as a hard error, so it substitutes Q2_K and needs no calibration data. v1 is the better artifact on both axes — 2.3 GiB smaller and closer to the Q8_0 reference (see below) — so prefer it unless you specifically want an artifact built without calibration data.

sha256
v1 (unsplit)	0e93f4bc41db6eb53c3520352ff7ec0be40749948a6608deb4cc2ad0818c94a1
pilot (unsplit)	2d840ee44b0e10cb2e14ec7cf58d2e7849615de1a92f58b1220790f42310ce39

Recipe

Quantization is assigned by what each tensor does, not by a global bit budget.

Tensor group	Type	Why
Token embedding, LM head	Q8_0	multilingual token fidelity; logit sensitivity
All norms (incl. QK-norm)	F32	tiny, and error accumulates through them
Router (ffn_gate_inp, exp_probs_b)	F32	a wrong expert choice costs more than any bit saved
Attention Q/K/V/O	Q8_0	long-context stability
Dense layer 0 MLP	Q8_0	every token passes through it
Shared expert	Q8_0	every token passes through it
Routed expert gate/up	IQ2_XXS (pilot: Q2_K)	~64 % of all parameters; where the compression has to come from
Routed expert down	Q3_K (pilot: Q2_K)	weighted accumulation, kept more conservative than gate/up
Edge MoE layers 1–4, 44–47	Q4_K	first and last sparse blocks protected
MTP block (blk.48)	Q8_0	draft quality drives speculative acceptance

Full recipe: quant-recipe-v1.yaml. Exact per-tensor assignments as fed to llama-quantize: *.tensor-types.txt. Per-tensor verification against the recipe: verify-*.json.

Calibration (v1 only)

The importance matrix was built from a corpus covering all six languages the model serves — Korean weighted heaviest, since routed gate/up goes to the most aggressive quant in the recipe and Korean capacity is what this artifact exists to protect. Sources: nvidia/Nemotron-SFT-Multilingual-v2 (ko, ja), -v1 (es, de), Nemotron-Cascade-SFT-Stage-1, Nemotron-SFT-Instruction-Following-Chat-v2, Nemotron-SFT-Competitive-Programming-v2, and Wikipedia for Vietnamese — neither Nemotron release covers it. Records are rendered with K-EXAONE's own chat template so the activations the matrix sees match serving time. Composition: calibration.composition.json.

Provenance

Source model	LGAI-EXAONE/K-EXAONE-236B-A23B @ 61e6d578eb102b578e5704e2916ac841df9eca0a
Source GGUF	LGAI-EXAONE/K-EXAONE-236B-A23B-GGUF @ 5bd0394e4f42c00df63e207b9c434387523a6b77
BF16 GGUF sha256	73be2da8653976df036bf9b6466b011f86cb10f78bab30a47025638ec999d3f8
llama.cpp (quantizer)	ggml-org/llama.cpp @ 6a32c29a746a2e44de463de647f9f6661eb5086b (build b10295)
Converter	Baekpica/k-exaone-mixed-ds4
Serving engine (measured below)	Baekpica/ds4 @ b2faf06f1ce2702efa53ac17145b1f56d3fb23b8
— upstream engine	antirez/ds4
— DGX Spark port	Entrpi/ds4-on-spark

Artifact sha256 and build parameters: *.manifest.json. Tensor-level verification against the recipe (verify-v1.json): 781 tensors, 85.558 GiB, 0 errors, 0 warnings, matching the BF16 source's own tensor count.

Model structure

48 transformer layers plus one MTP block stored as blk.48, so exaone-moe.block_count is 49. Hidden 6144, vocab 153 600, context 262 144. Attention is GQA — 64 query heads over 8 KV heads at head_dim 128 — with per-head RMSNorm on Q and K, on an LLLG sliding-window schedule (window 128; every fourth layer is full attention). Layer 0 is dense (18 432); layers 1–47 are MoE with 128 routed experts, top-8, sigmoid gating with normalized top-k probabilities, routed_scaling_factor 2.5, plus one 2048-wide shared expert. The MTP block is a dense layer with its own attention and eh_proj; it shares the base model's embedding and LM head.

How to run it

Two runtimes serve these files, and they are not interchangeable. ds4 is the one this artifact was sized for and the only one measured here at full context; llama.cpp runs the same file unmodified, but leaves the MTP block on the floor.

	ds4 (Baekpica/ds4, feature/exaone-model-loader)	llama.cpp
Runs the artifact	yes, unmodified	yes, unmodified
MTP block blk.48	executed, target-verified speculative decoding	ignored — stored, never executed
262 144-token context on one 128 GB device	measured — 103.95 GiB resident	not measured here
Multi-turn prefix reuse	yes — a continuation resumes at the divergence point	not measured here
Server API	OpenAI / Responses / Anthropic-compatible	llama.cpp HTTP API
Validated on GB10 / sm_121	yes — see below	no

ds4 — the engine this artifact was sized for

This section describes software outside this repository. Everything below requires Baekpica/ds4, branch feature/exaone-model-loader. Pin the commit below; the branch moves.

The lineage matters, because almost none of the engine is ours:

Layer	Repository	What it provides
Engine	antirez/ds4	the whole runtime — GGUF loader, sessions, KV, CUDA backend, MoE routing, the OpenAI/Responses/Anthropic server, and the NextN/MTP scheduling contract
GB10 port	Entrpi/ds4-on-spark	the sm_121 build target and the aligned-artifact tier that makes mixed-quant MoE weights fast on unified memory
This work	Baekpica/ds4	the exaone-moe model family: GQA + QK-norm attention, the LLLG sliding-window schedule, sigmoid/top-8 routing, and the blk.48 MTP graph

ds4 was an MLA-only engine; K-EXAONE is plain GQA, so that attention path had to be written. Neither antirez/ds4 nor Entrpi/ds4-on-spark serves this model as shipped — use the branch above.

The full serving walkthrough, with the measured numbers, is the next section.

llama.cpp

The artifact is a plain GGUF, so it also runs unmodified on stock llama.cpp — useful for a quick check, or on hardware where ds4 has no backend.

llama-server -m K-EXAONE-236B-A23B-MXQ-IQ2XXS-Q3K-Q4Edge-Q8Dense-MTPQ8-v1-00001-of-00003.gguf \
  -ngl 99 -c 8192

Point it at the first shard; it loads the other two automatically. A mixed-quant GGUF needs no special runtime: GGUF stores a type per tensor and ggml dispatches per tensor, which is how Q4_K_M — itself a mixture of Q4_K, Q6_K and Q8_0 — already works. This recipe just assigns that mixture more aggressively, and llama-quantize is what produced the file.

Measured, not assumed: the pilot artifact loaded in llama-server on 4 × RTX PRO 6000 in 10.2 s and generated 384 tokens of Korean at 78.1 tok/s with a broken-jamo ratio of 0.000.

Two caveats. llama.cpp ignores the MTP block — those tensors are preserved in the artifact, not executed. And nothing on this page about 256K context, resident memory or prefix reuse was measured on it; those are ds4 numbers.

Serving on DGX Spark (GB10 / sm_121) with ds4

Measured on a DGX Spark: NVIDIA GB10, sm_121, 121.6 GiB unified memory, driver 595.71.05, CUDA 13.3, Linux 6.17.

Engine	Baekpica/ds4
Branch	feature/exaone-model-loader
Commit	b2faf06f1ce2702efa53ac17145b1f56d3fb23b8
Weights	Baekpica/K-EXAONE-236B-A23B-Mixed-Quant-GGUF, variant v1
Converter / reports	Baekpica/k-exaone-mixed-ds4

Pin b2faf06 for the measurements on this page. Do not use anything earlier than 920427a for long prompts. Before 920427a, the exaone-moe sliding layers allocated a KV ring exactly the width of the attention window while prefill ran 2 048-token chunks. A chunk writes every row's KV before any row attends, so the ring was left holding only the chunk's last 128 positions and all but the final row of each chunk attended over slots a later position had overwritten. 36 of the 48 layers are sliding, so long-prompt comprehension was badly degraded — asked to summarise 7 000 tokens of Manzoni's Italian prose, the earlier build answered about "Logos" and, on a second passage, about pasta sauce. Short prompts (under ~128 tokens) were never affected, which is why the API validation suite passed throughout. The same defect gave the two-row MTP verify one stale key past depth 128, so the "committed only on an exact match against the target's own argmax" guarantee did not hold there either.

This is a serving-engine defect, not an artifact defect: the GGUF files are unchanged and the quality numbers below were measured on llama.cpp, not ds4.

1 — get the weights (85.56 GiB across three shards):

hf download Baekpica/K-EXAONE-236B-A23B-Mixed-Quant-GGUF \
  --include 'K-EXAONE-236B-A23B-MXQ-IQ2XXS-Q3K-Q4Edge-Q8Dense-MTPQ8-v1-*.gguf' \
  --local-dir ./K-EXAONE-mixed

2 — build the engine. make cuda-spark is the GB10 target; it forces CUDA_ARCH=sm_121 across every binary. Building for the wrong architecture is the single most common way to get wrong kernel results here.

git clone https://github.com/Baekpica/ds4
cd ds4
git checkout b2faf06f1ce2702efa53ac17145b1f56d3fb23b8
make cuda-spark

3 — serve. This is the exact command validated below:

./ds4-server \
  -m ../K-EXAONE-mixed/K-EXAONE-236B-A23B-MXQ-IQ2XXS-Q3K-Q4Edge-Q8Dense-MTPQ8-v1-00001-of-00003.gguf \
  --cuda \
  -c 262144 \
  --host 0.0.0.0 \
  --port 8001

The full 262 144-token context fits on one GB10 with the model resident. Cold start to listening is about 3 min 45 s, dominated by the one-time alignment repack. Give the machine ~119 GiB free before starting: the loader peaks higher than its steady state.

4 — call it. Any OpenAI client works; point base_url at http://<host>:8001/v1. /v1/models advertises the loaded model as k-exaone-236b-a23b (thinking on by default) and k-exaone-236b-a23b-chat (direct answer).

curl -sS http://127.0.0.1:8001/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "k-exaone-236b-a23b-chat",
    "messages": [{"role": "user", "content": "대한민국의 수도는 어디인가요?"}],
    "temperature": 0,
    "max_tokens": 64
  }'

Useful flags: --batched-session N keeps N resident sessions and batches decode-ready requests (concurrency); --exaone-mtp / --exaone-mtp-timing enable the MTP path; --kv-disk-dir enables disk KV checkpoints — not recommended for this model, keep K-EXAONE on in-memory KV.

Budget --batched-session carefully. Each resident session owns its KV and small session state, but the 1.60 GiB prefill graph workspace is shared once per server, not multiplied per slot. The previous per-slot-workspace formula is obsolete. --batched-session 8 -c 40960 has been boot-validated with about 8.4 GiB free on this host; still leave an operational margin because CUDA's retained unified-memory pool makes simple host-RAM accounting misleading.

Resident memory at -c 262144

The 85.56 GiB GGUF is mapped once and left unpinned; ds4 then materialises the weights the CUDA backend actually reads. Those two are alternatives, not additions — the mapping's pages are handed over, not duplicated:

Component	Size
Aligned CUDA artifacts (repacked at load)	39.09 GiB — 78 IQ2 tensors 30.16 GiB + 345 Q8 tensors 8.93 GiB
Raw expert cache payload	45.39 GiB
Weights resident on device	84.48 GiB
KV cache, 262 144 tokens (12 full + 36 sliding layers)	12.30 GiB
Graph workspace (shared across sessions)	1.60 GiB
Context buffers (prefill_chunk 2048)	104.22 MiB
nvtop GPU Mem, idle and ready at 256K	103.95 GiB / 121.6 GiB

That leaves roughly 18 GiB of headroom on a 121.6 GiB machine with the largest context the model supports already allocated.

The LLLG schedule is what makes 256K affordable: only 12 of 48 layers keep a full-context KV, the other 36 keep a 128-position sliding window, so KV costs 48 KiB/token instead of the ~192 KiB/token a fully global GQA stack would need.

Reading the memory numbers

GB10 is a coherent unified-memory device, so "GPU memory" and "host memory" are the same physical pool and the usual tools disagree about who owns it:

• nvidia-smi --query-gpu=memory.used reports [N/A] on GB10. Use nvtop -s and read the process's gpu_mem_bytes_alloc.

• Process VmRSS understates residency by design: ds4 leaves the 85.56 GiB model mmap unpinned and hands the pages to the CUDA cache, so most of the footprint is CUDA-owned rather than process-anonymous.

• A small-context run is not comparable to a 256K run. The same build with two 127-token sessions peaks near 90.17 GiB; the 256K server sits at 103.95 GiB. The difference is almost entirely the 12.30 GiB 256K KV.

• After a clean exit the driver keeps the memory, and that is fine. With 595.71.05, free reports roughly 14 GiB available after ds4-server exits, and it stays there: the whole of /proc/meminfo accounts for only ~17.7 GiB of the 127.5 GiB total, so the ~110 GiB is held by the NVIDIA kernel module, not by page cache. Dropping caches cannot reclaim it — there is nothing in the page cache to drop.

  It also does not need reclaiming. The next CUDA process reuses the driver's pool directly: a second 256K server booted normally in 230 s with MemAvailable still showing 14 GiB. The precondition for booting is that no other ds4-server is running — not a MemAvailable threshold. A readiness check that waits for free memory will wait forever.

Measured throughput

Greedy (temperature: 0), thinking disabled, 128 generated tokens per request, one cold prompt per measurement over /v1/chat/completions with streaming. Every frontier uses a disjoint corpus slice. Prefilled excludes the tiny template prefix already resident in the two short cells; prefill t/s is prefilled / TTFT. decode t/s is measured between the first and last content chunk.

Frontier	Prompt	Prefilled	Prefill t/s	Decode t/s	TTFT
2K	1 451	1 387	269.6	10.75	5.14 s
8K	7 925	7 923	276.5	10.46	28.66 s
32K	31 300	31 300	245.0	9.00	127.78 s
64K	64 663	64 663	207.3	7.27	311.90 s

The previous published engine was roughly 54–56 t/s through 34K. The measured 31.3K cell is now 4.5x faster, and 64K is measured rather than projected. The 2K/8K rows were remeasured on the pinned commit after server warm-up. The 32K/64K cells were taken on the final optimization candidate immediately before a safety-only invalid-token guard was added to batch embedding; valid token IDs use the same numerical path. Raw records and exact methodology are in reports/DGX-SPARK-PREFILL-OPT-2026-08-09.md.

Decode still declines with context depth because 12 full-attention layers read the growing KV history. Cold 256K prefill has not been measured end to end; this card intentionally no longer publishes a 256K time extrapolated from the old 55 t/s kernel stack.

What that means in practice

The 262 144-token context fits, is allocated, and is resident. A cold 31.3K prompt now reaches first token in 127.8 s and a cold 64.7K prompt in 311.9 s; decode is 9.0 and 7.3 t/s at those depths. The request-only profile now puts prefill attention at 2.6% and QK norm/RoPE at 0.6%. The dominant work is the quantized MoE and dense stack — IQ2 gate/up, Q3 down, dense/shared Q8, then Q4. Warm continuations still skip almost all of the prefill entirely (below).

Multi-turn chat reuses the prefix; a cold prompt does not. A continuation resumes at the point where it diverges from what the session already holds, so only the tail is prefilled:

Turn	Prompt tokens	Time to first token	Reused
1 — cold, ~7K document + question	6 978	165.7 s	0
2 — same history + the assistant's own reply + a follow-up	7 083	5.9 s	6 992
3 — a different document, cold	6 725	137.0 s	0

Turn 2 is 24× faster than the same request without reuse, and turns 1 and 3 are unchanged — an unrelated prompt is not falsely matched onto a live session.

This is worth spelling out because it is the case an all-or-nothing prefix test gets wrong, and ds4 used to have one. A chat client replays the assistant's previous reply as text, and re-tokenising it does not reproduce the token IDs the model sampled. The old test required the new prompt to contain the entire checkpoint, so a continuation sharing 6 984 of 7 086 tokens — 98.6 % — failed it and re-prefilled everything, at 143.9 s per turn. It now resumes at the divergence point instead.

How far back that can reach is a property of the sliding-window KV ring rather than a tunable: the ring is window + prefill chunk wide, so a divergence further back than about 2 000 tokens falls back to a cold prefill. Typical chat divergence is one assistant turn, well inside it. Requires ds4 at the commit pinned above. Concurrency now helps. ds4's cross-session row batching (2026-08-08) runs concurrent decode steps through one pass — the weight-bound stages are read once for all streams — so aggregate throughput rises with load instead of staying flat. Steady-state aggregate decode, --batched-session 8, short prompts, all streams decoding:

Concurrent streams	Summed decode t/s	Per stream	before row batching
1	11.5	11.5	11.1
2	14.8	~7.4	9.8
4	16.3	~4.1	10.0
8	18.5	~2.3	10.8

An operator serving several users sees ~15–18 tok/s of total output; a single user still sees the single-stream rate above. The remaining per-row floor is mostly the routed experts — concurrent tokens route to largely disjoint top-8-of-128 sets, so that read genuinely cannot amortise — plus the per-row attention, which is per-session by construction.

A prefill no longer blocks the batch either: a pending prefill quantum rides the decode batch's weight sweep (+prefill in the batch log), so admitting a new long prompt costs the running streams far less than alternating whole passes did.

One contract changed with row batching: greedy output across batch widths is not bit-stable at near-ties. A request decoded alongside seven others can pick a different token than the same request alone where the top-2 margin is tiny, deterministically per batch composition. Sequential (width-1) decode is unchanged, and the same batch always reproduces the same output.

Where the time goes

The 12 full-attention layers hold 49 152 bytes of KV per context position (GQA, 8 KV heads × 128 dims, K and V, f16). Decode adds 0.61 µs per context position — an effective 81 GB/s of KV read against roughly 273 GB/s of device bandwidth. An earlier engine paid 5.97 µs here (3 % of bandwidth, one attention block per head); the flash-decode split closed most of that, and what remains splits between the depth-independent floor (~94 ms/token, streaming the active weights, near the roofline) and the last ~3× of the attention read.

The request-only Nsight slice, immediately before the final 64-column-tail increment, is now: IQ2 aligned gate/up D2R 30.66%, Q3 routed down 23.65%, dense/shared Q8 14.97%, Q4 routed 10.77%, batch embedding 4.90%, prefill attention 2.62%, and QK norm/RoPE 0.60%. That is the next-work order; attention is no longer the first-order bottleneck.

For scale, the cited Spark Arena vLLM run reports about 1,233 t/s at pp2048, but it is a different 180B DeepSeek-V4-Flash checkpoint using MXFP4 experts, FP8 MLA KV, an 8,192-token batch budget, async scheduling, prefix cache, graphs/compilation, a custom model modification, and MTP. It is a useful ceiling, not an apples-to-apples runtime comparison with this 237B top-8 GQA IQ2/Q3/Q4 GGUF.

OpenAI-compatible API

/v1/chat/completions, /v1/completions, /v1/responses and /v1/messages are served; /v1/models advertises k-exaone-236b-a23b and k-exaone-236b-a23b-chat. The bare ID keeps thinking on by default; the chat ID answers directly. Both serve the same loaded GGUF. There is no llama.cpp-style /health or /props — probe /v1/models plus a real completion. Validated on this host, greedy (temperature: 0):

Check	Result
GET /v1/models	serves the bare and -chat K-EXAONE IDs
non-streaming chat completion	finish_reason=stop, correct Korean answer, usage populated
streaming chat completion	SSE chunks, finish_reason, and — with stream_options: {"include_usage": true} — a final usage chunk
streamed text == non-streamed text	identical under temperature: 0
thinking mode	reasoning_content arrives in its own delta field, never inlined into content
four sequential requests	no state carried between them; repeating the first request reproduces it byte-for-byte

Two behaviours worth knowing before you benchmark:

• Thinking is on by default for chat requests. With max_tokens: 64 the budget is spent inside reasoning_content and content comes back empty — that is correct, not a hang. Select k-exaone-236b-a23b-chat, or send "thinking": {"type": "disabled"} / "think": false, for short factual answers.
• Streaming usage follows the OpenAI rule: no stream_options.include_usage, no usage chunk.

Multi-token prediction (blk.48)

ds4 executes the trained MTP block from this same GGUF — no separate draft model, no second weight copy. It is opt-in and off by default: --exaone-mtp enables it, --exaone-mtp-timing adds per-cycle counters.

• Input ordering is the trained one, enorm(embed(x[p+1])) || hnorm(target_hidden[p]), with the decoder position explicitly shifted to p + 1.
• Every draft is verified against the target model's own argmax and committed only on an exact token-ID match, so speculation cannot change greedy output. A 64-token identity test passes with plain == MTP, mismatch = -1.
• Extra runtime state is 0.50 MiB (a 128-row private f16 KV ring).
• Speculation runs for greedy requests only (temperature: 0).
• An automatic loss quench watches the first 12 verifier cycles and disables speculation for the rest of the session when measured MTP work runs more than 3 % slower. DS4_EXAONE_MTP_NO_QUENCH=1 defeats it, for measurement only.

It is close to a wash now, and still not a win. Measured with the quench defeated so the whole generation is speculative, on the engine's current kernels (the two-row verify rides the same aligned-dispatch, small-batch and flash-decode tiers as everything else):

Context	Draft acceptance	MTP ms/token	Plain ms/token	vs plain
1 387	36.6 %	108.8	98.3	+11 %
7 752	60.8 %	102.5	99.7	+2.8 %
33 914	37.0 %	142.8	114.5	+25 %

The mechanism is a single ratio: a cycle runs one draft pass plus one two-row target verify pass; with k the verify's cost relative to a one-row decode and a the acceptance, a cycle commits 1 + a tokens and wins exactly when k < 1 + a. An earlier engine paid k ≈ 2 at shallow depth because its two-row pass re-read the weights per row; that k is now near its floor, which moved MTP from a 26–50 % loss to the table above. What remains is acceptance: at the 3 % quench threshold the 8K row is already a wash, and five to ten more points of acceptance — a warmed MTP ring instead of a cold 128-row one, or corpus luck — is the difference between off and on.

Acceptance numbers are a property of the text (they moved 20 points between corpus slices in these very measurements); compare k across runs, not acceptance.

MTP stays off by default. The auto-quench makes --exaone-mtp safe to try on workloads where drafts land often; nothing here changes greedy output either way.

Measured quality

32 fixtures, greedy (temperature=0, top_k=1), reasoning off, max_tokens 768, compared against the same fixtures run on the official Q8_0 build (234.7 GiB) as reference.

Both sides were run on llama.cpp, on 4 × RTX PRO 6000 (sm_120) — this table measures the artifact, not the ds4 serving path.

	pilot Q2_K · 87.84 GiB	v1 IQ2_XXS+Q3_K · 85.56 GiB
word-agreement vs Q8_0, mean	0.139	0.183
— json / tool-call	0.250	0.681
— long-context retrieval	0.364	0.450
identical outputs	2/32	3/32
JSON parses	4/4	4/4
needle retrieved	3/4	3/4
broken-jamo ratio	0.0001	0.0003
repetition (3-gram)	0.022	0.022
decode, 4 × RTX PRO 6000	66.5 tok/s	77.6 tok/s

v1 tracks the Q8_0 reference more closely than the pilot while being smaller — the importance matrix and Q3_K down are doing real work, most visibly on structured output. Absolute agreement is low for both because greedy long-form generation diverges after a single differing token; the pair track for about 11 words on average before separating. The task-level outcomes (JSON validity, retrieval, no jamo collapse, no repetition loops) match the Q8_0 reference's own scores.

Limitations

• IQ2_XXS on routed gate/up is aggressive. The recipe protects embeddings, attention, router, shared expert, dense layer 0, and the edge MoE layers specifically to offset it, but expect degradation relative to Q4_K_M on tasks that lean on rarely-activated experts.
• Evaluation is a 32-prompt fixture set plus the token-fidelity comparison above, not a full benchmark suite. Raw results, including the failures, ship in the converter repository.
• The full 256K context fits, but cold 256K prefill is not measured. The deepest cold cell here is 64,663 tokens at 207.3 prefill t/s and 7.27 decode t/s. Prefix reuse remains the practical way to reach deep contexts without paying the whole cold admission cost.
• The remaining prefill gap is mainly quantized linear algebra. In the request-only profile, IQ2 gate/up + Q3 down + dense/shared Q8 + Q4 account for about 80% of GPU kernel time; prefill attention is 2.6% and QK norm/RoPE is 0.6%.
• The MTP block only runs under ds4, on the pinned branch and commit above. Under llama.cpp it is inert. There is no third runtime that executes it.
• MTP is a mild loss (3–25 % by depth and corpus), so it ships off by default and auto-quenches when enabled. Its verify cost is now near its floor; the remaining limit is draft acceptance.
• MTP does not run under --batched-session. ds4 disables speculative decoding whenever native session batching is active, so concurrency > 1 is plain decode regardless of the MTP flags.
• Greedy output across batch widths is not bit-stable at near-ties. A request decoded alongside others can pick a different token than the same request alone where the top-2 margin is tiny; the same batch composition always reproduces the same output, and width-1 decode is unchanged. This is the standard batched-inference contract.
• Multi-turn reuse reaches back about 2 000 tokens. The sliding-window KV ring is window + prefill chunk wide, and a resumed prefill needs the window that preceded its restart point. A conversation that diverges further back than that — an edited early message, a re-ordered history — falls back to a cold prefill. One assistant turn of divergence, the normal case, is well inside it.
• Host memory accounting is not usable as a readiness signal. Driver 595.71.05 retains the unified allocation after a clean exit and does not return it to the kernel. Gate a restart on "no ds4-server process", not on free.

Acknowledgements

This artifact is only interesting because there is an engine that serves it, and that engine is almost entirely other people's work.

• antirez/ds4 — Salvatore Sanfilippo's DwarfStar/ds4, the original engine. Everything here is downstream of it: the GGUF loader, the session and KV machinery, the OpenAI/Responses/Anthropic server, the CUDA backend, the MoE routing path, and the NextN/MTP scheduling contract that the blk.48 work slots into. The exaone-moe family is a new model family added to his architecture, not a new engine.
• Entrpi/ds4-on-spark — the DGX Spark fork. The GB10 story rests on it: the sm_121 build target, and above all the aligned-artifact tier — repacking IQ2/Q8 tensors into alignment-correct CUDA artifacts — which is what makes a mixed-quant MoE actually run at speed on GB10's unified memory. That tier was adopted from this fork, along with its warning that single-stream MTP can lose on Spark, which our own measurements then confirmed.
• ggml-org/llama.cpp and GGML — Georgi Gerganov and contributors. GGUF, llama-quantize, the importance-matrix tooling, and the IQ2_XXS/Q3_K/Q4_K/Q8_0 formats are all theirs; this artifact is a llama-quantize output, and llama.cpp is still the reference runtime for it.
• LGAI-EXAONE — K-EXAONE-236B-A23B itself, including the trained MTP block that blk.48 executes.

Mistakes in the recipe, the exaone-moe port, and the measurements here are ours, not theirs.

License and attribution

Base model © LG Management Development Institute, under the K-EXAONE AI Model License Agreement (LICENSE, shipped alongside).

Per §2.1, this Agreement is distributed with the artifact and the derivative name begins with "K-EXAONE". Note §2.2: distributing or sublicensing the model or derivative works to third parties for commercial purposes requires a separate agreement with the Licensor. This repository is published as a research artifact; commercial redistribution is not granted by it.