jakeatx/Qwen3.6-35B-A3B-ATX-K3-Offload-GGUF

Qwen3.6-35B-A3B ATX K3 Offload GGUF

This repository contains experimental GGUF quantizations of Qwen3.6-35B-A3B intended for CUDA/RAM-offload testing. It now includes both the original trunk-only builds and MTP-capable builds.

The goal is to test whether a Q3-class model made with standard K-quants for the packed MoE expert tensors can run better under offload than the stock IQ-heavy Q3_K_XL style quantization.

These are not official Unsloth releases. They are experimental ATX builds for runtime and quantization research.

Scope note: files without MTP in the filename are trunk-only qwen35moe GGUFs. Files with MTP in the filename were rebuilt from unsloth/Qwen3.6-35B-A3B-MTP-GGUF and retain qwen35moe.nextn_predict_layers = 1, qwen35moe.block_count = 41, and the blk.40.nextn.* MTP tensors.

Base And Related Artifacts

Base model repository:

• unsloth/Qwen3.6-35B-A3B-GGUF
• unsloth/Qwen3.6-35B-A3B-MTP-GGUF

Reference files in the base repository:

• Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf: reference Q4_K_XL tensor policy used as the non-MoE trunk baseline.
• Qwen3.6-35B-A3B-UD-Q3_K_XL.gguf: IQ-heavy Q3-class comparison target.
• Qwen3.6-35B-A3B-UD-Q3_K_M.gguf: Q3_K_M comparison target.
• Qwen3.6-35B-A3B-UD-Q4_K_M.gguf: Q4_K_M comparison target.

Importance / saliency source data:

• sjakek/qwen36-moe-coding-expert-saliency

That dataset contains the SciCode and Terminal-Bench Hard agent saliency reports and portable residency policies used as the coding-task importance prior.

What Was Built

The trunk-only files preserve the Q4_K_XL non-MoE trunk layout:

• 361 F32 tensors
• 252 Q8_0 tensors

Only the 120 trunk packed MoE expert tensors were retargeted. These are tensors of the form:

• blk.N.ffn_gate_exps.weight
• blk.N.ffn_up_exps.weight
• blk.N.ffn_down_exps.weight

The quantization is at packed tensor granularity, not individual expert or layer-expert granularity. Standard GGUF does not support assigning different quant types to individual experts inside a packed expert tensor without changing the format/runtime.

For MTP files, the same 120 trunk packed MoE tensors are retargeted. The extra MTP draft layer, blk.40.*, is preserved from the BF16 MTP source as BF16/F32. That includes the three packed MTP MoE tensors:

• blk.40.ffn_gate_exps.weight
• blk.40.ffn_up_exps.weight
• blk.40.ffn_down_exps.weight

Files

File	Size	Packed MoE expert tensor policy	Intended use
Qwen3.6-35B-A3B-ATX-K3-hot10-q4.gguf	16.887 GB	top 10 packed MoE tensors Q4_K, remaining 110 Q3_K	Clean first CUDA/offload causal test near Q3_K_XL size
Qwen3.6-35B-A3B-ATX-K3-hot20-q4.gguf	17.243 GB	top 20 packed MoE tensors Q4_K, remaining 100 Q3_K	Larger quality guardrail while staying near Q3 size class
Qwen3.6-35B-A3B-ATX-K3-hot30-q4-cold30-q2.gguf	16.782 GB	top 30 Q4_K, middle 60 Q3_K, bottom 30 Q2_K	Saliency-shaped experiment; more high-rank precision funded by low-rank Q2_K
Qwen3.6-35B-A3B-MTP-ATX-k3_hot10_q4.gguf	18.576 GB	trunk top 10 Q4_K, remaining 110 Q3_K; blk.40.* MTP tensors BF16/F32	MTP-capable Hot10 variant
Qwen3.6-35B-A3B-MTP-ATX-k3_hot20_q4.gguf	18.933 GB	trunk top 20 Q4_K, remaining 100 Q3_K; blk.40.* MTP tensors BF16/F32	MTP-capable Hot20 variant
Qwen3.6-35B-A3B-MTP-ATX-k3_hot30_q4_cold30_q2.gguf	18.471 GB	trunk top 30 Q4_K, middle 60 Q3_K, bottom 30 Q2_K; blk.40.* MTP tensors BF16/F32	MTP-capable shaped-policy variant

No retained packed MoE expert tensor uses IQ3_XXS or IQ4_XS.

How The Ranking Was Derived

The ranking is tensor-level. We started from the combined coding saliency work and aggregated it into a packed-tensor priority score.

Inputs:

• Q4_K_XL tensor metadata and imatrix sums from local GGUF inspection.
• SciCode MoE saliency heatmap.
• Terminal-Bench Hard real-agent MoE saliency heatmap.
• Unsloth imatrix signal: imatrix_sum2 / in_sum2 style importance proxy.

Scoring formula used by the compiler:

score =
  0.48 * normalized_imatrix_sum2
+ 0.34 * normalized_coding_layer_importance
+ 0.15 * normalized_coding_layer_activation
+ 0.04 * ffn_down_exps_bonus
+ 0.03 * late_layer_tie_bonus_for_layers_30_39

The saliency data was not applied at individual expert-layer precision because the deliverable is a normal GGUF. The expert/layer-expert data was used as a prior that helps rank which packed MoE tensors are likely worth preserving at higher precision.

Candidate Differences

Hot10 Q4

This is the cleanest first test. It asks:

If we keep the Q4_K_XL non-MoE trunk and switch packed MoE experts to standard K-quants at Q3-class size, does CUDA/RAM offload improve versus stock IQ-heavy Q3_K_XL?

It keeps only the top 10 out of 120 packed MoE tensors at Q4_K and quantizes the other 110 to Q3_K.

Hot20 Q4

This is the practical quality guardrail. It keeps twice as many high-ranked packed MoE tensors at Q4_K, while still landing near the Q3 size class. If Hot10 is fast but degrades quality too much, Hot20 is the next likely candidate.

Hot30 Q4 Cold30 Q2

This is the most shaped policy. It protects more of the high-ranked tensors at Q4_K, leaves the middle at Q3_K, and pushes the bottom-ranked tensors to Q2_K to stay in the same size band. It is more aggressive and should be treated as experimental until quality is measured.

Validation Performed

Local validation was done on Apple Silicon / Metal using the ATX llama.cpp fork.

Static validation:

• Each trunk-only output has 733 tensors.
• Each MTP output has 753 tensors.
• MTP outputs preserve qwen35moe.block_count = 41 and qwen35moe.nextn_predict_layers = 1.
• MTP outputs preserve blk.40.* draft-layer tensors at BF16/F32.
• The Q4_K_XL non-MoE trunk layout is preserved for trunk tensors.
• Packed MoE tensor assignments exactly match the policy files.
• Packed MoE tensors have zero IQ-format tensors.

Smoke validation:

• llama-bench -p 1 -n 1 -ngl 99 -r 1 -o json exited successfully for all three models.
• MTP metadata/tensor validation was performed locally with gguf_dump / tensor summaries. A Metal runtime smoke for the MTP outputs could not be completed in this build pass because the local Metal backend failed to allocate a command queue in the active desktop session; CUDA/offload validation remains the intended test path.

These smoke numbers are load/eval checks, not causal CUDA offload results.

What Still Needs Testing

The intended decision gate is CUDA/RAM-offload benchmarking against:

• stock Q3_K_XL
• stock Q3_K_M
• stock Q4_K_M
• stock Q4_K_XL
• these three ATX K3 variants

Use identical prompts, context, -ngl, batch/ubatch, sampling, runtime build, and offload settings. The key question is whether standard K-quants avoid the suspected IQ-format/offload penalty.

Included Metadata

• policies/: quantization policy JSONs and tensor assignment CSVs.
• tensor_type_files/: llama-quantize --tensor-type-file inputs used to build the models.
• validation/: tensor histograms, build validation summary, and local llama-bench smoke results.

Runtime Notes

Example llama.cpp invocation:

./llama-cli \
  -m Qwen3.6-35B-A3B-ATX-K3-hot10-q4.gguf \
  -p "Write a Python function that parses a JSONL file." \
  -n 128 \
  --ctx-size 4096

CUDA/offload testing should use the target CUDA build and your real offload flags. These files do not require ATX-specific runtime changes to load as GGUFs, but the research motivation is CUDA/RAM-offload behavior.