ngquocvinh/AliceAI-T5-35B-A0.6B-GGUF

AliceAI-T5-35B-A0.6B GGUF

Community GGUF quantizations of yandex/AliceAI-T5-35B-A0.6B.

☕ If this GGUF made your day easier, a coffee would make mine.
Send a coffee ☕
I build and test these releases myself. Your coffee helps keep me going.
Thank you for supporting this work.

About AliceAI-T5-35B-A0.6B

AliceAI-T5-35B-A0.6B is a Yandex encoder-decoder language model with sparse mixture-of-experts layers. The upstream model card describes approximately 34.35B unique parameters, 512 experts per MoE layer with top-8 routing, a 1,536-wide hidden state, 16 encoder layers, 12 decoder layers, shared encoder/decoder embeddings, and RoPE with YaRN. The upstream card advertises a 128K-token context window; this GGUF release has not been validated at that maximum context length.

See the official upstream model card for the original model description, benchmark claims, dependencies, and Transformers usage. This is a custom encoder-decoder GGUF architecture, not a decoder-only model. The runtime must execute the encoder, cross-attention, and decoder generation path.

No training or fine-tuning was performed. Every published artifact was converted directly from the locked upstream BF16 snapshot; no quantized file was used as the source of another quantized file.

Published files

File	Size (GB)	Size (bytes)	Intended use
AliceAI-T5-35B-A0.6B-BF16.gguf	69.177	69,176,812,608	BF16 reference
AliceAI-T5-35B-A0.6B-Q8_0.gguf	36.796	36,795,843,648	Highest-fidelity quantization tier
AliceAI-T5-35B-A0.6B-Q6_K.gguf	28.431	28,430,760,320	High-quality lower-memory tier
AliceAI-T5-35B-A0.6B-Q5_K_M.gguf	24.624	24,623,656,832	Balanced quality/memory tier
AliceAI-T5-35B-A0.6B-Q4_K_M.gguf	21.041	21,040,500,608	General balanced tier
AliceAI-T5-35B-A0.6B-Q3_K_M.gguf	16.575	16,575,128,960	Smaller-memory tier
AliceAI-T5-35B-A0.6B-Q2_K.gguf	12.684	12,683,637,632	Experimental low-memory tier
AliceAI-T5-35B-A0.6B-IQ4_XS.gguf	18.507	18,506,585,984	Compact experimental tier

Q8_0 through Q3_K_M and IQ4_XS passed the encoder-decoder load and deterministic generation smoke test used for this release. Q2_K also loads and generates, but its factual smoke result was not semantically correct; treat it as a runtime/low-memory experiment, not as a quality recommendation. These smoke checks are not task benchmarks.

Three additional local candidates were not published: IQ2_XS produced no decoded token, IQ3_M produced only blank or control-token output, and IQ1_M entered a repeated <SPAN#511>Post loop. All three were excluded by the functional smoke gate.

Quick start

Use an AliceAI-aware llama.cpp build containing the llama-simple encoder- decoder test harness from the runtime used for this release:

./build/bin/llama-simple \
  -m ./AliceAI-T5-35B-A0.6B-Q4_K_M.gguf \
  -ngl 0 -n 1 \
  '[_S_]Question: What is the capital of France? Answer:<SPAN#0>'

The tested protocol prepends the upstream [_S_] mode token, appends <SPAN#0> to the encoder input, and seeds the decoder with [decoder_bos, <SPAN#0>]; the harness handles these encoder-decoder details. The Q4_K_M smoke returns Paris in the tested CPU configuration. A generic decoder-only llama-cli invocation is not sufficient for this architecture.

Reproducibility and validation

The input is pinned to upstream revision a0d71f58c40d6affe461797b30b35ff47f52a5f2 and the GGUF files were converted from the generated BF16 reference. The release used a model-specific AliceAI calibration set plus Wikitext training text to build a combined imatrix. The exact source hashes, imatrix hashes, converter/runtime state, quantization commands, file sizes, and smoke-test results are recorded in reproducibility/manifest.md.

Verify downloads with SHA256SUMS.txt. Raw converter, imatrix, quantization, and smoke-test logs remain local under reports/ and are not part of the public package.

No hold-out BF16 fidelity table is included yet. Mean KLD, Top-1 agreement, delta PPL, and RMS delta-p will be added only after a fixed encoder-decoder hold-out evaluator measures every published file against the same BF16 reference.

License and attribution

The upstream model is released under the Apache License 2.0; see the included LICENSE. These are community GGUF quantizations, not an official Yandex release or endorsement.