myroslavtryhubets/clef-omni-NVFP4

Clef-Omni — NVFP4 experts (runs on one 16 GB GPU)

An unofficial NVFP4 quantization of Cloudflare/clef-omni (revision 0db1cd2607d76a7bdb2a382f659e7b313079f84b), Cloudflare's 30B-A3B multimodal decision model (state + schema of typed questions in, one probability per option out, no text generation). Not affiliated with or endorsed by Cloudflare.

The original needs about 64 GB of GPU memory in bf16. Here the 18,432 thinker experts (97% of the parameters) are stored as 4-bit NVFP4 and everything else is unchanged bf16, which brings the checkpoint from ~70 GB to 21 GB and makes it run on a 16 GB card with the experts of part of the layers streamed from host RAM. It is not loadable with from_pretrained: the repo ships its own small runtime, clef_omni_rt.py (see "Use").

Same license as the original, Apache-2.0 (LICENSE is in this repo). Changed from the original: the expert weights (4-bit, stacked per layer in new experts-NN.safetensors files), the file layout and model.safetensors.index.json, a clef_nvfp4 block in config.json, and the speech-output weights (talker.*, code2wav.*) are not included (the release's own loader does not use them). joint_schema_model.py, the tokenizer, chat template, processor config and the joint head (joint_head.safetensors) are byte-identical copies of the original.

What you get

Measured on one RTX 5060 Ti 16 GB (Blackwell, sm_120), torch 2.14 / transformers 5.17, host with 45 GB RAM.

	original bf16	this checkpoint
Size on disk	~70 GB (incl. speech-output weights)	21 GB
GPU memory needed	~64 GB	~13 GiB peak on our card: ~3.1 GiB of non-expert weights (text only; +2.4 GiB for the vision and audio towers) + 0.32 GiB per layer whose experts are resident + activations and the streamed layer
Host RAM	-	up to ~11 GB pinned (experts of the layers that do not fit on the GPU)

Speed here is limited by moving the experts over PCIe and dequantizing them, not by the 3B active parameters. These are our numbers on a small card, not comparable to Cloudflare's 130 ms median on server GPUs:

One request at a time (100 BFCL requests, median 326 tokens), text only	1.46 s median, 1.49 s p95 (experts of 21 of the 48 layers resident on the GPU, the other 27 streamed from host RAM)
Batched (our benchmark runner, ~16k tokens per batch, right-padded), mixed 700-3,400-token requests	~3,100 tokens/s (about 1 request/s)

How close is it to the original

We have no bf16 run of our own (64 GB does not fit anywhere we have), so the comparison is with the numbers Cloudflare published for Clef-Omni (blog post) on the four benchmarks that are also in Decision Index 0.2.1. Requests were rebuilt with the official kit (apolinario/decision-index@87d4650, the same rows file we used for the other Clef checkpoints) and scored with the kit's own scorer. A gap therefore contains everything: 4-bit rounding, our runtime, benchmark-sample noise, and any difference between Cloudflare's setup and ours.

Benchmark	metric	requests scored	Cloudflare (bf16)	this checkpoint
BFCL	case exact accuracy	424 (every 4th of 1,694)	98.2	98.6
BANKING77	macro-F1	770 (every 4th of 3,080)	94.8	94.6
CLINC150+OOS	macro-F1	1,375 (every 4th of 5,500)	97.7	97.7
PhishNChips	accuracy	2,000 (all)	73.2	71.1

• Mean absolute gap 0.7 points; the one gap that stands out is PhishNChips, -2.1 points on all 2,000 rows (standard error of that accuracy is about 1 point). We cannot tell how much of it is the quantization.
• The other three are a systematic one-in-four sample of the benchmark rows (run time), so their numbers carry sampling error of roughly 1-2 points; every request was answered (3,069 and 2,000), none unsupported or errored.
• Not run: the other Clef-Omni benchmarks in Cloudflare's post (ToolRet, API-Bank, Home appliances, When2Call, BRIGHT, Amazon ESCI, TypeSafe), and the other seven Decision Index benchmarks (no Clef-Omni numbers published to compare with).
• Mean relative error of the quantized expert weights against the bf16 originals: 9.5% (L2, all 48 layers, quant_stats.json), the usual NVFP4 level.
• Multimodal: a smoke test only. With the towers on the GPU, 3 synthetic images (a red circle, a blue square, a green circle) were classified by color and shape at 0.96-0.97 confidence, and a 3-second 440 Hz tone vs 3 seconds of silence got P(sound) = 0.987 vs 0.015. Video was not tested (needs av, not in our environment) and no multimodal benchmark was run.

Use

Needs a Blackwell or other recent NVIDIA GPU with ~16 GB, torch with CUDA (we used 2.14), transformers 5.17 (Qwen3OmniMoe classes), torchvision (the processor imports it), pillow for images, av for video (untested). The runtime uses torch._grouped_mm and torch.compile (Triton) for a fused dequantization kernel; first request compiles for a few seconds.

import sys
from huggingface_hub import snapshot_download

path = snapshot_download("myroslavtryhubets/clef-omni-NVFP4")
sys.path.insert(0, path)
from clef_omni_rt import ClefOmni

model = ClefOmni(path, towers_on_gpu=False)   # True keeps the vision/audio towers on the GPU (needed for images/audio; fewer expert layers resident)

response = model.systemone({
    "model": "clef-omni",
    "state": "Our checkout started returning errors and orders are blocked.",
    "questions": {
        "department": {"type": "choice", "instructions": "Which team should handle the message?",
                       "criteria": {"billing": "Payments or invoices", "technical": "Bugs or outages"}},
        "urgency": {"type": "score", "criteria": ["Can wait", "This week", "Today"]},
        "outage": {"type": "noul", "instructions": "Is a service down?"},
    },
})
print(response["answers"])

systemone is the release's own function (Jev/SystemOne request and response bodies, images / audio / videos as in the original card); only the model execution differs. ClefOmni(..., resident_layers=N) sets how many layers keep their experts on the GPU (default: as many as fit with 5.5 GiB left over); model.forward_batch(collated) takes a batch built with joint_schema_model.collate_records. For text-only batches it passes no attention mask: batches are right-padded and attention is causal, so padding is never seen by a real token (we checked: the probabilities match the masked path within 0.003, the same size as the bf16 noise between a batch and single requests).

How it was made

• Streaming quantizer scripts/quantize_omni.py (needs a GPU, never holds the model in memory): reads the bf16 shards, copies every non-expert tensor unchanged (attention, router mlp.gate, norms, embeddings, lm_head, vision and audio towers), and for each thinker layer stacks the 128 experts and quantizes gate_proj+up_proj (concatenated, the layout transformers uses) and down_proj.
• Format per expert matrix: FP4 E2M1 weights packed two per byte, one FP8-E4M3 scale per 16 weights, one fp32 global scale (448*6/amax); weight_packed, weight_scale, weight_global_scale as in compressed-tensors NVFP4. Rounding is round-to-nearest-even on the FP4 grid; plain min-max scaling (no calibration, no clipping search).
• Weights only (W4A16): the runtime dequantizes the selected experts to bf16 on the GPU right before a grouped matmul; activations stay bf16. No activation calibration was done.
• scripts/di_batch.py is the batched Decision Index runner used for the table; scripts/clef_omni_engine.py is the one-request-at-a-time engine for the kit's own runner (the latency row).

Not tested

• Video input; any multimodal benchmark; contexts beyond 16k tokens; the benchmarks listed above; other GPUs (all numbers are from one RTX 5060 Ti); other torch / transformers versions.
• vLLM, SGLang (the original card says SGLang support is coming; the checkpoint layout here is custom and will not load there), llama.cpp, from_pretrained.
• Whether activation quantization (W4A4) would hold up; we did not try.

Credits

Model, training, joint schema head and joint_schema_model.py by Cloudflare; backbone Qwen3-Omni-30B-A3B-Instruct by the Qwen team. Quantization and runtime by @myroslavtryhubets. scripts/ and clef_omni_rt.py are Apache-2.0 like the rest of the repo.