| --- |
| license: apache-2.0 |
| library_name: transformers |
| pipeline_tag: image-text-to-text |
| base_model: Cloudflare/clef |
| base_model_relation: quantized |
| datasets: |
| - mit-han-lab/pile-val-backup |
| tags: |
| - clef |
| - qwen3.8 |
| - quark |
| - quantized |
| - mxfp4 |
| - awq |
| - amd |
| - rocm |
| - multimodal |
| - structured-output |
| - classification |
| - custom-code |
| --- |
| |
| # Clef MXFP4 |
|
|
| This is an MXFP4 quantization of [Cloudflare/clef](https://huggingface.co/Cloudflare/clef), a 27B multimodal decision |
| model post-trained from Qwen3.8-27B. It was quantized with [AMD Quark](https://quark.docs.amd.com/) 0.13. |
|
|
| The checkpoint is a standard Quark `hf_format` export. It loads in vLLM, and in Transformers with `amd-quark` |
| installed. It is not tied to one GPU family: it runs natively on hardware with MXFP4 matrix units (for example |
| MI350/MI355X) and emulated elsewhere. |
|
|
| | | BF16 original | This repo | |
| |---|---|---| |
| | Backbone weights on disk | 54.7 GB | 18.9 GB | |
| | Joint schema head | BF16 | BF16, byte-identical | |
| | vLLM weight memory | 50.2 GiB | 17.0 GiB | |
|
|
| ## Quantization |
|
|
| | | | |
| |---|---| |
| | Format | OCP MXFP4: E2M1 elements with one E8M0 scale per 32 values along the input dimension | |
| | Weights | MXFP4, static, `even` scale rounding | |
| | Activations | MXFP4, dynamic per 32-value block | |
| | Algorithm | AWQ (Quark's `qwen3_5` template) on the MLP projections | |
| | Calibration | 128 samples × 512 tokens from `pileval` (`mit-han-lab/pile-val-backup`) | |
| | Quantized | All language-model linear layers: full attention, Gated DeltaNet projections and MLP, in all 64 layers | |
| | Kept in BF16 | Vision encoder and merger, `lm_head`, embeddings, norms, the `conv1d` layers, and the joint schema head | |
|
|
| `lm_head` stays in BF16 deliberately. Clef's joint schema head reads the output-embedding matrix directly to embed answer |
| options. |
|
|
| The recipe is the same as AMD's own [amd/Qwen3.8-27B-Quark-AWQ-MXFP4](https://huggingface.co/amd/Qwen3.8-27B-Quark-AWQ-MXFP4) |
| for the base model: |
|
|
| ```bash |
| python3 quantize_quark.py \ |
| --model_dir Cloudflare/clef \ |
| --output_dir ./clef-MXFP4 \ |
| --quant_scheme mxfp4 \ |
| --quant_algo awq \ |
| --num_calib_data 128 \ |
| --seq_len 512 \ |
| --model_export hf_format \ |
| --data_type auto \ |
| --device cuda \ |
| --skip_evaluation |
| ``` |
|
|
| `quantize_quark.py` is the LLM PTQ example from the [Quark v0.13 repository](https://github.com/amd/Quark/tree/v0.13/examples/torch/language_modeling/llm_ptq). |
| After export, the backbone was resharded into five files with a safetensors index (bit-identical tensors), and the |
| joint head files were copied unchanged. `algo_config` is set to `null` in `config.json` because AWQ is already folded |
| into the weights and vLLM does not parse that field. The exported original is kept as `config.json.orig_with_algo_config`. |
|
|
| ## Usage |
|
|
| ### Clef decisions (Transformers) |
|
|
| Use the original Clef code unchanged; only the repository name changes. Loading needs `amd-quark`: |
|
|
| ```bash |
| pip install amd-quark transformers pillow |
| ``` |
|
|
| ```python |
| import sys |
| |
| import torch |
| from huggingface_hub import snapshot_download |
| |
| path = snapshot_download("EliovpAI/clef-MXFP4") |
| sys.path.insert(0, path) |
| from joint_schema_model import load_release_model, systemone |
| |
| model, processor = load_release_model(path, device="cuda") |
| |
| response = systemone(model, processor, { |
| "model": "clef", |
| "state": "Our checkout started returning errors and orders are blocked.", |
| "questions": { |
| "department": { |
| "type": "choice", |
| "instructions": "Which team should handle the message?", |
| "criteria": {"billing": "Payments or invoices", "technical": "Bugs or outages"}, |
| }, |
| "urgency": {"type": "score", "criteria": ["Can wait", "This week", "Today"]}, |
| "outage": {"type": "noul", "instructions": "Is a service down?"}, |
| }, |
| }) |
| print(response["answers"]) |
| ``` |
|
|
| See the [Clef model card](https://huggingface.co/Cloudflare/clef) for the input format, image and video inputs, and |
| batching. In Transformers, Quark simulates the MXFP4 arithmetic. This is the reference path for accuracy, not for |
| speed. |
|
|
| ### Backbone serving (vLLM) |
|
|
| vLLM loads the quantized backbone and runs native MXFP4 GEMMs where the hardware supports them. The Clef joint head is |
| not part of vLLM; use the Transformers path above for decisions. |
|
|
| ```bash |
| vllm serve EliovpAI/clef-MXFP4 --max-model-len 16384 |
| ``` |
|
|
| ## Evaluation |
|
|
| All runs were on one AMD Instinct MI355X (gfx950) with ROCm 7.2.3. BF16 is `Cloudflare/clef` at revision `2f3de3d`. |
| Both models used the same code and inputs. |
|
|
| ### Clef decisions: release code (Transformers 5.8.1, Quark 0.13) |
|
|
| | Task | Records | BF16 accuracy | MXFP4 accuracy | Top-1 agreement | Mean KL (BF16 ‖ MXFP4) | |
| |---|---|---|---|---|---| |
| | BANKING77 test, 77-way choice | 300 | 94.0 % | 93.3 % | 98.0 % | 0.022 | |
| | CLINC150 plus test, 151-way choice incl. out-of-scope | 300 | 96.7 % | 96.7 % | 98.7 % | 0.018 | |
| | Receipt image, 2 × noul + 3-way choice | 3 questions | – | – | 100 % | 0.0008 | |
|
|
| The records are a fixed random sample (seed 0). Each label is a choice option whose description is the |
| humanized label name. These are our own prompts, not the Decision Index protocol, so the absolute numbers are not |
| comparable to Cloudflare's published results. Use the BF16/MXFP4 difference. |
|
|
| The SystemOne example from the Clef model card gives the same answers. `technical` is chosen at 0.918 |
| (BF16 0.917), `urgency` peaks at "Today" with 0.920 (BF16 0.862), and `outage` is 0.841 (BF16 0.895). |
|
|
| ### Backbone language modelling (vLLM 0.21.0) |
|
|
| | | BF16 | MXFP4 | |
| |---|---|---| |
| | WikiText-2 perplexity (64 × 1,024 tokens) | 9.974 | 10.505 (+5.3 %) | |
| | Greedy chat answers (4 prompts) | coherent | coherent, same content | |
|
|
| ## Files |
|
|
| | File | Purpose | |
| |---|---| |
| | `model-0000X-of-00005.safetensors`, `model.safetensors.index.json` | Quantized backbone (MXFP4 language model, BF16 vision encoder) | |
| | `config.json` | Model and Quark quantization config | |
| | `config.json.orig_with_algo_config` | Config as exported by Quark, including the AWQ settings | |
| | `joint_head.safetensors`, `joint_head_config.json`, `joint_schema_model.py` | Clef joint schema head and code, unchanged from the original | |
| | `tokenizer*`, `chat_template.jinja`, `processor_config.json`, `preprocessor_config.json`, `generation_config.json` | Tokenizer and processors | |
| | `LICENSE` | Apache-2.0, from the original repository | |
| | `SHA256SUMS` | Checksums of every file above | |
|
|
| ## License and attribution |
|
|
| Apache-2.0, the same as [Cloudflare/clef](https://huggingface.co/Cloudflare/clef), which is post-trained from |
| [Qwen/Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B). All credit for the model goes to Cloudflare and the Qwen |
| team. This repository changes only the weight format of the backbone, as described above. It is not affiliated with or |
| endorsed by Cloudflare, Qwen or AMD. |
|
|
