akhaliq/drama-3-int4

Drama 3 · int4 (hybrid W4A16)

The smallest working Drama 3. Unofficial int4 quantization of fishaudio/drama-3 — Fish Audio's 35B (3B active) multi-speaker TTS model — with real eval numbers, one-command serving, and a loader patch for SGLang-Omni.

• 22.1 GB download (vs 41.5 GB original) — experts, attention and deltanet layers int4; shared-expert MLP kept bf16 (see What we quantize)
• Runs on Ampere (A100, sm80) — the FP8 original needs Hopper/Ada (fp8e4nv kernels); our int4 kernels (moe_wna16 + GPTQ-Marlin) don't
• First drama-3 quant with published WER + speaker-similarity eval (below)
• 5 verified sample types: plain TTS, voice cloning, [sighs]/[laughs] tags, long-form, Chinese

Samples (all generated by this checkpoint)

prompt	clip
plain TTS	int4_plain.wav
voice clone (reference clip)	int4_clone.wav
clone + [sighs]/[laughs] tags	int4_tags.wav
long-form	int4_long.wav
Chinese	int4_zh.wav

Eval (Whisper large-v3 WER + resemblyzer speaker similarity, 8 prompts)

Compared against the strongest community quant, darioooooo0o/drama-3-w4a16-32gb (int8 dense layers, 32 GB), same prompts, same scorer, one A100:

	mean WER ↓	mean WER excl. zh¹ ↓	speaker sim ↑
this repo (int4 hybrid, 22.1 GB)	0.197	0.082	0.846
theirs (int8 dense, 24.5 GB)	0.259	0.154	0.846

¹ The Chinese prompt scored WER 1.0 for both checkpoints despite nearly identical transcripts — a jiwer tokenization artifact on unspaced Chinese, not a model difference. Full per-prompt rows in eval/eval_results.json; clips in eval/.

Notable per-prompt outcomes: their build hallucinated an ending on the long paragraph (WER 0.667 vs our 0.028); ours truncated the tag line's tail on one take (0.333 vs their 0.25). Speaker similarity is a tie.

Quickstart

git clone https://github.com/sgl-project/sglang-omni.git && cd sglang-omni
uv venv -p 3.12 .venv && . .venv/bin/activate
uv pip install --prerelease=allow -e ".[fishaudio]"
uv pip install ninja   # Marlin repack JIT-builds on first load

hf download akhaliq/drama-3-int4 --local-dir drama-3-int4
python drama-3-int4/apply_patch.py .   # idempotent; registers the drama3-int4 quant method

sgl-omni serve --model-path drama-3-int4 --port 8000 \
  --tts_engine.gpu_memory_fraction 0.62 --vocoder.gpu_memory_fraction 0.2

Then the standard Drama 3 API: /v1/audio/speech with input, optional references for voice cloning, [sighs]/[laughs] tags, multi-speaker scenes, and previous_takes editing all work as on the original.

Measured serving memory: 57.3 GB GPU at startup, 57.9 GB peak across the five sample generations (A100-80GB, flags above). On smaller cards, lower tts_engine.gpu_memory_fraction and vocoder.gpu_memory_fraction proportionally; the weights themselves are 22.1 GB.

What we quantize

component	original	here	why
text embeddings, lm_head, router	bf16	bf16	quality-critical, small
attention q/k/v/o (full-attn layers)	bf16	int4	big, RTN-tolerant
deltanet in/out projections	bf16	int4	big, RTN-tolerant
shared-expert MLP	bf16	bf16	int4 passed all numerics checks but produced zero audio frames at serve time (see shared_expert_verdict.md)
routed MoE experts (30,720 matrices)	FP8 (block)	int4	the bulk: 32.2 GB → 16.6 GB
audio decoder + codec	bf16	bf16	the audio path stays lossless

Method: RTN + per-group clip search (group 128, symmetric), no calibration data. GPTQ packing for SGLang's moe_wna16 kernel (experts) and standard GPTQ v1 fields repacked by GPTQ-Marlin at load (dense). Reproduce with quantize.py; load with apply_patch.py — the patch also fixes fused-name matching (gate_up_proj/qkv_proj/in_proj_qkvz), without which fused modules silently miss their int4 weights and generation returns zero frames (the v3 fix). Stats in quant_stats.json.

Limitations

• RTN, no calibration — a GPTQ/AutoRound pass with real calibration data would likely close some of the remaining quality gap; the building blocks are in quantize.py.
• Tags prompts occasionally truncate a take (also happens on the original) — regenerate.
• FP8-vs-int4 A/B eval not included: the FP8 original needs Hopper (fp8e4nv), unavailable here.
• Serving tested on A100-80GB only; 48 GB and 32 GB cards should fit with the memory flags scaled down but are unverified.

License

Derivative Work of Fish Audio Materials under the Fish Audio Community License v1.0 (LICENSE) — same terms as the original model, including: research/non-commercial use free; commercial use only under USD 500k annual revenue; no hosted TTS/voice-cloning service without a separate license; no cloning real people's voices without consent; do not remove watermarks; do not use outputs to train other speech or foundation models.

Credits

• Model: Fish Audio — fishaudio/drama-3 · tech report
• Runtime: SGLang-Omni and SGLang int4 kernels
• Companion quants: darioooooo0o/drama-3-w4a16 (experts-only int4, first) · darioooooo0o/drama-3-w4a16-32gb (int8 dense, 32GB-card target)