Hugging Face's logo

scarxity
/
fish-speech-s2-pro-nf4

Text-to-Speech
Transformers
dual_ar
tts
voice-cloning
bitsandbytes
nf4
4-bit precision
quantized
Model card Files
xet
 Community
fish-speech-s2-pro-nf4 / README.md
scarxity's picture
scarxity
NF4 (bitsandbytes 4-bit) build of Fish Speech S2-Pro
279f07f verified
preview code
|
Raw History Blame Contribute Delete
4.15 kB
metadata 
license: other
license_name: fish-audio-research-license
license_link: https://huggingface.co/fishaudio/s2-pro/blob/main/LICENSE
base_model: fishaudio/s2-pro
pipeline_tag: text-to-speech
library_name: transformers
tags:
  - text-to-speech
  - tts
  - voice-cloning
  - bitsandbytes
  - nf4
  - 4-bit
  - quantized
language:
  - en
  - zh
  - ja
  - ko
  - id
  - de
  - fr
  - ar

Fish Speech S2-Pro — NF4 (bitsandbytes 4-bit)

A bitsandbytes NF4 4-bit quantization of fishaudio/s2-pro, the 4B-parameter flagship Fish Audio S2 text-to-speech model.

Roughly 4.9 GB on disk against 8.5 GB of bf16 safetensors, and it loads directly as 4-bit — no quantization step at startup.

Everything the base model does is preserved: voice cloning from a short reference clip, ~50 languages, and inline expressiveness tags such as [laugh], [whispers], [sighs].

Intended use

This checkpoint is built for scarxity/fish-speech-int4-patch, a fork whose server-4gb / webui-4gb Docker profiles run S2-Pro on a 4 GB laptop GPU (measured 3294 MiB peak on a simulated 4096 MiB budget, faster than realtime on an RTX 4060 Ti).

The 4 GB figure comes from the fork's serving code, not from these weights alone. The quantization is what fits ~4.9 GB on disk; getting into 4 GB of VRAM additionally needs the fork's decode-only codec, host-resident embedding table, sliced lm_head, and chunked codec decode. Stock upstream code with this checkpoint will need more VRAM than that.

git clone https://github.com/scarxity/fish-speech-int4-patch
cd fish-speech-int4-patch

huggingface-cli download scarxity/fish-speech-s2-pro-nf4 --local-dir checkpoints/s2-pro-nf4

docker compose --profile server-4gb build
docker compose --profile server-4gb up -d

Then POST to http://localhost:8880/v1/tts:

{ "text": "Hello, world.", "reference_id": "my-voice" }

An OpenAI-compatible /v1/audio/speech endpoint and a Gradio WebUI (port 7860) are included. See the fork's 4 GB guide for reference-voice setup and long-text handling.

Requirements

  • NVIDIA GPU (bitsandbytes NF4 is CUDA-only)
  • bitsandbytes >= 0.43
  • Loading flags --bnb4 --half

This is NF4, not GPTQ, AWQ, or int4 — those formats are not interchangeable, and loaders for them will not read this checkpoint.

Quantization details

Scheme bitsandbytes NF4 (normal-float 4-bit)
Double quantization enabled (compress_statistics=True)
Compute dtype fp16
Quantized the 36-layer slow backbone and the 4-layer depth transformer
Effective ~4.13 bits/weight

Architecture

S2-Pro is a Dual-AR model over an RVQ audio codec (10 codebooks, 21.53 Hz):

  • Slow AR (36 layers, dim 2560) predicts the semantic codebook along time — words, pronunciation, language, and the inline emotion tags.
  • Fast AR (4 layers, dim 2560) runs nine times per frame for the residual codebooks — timbre and fine acoustic texture.
  • Codec upsamples the codes ×2048 to a 44.1 kHz waveform.

Notes

  • tokenizer_config.json in this repo declares PreTrainedTokenizerFast. Some S2-Pro copies ship TokenizersBackend, a transformers v5 name that older loaders reject with UnboundLocalError: tokenizer.
  • Voice cloning is in-context: the reference clip's codes are prepended to every request rather than fine-tuned in. Keep reference clips near 10 seconds — a 31.7 s clip costs 683 prompt tokens on every request against 220 for a 10.2 s one.

License

Released under the Fish Audio Research License, inherited from the base model. Review it before any commercial or redistributive use.

Credits