bosonai

/

higgs-tts-3-4b

Like 760

Follow

Boson AI 671

Hi, thanks for your feedback. Examples are welcomed to help us locate the problem.

Can you also share your inference setting? Are u using sglang-omni?

Thanks for the reply. Give me some time to get the audio evidence and write a small report. I don't have the glitches saved, so I will have to try to reproduce them. They don't happen all the time, but enough to be a bit annoying.

Ok, here are my results attached. Please note that I am not an expert coder in any way. I am an amateur. So, forgive my ignorance. I am a AI Vibe Code Baby lol.

SYSTEM SPECS:
Linux Mint
RAM: 128GB DDR4-3600
GPU Primary: NVidia RTX 4090 24GB VRAM
GPU Secondary: NVidia RTX 3080Ti 12GB VRAM
CPU: AMD Ryzen 9 5950x 16-core
M.2 main drive that HiggsTTS is running on.

I am using sglang-omni. I will try to share the info I have to the best of my knowledge.
I have made a lot of changes to fit my project, so I kinda lost track of some things. So, I will post settings and other things I believe will be useful.

(docker-compose.yml:)-----------------------------------------------------------------------
services:
  higgs-tts:
    build: 
      context: .
      dockerfile: Dockerfile.higgs
    container_name: higgs_tts_server
    privileged: true
    ipc: host
    network_mode: host
    shm_size: '32gb'
    environment:
      - HF_HOME=/root/.cache/huggingface
      - CUDA_VISIBLE_DEVICES=0 
    volumes:
      - "./:/root/.cache/huggingface"
      - "/home/nezz/AI Agent/Data/TTS Audio:/home/nezz/AI Agent/Data/TTS Audio"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ["0"] 
              capabilities: [gpu]
    
    # The Dynamic Boot Script
    command:
      - /bin/bash
      - -c
      - |
        config_cls: HiggsTtsPipelineConfig
        model_path: bosonai/higgs-audio-v3-tts-4b
        stage_overrides:
          tts_engine:
            runtime:
              sglang_server_args:
                mem_fraction_static: 0.80
        EOF
        echo "Starting SGLang with memory fraction: 0.80"
        exec sgl-omni serve --model-path bosonai/higgs-audio-v3-tts-4b --port 8000 
-------------------------------------------------------------------------------------------

(Dockerfile.higgs:)------------------------------------------------------------------------
FROM lmsysorg/sglang-omni:dev
WORKDIR /workspace

# Clone and install SGLang-Omni
RUN git clone https://github.com/sgl-project/sglang-omni.git \
    && cd sglang-omni \
    && uv venv .venv -p 3.12 \
    && . .venv/bin/activate \
    && uv pip install -v -e .

# Expose the virtual environment to the system path
ENV PATH="/workspace/sglang-omni/.venv/bin:$PATH"

# Default execution command
CMD ["sgl-omni", "serve", "--model-path", "bosonai/higgs-audio-v3-tts-4b", "--port", "8000"]
----------------------------------------------------------------------------------------

I start up Higgs and my program in a script. Here is a snippet for Higgs:

HIGGS_GPU="0"       #Assign which GPU you want Higgs to run on - Default is 0 for one GPU
HIGGS_MEM="0.80"    #Set the amount of your cards total VRAM Higgs reserveres for itself - 1.0 is 100% - 0.50 is 50%
===========================================
# 3. STARTUP HIGGS TTS SERVER
#===========================================
echo "Starting Higgs TTS Server via Docker..."
# Navigate into the Docker folder
cd "Data/Docker"
# Spin up the container in detached mode
HIGGS_GPU="$HIGGS_GPU" HIGGS_MEM="$HIGGS_MEM" CUDA_VISIBLE_DEVICES="$HIGGS_GPU" NVIDIA_VISIBLE_DEVICES="$HIGGS_GPU" docker compose up -d

--------------------------------------------------------------------------------------

Here is a snippet of code in my python that calls Higgs:

def generate_tts(text, history=None, output_path=OUTPUT_AUDIO_PATH):
    """
    Sends text to the local HiggsTTS API and saves the resulting audio file.
    """
    text_to_speak = preprocess_text_for_tts(text)
    print(f"{AGENT.upper()} SPEAKING: {text_to_speak}")

    if not text_to_speak.strip():
        return False

    payload = {
        "input": text_to_speak,
        "max_new_tokens": 1024,
        "temperature": 0.5,
        "repetition_penalty": 1.15,
        "num_steps": 40,
        "references": [{
            # Pointing to the dynamic mutable state instead of the static constant
            "audio_path": ACTIVE_VOICE_STATE["audio"],
            "text": ACTIVE_VOICE_STATE["text"]
        }]
    }
-----------------------------------------------------------------------------------

RECORDINGS:

Description: TTS Audio loops the same phrase. Also, doesn't even repeat what the text was.
Text input: The Forest is a good game <|sfx:laughter|>Hehe. Maybe we should play that?
TTS Output: repetition.wav

Description: TTS Audio skips "<|sfx:sigh|>huh" at the end and instead there is a long silence.
Text input: <|emotion:sadness|><|prosody:speed_slow|>I just don't feel like doing anything today. Why don't you just go by yourself and leave me alone. I just need quiet and darkness today. I'm sorry, User. Thank you for understanding <|sfx:sigh|>huh.
TTS Output: long_silence.wav

Description: Glitched laughter at the end.
Text input: <|emotion:amusement|><|prosody:pitch_low|>Oh, really, User? Is that so? Well, I bet you wouldn't be laughing if I deleted all your files <|sfx:laughter|>Hehe!
TTS Output: glitched_laughter.wav

Description: Incomplete reply. TTS begins reply, but then falls silent and doesn't finish the rest of the sentence.
Text input: <|emotion:arousal|><|style:whispering|><|prosody:speed_slow|><|prosody:pitch_low|>Everything is going to be ok. Just close your eyes and relax. Take a deep breath, hold it, then exhale. Let the stress of the day melt away with each deep breath. Shhh... Close your eyes and breath. Relax your jaw and shoulders.
TTS Output: incomplete.wav

Description: Sound Glitch. Same phrase as 'incomplete.wav' but sound glitches on 'Shhh...'
Text input: <|emotion:arousal|><|style:whispering|><|prosody:speed_slow|><|prosody:pitch_low|>Everything is going to be ok. Just close your eyes and relax. Take a deep breath, hold it, then exhale. Let the stress of the day melt away with each deep breath. Shhh... Close your eyes and breath. Relax your jaw and shoulders.
TTS Output: sound_glitch.wav

Have you tested with "temperature": 0.8, "top_k": 50 before? That could be a factor

I will give it a try.

Same Issue....😥

So, I tried a temp of 0.7 and top_k of 50 and it did better. Then I turned the top_k down more to 10 and it did even better. Still has it's hic-ups. Going to try top_k=0 and just turn it off and see how that goes.