Use Audio8 native compile batch policy
Browse files- Dockerfile +1 -2
- README.md +2 -2
Dockerfile
CHANGED
|
@@ -13,7 +13,7 @@ RUN apt-get update && \
|
|
| 13 |
WORKDIR /home/user/app
|
| 14 |
|
| 15 |
ARG SGLANG_OMNI_REVISION=68a572348837f7b004857b4b07993c20ade4c017
|
| 16 |
-
ARG AUDIO8_TTS_REVISION=
|
| 17 |
|
| 18 |
RUN git init /opt/sglang-omni && \
|
| 19 |
git -C /opt/sglang-omni remote add origin https://github.com/sgl-project/sglang-omni.git && \
|
|
@@ -71,7 +71,6 @@ ENV HOME=/home/user \
|
|
| 71 |
AUDIO8_TTS_MAX_RUNNING_REQUESTS=8 \
|
| 72 |
AUDIO8_TTS_DISABLE_CUDA_GRAPH=0 \
|
| 73 |
AUDIO8_TTS_ENABLE_TORCH_COMPILE=1 \
|
| 74 |
-
AUDIO8_TTS_TORCH_COMPILE_MAX_BS=1 \
|
| 75 |
AUDIO8_TTS_VERIFY_FAST_KV=1 \
|
| 76 |
FLASHINFER_WORKSPACE_BASE=/tmp/audio8-flashinfer \
|
| 77 |
TORCHINDUCTOR_CACHE_DIR=/tmp/audio8-torchinductor \
|
|
|
|
| 13 |
WORKDIR /home/user/app
|
| 14 |
|
| 15 |
ARG SGLANG_OMNI_REVISION=68a572348837f7b004857b4b07993c20ade4c017
|
| 16 |
+
ARG AUDIO8_TTS_REVISION=59ca6d27d441302eaa568bcb140b63c50a879859
|
| 17 |
|
| 18 |
RUN git init /opt/sglang-omni && \
|
| 19 |
git -C /opt/sglang-omni remote add origin https://github.com/sgl-project/sglang-omni.git && \
|
|
|
|
| 71 |
AUDIO8_TTS_MAX_RUNNING_REQUESTS=8 \
|
| 72 |
AUDIO8_TTS_DISABLE_CUDA_GRAPH=0 \
|
| 73 |
AUDIO8_TTS_ENABLE_TORCH_COMPILE=1 \
|
|
|
|
| 74 |
AUDIO8_TTS_VERIFY_FAST_KV=1 \
|
| 75 |
FLASHINFER_WORKSPACE_BASE=/tmp/audio8-flashinfer \
|
| 76 |
TORCHINDUCTOR_CACHE_DIR=/tmp/audio8-torchinductor \
|
README.md
CHANGED
|
@@ -17,8 +17,8 @@ The service uses the official Audio8 adapter with SGLang Omni 0.1.0 and
|
|
| 17 |
SGLang 0.5.8.
|
| 18 |
|
| 19 |
Reference audio uploaded through the web interface is limited to 30 seconds.
|
| 20 |
-
The A10G runtime enables `torch.compile`
|
| 21 |
-
|
| 22 |
|
| 23 |
## API
|
| 24 |
|
|
|
|
| 17 |
SGLang 0.5.8.
|
| 18 |
|
| 19 |
Reference audio uploaded through the web interface is limited to 30 seconds.
|
| 20 |
+
The A10G runtime enables `torch.compile` with SGLang's native batch policy and
|
| 21 |
+
uses CUDA Graph without an Audio8-specific compile batch-size limit.
|
| 22 |
|
| 23 |
## API
|
| 24 |
|