Upload folder using huggingface_hub

README.md

@@ -931,7 +931,7 @@ For more details, please refer to our blog post [Qwen3.5](https://qwen.ai/blog?i
 > To disable thinking content and obtain direct response, refer to the examples [here](#instruct-or-non-thinking-mode).
 
 
-For streamlined integration, we recommend using Qwen3.5 via APIs. Below is a guide to use Qwen3.5 via OpenAI-compatible API. For programmatic inference or offline batch processing, please consult our [documentation](https://qwenlm.github.io/Qwen3.5).
+For streamlined integration, we recommend using Qwen3.5 via APIs. Below is a guide to use Qwen3.5 via OpenAI-compatible API. 
 
 ### Serving Qwen3.5
 
@@ -963,19 +963,19 @@ The following will create API endpoints at `http://localhost:8000/v1`:
 - **Standard Version**: The following command can be used to create an API endpoint with maximum context length 262,144 tokens using tensor parallel on 8 GPUs.
     
     ```shell
-    python -m sglang.launch_server --model-path Qwen/Qwen3.5-397B-A17B --port 8000 --tp-size 8 --mem-fraction-static 0.8 --context-length 262144 --mamba-ssm-dtype float32 --reasoning-parser qwen3
+    python -m sglang.launch_server --model-path Qwen/Qwen3.5-397B-A17B --port 8000 --tp-size 8 --mem-fraction-static 0.8 --context-length 262144 --reasoning-parser qwen3
     ```
 
 - **Tool Use**: To support tool use, you can use the following command.
     
     ```shell
-    python -m sglang.launch_server --model-path Qwen/Qwen3.5-397B-A17B --port 8000 --tp-size 8 --mem-fraction-static 0.8 --context-length 262144 --mamba-ssm-dtype float32 --reasoning-parser qwen3 --tool-call-parser qwen3_coder
+    python -m sglang.launch_server --model-path Qwen/Qwen3.5-397B-A17B --port 8000 --tp-size 8 --mem-fraction-static 0.8 --context-length 262144 --reasoning-parser qwen3 --tool-call-parser qwen3_coder
     ```
 
 - **Multi-Token Prediction (MTP)**: The following command is recommended for MTP:
     
     ```shell
-    python -m sglang.launch_server --model-path Qwen/Qwen3.5-397B-A17B --port 8000 --tp-size 8 --mem-fraction-static 0.8 --context-length 262144 --mamba-ssm-dtype float32 --reasoning-parser qwen3 --speculative-algo NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4
+    python -m sglang.launch_server --model-path Qwen/Qwen3.5-397B-A17B --port 8000 --tp-size 8 --mem-fraction-static 0.8 --context-length 262144 --reasoning-parser qwen3 --speculative-algo NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4
     ```
 
 #### vLLM
@@ -994,30 +994,27 @@ The following will create API endpoints at `http://localhost:8000/v1`:
 - **Standard Version**: The following command can be used to create an API endpoint with maximum context length 262,144 tokens using tensor parallel on 8 GPUs.
 
     ```shell
-    vllm serve Qwen/Qwen3.5-397B-A17B --port 8000 --tensor-parallel-size 8 --max-model-len 262144 --mamba-ssm-cache-dtype float32 --reasoning-parser qwen3 
+    vllm serve Qwen/Qwen3.5-397B-A17B --port 8000 --tensor-parallel-size 8 --max-model-len 262144 --reasoning-parser qwen3 
     ```
 
 - **Tool Call**: To support tool use, you can use the following command.
     
     ```shell
-    vllm serve Qwen/Qwen3.5-397B-A17B --port 8000 --tensor-parallel-size 8 --max-model-len 262144 --mamba-ssm-cache-dtype float32 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder 
+    vllm serve Qwen/Qwen3.5-397B-A17B --port 8000 --tensor-parallel-size 8 --max-model-len 262144 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder 
     ```
 
 - **Multi-Token Prediction (MTP)**: The following command is recommended for MTP:
 
     ```shell
-    vllm serve Qwen/Qwen3.5-397B-A17B --port 8000 --tensor-parallel-size 8 --max-model-len 262144 --mamba-ssm-cache-dtype float32 --reasoning-parser qwen3 --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":2}'
+    vllm serve Qwen/Qwen3.5-397B-A17B --port 8000 --tensor-parallel-size 8 --max-model-len 262144 --reasoning-parser qwen3 --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":2}'
     ```
 
 - **Text-Only**: The following command skips the vision encoder and multimodal profiling to free up memory for additional KV cache:
     
     ```shell
-    vllm serve Qwen/Qwen3.5-397B-A17B --port 8000 --tensor-parallel-size 8 --max-model-len 262144 --mamba-ssm-cache-dtype float32 --reasoning-parser qwen3 --limit-mm-per-prompt.video 0 --limit-mm-per-prompt.image 0
+    vllm serve Qwen/Qwen3.5-397B-A17B --port 8000 --tensor-parallel-size 8 --max-model-len 262144 --reasoning-parser qwen3 --language-model-only
     ```
 
-> [!Tip]
-> Because vLLM defaults to `--mamba-ssm-cache-dtype auto`, which resolves to `bfloat16` for Qwen3.5, the model's generation quality may suffer unless you explicitly set it to higher precision, i.e., `--mamba-ssm-cache-dtype float32`.
-
 #### Hugging Face Transformers
 
 Hugging Face Transformers contains a _lightweight_ server which can be used for quick testing and moderate load deployment.

generation_config.json

@@ -0,0 +1,13 @@
+{
+    "bos_token_id": 248044,
+    "do_sample": true,
+    "eos_token_id": [
+        248046,
+        248044
+    ],
+    "pad_token_id": 248044,
+    "temperature": 0.6,
+    "top_k": 20,
+    "top_p": 0.95,
+    "transformers_version": "4.57.0.dev0"
+}