Text Generation
Transformers
Safetensors
English
Arabic
quasar_long
silx-ai
quasar-preview
quasar
foundation-model
Mixture of Experts
18b
2b-active
long-context
bittensor
sn24
decentralized-training
distillation
hybrid-transformer
loop-transformer
safe-nope
drope
conversational
custom_code
Instructions to use silx-ai/Quasar-Preview with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use silx-ai/Quasar-Preview with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="silx-ai/Quasar-Preview", trust_remote_code=True) messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# Load model directly from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("silx-ai/Quasar-Preview", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use silx-ai/Quasar-Preview with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "silx-ai/Quasar-Preview" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "silx-ai/Quasar-Preview", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/silx-ai/Quasar-Preview
- SGLang
How to use silx-ai/Quasar-Preview with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "silx-ai/Quasar-Preview" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "silx-ai/Quasar-Preview", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "silx-ai/Quasar-Preview" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "silx-ai/Quasar-Preview", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use silx-ai/Quasar-Preview with Docker Model Runner:
docker model run hf.co/silx-ai/Quasar-Preview
Fix FLA fused recurrent generation cache state
Browse files
fla/ops/gsa/fused_recurrent.py
CHANGED
|
@@ -101,12 +101,13 @@ def fused_recurrent_gsa_inference(
|
|
| 101 |
else:
|
| 102 |
hk0, hv0 = q.new_zeros(B, H, K, M, dtype=torch.float), q.new_zeros(B, H, M, V, dtype=torch.float)
|
| 103 |
|
| 104 |
-
|
| 105 |
-
|
| 106 |
-
|
| 107 |
-
|
| 108 |
-
|
| 109 |
-
|
|
|
|
| 110 |
|
| 111 |
o = v.new_empty(B, T, HQ, V)
|
| 112 |
grid = (B * HQ,)
|
|
|
|
| 101 |
else:
|
| 102 |
hk0, hv0 = q.new_zeros(B, H, K, M, dtype=torch.float), q.new_zeros(B, H, M, V, dtype=torch.float)
|
| 103 |
|
| 104 |
+
# The Triton inference kernel stores final recurrent state for the first
|
| 105 |
+
# query head in each KV group. It therefore needs valid hkt/hvt pointers
|
| 106 |
+
# even when the caller does not request returning the final state.
|
| 107 |
+
if NG == 1:
|
| 108 |
+
hkt, hvt = hk0, hv0
|
| 109 |
+
else:
|
| 110 |
+
hkt, hvt = q.new_empty(B, H, K, M, dtype=torch.float), q.new_empty(B, H, M, V, dtype=torch.float)
|
| 111 |
|
| 112 |
o = v.new_empty(B, T, HQ, V)
|
| 113 |
grid = (B * HQ,)
|