Spaces:
Running
Reflex — a Rust/CUDA inference engine built for cold-start, with day-one Qwen3/Qwen3.5 support
Sharing an open-source engine we've been building that takes a different angle than llama.cpp/vLLM: instead of chasing steady-state throughput, it's optimized for cold-start latency and energy — process launch to first token. That's the metric that matters for serverless/scale-to-zero GPU deployments, where you're paying for a cold process spin-up on every invocation, not a long-lived server.
The core trick: every CUDA kernel is compiled ahead-of-time by nvcc at build time, not JIT'd via NVRTC at runtime. That avoids a real multi-second JIT tax that a naive runtime-compilation design pays on every single cold start.
Qwen coverage (all real-hardware-verified, byte-exact against llama.cpp):
- Dense Qwen3
- Qwen3-MoE (top-k expert routing)
- Qwen3.5's hybrid Gated DeltaNet mixer architecture
- DeepSeek-V2/V3 MLA, for comparison
Also supports LoRA adapters (--lora, including per-expert MoE adapters), a C-FFI + Python binding surface for embedding, and non-network IPC modes (stdio/UDS) plus an optional OpenAI-compatible HTTP sidecar for local use.
Numbers: on real GPU hardware (T4/A6000/A100), currently measuring ~1.3–1.4x faster cold-start than llama.cpp on Qwen3 dense/MoE. Our "System1" cold-start phase benchmark (reflex system1) runs against unsloth/Qwen3-0.6B-GGUF's Qwen3-0.6B-Q4_K_M.gguf specifically, so it's easy to pull the exact same quant and reproduce the numbers yourself. Every benchmark run in the repo's history is against real hardware, not simulated — including a couple of regressions we found and fixed along the way, disclosed rather than buried.
What it's not: no batching, no multi-tenant serving, no internal HTTP server in the core engine — those are explicitly out of scope; this is a single-process engine meant to sit behind an orchestrator, not replace one.
Repo: https://github.com/lateos-ai/reflex
Would love feedback from anyone running Qwen3/Qwen3.5 in serverless/cold-start-sensitive setups — especially bug reports against real checkpoints, since that's the whole verification methodology this project leans on.