garnermccloud/Qwen3.8-Flash-Next-NVFP4-SSD-Stream · Running on RTX 5090 + 96 GB RAM

Aug 31

Tell me, is it possible to run this model with an RTX 5090 by compensating for the insufficient VRAM with system RAM? And if so, could you tell me the launch command?

garnermccloud

Owner Aug 31

Just added this, it should work for your setup: https://huggingface.co/garnermccloud/Qwen3.8-Flash-Next-NVFP4-SSD-Stream#24-32-gb-nvidia-gpus-experimental

Skillerok

Aug 31

Thank you for your work! I'm starting the download now!

jetstream131

Sep 1

Just added this, it should work for your setup: https://huggingface.co/garnermccloud/Qwen3.8-Flash-Next-NVFP4-SSD-Stream#24-32-gb-nvidia-gpus-experimental

Thanks for putting this together!

Thank you for your work! I'm starting the download now!

Any performance results? Also have a 5090 am curious to see if this improves things

Skillerok

Sep 2

@jetstream131
I’ll only be able to start testing today. The artifact was downloading all day yesterday.

Skillerok

Sep 3

Hi everyone. I couldn't get the build running with this artifact. The crash is caused by insufficient RAM. I tried using WSL first, then installed Ubuntu separately, but the artifact still doesn't load completely in either case, and the system freezes.
My setup : R9 9950x3d, 96GB RAM, RTX5090 32GB, NVME PCIe4.

log segment (stage where the failure occurs):
'w13_blockscale_swizzled', 'w2_blockscale_swizzled'] memory_allocated=16499251200[2026-09-03 00:22:41] [offloader] offload module_index=41 submodule=<class 'sglang.srt.layers.moe.fused_moe_triton.layer.FusedMoE'> params=['w13_weight', 'w2_weight', 'w13_blockscale_swizzled', 'w2_blockscale_swizzled'] memory_allocated=16811344384[2026-09-03 00:22:43] [offloader] offload module_index=42 submodule=<class 'sglang.srt.layers.moe.fused_moe_triton.layer.FusedMoE'> params=['w13_weight', 'w2_weight', 'w13_blockscale_swizzled', 'w2_blockscale_swizzled'] memory_allocated=17123437568[2026-09-03 00:22:45] [offloader] offload module_index=43 submodule=<class 'sglang.srt.layers.moe.fused_moe_triton.layer.FusedMoE'> params=['w13_weight', 'w2_weight', 'w13_blockscale_swizzled', 'w2_blockscale_swizzled'] memory_allocated=17422529536[2026-09-03 00:22:46] Scheduler hit an exception: Traceback (most recent call last): File "/home/skill/.local/share/sglang-ssd-stream/venv/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 5052, in run_scheduler_process scheduler = Scheduler( ^^^^^^^^^^ File "/home/skill/.local/share/sglang-ssd-stream/venv/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 520, in init self.init_model_worker() File "/home/skill/.local/share/sglang-ssd-stream/venv/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 998, in init_model_worker self.init_tp_model_worker() File "/home/skill/.local/share/sglang-ssd-stream/venv/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 924, in init_tp_model_worker self.tp_worker = TpModelWorker(**worker_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/skill/.local/share/sglang-ssd-stream/venv/lib/python3.12/site-packages/sglang/srt/managers/tp_worker.py", line 337, in init self._init_model_runner() File "/home/skill/.local/share/sglang-ssd-stream/venv/lib/python3.12/site-packages/sglang/srt/managers/tp_worker.py", line 466, in _init_model_runner self._model_runner = ModelRunner( ^^^^^^^^^^^^ File "/home/skill/.local/share/sglang-ssd-stream/venv/lib/python3.12/site-packages/sglang/srt/model_executor/model_runner.py", line 436, in init self.initialize() File "/home/skill/.local/share/sglang-ssd-stream/venv/lib/python3.12/site-packages/sglang/srt/model_executor/model_runner.py", line 630, in initialize self.load_model() File "/home/skill/.local/share/sglang-ssd-stream/venv/lib/python3.12/site-packages/sglang/srt/model_executor/model_runner.py", line 1101, in load_model loaded = load_model_with_memory_saver( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/skill/.local/share/sglang-ssd-stream/venv/lib/python3.12/site-packages/sglang/srt/model_executor/model_runner_components/load_model_utils.py", line 334, in load_model_with_memory_saver model = loader.load_model( ^^^^^^^^^^^^^^^^^^ File "/home/skill/.local/share/sglang-ssd-stream/venv/lib/python3.12/site-packages/sglang/srt/model_loader/loader.py", line 3949, in load_model return super().load_model( ^^^^^^^^^^^^^^^^^^^ File "/home/skill/.local/share/sglang-ssd-stream/venv/lib/python3.12/site-packages/sglang/srt/model_loader/loader.py", line 985, in load_model model = _initialize_model( ^^^^^^^^^^^^^^^^^^ File "/home/skill/.local/share/sglang-ssd-stream/venv/lib/python3.12/site-packages/sglang/srt/model_loader/loader.py", line 345, in _initialize_model return model_class(**kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/skill/.local/share/sglang-ssd-stream/venv/lib/python3.12/site-packages/sglang/srt/models/qwen4_exp.py", line 1727, in init super().init(config, quant_config, prefix, language_model_cls) File "/home/skill/.local/share/sglang-ssd-stream/venv/lib/python3.12/site-packages/sglang/srt/models/qwen3_vl.py", line 1272, in init self.model = language_model_cls( ^^^^^^^^^^^^^^^^^^^ File "/home/skill/.local/share/sglang-ssd-stream/venv/lib/python3.12/site-packages/sglang/srt/models/qwen4_exp.py", line 1684, in init super().init(config=config, quant_config=quant_config, prefix=prefix) File "/home/skill/.local/share/sglang-ssd-stream/venv/lib/python3.12/site-packages/sglang/srt/models/qwen4_exp.py", line 1601, in init super().init(config, quant_config, prefix, is_nextn) File "/home/skill/.local/share/sglang-ssd-stream/venv/lib/python3.12/site-packages/sglang/srt/models/qwen3_5.py", line 1435, in init self.layers, self._start_layer, self._end_layer = make_layers( ^^^^^^^^^^^^ File "/home/skill/.local/share/sglang-ssd-stream/venv/lib/python3.12/site-packages/sglang/srt/plugins/hook_registry.py", line 390, in wrapper return hook(original_fn, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File

Ubuntu has slightly more free memory; consequently, the freeze occurs at offload module_index=44 submodule.

Skillerok

Sep 5

@garnermccloud
Is there any way to do something about this?

Andrew1117

Sep 7

memory_allocated is just 17422529536, and in your log part, the actual module with error is cut

Skillerok

Sep 10

There are no error messages when running it on Linux; it simply hangs, and the security system terminates the process. I’m running out of memory—I have 96 GB of RAM—and I would appreciate it if the engine were improved in this regard. Is there anything that can be done about this?

garnermccloud

Owner Sep 10

There are no error messages when running it on Linux; it simply hangs, and the security system terminates the process. I’m running out of memory—I have 96 GB of RAM—and I would appreciate it if the engine were improved in this regard. Is there anything that can be done about this?

@Skillerok Thanks for the screenshot. I checked the loader and found two things worth addressing: pinned-memory allocation overhead and a CPU/GPU split that currently offloads every expert layer.

The expert weights and scales total about 63.3 GiB, but PyTorch’s default pinned allocator rounds these individual allocations up to powers of two. That can raise their host-memory allocation to approximately 81 GiB, before other memory usage. This is a plausible explanation for the failure, not yet a confirmed diagnosis.

First, try removing that allocation overhead:

PYTORCH_ALLOC_CONF='pinned_max_round_threshold_mb:16,pinned_max_cached_size_mb:16' \
  ~/.local/bin/sglang-ssd-stream serve --context 8192

Check env | grep ALLOC_CONF first: existing PYTORCH_CUDA_ALLOC_CONF or PYTORCH_HIP_ALLOC_CONF settings can override this. The allocator options are supported by the installer’s pinned PyTorch 2.13 version.

Once that starts and completes a short request, we can use more of your 5090. This next configuration keeps four of the 48 expert layers permanently on the GPU and offloads the other 44:

PYTORCH_ALLOC_CONF='pinned_max_round_threshold_mb:16,pinned_max_cached_size_mb:16' \
  ~/.local/bin/sglang-ssd-stream serve --context 8192 -- \
  --offload-group-size 12 \
  --offload-num-in-group 11

That shifts approximately 5.3 GiB of expert tensors from RAM to VRAM and avoids transferring those four layers for every token. It is a conservative tuning candidate, not a measured safe maximum for your card. If it produces a CUDA OOM, revert to the first command.

Please run only one server at a time and report:

Whether the first command reaches readiness and completes a short response.
RAM and VRAM usage after startup and during that response.
The same results for the second command, if the first succeeds.

That will let us distinguish the host-allocation problem from GPU headroom and tune the split using your actual measurements. No model redownload is needed.

Skillerok

Sep 10

Thanks for the recommendations; I’m away for a couple of days, but I’ll give them a try as soon as I get back and let you know how it goes.

Skillerok

28 days ago

Hi everyone! There’s no longer a lack of RAM, but now errors are occurring with VRAM allocation.
@garnermccloud
log_part1: ai@ai-station:~/.local/share/sglang-ssd-stream/venv/lib/python3.12/site-packages/sglang/srt$ PYTORCH_ALLOC_CONF='pinned_max_round_threshold_mb:16,pinned_max_cached_size_mb:16'
~/.local/bin/sglang-ssd-stream serve --context 8192
Fetching 227 files: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 227/227 [00:00<00:00, 7164.46it/s]
Download complete: : | 0.00B Free 48 GB of RAM. Keep the speed. | 0.00B / 0.00B
GPU: NVIDIA GeForce RTX 5090
Compute capability: 12.0
CPU expert offload: enabled (experimental)
Model: garnermccloud/Qwen3.8-Flash-Next-NVFP4-SSD-Stream@7ac24134c763
Data: /home/ai/.cache/sglang-ssd-stream
API: http://127.0.0.1:30000/v1
/home/ai/.local/share/sglang-ssd-stream/venv/lib/python3.12/site-packages/sglang/launch_server.py:56: UserWarning: 'python -m sglang.launch_server' is still supported, but 'sglang serve' is the recommended entrypoint.
Example: sglang serve --model-path [options]
warnings.warn(
[transformers] Unrecognized keys in rope_parameters for 'rope_type'='default': {'mrope_section', 'mrope_interleaved'}
[transformers] Unrecognized keys in rope_parameters for 'rope_type'='default': {'mrope_section', 'mrope_interleaved'}
[2026-09-14 22:56:24] Setting page size to 64 for compressed QSA (full//ratio compressed addressing).
[2026-09-14 22:56:24] Attention backend not specified. Use flashinfer backend by default.
[2026-09-14 22:56:24] Use flashinfer_cutlass as MoE runner backend on SM120 for modelopt_fp4 (trtllm-gen MoE kernels are SM100-only)
[2026-09-14 22:56:25] server_args=ServerArgs(model_path='/home/ai/.cache/sglang-ssd-stream/huggingface/hub/models--garnermccloud--Qwen3.8-Flash-Next-NVFP4-SSD-Stream/snapshots/7ac24134c76320339cb503c3b4071a312050b1fc', tokenizer_path='/home/ai/.cache/sglang-ssd-stream/huggingface/hub/models--garnermccloud--Qwen3.8-Flash-Next-NVFP4-SSD-Stream/snapshots/7ac24134c76320339cb503c3b4071a312050b1fc', tokenizer_mode='auto', tokenizer_backend='huggingface', tokenizer_worker_num=1, detokenizer_worker_num=1, skip_tokenizer_init=False, load_format='auto', model_loader_extra_config='{}', trust_remote_code=True, context_length=8192, is_embedding=False, enable_multimodal=True, revision=None, model_impl='auto', model_config_parser='auto', json_model_override_args='{}', dtype='auto', quantization='modelopt_fp4', quantization_param_path=None, kv_cache_dtype='fp8_e4m3', enable_fp32_lm_head=False, modelopt_quant=None, modelopt_checkpoint_restore_path=None, modelopt_checkpoint_save_path=None, modelopt_export_path=None, quantize_and_serve=False, rl_quant_profile=None, enable_tf32_matmul=False, mem_fraction_static=0.8, max_running_requests=1, max_queued_requests=None, max_total_tokens=8192, chunked_prefill_size=1024, enable_dynamic_chunking=False, max_prefill_tokens=16384, prefill_max_requests=None, schedule_policy='fcfs', enable_priority_scheduling=False, disable_priority_preemption=False, default_priority_value=None, abort_on_priority_when_disabled=False, schedule_low_priority_values_first=False, priority_scheduling_preemption_threshold=10, retraction_policy='length', schedule_conservativeness=1.0, page_size=64, c128_page_size=16, swa_full_tokens_ratio=0.8, disable_hybrid_swa_memory=False, radix_eviction_policy='lru', prefill_only_disable_kv_cache=False, disable_radix_cache=False, enable_page_major_kv_layout=False, enable_unified_memory=False, disable_chunked_prefix_cache=False, disable_overlap_schedule=False, num_continuous_decode_steps=1, scheduler_recv_interval=1, enable_mixed_chunk=False, nccl_port=None, dist_timeout=None, dist_init_addr=None, nnodes=1, node_rank=0, tp_size=1, dcp_size=1, pp_size=1, pp_max_micro_batch_size=None, pp_async_batch_depth=0, dp_size=1, load_balance_method='round_robin', attn_cp_size=1, moe_dp_size=1, dwdp_size=1, dcp_comm_backend='ag_rs', dcp_replicate_q_proj=None, enable_prefill_cp=False, cp_strategy=None, enable_dsa_cache_layer_split=False, enable_dsa_prefill_context_parallel=False, dsa_prefill_cp_mode='round-robin-split', enable_prefill_context_parallel=False, prefill_cp_mode='in-seq-split', enable_cp_decode_attn_tp=False, enable_dp_attention=False, enable_dp_attention_local_control_broadcast=False, enable_dp_lm_head=False, enable_attn_tp_input_scattered=False, disable_attn_tp_gather=False, enable_p2p_check=False, device='cuda', base_gpu_id=0, gpu_id_step=1, random_seed=193886304, mlx_enable_sampling=False, watchdog_timeout=300, soft_watchdog_timeout=None, sleep_on_idle=False, use_ray=False, custom_sigquit_handler=None, numa_node=None, gc_threshold=None, host='127.0.0.1', port=30000, fastapi_root_path='', smg_grpc_mode=False, grpc_mode=False, grpc_port=None, sidecar=None, sidecar_args=None, skip_server_warmup=False, warmups=None, enable_http2=False, http2_max_concurrent_streams=200, ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_keyfile_password=None, enable_ssl_refresh=False, api_key=None, admin_api_key=None, served_model_name='Qwen3.8-Flash-Next-NVFP4-SSD-Stream', weight_version='default', chat_template=None, hf_chat_template_name=None, completion_template=None, file_storage_path='sglang_storage', enable_cache_report=False, reasoning_parser='auto', default_chat_template_kwargs=None, strip_thinking_cache=False, enable_strict_thinking=False, tool_call_parser='qwen3_coder', tool_server=None, sampling_defaults='model', asr_max_buffer_seconds=60, asr_max_concurrent_sessions=32, preferred_sampling_params=None, allow_auto_truncate=True, stream_interval=1, batch_notify_size=16, stream_response_default_include_usage=False, incremental_streaming_output=False, enable_streaming_session=False, enable_session_radix_cache=False, log_level='info', log_level_http=None, log_requests=False, log_requests_level=2, log_requests_format='text', log_requests_target=None, uvicorn_access_log_exclude_prefixes=[], crash_dump_folder=None, show_time_cost=False, enable_metrics=False, smg_http_sidecar_port=None, enable_mfu_metrics=False, enable_metrics_for_all_schedulers=False, load_snapshot_publish_interval=15, tokenizer_metrics_custom_labels_header='x-custom-labels', tokenizer_metrics_allowed_custom_labels=None, extra_metric_labels=None, bucket_time_to_first_token=None, bucket_inter_token_latency=None, bucket_e2e_request_latency=None, prompt_tokens_buckets=None, generation_tokens_buckets=None, gc_warning_threshold_secs=0.0, decode_log_interval=40, enable_request_time_stats_logging=False, kv_events_config=None, enable_forward_pass_metrics=False, forward_pass_metrics_worker_id='', forward_pass_metrics_ipc_name=None, enable_trace=False, trace_modules='request', otlp_traces_endpoint='localhost:4317', export_metrics_to_file=False, export_metrics_to_file_dir=None, stat_loggers=None, constrained_json_whitespace_pattern=None, constrained_json_disable_any_whitespace=False, attention_backend='flashinfer', decode_attention_backend=None, prefill_attention_backend=None, sampling_backend='flashinfer', grammar_backend='xgrammar', radix_cache_backend=None, mm_attention_backend=None, fp8_gemm_runner_backend='auto', fp4_gemm_runner_backend='auto', bf16_gemm_backend='auto', dsa_prefill_backend=None, dsa_decode_backend=None, dsa_paged_mqa_logits_backend='auto', dsa_topk_backend='sgl-kernel', disable_flashinfer_autotune=False, flashinfer_autotune_skip_ops=None, mamba_backend='triton', cuda_graph_config=CudaGraphConfig(decode=PhaseConfig(backend='disabled', max_bs=24, bs=[1, 2, 4, 8, 12, 16, 24], tc_compiler='eager', full_prefill_max_req=None, full_prefill_prefix_chunk_tokens=None), prefill=PhaseConfig(backend='disabled', max_bs=1024, bs=[4, 8, 12, 16, 20, 24, 28, 32, 48, 64, 80, 96, 112, 128, 144, 160, 176, 192, 208, 224, 240, 256, 288, 320, 352, 384, 416, 448, 480, 512, 576, 640, 704, 768, 832, 896, 960, 1024], tc_compiler='eager', full_prefill_max_req=None, full_prefill_prefix_chunk_tokens=None)), cuda_graph_backend_decode='disabled', cuda_graph_backend_prefill='disabled', cuda_graph_max_bs_decode=None, cuda_graph_max_bs_prefill=None, cuda_graph_bs_decode=None, cuda_graph_bs_prefill=None, cuda_graph_tc_compiler=None, disable_prefill_cuda_graph=False, disable_decode_cuda_graph=False, disable_cuda_graph=False, disable_cuda_graph_padding=False, enable_profile_cuda_graph=False, enable_cudagraph_gc=False, debug_cuda_graph=False, enable_layerwise_nvtx_marker=False, enable_nccl_nvls=False, enable_symm_mem=False, triton_attention_reduce_in_fp32=False, triton_attention_num_kv_splits=8, triton_attention_split_tile_size=None, flashinfer_mla_disable_ragged=False, enable_fused_qk_norm_rope=False, enable_precise_embedding_interpolation=False, enable_fused_moe_sum_all_reduce=False, enable_deepseek_v4_fp4_indexer=False, disable_custom_all_reduce=False, enable_mscclpp=False, enable_torch_symm_mem=False, enable_scattered_sconv=False, pre_warm_nccl=False, enable_quant_communications=False, enable_flashinfer_allreduce_fusion=False, enforce_disable_flashinfer_allreduce_fusion=False, flashinfer_allreduce_fusion_backend=None, enable_aiter_allreduce_fusion=False, enable_torch_compile=False, enable_torch_compile_debug_mode=False, torch_compile_max_bs=32, speculative_algorithm=None, speculative_draft_model_path=None, speculative_draft_model_revision=None, speculative_draft_load_format=None, speculative_num_steps=None, speculative_eagle_topk=None, speculative_num_draft_tokens=None, speculative_dflash_block_size=None, speculative_dspark_block_size=None, speculative_dspark_sps_table_path=None, speculative_dspark_confidence_sts_path=None, speculative_dspark_align_verify_tokens_to_graph_tier=False, speculative_accept_threshold_single=1.0, speculative_accept_threshold_acc=1.0, speculative_use_rejection_sampling=False, speculative_token_map=None, speculative_attention_mode='prefill', speculative_draft_attention_backend=None, speculative_draft_kv_cache_dtype=None, speculative_draft_window_size=None, speculative_moe_runner_backend='flashinfer_cutlass', speculative_moe_a2a_backend=None, speculative_draft_model_quantization='modelopt_fp4', _speculative_draft_quantization_explicitly_set=False, speculative_skip_dp_mlp_sync=False, enable_multi_layer_eagle=False, speculative_adaptive=False, speculative_adaptive_config=None, decoupled_spec_bind_endpoint=None, decoupled_spec_connect_endpoints=None, decoupled_spec_rank=None, decoupled_spec_role='null', spec_trace_dir=None, speculative_ngram_min_bfs_breadth=1, speculative_ngram_max_bfs_breadth=10, speculative_ngram_match_type='BFS', speculative_ngram_max_trie_depth=18, speculative_ngram_capacity=10000000, speculative_ngram_external_corpus_path=None, speculative_ngram_external_sam_budget=0, speculative_ngram_external_corpus_max_tokens=10000000, ep_size=1, moe_a2a_backend='none', moe_runner_backend='flashinfer_cutlass', flashinfer_mxfp4_moe_precision='default', deepep_mode='auto', fuseep_mode=2, deepep_dispatcher_output_dtype='auto', ep_num_redundant_experts=0, ep_dispatch_algorithm=None, init_expert_location='trivial', enable_eplb=False, eplb_algorithm='auto', eplb_rebalance_num_iterations=1000, eplb_rebalance_layers_per_chunk=None, eplb_min_rebalancing_utilization_threshold=1.0, expert_distribution_recorder_mode=None, expert_distribution_recorder_buffer_size=1000, expert_balancedness_report_mode='off', deepep_config=None, moe_dense_tp_size=None, elastic_ep_backend=None, enable_elastic_expert_backup=False, mooncake_ib_device=None, enable_waterfill=False, ep_join_mode=None, ep_join_rank_offset=0, elastic_ep_initial_size=None, max_ep_size=None, elastic_ep_scale_timeout=600, elastic_ep_rejoin=False, disable_flashinfer_cutlass_moe_fp4_allgather=False, disable_shared_experts_fusion=False, enforce_shared_experts_fusion=False, max_mamba_cache_size=4, mamba_ssm_dtype='float32', mamba_max_states_per_path=-1, enable_mamba_cache_stochastic_rounding=False, mamba_cache_philox_rounds=0, mamba_full_memory_ratio=0.9, mamba_radix_cache_strategy='extra_buffer_lazy', uses_mamba_radix_cache=True, mamba_track_interval=64, enable_int8_mamba_checkpoint=False, int8_mamba_ckpt_size=None, linear_attn_backend='triton', linear_attn_decode_backend=None, linear_attn_prefill_backend=None, ple_offload_embedding=False, linear_attn_verify_backend=None, enable_linear_replayssm=False, linear_replayssm_cache_len=16, enable_linear_replayssm_spec=False, enable_hierarchical_cache=False, hicache_ratio=2.0, hicache_size=0, hicache_write_policy='write_through', hicache_io_backend='kernel', hicache_mem_layout='page_first', hicache_storage_backend=None, hicache_storage_prefetch_policy='timeout', hicache_storage_backend_extra_config=None, enable_hisparse=False, hisparse_config=None, enable_broadcast_mm_inputs_process=False, enable_prefix_mm_cache=False, mm_enable_dp_encoder=False, mm_process_config={}, mm_processor_worker_num=0, mm_io_worker_num=0, allowed_media_domains=[], media_url_max_file_size_mb=64, mm_preprocess_cache_size_mb=None, trust_mm_content_hashes=False, limit_mm_data_per_request=None, enable_mm_global_cache=False, image_processor_backend='auto', mm_global_cache_backend='mooncake', disable_fast_image_processor=False, mm_feature_transport='cpu', keep_mm_feature_on_device=False, enable_lora=None, enable_lora_overlap_loading=None, max_lora_rank=None, lora_target_modules=None, lora_paths=None, max_loaded_loras=None, max_loras_per_batch=8, lora_eviction_policy='lru', lora_backend='csgmv', max_lora_chunk_size=16, experts_shared_outer_loras=None, lora_use_virtual_experts=False, lora_strict_loading=False, lora_drain_wait_threshold=0.0, enable_two_batch_overlap=False, enable_single_batch_overlap=False, tbo_token_distribution_threshold=0.48, cpu_offload_gb=0, offload_group_size=1, offload_num_in_group=1, offload_prefetch_step=1, offload_mode='cpu', enable_lmcache=False, lmcache_config_file=None, enable_flexkv=False, flexkv_config_file=None, kt_weight_path=None, kt_method='AMXINT4', kt_cpuinfer=None, kt_threadpool_count=2, kt_num_gpu_experts=None, kt_max_deferred_experts_per_token=None, dllm_algorithm=None, dllm_algorithm_config=None, dllm_fdfo=True, disaggregation_mode='null', disaggregation_transfer_backend='mooncake', disaggregation_bootstrap_port=8998, disaggregation_ib_device=None, disaggregation_decode_enable_radix_cache=False, disaggregation_decode_enable_offload_kvcache=False, disaggregation_decode_retraction_backup=None, num_reserved_decode_tokens=512, disaggregation_decode_extra_slots=None, disaggregation_decode_polling_interval=1, optimistic_prefill_attempts=0, encoder_only=False, language_only=False, language_model_only=False, encoder_transfer_backend='zmq_to_scheduler', encoder_urls=[], encoder_bootstrap_port=8997, encoder_register_urls=[], enable_adaptive_dispatch_to_encoder=False, enable_pdmux=False, pdmux_config_path=None, sm_group_num=8, startup_weight_load_mode='serial', custom_weight_loader=[], weight_loader_disable_mmap=False, weight_loader_prefetch_checkpoints=False, weight_loader_prefetch_num_threads=4, weight_loader_drop_cache_after_load=False, remote_instance_weight_loader_seed_instance_ip=None, remote_instance_weight_loader_seed_instance_service_port=None, remote_instance_weight_loader_send_weights_group_ports=None, remote_instance_weight_loader_backend='nccl', remote_instance_weight_loader_start_seed_via_transfer_engine=False, engine_info_bootstrap_port=6789, modelexpress_config=None, download_dir=None, model_checksum=None, delete_ckpt_after_loading=False, decrypted_config_file=None, decrypted_draft_config_file=None, checkpoint_engine_wait_weights_before_ready=False, enable_prefill_delayer=False, prefill_delayer_max_delay_passes=30, prefill_delayer_token_usage_low_watermark=None, prefill_delayer_forward_passes_buckets=None, prefill_delayer_wait_seconds_buckets=None, prefill_delayer_queue_min_ratio=None, prefill_delayer_max_delay_ms=None, min_free_slots_delay=None, enable_deterministic_inference=False, rl_on_policy_target=None, kv_canary='none', kv_canary_real_data='none', kv_canary_sweep_interval=0, enable_dynamic_batch_tokenizer=False, dynamic_batch_tokenizer_batch_size=32, dynamic_batch_tokenizer_batch_timeout=0.002, enable_tokenizer_batch_encode=False, disable_tokenizer_batch_decode=False, debug_tensor_dump_output_folder=None, debug_tensor_dump_layers=None, debug_tensor_dump_input_file=None, enable_memory_saver=False, enable_weights_cpu_backup=False, enable_draft_weights_cpu_backup=False, enable_custom_logit_processor=False, enable_return_hidden_states=False, return_hidden_states_mode=None, enable_return_routed_experts=False, enable_return_indexer_topk=False, disable_outlines_disk_cache=False, enable_mis=False, weight_cache_mode='off', weight_cache_socket=None, weight_cache_timeout=1800, forward_hooks=None, msprobe_dump_config=None)
[transformers] Unrecognized keys in rope_parameters for 'rope_type'='default': {'mrope_section', 'mrope_interleaved'}
[2026-09-14 22:56:25] Auto-detected --reasoning-parser as 'qwen3' from chat template
[2026-09-14 22:56:26] Ignore import error when loading sglang.srt.multimodal.processors.mimo_audio: Could not load libtorchcodec. Likely causes:
1. FFmpeg is not properly installed in your environment. We support
versions 4, 5, 6, 7, and 8, and we attempt to load libtorchcodec
for each of those versions. Errors for versions not installed on
your system are expected; only the error for your installed FFmpeg
version is relevant. On Windows, ensure you've installed the
"full-shared" version which ships DLLs.
2. The PyTorch version (2.13.0+cu130) is not compatible with
this version of TorchCodec. Refer to the version compatibility
table:
https://github.com/pytorch/torchcodec?tab=readme-ov-file#installing-torchcodec.
3. Another runtime dependency; see exceptions below.

    The following exceptions were raised as we tried to load libtorchcodec:

[start of libtorchcodec loading traceback]
FFmpeg version 8:
Traceback (most recent call last):
File "/home/ai/.local/share/sglang-ssd-stream/venv/lib/python3.12/site-packages/torch/_ops.py", line 1516, in load_library
ctypes.CDLL(path)
File "/home/ai/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/lib/python3.12/ctypes/init.py", line 379, in init
self._handle = _dlopen(self._name, mode)
^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: libavdevice.so.62: cannot open shared object file: No such file or directory

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/ai/.local/share/sglang-ssd-stream/venv/lib/python3.12/site-packages/torchcodec/_internally_replaced_utils.py", line 93, in load_torchcodec_shared_libraries
torch.ops.load_library(core_library_path)
File "/home/ai/.local/share/sglang-ssd-stream/venv/lib/python3.12/site-packages/torch/_ops.py", line 1518, in load_library
raise OSError(f"Could not load this library: {path}") from e
OSError: Could not load this library: /home/ai/.local/share/sglang-ssd-stream/venv/lib/python3.12/site-packages/torchcodec/libtorchcodec_core8.so

FFmpeg version 7:
Traceback (most recent call last):
File "/home/ai/.local/share/sglang-ssd-stream/venv/lib/python3.12/site-packages/torch/_ops.py", line 1516, in load_library
ctypes.CDLL(path)
File "/home/ai/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/lib/python3.12/ctypes/init.py", line 379, in init
self._handle = _dlopen(self._name, mode)
^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: libavutil.so.59: cannot open shared object file: No such file or directory

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/ai/.local/share/sglang-ssd-stream/venv/lib/python3.12/site-packages/torchcodec/_internally_replaced_utils.py", line 93, in load_torchcodec_shared_libraries
torch.ops.load_library(core_library_path)
File "/home/ai/.local/share/sglang-ssd-stream/venv/lib/python3.12/site-packages/torch/_ops.py", line 1518, in load_library
raise OSError(f"Could not load this library: {path}") from e
OSError: Could not load this library: /home/ai/.local/share/sglang-ssd-stream/venv/lib/python3.12/site-packages/torchcodec/libtorchcodec_core7.so

FFmpeg version 6:
Traceback (most recent call last):
File "/home/ai/.local/share/sglang-ssd-stream/venv/lib/python3.12/site-packages/torch/_ops.py", line 1516, in load_library
ctypes.CDLL(path)
File "/home/ai/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/lib/python3.12/ctypes/init.py", line 379, in init
self._handle = _dlopen(self._name, mode)
^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: libavutil.so.58: cannot open shared object file: No such file or directory

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/ai/.local/share/sglang-ssd-stream/venv/lib/python3.12/site-packages/torchcodec/_internally_replaced_utils.py", line 93, in load_torchcodec_shared_libraries
torch.ops.load_library(core_library_path)
File "/home/ai/.local/share/sglang-ssd-stream/venv/lib/python3.12/site-packages/torch/_ops.py", line 1518, in load_library
raise OSError(f"Could not load this library: {path}") from e
OSError: Could not load this library: /home/ai/.local/share/sglang-ssd-stream/venv/lib/python3.12/site-packages/torchcodec/libtorchcodec_core6.so

FFmpeg version 5:
Traceback (most recent call last):
File "/home/ai/.local/share/sglang-ssd-stream/venv/lib/python3.12/site-packages/torch/_ops.py", line 1516, in load_library
ctypes.CDLL(path)
File "/home/ai/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/lib/python3.12/ctypes/init.py", line 379, in init
self._handle = _dlopen(self._name, mode)
^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: libavutil.so.57: cannot open shared object file: No such file or directory

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/ai/.local/share/sglang-ssd-stream/venv/lib/python3.12/site-packages/torchcodec/_internally_replaced_utils.py", line 93, in load_torchcodec_shared_libraries
torch.ops.load_library(core_library_path)
File "/home/ai/.local/share/sglang-ssd-stream/venv/lib/python3.12/site-packages/torch/_ops.py", line 1518, in load_library
raise OSError(f"Could not load this library: {path}") from e
OSError: Could not load this library: /home/ai/.local/share/sglang-ssd-stream/venv/lib/python3.12/site-packages/torchcodec/libtorchcodec_core5.so

FFmpeg version 4:
Traceback (most recent call last):
File "/home/ai/.local/share/sglang-ssd-stream/venv/lib/python3.12/site-packages/torch/_ops.py", line 1516, in load_library
ctypes.CDLL(path)
File "/home/ai/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/lib/python3.12/ctypes/init.py", line 379, in init
self._handle = _dlopen(self._name, mode)
^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: libavutil.so.56: cannot open shared object file: No such file or directory

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/ai/.local/share/sglang-ssd-stream/venv/lib/python3.12/site-packages/torchcodec/_internally_replaced_utils.py", line 93, in load_torchcodec_shared_libraries
torch.ops.load_library(core_library_path)
File "/home/ai/.local/share/sglang-ssd-stream/venv/lib/python3.12/site-packages/torch/_ops.py", line 1518, in load_library
raise OSError(f"Could not load this library: {path}") from e
OSError: Could not load this library: /home/ai/.local/share/sglang-ssd-stream/venv/lib/python3.12/site-packages/torchcodec/libtorchcodec_core4.so
[end of libtorchcodec loading traceback].
[2026-09-14 22:56:26] Ignore import error when loading sglang.srt.multimodal.processors.mimo_v2: Could not load libtorchcodec. Likely causes:
1. FFmpeg is not properly installed in your environment. We support
versions 4, 5, 6, 7, and 8, and we attempt to load libtorchcodec
for each of those versions. Errors for versions not installed on
your system are expected; only the error for your installed FFmpeg
version is relevant. On Windows, ensure you've installed the
"full-shared" version which ships DLLs.
2. The PyTorch version (2.13.0+cu130) is not compatible with
this version of TorchCodec. Refer to the version compatibility
table:
https://github.com/pytorch/torchcodec?tab=readme-ov-file#installing-torchcodec.
3. Another runtime dependency; see exceptions below.

    The following exceptions were raised as we tried to load libtorchcodec: