Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,6 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+Transformer.onnx.data filter=lfs diff=lfs merge=lfs -text
+Vision.onnx.data filter=lfs diff=lfs merge=lfs -text
+tokenizer/tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,88 @@
+---
+license: apache-2.0
+base_model: Qwen/Qwen3-VL-Embedding-2B
+pipeline_tag: feature-extraction
+tags:
+  - onnx
+  - qwen3
+  - qwen3-vl
+  - embedding
+  - multimodal
+  - vision-language
+---
+
+# Qwen3-VL-Embedding-2B — ONNX
+
+ONNX export of [Qwen/Qwen3-VL-Embedding-2B](https://huggingface.co/Qwen/Qwen3-VL-Embedding-2B), split into a separate vision encoder and transformer decoder so each component can be consumed independently by ONNX Runtime or built into a TensorRT engine.
+
+The image resolution and temporal size are **baked into the ONNX graph** (see [Configuration](#configuration)). Only `seq_len` on the transformer is dynamic.
+
+## Repository contents
+
+| File / Dir | Purpose |
+|------------|---------|
+| `Vision.onnx` (+ `.onnx.data`) | Vision encoder. Fixed input resolution and temporal size baked in. |
+| `Transformer.onnx` (+ `.onnx.data`) | Transformer decoder layers, embedding mode (no KV cache), dynamic `seq_len`. |
+| `rotary_params.npz` | mRoPE parameters (`inv_freq`, `mrope_section`), token embedding weights, and image/grid config (`image_height/width`, `height_factor/width_factor`, `patch_size`, `merge_size`, `hidden_size`, `head_dim`, …). Required alongside the ONNX files at inference time. |
+| `tokenizer/` | HF `Qwen3VLProcessor` / tokenizer files for text + image preprocessing. |
+| `export_script/` | Scripts used to produce the ONNX files (see [Reproducing the export](#reproducing-the-export)). |
+| `text_prompt_APOv2.1_BF16/` | Sample text prompts (fire / smoke detection) used for downstream evaluation — not required for loading the model. |
+
+## Configuration
+
+The export was produced with:
+
+| Variable | Value | Notes |
+|----------|-------|-------|
+| `IMG_SIZE` | `(768, 768)` | Fixed input resolution — must be a multiple of `patch_size × merge_size`. |
+| `TEMPORAL_SIZE` | `1` | Frames per clip. |
+
+Changing either of these requires re-running the full export pipeline.
+
+## Usage
+
+Load the ONNX files with ONNX Runtime and apply the mRoPE / token-embedding parameters from `rotary_params.npz` around the transformer. High-level flow:
+
+1. Preprocess inputs with the tokenizer / processor in `tokenizer/`.
+2. Run image pixels through `Vision.onnx` to obtain visual tokens.
+3. Look up text token embeddings using the weights saved in `rotary_params.npz`.
+4. Concatenate visual + text embeddings into the transformer input sequence.
+5. Run the sequence through `Transformer.onnx` with mRoPE parameters from `rotary_params.npz`.
+6. The final pooled hidden state is the multimodal embedding.
+
+### TensorRT
+
+These ONNX files are designed to be converted to TensorRT engines. Notes for the TRT build:
+
+- FP16 is safe globally, except for normalization-sensitive layers — force those to FP32 to avoid overflow in the RMSNorm `x * rsqrt(square(x).sum())` pattern.
+- Engines are **not portable** across GPU architectures or TRT versions and must be built on the target machine.
+
+## Reproducing the export
+
+Scripts in `export_script/` regenerate the ONNX assets from the HF PyTorch checkpoint:
+
+```bash
+# Install dependencies
+pip install -r export_script/requirements.txt
+
+# Full pipeline
+bash export_script/run_all.sh
+
+# Or run the stages individually
+python export_script/a_export_to_onnx.py
+python export_script/b_export_onnx_vision.py
+```
+
+- **`a_export_to_onnx.py`** — Exports `Transformer.onnx` and saves `rotary_params.npz`. Also produces an initial `Vision.onnx` via a manual path (norm fusion, GELU replacement), which gets overwritten in step b.
+- **`b_export_onnx_vision.py`** — Re-exports `Vision.onnx` by wrapping the HF `Qwen3VLVisionModel` directly. Traces the exact PyTorch code path, so numerics match HF.
+- **`qwen3_vl_embedding.py`** — Model wrapper used by the export scripts.
+
+The TensorRT build step is not included in this repo; it depends on your local GPU / TRT installation.
+
+## Known limitations
+
+The export pipeline currently only produces reliable engines for `TEMPORAL_SIZE ≤ 2`. Beyond that, the Torch vs TRT cosine similarity drops below 0.99 and parity can no longer be guaranteed. If you need longer temporal contexts, the export path will need further investigation (likely around the temporal patching / rotary handling).
+
+## License
+
+Apache-2.0, inherited from the base model. See [Qwen/Qwen3-VL-Embedding-2B](https://huggingface.co/Qwen/Qwen3-VL-Embedding-2B) for upstream terms.

Transformer.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e2bad7dc0bf0539edcb93cf2ebe224129309b6095e217356d91df82051fbeaa9
+size 260402

Transformer.onnx.data

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:08c8f43cd7efdea87386b879317f1ae5ff83fd32b8b28438d46c44c3e358a19b
+size 5637496832

Vision.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d0c503e408a564218370c3ff09c157d2f80928c4fa26ab39ce3c51e99386ada2
+size 731088

Vision.onnx.data

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f276a72b814c0c5de573f7ed3a4e5196ea714dfa4cac2d2aa55cc4ec566137f7
+size 1627828224

export_script/a_export_to_onnx.py

@@ -0,0 +1,727 @@
+"""
+Export Qwen3-VL-Embedding model to ONNX files for inference.
+
+Produces 2 ONNX files:
+  1. Vision.onnx      — Vision encoder (fixed image resolution)
+  2. Transformer.onnx — Transformer decoder layers (embedding mode, no KV cache)
+
+Also saves rotary_params.npz, which contains:
+  • inv_freq + mrope_section — for Python-side mRoPE computation
+  • embed_weight             — token-embedding lookup table (replaces Embed.onnx)
+  • image dimensions / config values
+
+Usage:
+    python 1_export_to_onnx.py
+    python 1_export_to_onnx.py --model_path /path/to/model --output_dir /path/to/output
+"""
+
+import math
+import os
+import gc
+import glob
+import argparse
+import torch
+import onnx
+import numpy as np
+from transformers.models.qwen3_vl.modeling_qwen3_vl import (
+    Qwen3VLVisionModel,
+)
+from qwen3_vl_embedding import Qwen3VLForEmbedding, MAX_TOTAL_PIXELS
+
+
+def consolidate_external_data(onnx_path: str):
+    """
+    Re-save an ONNX model so all external weights live in one .onnx.data file
+    instead of hundreds of per-tensor files.
+    """
+    out_dir  = os.path.dirname(onnx_path)
+    base     = os.path.basename(onnx_path)
+    data_rel = base + ".data"                      # sibling file: Transformer.onnx.data
+    model    = onnx.load(onnx_path, load_external_data=True)
+
+    # Delete ONNX per-tensor external files (e.g.
+    # `model.model.language_model.layers.0.self_attn.qk_norm_weight`). They can
+    # be named with dots, so we use an allow-list of known suffixes we want to
+    # keep. Subdirectories (e.g. `tokenizer/`) are skipped via the isfile check.
+    PRESERVE_SUFFIXES = (".onnx", ".onnx.data", ".npz", ".bin")
+    for fname in list(os.listdir(out_dir)):
+        fpath = os.path.join(out_dir, fname)
+        if not os.path.isfile(fpath):
+            continue
+        if any(fname.endswith(sfx) for sfx in PRESERVE_SUFFIXES):
+            continue
+        try:
+            os.remove(fpath)
+        except OSError:
+            pass
+
+    # Remove the target .data file if it already exists (stale)
+    target_data = os.path.join(out_dir, data_rel)
+    if os.path.exists(target_data):
+        os.remove(target_data)
+
+    onnx.save_model(
+        model, onnx_path,
+        save_as_external_data=True,
+        all_tensors_to_one_file=True,
+        location=data_rel,
+        size_threshold=1024,
+        convert_attribute=False,
+    )
+    del model
+    gc.collect()
+
+
+# ══════════════════════════════════════════════════════════════════════════════
+# Configuration — reads from the module's config.py
+# ══════════════════════════════════════════════════════════════════════════════
+IMG_SIZE = (768,768) # (h,w)
+QWEN3VLE_MODEL_HF_PATH = os.getenv(
+    "MODEL_QWEN3VLE_MODEL_HF_PATH", "/home/jordan/Research/Product-AI-mono/assets/model/Qwen3-VL-Embedding-2B"
+)
+QWEN3VLE_TRT_DIR_PATH = os.getenv(
+    "MODEL_QWEN3VLE_TRT_DIR_PATH", "assets/model/Qwen3-VL-Embedding-2B-onnx"
+)
+TEMPORAL_SIZE=1
+
+DEFAULT_MODEL_PATH = QWEN3VLE_MODEL_HF_PATH
+DEFAULT_OUTPUT_DIR = QWEN3VLE_TRT_DIR_PATH
+
+# Vision — fixed image resolution at export time.
+# Must be multiples of (patch_size * merge_size), validated after model loading.
+IMAGE_HEIGHT     = IMG_SIZE[0]
+IMAGE_WIDTH      = IMG_SIZE[1]
+
+MAX_SEQ_LEN      = MAX_TOTAL_PIXELS                           # 8192
+OPSET            = 17
+
+
+# ══════════════════════════════════════════════════════════════════════════════
+# Token Embedding Module
+# ══════════════════════════════════════════════════════════════════════════════
+class LLM_EMBED(torch.nn.Module):
+    """Extract the token-embedding layer and run it in float32."""
+
+    def __init__(self, model):
+        super().__init__()
+        self.embed_tokens = model.model.language_model.embed_tokens.float()
+
+    def forward(self, input_ids):
+        return self.embed_tokens(input_ids)
+
+
+# ══════════════════════════════════════════════════════════════════════════════
+# Vision Encoder Module
+# ══════════════════════════════════════════════════════════════════════════════
+class LLM_VISION_EMBED(torch.nn.Module):
+    """
+    ONNX-exportable vision encoder with baked-in positional / rotary
+    embeddings for a fixed image resolution.
+
+    Takes the HF processor's pixel_values directly (already CLIP-normalised
+    and patched) so the patch ordering, normalisation, and Conv3d behaviour
+    are identical to the PyTorch model.
+
+    Input : float32 pixel_values [total_patches, flatten_dim]
+    Output: (deepstack_feature_0, …, vision_hidden_states)
+    """
+
+    def __init__(self, model, height_factor, width_factor, temporal_patches=1):
+        super().__init__()
+        visual        = model.model.visual
+        vision_config = model.config.vision_config
+
+        self.num_heads     = vision_config.num_heads
+        self.head_dim      = vision_config.hidden_size // self.num_heads
+        self.head_dim_half = self.head_dim // 2
+        self.embed_dim     = visual.patch_embed.embed_dim
+        self.merge_size    = visual.spatial_merge_size
+        self.t_patches     = temporal_patches
+
+        ms = self.merge_size
+        self.grid_h = height_factor * ms
+        self.grid_w = width_factor  * ms
+
+        # ── pre-compute positional / rotary embeddings for the fixed grid ─
+        grid_thw = torch.tensor(
+            [[self.t_patches, self.grid_h, self.grid_w]], dtype=torch.int32
+        )
+
+        pos_embeds = Qwen3VLVisionModel.fast_pos_embed_interpolate(
+            visual, grid_thw
+        ).unsqueeze(0)                                  # [1, N, embed_dim]
+        self.register_buffer("pos_embeds", pos_embeds)
+
+        rot_pos_emb = (
+            Qwen3VLVisionModel.rot_pos_emb(visual, grid_thw)
+            .float()
+            .unsqueeze(0).unsqueeze(0).unsqueeze(0)     # [1, 1, 1, N, dim/2]
+        )
+        cos = rot_pos_emb.cos()
+        sin = rot_pos_emb.sin()
+        self.register_buffer("rotary_cos", torch.cat([cos, cos], dim=-1))
+        self.register_buffer("rotary_sin", torch.cat([-sin, sin], dim=-1))
+
+        # ── ONNX-friendly GELU ──────────────────────────────────────────
+        self._replace_gelu(visual)
+
+        # ── fuse layer-norms into the subsequent linear layers ──────────
+        scaling = self.head_dim ** -0.25
+        for blk in visual.blocks:
+            blk.attn.qkv.weight.data[: -self.embed_dim] *= scaling
+            blk.attn.qkv.bias.data[: -self.embed_dim]   *= scaling
+            self._fuse_norm(blk.norm1, blk.attn.qkv)
+            self._fuse_norm(blk.norm2, blk.mlp.linear_fc1)
+
+        for ds_layer in visual.deepstack_merger_list:
+            self._fuse_norm(ds_layer.norm, ds_layer.linear_fc1)
+
+        self._fuse_norm(visual.merger.norm, visual.merger.linear_fc1)
+
+        self.visual = visual
+
+    # ── helpers ──────────────────────────────────────────────────────────
+    @staticmethod
+    def _fuse_norm(norm, linear):
+        norm_bias   = norm.bias.data
+        norm_weight = norm.weight.data
+        if linear.weight.shape[1] != norm_bias.shape[0]:
+            repeat_factor = linear.weight.shape[1] // norm_bias.shape[0]
+            norm_bias     = norm_bias.repeat(repeat_factor)
+            norm_weight   = norm_weight.repeat(repeat_factor)
+        linear.bias.data.add_(torch.matmul(linear.weight.data, norm_bias))
+        linear.weight.data.mul_(norm_weight.unsqueeze(0))
+        norm.elementwise_affine = False
+        norm.weight = None
+        norm.bias   = None
+
+    @staticmethod
+    def _replace_gelu(module):
+        for name, child in module.named_children():
+            if isinstance(child, torch.nn.GELU):
+                setattr(module, name, torch.nn.GELU(approximate="tanh"))
+            else:
+                LLM_VISION_EMBED._replace_gelu(child)
+
+    def _rotate_half(self, x, batch_size):
+        x = x.view(2, batch_size, self.num_heads, -1, 2, self.head_dim_half)
+        x = x.flip(-2)
+        return x.view(2, batch_size, self.num_heads, -1, self.head_dim)
+
+    def _spatial_merge(self, x, target_hidden_size):
+        """Explicit spatial reshape: group merge_size×merge_size patches per frame."""
+        ms = self.merge_size
+        x = x.view(1, self.t_patches, self.grid_h, self.grid_w, -1)
+        x = x.view(1, self.t_patches, self.grid_h // ms, ms, self.grid_w // ms, ms, -1)
+        x = x.permute(0, 1, 2, 4, 3, 5, 6)
+        return x.reshape(1, -1, target_hidden_size)
+
+    # ── forward ──────────────────────────────────────────────────────────
+    def forward(self, pixel_values):
+        """
+        pixel_values : float32 [total_patches, flatten_dim]
+            Already CLIP-normalised, from the HF processor.
+        """
+        batch_size = 1
+
+        # patch embedding (original HF module)
+        hidden = self.visual.patch_embed(pixel_values)           # [N, embed_dim]
+        hidden = hidden.unsqueeze(0)                             # [1, N, embed_dim]
+        hidden = hidden + self.pos_embeds
+
+        # transformer blocks
+        deepstack_features = []
+        ds_indices = self.visual.deepstack_visual_indexes
+        ds_modules = self.visual.deepstack_merger_list
+
+        for layer_num, blk in enumerate(self.visual.blocks):
+            # --- self-attention (manual, ONNX-safe) ---
+            h_norm = blk.norm1(hidden)
+            qkv    = blk.attn.qkv(h_norm)
+            qkv    = qkv.reshape(batch_size, -1, 3, self.num_heads, self.head_dim)
+            qkv    = qkv.permute(2, 0, 3, 1, 4)          # [3, B, heads, seq, dim]
+            qk, v  = qkv.split([2, 1], dim=0)
+
+            qk_rot = (
+                qk * self.rotary_cos
+                + self._rotate_half(qk, batch_size) * self.rotary_sin
+            )
+            q_rot, k_rot = qk_rot.split([1, 1], dim=0)
+            attn = torch.matmul(q_rot, k_rot.transpose(-1, -2))
+            attn = torch.softmax(attn, dim=-1)
+            attn = torch.matmul(attn, v)
+            attn = attn.transpose(2, 3).reshape(
+                batch_size, -1, blk.attn.proj.in_features
+            )
+            hidden = hidden + blk.attn.proj(attn)
+
+            # --- feed-forward ---
+            mlp_out = blk.mlp.linear_fc1(blk.norm2(hidden))
+            mlp_out = blk.mlp.act_fn(mlp_out)
+            mlp_out = blk.mlp.linear_fc2(mlp_out)
+            hidden  = hidden + mlp_out
+
+            # --- deepstack (explicit spatial merge) ---
+            if layer_num in ds_indices:
+                idx      = ds_indices.index(layer_num)
+                ds_layer = ds_modules[idx]
+                x_ds = self._spatial_merge(hidden, ds_layer.hidden_size)
+                x_ds = ds_layer.norm(x_ds)
+                x_ds = ds_layer.linear_fc1(x_ds)
+                x_ds = ds_layer.act_fn(x_ds)
+                x_ds = ds_layer.linear_fc2(x_ds)
+                deepstack_features.append(x_ds)
+
+        # merger (explicit spatial merge)
+        hidden = self.visual.merger.norm(hidden)
+        hidden = self._spatial_merge(hidden, self.visual.merger.hidden_size)
+        hidden = self.visual.merger.linear_fc1(hidden)
+        hidden = self.visual.merger.act_fn(hidden)
+        hidden = self.visual.merger.linear_fc2(hidden)
+
+        return *deepstack_features, hidden
+
+
+# ══════════════════════════════════════════════════════════════════════════════
+# Transformer Module (embedding mode — no KV cache, no lm_head)
+# ══════════════════════════════════════════════════════════════════════════════
+class LLM_MAIN_EMBED(torch.nn.Module):
+    """
+    All decoder layers in a single ONNX graph.
+
+    Differences from the generation-oriented LLM_MAIN:
+      • No KV-cache inputs / outputs  (single forward pass)
+      • No lm_head projection         (we need hidden states, not logits)
+      • Final RMSNorm is kept         (not fused into lm_head)
+      • Outputs the full sequence      (not just last token)
+
+    Input layout  (via *all_inputs):
+        hidden_states, ds_feat_0 … ds_feat_N, rotary_cos, rotary_sin, attn_mask
+
+    Output:
+        last_hidden_state  [batch, seq_len, hidden_size]
+    """
+
+    def __init__(
+        self,
+        model,
+        num_heads,
+        num_key_value_heads,
+        head_dim,
+        num_layers,
+        hidden_size,
+        deepstack_features_len,
+    ):
+        super().__init__()
+        self.model = model
+
+        self.head_dim            = head_dim
+        self.head_dim_half       = head_dim // 2
+        self.num_heads           = num_heads
+        self.num_key_value_heads = num_key_value_heads
+        self.num_key_value_groups = num_heads // num_key_value_heads
+        self.qk_heads            = num_heads + num_key_value_heads
+        self.num_layers          = num_layers
+        self.deepstack_features_len = deepstack_features_len
+
+        # 3 trailing args after deepstack features: cos, sin, mask
+        self._ds_offset = 3 + deepstack_features_len
+
+        # ── save the final RMSNorm (before weight fusion touches layers) ─
+        lang = model.model.language_model
+        self.register_buffer(
+            "final_norm_weight", lang.norm.weight.data.clone()
+        )
+        self.final_norm_eps = float(lang.norm.variance_epsilon)
+
+        # ── ONNX-friendly GELU ──────────────────────────────────────────
+        self._replace_gelu(model)
+
+        # ── fuse weights (same optimisations as reference) ──────────────
+        self._fuse_weights(hidden_size)
+
+    # ═══════════��══════════════════════════════════════════════════════════
+    # Weight fusion (runs once at init)
+    # ══════════════════════════════════════════════════════════════════════
+    def _fuse_weights(self, hidden_size):
+        scale_factor   = self.head_dim ** -0.25
+        norm_factor    = hidden_size ** 0.5
+        norm_factor_qk = self.head_dim ** 0.5
+
+        with torch.no_grad():
+            for layer in self.model.model.language_model.layers:
+                self._fuse_qkv(layer, scale_factor, norm_factor, norm_factor_qk)
+                self._fuse_gate_up(layer, norm_factor)
+            # NOTE: we do NOT fuse the final norm into lm_head (no lm_head).
+
+    def _fuse_qkv(self, layer, scale_factor, norm_factor, norm_factor_qk):
+        attn = layer.self_attn
+        q, k, v = attn.q_proj, attn.k_proj, attn.v_proj
+
+        in_f  = int(q.in_features)
+        out_f = int(q.out_features + k.out_features + v.out_features)
+        has_bias = any(p.bias is not None for p in (q, k, v))
+
+        qkv = torch.nn.Linear(in_f, out_f, bias=has_bias)
+        qkv.weight.copy_(torch.cat([q.weight, k.weight, v.weight], dim=0))
+        if has_bias:
+            def _b(p):
+                return p.bias if p.bias is not None else torch.zeros(
+                    p.out_features, dtype=qkv.weight.dtype
+                )
+            qkv.bias.copy_(torch.cat([_b(q), _b(k), _b(v)], dim=0))
+
+        attn.q_out_features = int(q.out_features)
+        attn.k_out_features = int(k.out_features)
+        attn.v_out_features = int(v.out_features)
+        del attn.q_proj, attn.k_proj, attn.v_proj
+
+        # fuse QK norms + attention scaling
+        combined_scale = scale_factor * norm_factor_qk
+        attn.q_norm.weight.mul_(combined_scale)
+        attn.k_norm.weight.mul_(combined_scale)
+        q_norm_rep = attn.q_norm.weight.repeat(self.num_heads)
+        k_norm_rep = attn.k_norm.weight.repeat(self.num_key_value_heads)
+        attn.qk_norm_weight = torch.nn.Parameter(
+            torch.cat([q_norm_rep, k_norm_rep], dim=0).view(
+                1, 1, 1, -1, self.head_dim
+            )
+        )
+        del attn.q_norm, attn.k_norm
+
+        # absorb input LayerNorm into QKV
+        input_norm_w = layer.input_layernorm.weight.unsqueeze(0) * norm_factor
+        qkv.weight.mul_(input_norm_w)
+        attn.qkv = qkv
+        del layer.input_layernorm
+
+    def _fuse_gate_up(self, layer, norm_factor):
+        post_norm_w = layer.post_attention_layernorm.weight.unsqueeze(0) * norm_factor
+        gate, up = layer.mlp.gate_proj, layer.mlp.up_proj
+        gate_up = torch.nn.Linear(
+            gate.in_features, gate.out_features + up.out_features, bias=False
+        )
+        gate_up.weight.copy_(
+            torch.cat([gate.weight * post_norm_w, up.weight * post_norm_w], dim=0)
+        )
+        layer.mlp.gate_up_proj = gate_up
+        del layer.mlp.gate_proj, layer.mlp.up_proj, layer.post_attention_layernorm
+
+    # ══════════════════════════════════════════════════════════════════════
+    # Utility
+    # ══════════════════════════════════════════════════════════════════════
+    @staticmethod
+    def _replace_gelu(module):
+        for name, child in module.named_children():
+            if isinstance(child, torch.nn.GELU):
+                setattr(module, name, torch.nn.GELU(approximate="tanh"))
+            else:
+                LLM_MAIN_EMBED._replace_gelu(child)
+
+    def _rms_norm(self, x):
+        """Sum-based RMS norm (weight is absorbed into the subsequent linear)."""
+        return x * torch.rsqrt(x.square().sum(-1, keepdim=True))
+
+    def _rotate_half(self, x, batch_size):
+        x = x.view(batch_size, -1, 1, self.qk_heads, 2, self.head_dim_half)
+        x = x.flip(-2)
+        return x.view(batch_size, -1, 1, self.qk_heads, self.head_dim)
+
+    # ══════════════════════════════════════════════════════════════════════
+    # Forward
+    # ══════════════════════════════════════════════════════════════════════
+    def forward(self, *all_inputs):
+        """
+        all_inputs layout:
+            hidden_states          [batch, seq, hidden]
+            deepstack_feat_0 …     [1, seq, hidden]   (×deepstack_features_len)
+            rotary_cos             [1, seq, 1, 1, head_dim]
+            rotary_sin             [1, seq, 1, 1, head_dim]
+            attention_mask         [1, 1, 1, seq, seq]
+        """
+        hidden_states  = all_inputs[0]
+        rotary_cos     = all_inputs[-3]
+        rotary_sin     = all_inputs[-2]
+        attention_mask  = all_inputs[-1]
+        batch_size     = hidden_states.shape[0]
+
+        for i, layer in enumerate(self.model.model.language_model.layers):
+            # ── self-attention ───────────────────────────────────────
+            residual      = hidden_states
+            hidden_states = self._rms_norm(hidden_states)
+
+            qkv = layer.self_attn.qkv(hidden_states)
+            qkv = qkv.reshape(
+                batch_size, -1, 1,
+                self.qk_heads + self.num_key_value_heads,
+                self.head_dim,
+            )
+            qk, v = torch.split(
+                qkv, [self.qk_heads, self.num_key_value_heads], dim=-2
+            )
+
+            qk     = self._rms_norm(qk) * layer.self_attn.qk_norm_weight
+            qk_rot = (
+                qk * rotary_cos
+                + self._rotate_half(qk, batch_size) * rotary_sin
+            )
+
+            q, k = torch.split(
+                qk_rot, [self.num_heads, self.num_key_value_heads], dim=-2
+            )
+            q = q.reshape(
+                batch_size, -1,
+                self.num_key_value_heads,
+                self.num_key_value_groups,
+                self.head_dim,
+            )
+            q = q.permute(0, 2, 3, 1, 4)      # [B, kv_h, g, seq, dim]
+            k = k.permute(0, 3, 2, 4, 1)       # [B, kv_h, 1, dim, seq]
+            v = v.transpose(1, 3)               # [B, kv_h, 1, seq, dim]
+
+            # no KV cache — direct attention over the full sequence
+            attn = torch.matmul(q, k) + attention_mask
+            attn = torch.softmax(attn, dim=-1)
+            attn = torch.matmul(attn, v)
+
+            attn = attn.permute(0, 3, 1, 2, 4).reshape(
+                batch_size, -1, layer.self_attn.o_proj.in_features
+            )
+            hidden_states = residual + layer.self_attn.o_proj(attn)
+
+            # ── feed-forward ─────────────────────────────────────────
+            residual      = hidden_states
+            hidden_states = self._rms_norm(hidden_states)
+
+            gate_up = layer.mlp.gate_up_proj(hidden_states)
+            gate, up = torch.split(
+                gate_up,
+                [layer.mlp.down_proj.in_features, layer.mlp.down_proj.in_features],
+                dim=-1,
+            )
+            hidden_states = residual + layer.mlp.down_proj(
+                layer.mlp.act_fn(gate) * up
+            )
+
+            # ── deepstack feature injection ──────────────────────────
+            if i < self.deepstack_features_len:
+                hidden_states = all_inputs[1 + i] + hidden_states
+
+        # ── final RMSNorm (standard, not fused) ─────────────────────────
+        variance = hidden_states.pow(2).mean(-1, keepdim=True)
+        hidden_states = hidden_states * torch.rsqrt(
+            variance + self.final_norm_eps
+        )
+        hidden_states = hidden_states * self.final_norm_weight
+
+        return hidden_states
+
+
+# ══════════════════════════════════════════════════════════════════════════════
+# Export
+# ══════════════════════════════════════════════════════════════════════════════
+def main(model_path, output_dir):
+    os.makedirs(output_dir, exist_ok=True)
+
+    onnx_vision      = os.path.join(output_dir, "Vision.onnx")
+    onnx_transformer = os.path.join(output_dir, "Transformer.onnx")
+    rotary_params    = os.path.join(output_dir, "rotary_params.npz")
+
+    # ── Save processor files (tokenizer + image processor config) ───────
+    # Stored in a `tokenizer/` subdirectory so the output layout stays tidy
+    # and the directory remains self-contained — no HF repo pull at inference.
+    print("Saving processor files …")
+    from transformers.models.qwen3_vl.processing_qwen3_vl import Qwen3VLProcessor
+    tokenizer_dir = os.path.join(output_dir, "tokenizer")
+    os.makedirs(tokenizer_dir, exist_ok=True)
+    Qwen3VLProcessor.from_pretrained(model_path).save_pretrained(tokenizer_dir)
+    print(f"  Processor saved to {tokenizer_dir}")
+
+    print("Loading model …")
+    model = Qwen3VLForEmbedding.from_pretrained(
+        model_path, torch_dtype=torch.float32, device_map="cpu", low_cpu_mem_usage=True
+    ).eval()
+
+    # ── extract config ──��───────────────────────────────────────────────
+    text_cfg = model.config.text_config
+    num_heads           = text_cfg.num_attention_heads
+    num_key_value_heads = text_cfg.num_key_value_heads
+    head_dim            = text_cfg.head_dim
+    num_layers          = text_cfg.num_hidden_layers
+    hidden_size         = text_cfg.hidden_size
+    deepstack_features_len = len(model.model.visual.deepstack_visual_indexes)
+
+    patch_size  = model.model.visual.patch_size
+    merge_size  = model.model.visual.spatial_merge_size
+    temporal_patch_size = model.model.visual.patch_embed.temporal_patch_size
+    unit        = patch_size * merge_size
+
+    # Compute or validate IMAGE_HEIGHT / IMAGE_WIDTH
+    if IMAGE_HEIGHT is None:
+        IMAGE_HEIGHT_val = 16 * unit   # sensible default
+    else:
+        IMAGE_HEIGHT_val = IMAGE_HEIGHT
+    if IMAGE_WIDTH is None:
+        IMAGE_WIDTH_val = 16 * unit
+    else:
+        IMAGE_WIDTH_val = IMAGE_WIDTH
+
+    assert IMAGE_HEIGHT_val % unit == 0, (
+        f"IMAGE_HEIGHT ({IMAGE_HEIGHT_val}) must be a multiple of "
+        f"patch_size * merge_size = {unit}"
+    )
+    assert IMAGE_WIDTH_val % unit == 0, (
+        f"IMAGE_WIDTH ({IMAGE_WIDTH_val}) must be a multiple of "
+        f"patch_size * merge_size = {unit}"
+    )
+
+    HEIGHT_FACTOR     = IMAGE_HEIGHT_val // unit
+    WIDTH_FACTOR      = IMAGE_WIDTH_val  // unit
+
+    # Number of temporal patches after Conv3d with stride=temporal_patch_size.
+    # The HF processor pads frames to multiples of temporal_patch_size.
+    temporal_patches  = math.ceil(TEMPORAL_SIZE / temporal_patch_size)
+    vision_embed_size = temporal_patches * HEIGHT_FACTOR * WIDTH_FACTOR
+
+    print(f"  text  : layers={num_layers}  heads={num_heads}  kv_heads={num_key_value_heads}  "
+          f"head_dim={head_dim}  hidden={hidden_size}")
+    print(f"  vision: deepstack={deepstack_features_len}  patch={patch_size}  merge={merge_size}")
+    print(f"  TEMPORAL_SIZE={TEMPORAL_SIZE}  temporal_patch_size={temporal_patch_size}"
+          f"  →  temporal_patches={temporal_patches}")
+    print(f"  image size: {IMAGE_HEIGHT_val}x{IMAGE_WIDTH_val}  →  {vision_embed_size} vision tokens")
+
+    with torch.inference_mode():
+
+        # ══════════════════════════════════════════════════════════════════
+        # 1. Save rotary parameters + token embedding weight (to skip Embed.onnx)
+        # ══════════════════════════════════════════════════════════════════
+        rotary_emb = model.model.language_model.rotary_emb
+        inv_freq      = rotary_emb.inv_freq.cpu().numpy()
+        mrope_section = np.array(rotary_emb.mrope_section, dtype=np.int64)
+        embed_weight = (
+            model.model.language_model.embed_tokens.weight
+            .detach().cpu().float().numpy()
+        )
+        np.savez(
+            rotary_params,
+            inv_freq=inv_freq,
+            mrope_section=mrope_section,
+            embed_weight=embed_weight,
+            image_height=IMAGE_HEIGHT_val,
+            image_width=IMAGE_WIDTH_val,
+            height_factor=HEIGHT_FACTOR,
+            width_factor=WIDTH_FACTOR,
+            patch_size=patch_size,
+            merge_size=merge_size,
+            hidden_size=hidden_size,
+            head_dim=head_dim,
+            max_seq_len=MAX_SEQ_LEN,
+            temporal_patch_size=temporal_patch_size,
+            temporal_patches=temporal_patches,
+            temporal_size=TEMPORAL_SIZE,
+        )
+        print(f"Saved rotary params → {rotary_params}")
+
+        # ══════════════════════════════════════════════════════════════════
+        # 2. Export Vision
+        # ══════════════════════════════════════════════════════════════════
+        print("Exporting Vision …")
+        total_patches = temporal_patches * HEIGHT_FACTOR * merge_size * WIDTH_FACTOR * merge_size
+        flatten_dim   = 3 * temporal_patch_size * patch_size * patch_size
+        dummy_pixels  = torch.randn(total_patches, flatten_dim, dtype=torch.float32)
+
+        vision_output_names = []
+        vision_dynamic_axes = {}
+        for i in range(deepstack_features_len):
+            name = f"deepstack_feature_{i}"
+            vision_output_names.append(name)
+        vision_output_names.append("vision_hidden_states")
+
+        torch.onnx.export(
+            LLM_VISION_EMBED(model, HEIGHT_FACTOR, WIDTH_FACTOR, temporal_patches),
+            (dummy_pixels,),
+            onnx_vision,
+            input_names=["pixel_values"],
+            output_names=vision_output_names,
+            dynamic_axes=vision_dynamic_axes if vision_dynamic_axes else None,
+            opset_version=OPSET,
+            dynamo=False,
+        )
+        del dummy_pixels
+        gc.collect()
+        print(f"  → {onnx_vision}")
+        print("  Consolidating external weights …")
+        consolidate_external_data(onnx_vision)
+
+        # ══════════════════════════════════════════════════════════════════
+        # 3. Export Transformer
+        # ══════════════════════════════════════════════════════════════════
+        print("Exporting Transformer …")
+
+        # dummy sequence length for tracing
+        dummy_seq   = 10 + vision_embed_size
+        dummy_hidden = torch.ones((1, dummy_seq, hidden_size), dtype=torch.float32)
+        dummy_ds     = torch.ones((1, dummy_seq, hidden_size), dtype=torch.float32)
+        dummy_cos    = torch.zeros((1, dummy_seq, 1, 1, head_dim), dtype=torch.float32)
+        dummy_sin    = torch.zeros((1, dummy_seq, 1, 1, head_dim), dtype=torch.float32)
+        dummy_mask   = torch.zeros((1, 1, 1, dummy_seq, dummy_seq), dtype=torch.float32)
+
+        all_inputs  = [dummy_hidden]
+        input_names = ["hidden_states"]
+        dynamic_axes = {
+            "hidden_states":     {0: "batch", 1: "seq_len"},
+            "last_hidden_state": {0: "batch", 1: "seq_len"},
+            "rotary_cos":        {1: "seq_len"},
+            "rotary_sin":        {1: "seq_len"},
+            "attention_mask":    {3: "seq_len", 4: "seq_len"},
+        }
+
+        for i in range(deepstack_features_len):
+            name = f"deepstack_features_{i}"
+            input_names.append(name)
+            all_inputs.append(dummy_ds)
+            dynamic_axes[name] = {1: "seq_len"}
+
+        all_inputs.extend([dummy_cos, dummy_sin, dummy_mask])
+        input_names.extend(["rotary_cos", "rotary_sin", "attention_mask"])
+
+        model_main = LLM_MAIN_EMBED(
+            model, num_heads, num_key_value_heads, head_dim,
+            num_layers, hidden_size, deepstack_features_len,
+        )
+        del model
+        gc.collect()
+
+        torch.onnx.export(
+            model_main,
+            tuple(all_inputs),
+            onnx_transformer,
+            input_names=input_names,
+            output_names=["last_hidden_state"],
+            dynamic_axes=dynamic_axes,
+            opset_version=OPSET,
+            dynamo=False,
+        )
+        del model_main, all_inputs
+        gc.collect()
+        print(f"  → {onnx_transformer}")
+        print("  Consolidating external weights …")
+        consolidate_external_data(onnx_transformer)
+
+    print("Export complete.")
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser(
+        description="Export Qwen3-VL-Embedding model to ONNX files"
+    )
+    parser.add_argument(
+        "--model_path", type=str, default=DEFAULT_MODEL_PATH,
+        help=f"Path to the HF model (default: {DEFAULT_MODEL_PATH})",
+    )
+    parser.add_argument(
+        "--output_dir", type=str, default=DEFAULT_OUTPUT_DIR,
+        help=f"Directory to save ONNX files (default: {DEFAULT_OUTPUT_DIR})",
+    )
+    args = parser.parse_args()
+
+    main(args.model_path, args.output_dir)

export_script/b_export_onnx_vision.py

@@ -0,0 +1,212 @@
+"""
+Direct HF export of the Qwen3-VL vision encoder to ONNX.
+
+Instead of manually reimplementing the vision encoder (with norm fusion,
+GELU replacement, etc.), this script wraps the HF model's own
+Qwen3VLVisionModel and exports it via torch.onnx.export.
+
+This produces a Vision.onnx that is numerically identical to PyTorch
+because it traces through the exact same code path.
+
+Usage:
+    python 2_export_onnx_vision.py
+    python 2_export_onnx_vision.py --model_path /path/to/model --output_dir /path/to/output
+
+This ONLY re-exports Vision.onnx.  The Embed.onnx, Transformer.onnx,
+and rotary_params.npz from export_embedding_onnx.py are reused as-is.
+"""
+
+import math
+import os
+import gc
+import argparse
+import torch
+import onnx
+import numpy as np
+from qwen3_vl_embedding import Qwen3VLForEmbedding
+
+def consolidate_external_data(onnx_path: str):
+    """
+    Re-save an ONNX model so all external weights live in one .onnx.data file
+    instead of hundreds of per-tensor files.
+    """
+    out_dir  = os.path.dirname(onnx_path)
+    base     = os.path.basename(onnx_path)
+    data_rel = base + ".data"                      # sibling file: Transformer.onnx.data
+    model    = onnx.load(onnx_path, load_external_data=True)
+
+    # Delete ONNX per-tensor external files (e.g.
+    # `model.model.language_model.layers.0.self_attn.qk_norm_weight`). They can
+    # be named with dots, so we use an allow-list of known suffixes we want to
+    # keep. Subdirectories (e.g. `tokenizer/`) are skipped via the isfile check.
+    PRESERVE_SUFFIXES = (".onnx", ".onnx.data", ".npz", ".bin")
+    for fname in list(os.listdir(out_dir)):
+        fpath = os.path.join(out_dir, fname)
+        if not os.path.isfile(fpath):
+            continue
+        if any(fname.endswith(sfx) for sfx in PRESERVE_SUFFIXES):
+            continue
+        try:
+            os.remove(fpath)
+        except OSError:
+            pass
+
+    # Remove the target .data file if it already exists (stale)
+    target_data = os.path.join(out_dir, data_rel)
+    if os.path.exists(target_data):
+        os.remove(target_data)
+
+    onnx.save_model(
+        model, onnx_path,
+        save_as_external_data=True,
+        all_tensors_to_one_file=True,
+        location=data_rel,
+        size_threshold=1024,
+        convert_attribute=False,
+    )
+    del model
+    gc.collect()
+
+# ══════════════════════════════════════════════════════════════════════════════
+# Configuration  (must match export_embedding_onnx.py)
+# ══════════════════════════════════════════════════════════════════════════════
+QWEN3VLE_MODEL_HF_PATH = os.getenv(
+    "MODEL_QWEN3VLE_MODEL_HF_PATH", "/home/jordan/Research/Product-AI-mono/assets/model/Qwen3-VL-Embedding-2B"
+)
+QWEN3VLE_TRT_DIR_PATH = os.getenv(
+    "MODEL_QWEN3VLE_TRT_DIR_PATH", "assets/model/Qwen3-VL-Embedding-2B-onnx"
+)
+TEMPORAL_SIZE=1
+
+DEFAULT_MODEL_PATH = QWEN3VLE_MODEL_HF_PATH
+DEFAULT_OUTPUT_DIR = QWEN3VLE_TRT_DIR_PATH
+OPSET              = 17
+
+
+# ══════════════════════════════════════════════════════════════════════════════
+# Vision Wrapper — thin shell around the HF model
+# ══════════════════════════════════════════════════════════════════════════════
+class VisionEncoderWrapper(torch.nn.Module):
+    """
+    Wraps Qwen3VLVisionModel for ONNX export.
+
+    The HF vision model's forward() takes (hidden_states, grid_thw) and
+    returns (merged_features, deepstack_features_list).
+
+    For ONNX we bake in a fixed grid_thw and return flat outputs.
+    """
+
+    def __init__(self, visual, grid_thw: torch.Tensor):
+        super().__init__()
+        self.visual = visual
+        # Bake in the fixed grid as a buffer so it's part of the ONNX graph
+        self.register_buffer("grid_thw", grid_thw)
+
+    def forward(self, pixel_values):
+        """
+        pixel_values : float32 [total_patches, flatten_dim]
+            From the HF processor (already CLIP-normalised).
+
+        Returns: (deepstack_feature_0, …, vision_hidden_states)
+        """
+        hidden_states, deepstack_features = self.visual(
+            pixel_values, grid_thw=self.grid_thw
+        )
+        # Return deepstack features first, then merged vision features
+        return *deepstack_features, hidden_states
+
+
+# ══════════════════════════════════════════════════════════��═══════════════════
+# Export
+# ══════════════════════════════════════════════════════════════════════════════
+def main(model_path, output_dir):
+    os.makedirs(output_dir, exist_ok=True)
+    onnx_vision = os.path.join(output_dir, "Vision.onnx")
+
+    # ── Load rotary_params to get the grid config ────────────────────────
+    rp = np.load(os.path.join(output_dir, "rotary_params.npz"))
+    height_factor = int(rp["height_factor"])
+    width_factor  = int(rp["width_factor"])
+    patch_size    = int(rp["patch_size"])
+    merge_size    = int(rp["merge_size"])
+
+    grid_h = height_factor * merge_size
+    grid_w = width_factor  * merge_size
+    image_height  = int(rp["image_height"])
+    image_width   = int(rp["image_width"])
+
+    temporal_patch_size_npz = int(rp["temporal_patch_size"]) if "temporal_patch_size" in rp else 2
+    temporal_patches = math.ceil(TEMPORAL_SIZE / temporal_patch_size_npz)
+    total_patches = temporal_patches * grid_h * grid_w
+
+    print(f"Grid: {temporal_patches}x{grid_h}x{grid_w} = {total_patches} patches")
+    print(f"Image: {image_height}x{image_width}  TEMPORAL_SIZE={TEMPORAL_SIZE}")
+
+    # ── Load model ──────────────────────────────────────────────────────
+    print("Loading model …")
+    model = Qwen3VLForEmbedding.from_pretrained(
+        model_path,
+        torch_dtype=torch.float32,
+        device_map="cpu",
+        low_cpu_mem_usage=True,
+        _attn_implementation="eager",
+    ).eval()
+
+    visual = model.model.visual
+    temporal_patch_size = visual.patch_embed.temporal_patch_size
+    flatten_dim = 3 * temporal_patch_size * patch_size * patch_size
+
+    deepstack_features_len = len(visual.deepstack_visual_indexes)
+    print(f"  deepstack features: {deepstack_features_len}")
+    print(f"  temporal_patch_size: {temporal_patch_size}")
+    print(f"  flatten_dim: {flatten_dim}")
+
+    # ── Build wrapper ───────────────────────────────────────────────────
+    grid_thw = torch.tensor([[temporal_patches, grid_h, grid_w]], dtype=torch.int64)
+    wrapper = VisionEncoderWrapper(visual, grid_thw)
+
+    # ── Build dummy input ───────────────────────────────────────────────
+    dummy_pixels = torch.randn(total_patches, flatten_dim, dtype=torch.float32)
+
+    # ── Output names ────────────────────────────────────────────────────
+    output_names = []
+    for i in range(deepstack_features_len):
+        output_names.append(f"deepstack_feature_{i}")
+    output_names.append("vision_hidden_states")
+
+    # ── Export ──────────────────────────────────────────────────────────
+    print("Exporting Vision (direct HF) …")
+    with torch.inference_mode():
+        torch.onnx.export(
+            wrapper,
+            (dummy_pixels,),
+            onnx_vision,
+            input_names=["pixel_values"],
+            output_names=output_names,
+            opset_version=OPSET,
+            dynamo=False,
+        )
+
+    del wrapper, model
+    gc.collect()
+    print(f"  → {onnx_vision}")
+    print("Consolidating external weights …")
+    consolidate_external_data(onnx_vision)
+    print("Done.")
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser(
+        description="Export Qwen3-VL vision encoder to ONNX (direct HF)"
+    )
+    parser.add_argument(
+        "--model_path", type=str, default=DEFAULT_MODEL_PATH,
+        help=f"Path to the HF model (default: {DEFAULT_MODEL_PATH})",
+    )
+    parser.add_argument(
+        "--output_dir", type=str, default=DEFAULT_OUTPUT_DIR,
+        help=f"Directory to save ONNX files (default: {DEFAULT_OUTPUT_DIR})",
+    )
+    args = parser.parse_args()
+
+    main(args.model_path, args.output_dir)

export_script/qwen3_vl_embedding.py

@@ -0,0 +1,393 @@
+import os
+import torch
+import torch.nn.functional as F
+import unicodedata
+import numpy as np
+import logging
+
+from PIL import Image
+from urllib.parse import urlparse
+from dataclasses import dataclass
+from typing import Optional, List, Union, Dict, Any
+from transformers.models.qwen3_vl.modeling_qwen3_vl import Qwen3VLPreTrainedModel, Qwen3VLModel, Qwen3VLConfig
+from transformers.models.qwen3_vl.processing_qwen3_vl import Qwen3VLProcessor
+from transformers.modeling_outputs import ModelOutput
+from transformers.processing_utils import Unpack
+from transformers.utils import TransformersKwargs
+from transformers.cache_utils import Cache
+from transformers.utils.generic import check_model_inputs
+from qwen_vl_utils.vision_process import process_vision_info
+
+logger = logging.getLogger(__name__)
+
+# Constants for configuration
+MAX_LENGTH = 8192
+IMAGE_BASE_FACTOR = 16
+IMAGE_FACTOR = IMAGE_BASE_FACTOR * 2
+MIN_PIXELS = 4 * IMAGE_FACTOR * IMAGE_FACTOR
+MAX_PIXELS = 1800 * IMAGE_FACTOR * IMAGE_FACTOR
+FPS = 1
+MAX_FRAMES = 64
+FRAME_MAX_PIXELS = 768 * IMAGE_FACTOR * IMAGE_FACTOR
+MAX_TOTAL_PIXELS = 10 * FRAME_MAX_PIXELS
+PAD_TOKEN = "<|endoftext|>"
+
+# Define output structure for embeddings
+@dataclass
+class Qwen3VLForEmbeddingOutput(ModelOutput):
+    last_hidden_state: Optional[torch.FloatTensor] = None
+    attention_mask: Optional[torch.Tensor] = None
+
+# Define model class to compute embeddings
+class Qwen3VLForEmbedding(Qwen3VLPreTrainedModel):
+    _checkpoint_conversion_mapping = {}
+    accepts_loss_kwargs = False
+    config: Qwen3VLConfig
+
+    def __init__(self, config):
+        super().__init__(config)
+        self.model = Qwen3VLModel(config)
+        self.post_init()
+
+    def get_input_embeddings(self):
+        return self.model.get_input_embeddings()
+
+    def set_input_embeddings(self, value):
+        self.model.set_input_embeddings(value)
+
+    def set_decoder(self, decoder):
+        self.model.set_decoder(decoder)
+
+    def get_decoder(self):
+        return self.model.get_decoder()
+
+    # Extract video features from model
+    def get_video_features(self, pixel_values_videos: torch.FloatTensor,
+                           video_grid_thw: Optional[torch.LongTensor] = None):
+        return self.model.get_video_features(pixel_values_videos, video_grid_thw)
+
+    # Extract image features from model
+    def get_image_features(self, pixel_values: torch.FloatTensor,
+                           image_grid_thw: Optional[torch.LongTensor] = None):
+        return self.model.get_image_features(pixel_values, image_grid_thw)
+
+    # Make modules accessible through properties
+    @property
+    def language_model(self):
+        return self.model.language_model
+
+    @property
+    def visual(self):
+        return self.model.visual
+
+    # Forward pass through model with input parameters
+    # @check_model_inputs
+    def forward(self,
+                input_ids: torch.LongTensor = None,
+                attention_mask: Optional[torch.Tensor] = None,
+                position_ids: Optional[torch.LongTensor] = None,
+                past_key_values: Optional[Cache] = None,
+                inputs_embeds: Optional[torch.FloatTensor] = None,
+                pixel_values: Optional[torch.Tensor] = None,
+                pixel_values_videos: Optional[torch.FloatTensor] = None,
+                image_grid_thw: Optional[torch.LongTensor] = None,
+                video_grid_thw: Optional[torch.LongTensor] = None,
+                cache_position: Optional[torch.LongTensor] = None,
+                logits_to_keep: Union[int, torch.Tensor] = 0,
+                **kwargs: Unpack[TransformersKwargs],
+    ) -> Union[tuple, Qwen3VLForEmbeddingOutput]:
+        # Pass inputs through the model
+        outputs = self.model(
+            input_ids=input_ids,
+            pixel_values=pixel_values,
+            pixel_values_videos=pixel_values_videos,
+            image_grid_thw=image_grid_thw,
+            video_grid_thw=video_grid_thw,
+            position_ids=position_ids,
+            attention_mask=attention_mask,
+            past_key_values=past_key_values,
+            inputs_embeds=inputs_embeds,
+            cache_position=cache_position,
+            **kwargs,
+        )
+        # Return the model output
+        return Qwen3VLForEmbeddingOutput(
+            last_hidden_state=outputs.last_hidden_state,
+            attention_mask=attention_mask,
+        )
+
+def sample_frames(frames: List[Union[str, Image.Image]], max_segments: int) -> List[Union[str, Image.Image]]:
+    duration = len(frames)
+    if duration <= max_segments:
+        return frames
+
+    frame_id_array = np.linspace(0, duration - 1, max_segments, dtype=int)
+    frame_id_list = frame_id_array.tolist()
+    sampled_frames = [ frames[frame_idx] for frame_idx in frame_id_list ]
+    return sampled_frames
+
+def is_image_path(path: str) -> bool:
+    image_extensions = {'.jpg', '.jpeg', '.png', '.gif', '.bmp', '.webp', '.tiff', '.svg'}
+    
+    if path.startswith(('http://', 'https://')):
+        # Parse URL to remove query parameters
+        parsed_url = urlparse(path)
+        clean_path = parsed_url.path
+    else:
+        clean_path = path
+    
+    # Check file extension
+    _, ext = os.path.splitext(clean_path.lower())
+    return ext in image_extensions
+
+def is_video_input(video) -> bool:
+    if isinstance(video, str):
+        return True
+    
+    if isinstance(video, list) and len(video) > 0:
+        # Check first element to determine the type
+        first_elem = video[0]
+        
+        if isinstance(first_elem, Image.Image):
+            return True
+        
+        if isinstance(first_elem, str):
+            return is_image_path(first_elem)
+    
+    return False
+
+# Define embedder class for processing inputs and generating embeddings
+class Qwen3VLEmbedder():
+    def __init__(
+        self, 
+        model_name_or_path: str, 
+        max_length: int = MAX_LENGTH,
+        min_pixels: int = MIN_PIXELS,
+        max_pixels: int = MAX_PIXELS,
+        total_pixels: int = MAX_TOTAL_PIXELS,
+        fps: float = FPS,
+        max_frames: int = MAX_FRAMES,
+        default_instruction: str = "Represent the user's input.",
+        **kwargs
+    ):
+        device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+
+        self.max_length = max_length
+        self.min_pixels = min_pixels
+        self.max_pixels = max_pixels
+        self.total_pixels = total_pixels
+        self.fps = fps
+        self.max_frames = max_frames
+
+        self.default_instruction = default_instruction
+
+        self.model = Qwen3VLForEmbedding.from_pretrained(
+            model_name_or_path, trust_remote_code=True, **kwargs
+        ).to(device)
+        self.processor = Qwen3VLProcessor.from_pretrained(
+            model_name_or_path, padding_side='right'
+        )
+        self.model.eval()
+
+    @torch.no_grad()
+    def forward(self, inputs: Dict[str, Any]) -> Dict[str, torch.Tensor]:
+        outputs = self.model(**inputs)
+        return {
+            'last_hidden_state': outputs.last_hidden_state,
+            'attention_mask': inputs.get('attention_mask')
+        }
+
+    # Truncate token sequence to a specified max length
+    def _truncate_tokens(self, token_ids: List[int], max_length: int) -> List[int]:
+        if len(token_ids) <= max_length:
+            return token_ids
+
+        special_token_ids = set(self.processor.tokenizer.all_special_ids)
+        num_special = sum(1 for token_idx in token_ids if token_idx in special_token_ids)
+        num_non_special_to_keep = max_length - num_special
+
+        final_token_ids = []
+        non_special_kept_count = 0
+        # Ensure retention of special tokens while truncating the rest
+        for token_idx in token_ids:
+            if token_idx in special_token_ids:
+                final_token_ids.append(token_idx)
+            elif non_special_kept_count < num_non_special_to_keep:
+                final_token_ids.append(token_idx)
+                non_special_kept_count += 1
+        return final_token_ids
+
+    def format_model_input(
+        self, 
+        text: Optional[Union[List[str], str]] = None,
+        image: Optional[Union[List[Union[str, Image.Image]], str, Image.Image]] = None,
+        video: Optional[Union[List[Union[str, List[Union[str, Image.Image]]]], str, List[Union[str, Image.Image]]]] = None,
+        instruction: Optional[str] = None,
+        fps: Optional[float] = None,
+        max_frames: Optional[int] = None
+    ) -> List[Dict]:
+
+        # Ensure instruction ends with punctuation
+        if instruction:
+            instruction = instruction.strip()
+            if instruction and not unicodedata.category(instruction[-1]).startswith('P'):
+                instruction = instruction + '.'
+
+        # Initialize conversation with system prompts
+        content = []
+        conversation = [
+            {"role": "system", "content": [{"type": "text", "text": instruction or self.default_instruction}]},
+            {"role": "user", "content": content}
+        ]
+
+        # Normalize text input to list
+        if text is None:
+            texts = []
+        elif isinstance(text, str):
+            texts = [text]
+        else:
+            texts = text
+        
+        # Normalize image input to list
+        if image is None:
+            images = []
+        elif not isinstance(image, list):
+            images = [image]
+        else:
+            images = image
+        
+        # Normalize video input to list
+        if video is None:
+            videos = []
+        elif is_video_input(video):
+            videos = [video]
+        else:
+            # Assume it's a list of videos
+            videos = video
+
+        # Add text, image, or video content to conversation
+        if not texts and not images and not videos:
+            content.append({'type': 'text', 'text': "NULL"})
+            return conversation
+
+        # Process each video
+        for vid in videos:
+            video_content = None
+            video_kwargs = {'total_pixels': self.total_pixels}
+            
+            if isinstance(vid, list):
+                # Video as frame sequence
+                video_content = vid
+                if self.max_frames is not None:
+                    video_content = sample_frames(video_content, self.max_frames)
+                video_content = [
+                    ('file://' + ele if isinstance(ele, str) else ele) 
+                    for ele in video_content
+                ]
+            elif isinstance(vid, str):
+                # Video as file path
+                video_content = vid if vid.startswith(('http://', 'https://')) else 'file://' + vid
+                video_kwargs = {'fps': fps or self.fps, 'max_frames': max_frames or self.max_frames}
+            else:
+                raise TypeError(f"Unrecognized video type: {type(vid)}")
+
+            # Add video input to content
+            if video_content:
+                content.append({
+                    'type': 'video', 
+                    'video': video_content,
+                    **video_kwargs
+                })
+
+        # Process each image
+        for img in images:
+            image_content = None
+            
+            if isinstance(img, Image.Image):
+                image_content = img
+            elif isinstance(img, str):
+                image_content = img if img.startswith(('http://', 'https://')) else 'file://' + img
+            else:
+                raise TypeError(f"Unrecognized image type: {type(img)}")
+
+            # Add image input to content
+            if image_content:
+                content.append({
+                    'type': 'image', 
+                    'image': image_content,
+                    "min_pixels": self.min_pixels,
+                    "max_pixels": self.max_pixels
+                })
+
+        # Process each text
+        for txt in texts:
+            content.append({'type': 'text', 'text': txt})
+
+        return conversation
+
+    # Preprocess input conversations for model consumption
+    def _preprocess_inputs(self, conversations: List[List[Dict]]) -> Dict[str, torch.Tensor]:
+        text = self.processor.apply_chat_template(
+            conversations, add_generation_prompt=True, tokenize=False
+        )
+
+        try:
+            images, video_inputs, video_kwargs = process_vision_info(
+                conversations, image_patch_size=16,
+                return_video_metadata=True, return_video_kwargs=True
+            )
+        except Exception as e:
+            logger.error(f"Error in processing vision info: {e}")
+            images = None
+            video_inputs = None
+            video_kwargs = {'do_sample_frames': False}
+            text = self.processor.apply_chat_template(
+                [{'role': 'user', 'content': [{'type': 'text', 'text': 'NULL'}]}], 
+                add_generation_prompt=True, tokenize=False
+            )
+
+        if video_inputs is not None:
+            videos, video_metadata = zip(*video_inputs)
+            videos = list(videos)
+            video_metadata = list(video_metadata)
+        else:
+            videos, video_metadata = None, None
+
+        inputs = self.processor(
+            text=text, images=images, videos=videos, video_metadata=video_metadata, truncation=True, 
+            max_length=self.max_length, padding=True, do_resize=False, return_tensors='pt',
+            **video_kwargs
+        )
+        return inputs
+
+    # Pool the last hidden state by attention mask for embeddings
+    @staticmethod
+    def _pooling_last(hidden_state: torch.Tensor, attention_mask: torch.Tensor) -> torch.Tensor:
+        flipped_tensor = attention_mask.flip(dims=[1])
+        last_one_positions = flipped_tensor.argmax(dim=1)
+        col = attention_mask.shape[1] - last_one_positions - 1
+        row = torch.arange(hidden_state.shape[0], device=hidden_state.device)
+        return hidden_state[row, col]
+
+    # Process inputs to generate normalized embeddings
+    def process(self, inputs: List[Dict[str, Any]], normalize: bool = True) -> tuple:
+        conversations = [self.format_model_input(
+            text=ele.get('text'),
+            image=ele.get('image'),
+            video=ele.get('video'),
+            instruction=ele.get('instruction'),
+            fps=ele.get('fps'),
+            max_frames=ele.get('max_frames')
+        ) for ele in inputs]
+
+        processed_inputs = self._preprocess_inputs(conversations)
+        processed_inputs = {k: v.to(self.model.device) for k, v in processed_inputs.items()}
+
+        outputs = self.forward(processed_inputs)
+        embeddings = self._pooling_last(outputs['last_hidden_state'], outputs['attention_mask'])
+
+        # Normalize the embeddings if specified
+        if normalize:
+            embeddings = F.normalize(embeddings, p=2, dim=-1)
+
+        return embeddings

export_script/requirements.txt

@@ -0,0 +1,6 @@
+torch>=2.4
+onnx>=1.16
+numpy>=1.26
+transformers>=4.57
+Pillow>=10.0
+qwen-vl-utils>=0.0.11

export_script/run_all.sh

@@ -0,0 +1,10 @@
+echo "Installing Dependencies"
+pip install -r requirements.txt
+
+echo "First Stage - a_export_to_onnx"
+python export_script/a_export_to_onnx.py
+
+echo "Second Stage - b_export_onnx_vision"
+python export_script/b_export_onnx_vision
+
+echo "Finished!"

rotary_params.npz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:54f94155bdc24961992f8098b355f1edd606aad4d533bd66a8aae3de3122a19d
+size 1244663948

text_prompt_APOv2.1_BF16/fire_pred_prompts/fire/fire.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4b72141c31896aa7490c9fee6ed34eab0def685cd655011857967e2cf22dec70
+size 9684

text_prompt_APOv2.1_BF16/fire_pred_prompts/normal/normal.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3437e019f4161d510c6ae29482056b5ffe830c7c31cc9a23e7ec53184fcd9429
+size 9698

text_prompt_APOv2.1_BF16/smoke_pred_prompts/normal/normal.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8f52892acef497521ce6e647ddd8666c3627b80f4cd1f1796657c5f0c5d3cc1e
+size 9698

text_prompt_APOv2.1_BF16/smoke_pred_prompts/smoke/smoke.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:696ca3f417de363201f5ee9f0332105e6e1e92b58694abdfea95f6f919ee382f
+size 9691

tokenizer/added_tokens.json

@@ -0,0 +1,28 @@
+{
+  "</think>": 151668,
+  "</tool_call>": 151658,
+  "</tool_response>": 151666,
+  "<think>": 151667,
+  "<tool_call>": 151657,
+  "<tool_response>": 151665,
+  "<|box_end|>": 151649,
+  "<|box_start|>": 151648,
+  "<|endoftext|>": 151643,
+  "<|file_sep|>": 151664,
+  "<|fim_middle|>": 151660,
+  "<|fim_pad|>": 151662,
+  "<|fim_prefix|>": 151659,
+  "<|fim_suffix|>": 151661,
+  "<|im_end|>": 151645,
+  "<|im_start|>": 151644,
+  "<|image_pad|>": 151655,
+  "<|object_ref_end|>": 151647,
+  "<|object_ref_start|>": 151646,
+  "<|quad_end|>": 151651,
+  "<|quad_start|>": 151650,
+  "<|repo_name|>": 151663,
+  "<|video_pad|>": 151656,
+  "<|vision_end|>": 151653,
+  "<|vision_pad|>": 151654,
+  "<|vision_start|>": 151652
+}

tokenizer/chat_template.jinja

@@ -0,0 +1,125 @@
+{%- set default_system_message = 'Represent the user\'s input.' -%}
+{%- if tools %}
+    {{- '<|im_start|>system\n' }}
+    {%- if messages[0].role == 'system' %}
+        {%- if messages[0].content is string %}
+            {{- messages[0].content }}
+        {%- else %}
+            {%- for content in messages[0].content %}
+                {%- if 'text' in content %}
+                    {{- content.text }}
+                {%- endif %}
+            {%- endfor %}
+        {%- endif %}
+        {{- '\n\n' }}
+    {%- else %}
+        {{- default_system_message + '\n\n' }}
+    {%- endif %}
+    {{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
+{%- else %}
+    {%- if messages[0].role == 'system' %}
+        {{- '<|im_start|>system\n' }}
+        {%- if messages[0].content is string %}
+            {{- messages[0].content }}
+        {%- else %}
+            {%- for content in messages[0].content %}
+                {%- if 'text' in content %}
+                    {{- content.text }}
+                {%- endif %}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|im_end|>\n' }}
+    {%- else %}
+        {{- '<|im_start|>system\n' + default_system_message + '<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- set image_count = namespace(value=0) %}
+{%- set video_count = namespace(value=0) %}
+{%- for message in messages %}
+    {%- if message.role == "user" %}
+        {{- '<|im_start|>' + message.role + '\n' }}
+        {%- if message.content is string %}
+            {{- message.content }}
+        {%- else %}
+            {%- for content in message.content %}
+                {%- if content.type == 'image' or 'image' in content or 'image_url' in content %}
+                    {%- set image_count.value = image_count.value + 1 %}
+                    {%- if add_vision_id %}Picture {{ image_count.value }}: {% endif -%}
+                    <|vision_start|><|image_pad|><|vision_end|>
+                {%- elif content.type == 'video' or 'video' in content %}
+                    {%- set video_count.value = video_count.value + 1 %}
+                    {%- if add_vision_id %}Video {{ video_count.value }}: {% endif -%}
+                    <|vision_start|><|video_pad|><|vision_end|>
+                {%- elif 'text' in content %}
+                    {{- content.text }}
+                {%- endif %}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "assistant" %}
+        {{- '<|im_start|>' + message.role + '\n' }}
+        {%- if message.content is string %}
+            {{- message.content }}
+        {%- else %}
+            {%- for content_item in message.content %}
+                {%- if 'text' in content_item %}
+                    {{- content_item.text }}
+                {%- endif %}
+            {%- endfor %}
+        {%- endif %}
+        {%- if message.tool_calls %}
+            {%- for tool_call in message.tool_calls %}
+                {%- if (loop.first and message.content) or (not loop.first) %}
+                    {{- '\n' }}
+                {%- endif %}
+                {%- if tool_call.function %}
+                    {%- set tool_call = tool_call.function %}
+                {%- endif %}
+                {{- '<tool_call>\n{"name": "' }}
+                {{- tool_call.name }}
+                {{- '", "arguments": ' }}
+                {%- if tool_call.arguments is string %}
+                    {{- tool_call.arguments }}
+                {%- else %}
+                    {{- tool_call.arguments | tojson }}
+                {%- endif %}
+                {{- '}\n</tool_call>' }}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {%- if message.content is string %}
+            {{- message.content }}
+        {%- else %}
+            {%- for content in message.content %}
+                {%- if content.type == 'image' or 'image' in content or 'image_url' in content %}
+                    {%- set image_count.value = image_count.value + 1 %}
+                    {%- if add_vision_id %}Picture {{ image_count.value }}: {% endif -%}
+                    <|vision_start|><|image_pad|><|vision_end|>
+                {%- elif content.type == 'video' or 'video' in content %}
+                    {%- set video_count.value = video_count.value + 1 %}
+                    {%- if add_vision_id %}Video {{ video_count.value }}: {% endif -%}
+                    <|vision_start|><|video_pad|><|vision_end|>
+                {%- elif 'text' in content %}
+                    {{- content.text }}
+                {%- endif %}
+            {%- endfor %}
+        {%- endif %}
+        {{- '\n</tool_response>' }}
+        {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+{%- endif %}

tokenizer/preprocessor_config.json

@@ -0,0 +1,39 @@
+{
+  "crop_size": null,
+  "data_format": "channels_first",
+  "default_to_square": true,
+  "device": null,
+  "disable_grouping": null,
+  "do_center_crop": null,
+  "do_convert_rgb": true,
+  "do_normalize": true,
+  "do_pad": null,
+  "do_rescale": true,
+  "do_resize": true,
+  "image_mean": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "image_processor_type": "Qwen2VLImageProcessorFast",
+  "image_std": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "input_data_format": null,
+  "max_pixels": 1310720,
+  "merge_size": 2,
+  "min_pixels": 4096,
+  "pad_size": null,
+  "patch_size": 16,
+  "processor_class": "Qwen3VLProcessor",
+  "resample": 3,
+  "rescale_factor": 0.00392156862745098,
+  "return_tensors": null,
+  "size": {
+    "longest_edge": 1310720,
+    "shortest_edge": 4096
+  },
+  "temporal_patch_size": 2
+}

tokenizer/special_tokens_map.json

@@ -0,0 +1,31 @@
+{
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "eos_token": {
+    "content": "<|im_end|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer/tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:def76fb086971c7867b829c23a26261e38d9d74e02139253b38aeb9df8b4b50a
+size 11423705

tokenizer/tokenizer_config.json

@@ -0,0 +1,240 @@
+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "151643": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151644": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151645": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151646": {
+      "content": "<|object_ref_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151647": {
+      "content": "<|object_ref_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151648": {
+      "content": "<|box_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151649": {
+      "content": "<|box_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151650": {
+      "content": "<|quad_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151651": {
+      "content": "<|quad_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151652": {
+      "content": "<|vision_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151653": {
+      "content": "<|vision_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151654": {
+      "content": "<|vision_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151655": {
+      "content": "<|image_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151656": {
+      "content": "<|video_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151657": {
+      "content": "<tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151658": {
+      "content": "</tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151659": {
+      "content": "<|fim_prefix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151660": {
+      "content": "<|fim_middle|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151661": {
+      "content": "<|fim_suffix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151662": {
+      "content": "<|fim_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151663": {
+      "content": "<|repo_name|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151664": {
+      "content": "<|file_sep|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151665": {
+      "content": "<tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151666": {
+      "content": "</tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151667": {
+      "content": "<think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151668": {
+      "content": "</think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "extra_special_tokens": {},
+  "model_max_length": 262144,
+  "pad_token": "<|endoftext|>",
+  "processor_class": "Qwen3VLProcessor",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}

tokenizer/video_preprocessor_config.json

@@ -0,0 +1,41 @@
+{
+  "crop_size": null,
+  "data_format": "channels_first",
+  "default_to_square": true,
+  "device": null,
+  "do_center_crop": null,
+  "do_convert_rgb": true,
+  "do_normalize": true,
+  "do_rescale": true,
+  "do_resize": true,
+  "do_sample_frames": true,
+  "fps": 2,
+  "image_mean": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "image_std": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "input_data_format": null,
+  "max_frames": 768,
+  "merge_size": 2,
+  "min_frames": 4,
+  "num_frames": null,
+  "pad_size": null,
+  "patch_size": 16,
+  "processor_class": "Qwen3VLProcessor",
+  "resample": 3,
+  "rescale_factor": 0.00392156862745098,
+  "return_metadata": false,
+  "size": {
+    "longest_edge": 25165824,
+    "shortest_edge": 4096
+  },
+  "temporal_patch_size": 2,
+  "video_metadata": null,
+  "video_processor_type": "Qwen3VLVideoProcessor"
+}