Repackage into ax-llm model-directory format (AX650)

Repo is now a ready-to-run ax-llm model dir: config.json (vlm_type=InternVL3, tokenizer_type=InternVL3_5), internvl3-5-2b_tokenizer.txt, post_config.json, internvl3-5_axmodel (Qwen3-1.7B, 28 layers) + vit-models. Removed legacy layout / demo scripts.

.gitattributes

@@ -42,3 +42,4 @@ main_axcl_x86 filter=lfs diff=lfs merge=lfs -text
 *.jpg filter=lfs diff=lfs merge=lfs -text
 *.mp4 filter=lfs diff=lfs merge=lfs -text
 internvl3-5_tokenizer/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+*_tokenizer.txt filter=lfs diff=lfs merge=lfs -text

.gitignore

@@ -1,3 +0,0 @@
-__pycache__
-*tmp/
-

README.md

@@ -7,7 +7,6 @@ tags:
 - InternVL3
 - InternVL3_5-2B
 - InternVL3_5-2B_GPTQ_INT4
-- Int8
 - VLM
 pipeline_tag: image-text-to-text
 language:
@@ -16,133 +15,49 @@ language:
 
 # InternVL3_5-2B_GPTQ_INT4
 
-This version of InternVL3_5-2B_GPTQ_INT4 has been converted to run on the Axera NPU using **w4a16** quantization.
+InternVL3.5-2B (InternViT + Qwen3-1.7B) converted to run on the Axera NPU with **w4a16** quantization, packaged in **[ax-llm](https://github.com/AXERA-TECH/ax-llm)** model-directory format.
 
-This model has been optimized with the following LoRA: 
+- Compatible with Pulsar2 5.1-patch1
+- Context length: 2k, max prefill: 1k, image input: 448×448
+- Target chip: **AX650** (AX650N host & AXCL M.2 accelerator)
 
-Compatible with Pulsar2 version: 5.1-patch1.
-
-Please note that the context of the model is 2k and the maximum prefill length is 1k.
-
-## Convert tools links:
-
-For those who are interested in model conversion, you can try to export axmodel through the original repo:
-
-https://huggingface.co/OpenGVLab/InternVL3_5-2B
-
-[How to Convert LLM from Huggingface to axmodel](https://github.com/AXERA-TECH/InternVL3_5-2B_GPTQ_INT4.axera/tree/main/model_convert)
-
-[AXera NPU HOST LLM Runtime](https://github.com/AXERA-TECH/ax-llm/tree/ax-internvl) 
-
-[AXera NPU AXCL LLM Runtime](https://github.com/AXERA-TECH/ax-llm/tree/axcl-internvl)
-
-## Support Platform
-
-- AX650
-  - AX650N DEMO Board
-  - [M4N-Dock(爱芯派Pro)](https://wiki.sipeed.com/hardware/zh/maixIV/m4ndock/m4ndock.html)
-  - [M.2 Accelerator card](https://axcl-docs.readthedocs.io/zh-cn/latest/doc_guide_hardware.html)
- 
-|Chips|image encoder 448|ttft|w8a16|
-|--|--|--|--|
-|AX650| 364.412 ms | 4951.50 ms | 28.07 tokens/sec|
-
-
-## How to use
-
-Download all files from this repository to the device
+This repository **is** a ready-to-run ax-llm model directory:
 
 ```
-$ tree -L 1
 .
-├── assets
-├── config.json
-├── examples
-├── gradio_demo.py
-├── infer_axmodel.py
-├── infer_torch.py
-├── internvl3-5_axmodel
-├── internvl3-5_tokenizer
-├── README.md
-├── utils
-└── vit-models
-
-6 directories, 5 files
-```
-
-#### Install transformer
-
-```
-pip install transformers==4.57.1
+├── config.json                    # ax-llm runtime config (vlm_type=InternVL3, tokenizer_type=InternVL3_5)
+├── internvl3-5-2b_tokenizer.txt   # tokenizer
+├── post_config.json               # sampling config
+├── internvl3-5_axmodel/           # LLM: qwen3_p128_l0..l27 (28) + qwen3_post + embed
+└── vit-models/                    # InternViT image encoder (internvl_vit_model_1x3x448x448.axmodel)
 ```
 
-#### Inference with AX650 Host, such as M4N-Dock(爱芯派Pro) or AX650 DEMO Board
-
-Interactive conversations using the `Gradio API`:
-
-```bash
-$ python3 gradio_demo.py --hf_model internvl3-5_tokenizer/ --axmodel_path internvl3-5_axmodel/ --vit_model vit-models/internvl_vit_model_1x3x448x448.axmodel
-```
-
-Plain text dialogue:
-
-![demo_1](assets/demo_1.png)
+## Support Platform
 
-Image understanding:
+- **AX650**
+  - AX650N DEMO Board
+  - [M4N-Dock (爱芯派Pro)](https://wiki.sipeed.com/hardware/zh/maixIV/m4ndock/m4ndock.html)
+  - [M.2 Accelerator card (AXCL)](https://axcl-docs.readthedocs.io/zh-cn/latest/doc_guide_hardware.html)
 
-![demo_2](assets/demo_2.png)
+| Chip | image encoder 448 | ttft | w4a16 |
+|--|--|--|--|
+| AX650 | 364.412 ms | 4951.50 ms | 28.07 tokens/sec |
 
----
+## How to use
 
-Run the following command on the Axera board to start a chat conversation:
+Build/obtain the ax-llm runtime: [AXera NPU LLM Runtime (ax-llm)](https://github.com/AXERA-TECH/ax-llm)
 
 ```sh
-$ python3 infer_axmodel.py --hf_model internvl3-5_tokenizer/ --axmodel_path internvl3-5_axmodel/ --question "请计算函数[y=2x^2+2]的导数, 并提供 markdown 格式的推理过程"
-```
+# interactive chat
+./axllm run   InternVL3_5-2B_GPTQ_INT4
 
-output:
-
-```bash
-[INFO] Using provider: AxEngineExecutionProvider
-[INFO] Model type: 2 (triple core)
-[INFO] Compiler version: 5.1-dirty 0fdbfe15-dirty
-Model loaded successfully!
-slice_indices: [0]
-Slice prefill done: 0
-answer >> 函数 \( y = 2x^2 + 2 \) 的导数可以通过求导法则来计算。首先，我们对函数中的每一项分别求导：
-
-1. 对于 \( 2x^2 \)，使用幂法则求导：
-   \[
-   \frac{d}{dx}(2x^2) = 2 \cdot 2x = 4x
-   \]
-
-2. 对于常数项 \( 2 \)，其导数为 0，因为常数的导数为 0。
-
-将这两部分的结果相加，得到函数 \( y \) 的导数：
-\[
-y' = 4x
-\]
-
-因此，函数 \( y = 2x^2 + 2 \) 的导数为 \( y' = 4x \)。
+# OpenAI-compatible HTTP server (default port 8000)
+./axllm serve InternVL3_5-2B_GPTQ_INT4
 ```
 
-Enter the following command to perform the single-image understanding task:
+On AXCL, choose the card via `AXLLM_DEVICES=<id>` or the `"devices"` field in `config.json`.
 
-```sh
-$ python3 infer_axmodel.py --hf_model internvl3-5_tokenizer/ --axmodel_path internvl3-5_axmodel/ --question "请描述这幅图" -i examples/image_0.jpg --vit_model vit-models/internvl_vit_model_1x3x448x448.axmodel
-```
-
-![image_0.jpg](examples/image_0.jpg)
+## Source & conversion
 
-output:
-
-```bash
-[INFO] Model type: 2 (triple core)
-[INFO] Compiler version: 5.1-dirty 0fdbfe15-dirty
-Model loaded successfully!
-slice_indices: [0, 1, 2]
-Slice prefill done: 0
-Slice prefill done: 1
-Slice prefill done: 2
-answer >> 这是一张红熊猫的照片。红熊猫是一种红棕色的哺乳动物，通常生活在亚洲的森林中。它们以捕食昆虫和小型无脊椎动物为生。图片中，红熊猫正坐在一个木制的平台上，背景是绿色的树木和植被，显得非常自然和生动。红熊猫的表情看起来很友好，似乎在观察或等待什么。
-```
+- Base model: [OpenGVLab/InternVL3_5-2B](https://huggingface.co/OpenGVLab/InternVL3_5-2B)
+- License: BSD-3-Clause

config.json

@@ -0,0 +1,16 @@
+{
+  "model_name": "AXERA-TECH/InternVL3_5-2B_GPTQ_INT4 (AX650)",
+  "url_tokenizer_model": "internvl3-5-2b_tokenizer.txt",
+  "tokenizer_type": "InternVL3_5",
+  "post_config_path": "post_config.json",
+  "template_filename_axmodel": "internvl3-5_axmodel/qwen3_p128_l%d_together.axmodel",
+  "axmodel_num": 28,
+  "filename_post_axmodel": "internvl3-5_axmodel/qwen3_post.axmodel",
+  "filename_tokens_embed": "internvl3-5_axmodel/model.embed_tokens.weight.bfloat16.bin",
+  "tokens_embed_num": 151936,
+  "tokens_embed_size": 2048,
+  "b_use_mmap_load_embed": true,
+  "vlm_type": "InternVL3",
+  "filename_image_encoder_axmodel": "vit-models/internvl_vit_model_1x3x448x448.axmodel",
+  "devices": [0]
+}

examples/laorenshuaidao.mp4

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:8f5c00b37b23af3d01d133da880eb7f6e50d4af608e3575784be7063eb137011
-size 2704112

examples/red-panda.mp4

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:d921c07bb97224d65a37801541d246067f0d506f08723ffa1ad85c217907ccb8
-size 1867237

examples/tuboshu.mp4

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:ced4d95877b9a7f8b48f79bdfe4287eff8837f20348daec2f2e2987459ec1712
-size 5952043

gradio_demo.py

@@ -1,392 +0,0 @@
-import argparse
-import os
-import time
-from typing import Any, Dict, List, Optional, Generator, Tuple
-
-import gradio as gr
-import numpy as np
-import torch
-import torchvision.transforms as T
-from ml_dtypes import bfloat16
-from PIL import Image
-from torchvision.transforms.functional import InterpolationMode
-from transformers import AutoConfig, AutoTokenizer
-
-from utils.infer_func import InferManager
-from axengine import InferenceSession
-
-IMAGENET_MEAN = (0.485, 0.456, 0.406)
-IMAGENET_STD = (0.229, 0.224, 0.225)
-IMG_PLACEHOLDER_TOKEN_ID = 151669  # <img>
-IMG_CONTEXT_REPEAT = 256  # number of image context tokens expected by the model
-
-
-SYSTEM_PROMPT = (
-    "<|im_start|>system\n"
-    "你是由上海人工智能实验室联合商汤科技开发的书生多模态大模型, 英文名叫 InternVL3, "
-    "是一个有用无害的人工智能助手, 擅长思考和回答用户的问题. 请你在回答问题时使用简体中文."
-    "<|im_end|>\n"
-)
-
-
-def build_transform(input_size: int):
-    transform = T.Compose([
-        T.Lambda(lambda img: img.convert("RGB") if img.mode != "RGB" else img),
-        T.Resize((input_size, input_size), interpolation=InterpolationMode.BICUBIC),
-        T.ToTensor(),
-        T.Normalize(mean=IMAGENET_MEAN, std=IMAGENET_STD),
-    ])
-    return transform
-
-
-def dynamic_preprocess(image: Image.Image, min_num: int = 1, max_num: int = 12, image_size: int = 448,
-                       use_thumbnail: bool = False):
-    orig_width, orig_height = image.size
-    aspect_ratio = orig_width / orig_height
-
-    target_ratios = set(
-        (i, j)
-        for n in range(min_num, max_num + 1)
-        for i in range(1, n + 1)
-        for j in range(1, n + 1)
-        if i * j <= max_num and i * j >= min_num
-    )
-    target_ratios = sorted(target_ratios, key=lambda x: x[0] * x[1])
-
-    def find_closest_aspect_ratio(ar: float, ratios: List[tuple]):
-        best_ratio_diff = float("inf")
-        best_ratio = (1, 1)
-        area = orig_width * orig_height
-        for ratio in ratios:
-            target_aspect_ratio = ratio[0] / ratio[1]
-            ratio_diff = abs(ar - target_aspect_ratio)
-            if ratio_diff < best_ratio_diff:
-                best_ratio_diff = ratio_diff
-                best_ratio = ratio
-            elif ratio_diff == best_ratio_diff:
-                if area > 0.5 * image_size * image_size * ratio[0] * ratio[1]:
-                    best_ratio = ratio
-        return best_ratio
-
-    target_aspect_ratio = find_closest_aspect_ratio(aspect_ratio, target_ratios)
-    target_width = image_size * target_aspect_ratio[0]
-    target_height = image_size * target_aspect_ratio[1]
-    blocks = target_aspect_ratio[0] * target_aspect_ratio[1]
-
-    resized_img = image.resize((target_width, target_height))
-    processed_images = []
-    for i in range(blocks):
-        box = (
-            (i % (target_width // image_size)) * image_size,
-            (i // (target_width // image_size)) * image_size,
-            ((i % (target_width // image_size)) + 1) * image_size,
-            ((i // (target_width // image_size)) + 1) * image_size,
-        )
-        split_img = resized_img.crop(box)
-        processed_images.append(split_img)
-    assert len(processed_images) == blocks
-    if use_thumbnail and len(processed_images) != 1:
-        processed_images.append(image.resize((image_size, image_size)))
-    return processed_images
-
-
-def load_image(image_file: Image.Image, input_size: int = 448, max_num: int = 12):
-    transform = build_transform(input_size=input_size)
-    images = dynamic_preprocess(image_file, image_size=input_size, use_thumbnail=True, max_num=max_num)
-    pixel_values = [transform(img) for img in images]
-    pixel_values = torch.stack(pixel_values)
-    return pixel_values
-
-
-class InternVLGradioDemo:
-    def __init__(self, hf_model: str, axmodel_dir: str, vit_axmodel: str, max_seq_len: int = 2047):
-        self.hf_model = hf_model
-        self.axmodel_dir = axmodel_dir
-        self.vit_axmodel = vit_axmodel
-        self.device = "cuda" if torch.cuda.is_available() else "cpu"
-
-        self.embeds = np.load(os.path.join(axmodel_dir, "model.embed_tokens.weight.npy"))
-        self.tokenizer = AutoTokenizer.from_pretrained(self.hf_model)
-        config = AutoConfig.from_pretrained(self.hf_model, trust_remote_code=True)
-        if hasattr(config, 'llm_config') and config.llm_config is not None:
-            self.cfg = config.llm_config
-        else:
-            self.cfg = config
-
-        self.vit_session = InferenceSession(self.vit_axmodel)
-        self.infer_manager = InferManager(self.cfg, self.axmodel_dir, max_seq_len=max_seq_len)
-
-    def _build_single_turn_prompt(self, user_text: str, vit_features: List[np.ndarray]):
-        prompt = SYSTEM_PROMPT
-        prompt += f"<|im_start|>user\n{user_text}"
-        for _ in vit_features:
-            prompt += "\n<img>" + "<IMG_CONTEXT>" * IMG_CONTEXT_REPEAT + "</img>"
-        prompt += "<|im_end|>\n<|im_start|>assistant\n"
-        return prompt
-
-    def _insert_vision_features(self, token_ids: List[int], prefill_data: np.ndarray, vit_features: List[np.ndarray]):
-        image_start_indices = np.where(np.array(token_ids) == IMG_PLACEHOLDER_TOKEN_ID)[0].tolist()
-        if len(image_start_indices) != len(vit_features):
-            raise ValueError("图片数量与占位符数量不一致, 请检查输入和模板生成逻辑")
-        for idx, image_start_index in enumerate(image_start_indices):
-            insert_pos = image_start_index + 1
-            prefill_data[insert_pos: insert_pos + IMG_CONTEXT_REPEAT] = vit_features[idx][0, :, :]
-        return prefill_data
-
-    def _run_model(self, prompt: str, vit_features: List[np.ndarray]):
-        """Non-streaming推理，保留以防需要一次性结果。"""
-        for k_cache in self.infer_manager.k_caches:
-            k_cache.fill(0)
-        for v_cache in self.infer_manager.v_caches:
-            v_cache.fill(0)
-
-        token_ids = self.tokenizer.encode(prompt)
-        prefill_data = np.take(self.embeds, token_ids, axis=0).astype(bfloat16)
-        if vit_features:
-            prefill_data = self._insert_vision_features(token_ids, prefill_data, vit_features)
-
-        eos_token_id = None
-        if isinstance(self.cfg.eos_token_id, list) and len(self.cfg.eos_token_id) > 1:
-            eos_token_id = self.cfg.eos_token_id
-
-        slice_len = 128
-        token_ids = self.infer_manager.prefill(self.tokenizer, token_ids, prefill_data, slice_len=slice_len)
-        return self.infer_manager.decode(
-            self.tokenizer,
-            token_ids,
-            self.embeds,
-            slice_len=slice_len,
-            eos_token_id=eos_token_id,
-            stream=False,
-        )
-
-    def _stream_generate(self, prompt: str, vit_features: List[np.ndarray]):
-        """流式生成，逐 token 产出累积文本与计时信息 (TTFT 与平均 decode ms/token)。"""
-        # reset kv cache per request
-        for k_cache in self.infer_manager.k_caches:
-            k_cache.fill(0)
-        for v_cache in self.infer_manager.v_caches:
-            v_cache.fill(0)
-
-        token_ids = self.tokenizer.encode(prompt)
-        prefill_data = np.take(self.embeds, token_ids, axis=0).astype(bfloat16)
-        if vit_features:
-            prefill_data = self._insert_vision_features(token_ids, prefill_data, vit_features)
-
-        eos_token_id = None
-        if isinstance(self.cfg.eos_token_id, list) and len(self.cfg.eos_token_id) > 1:
-            eos_token_id = self.cfg.eos_token_id
-
-        slice_len = 128
-        t_start = time.time()
-        token_ids = self.infer_manager.prefill(self.tokenizer, token_ids, prefill_data, slice_len=slice_len)
-
-        # copy decode逻辑，实现手动流式输出
-        mask = np.zeros((1, 1, self.infer_manager.max_seq_len + 1), dtype=np.float32).astype(bfloat16)
-        mask[:, :, :self.infer_manager.max_seq_len] -= 65536
-        seq_len = len(token_ids) - 1
-        if slice_len > 0:
-            mask[:, :, :seq_len] = 0
-
-        ttft_ms: Optional[float] = None
-        decode_tokens = 0
-        decode_elapsed_ms: float = 0.0
-        generated_text = ""
-        yield generated_text, ttft_ms, None, None, False
-
-        for step_idx in range(self.infer_manager.max_seq_len):
-            if slice_len > 0 and step_idx < seq_len:
-                continue
-            cur_token = token_ids[step_idx]
-            indices = np.array([step_idx], np.uint32).reshape((1, 1))
-            data = self.embeds[cur_token, :].reshape((1, 1, self.cfg.hidden_size)).astype(bfloat16)
-            for layer_idx in range(self.cfg.num_hidden_layers):
-                input_feed = {
-                    "K_cache": self.infer_manager.k_caches[layer_idx],
-                    "V_cache": self.infer_manager.v_caches[layer_idx],
-                    "indices": indices,
-                    "input": data,
-                    "mask": mask,
-                }
-                outputs = self.infer_manager.decoder_sessions[layer_idx].run(None, input_feed, shape_group=0)
-                self.infer_manager.k_caches[layer_idx][:, step_idx, :] = outputs[0][:, :, :]
-                self.infer_manager.v_caches[layer_idx][:, step_idx, :] = outputs[1][:, :, :]
-                data = outputs[2]
-            mask[..., step_idx] = 0
-            if step_idx < seq_len - 1:
-                continue
-            post_out = self.infer_manager.post_process_session.run(None, {"input": data})[0]
-            next_token, possible_tokens, possible_probs = self.infer_manager.post_process(post_out, temperature=0.7)
-            if eos_token_id is not None and next_token in eos_token_id:
-                ttft_ms = ttft_ms or (time.time() - t_start) * 1000
-                break
-            if next_token == self.tokenizer.eos_token_id:
-                ttft_ms = ttft_ms or (time.time() - t_start) * 1000
-                break
-
-            token_ids.append(next_token)
-            # 使用完整 token 列表解码，避免多字节 UTF-8 字符被截断显示为乱码
-            # 只解码新生成的 tokens（从 seq_len 开始）
-            generated_text = self.tokenizer.decode(token_ids[seq_len:], skip_special_tokens=True)
-
-            if ttft_ms is None:
-                ttft_ms = (time.time() - t_start) * 1000
-            else:
-                decode_tokens += 1
-                decode_elapsed_ms = (time.time() - t_start) * 1000 - ttft_ms
-
-            avg_decode = (decode_elapsed_ms / decode_tokens) if decode_tokens > 0 else None
-            yield generated_text, ttft_ms, avg_decode, decode_tokens, False
-
-        total_ms = (time.time() - t_start) * 1000
-        avg_decode = (decode_elapsed_ms / decode_tokens) if decode_tokens > 0 else None
-        yield generated_text, ttft_ms, avg_decode, decode_tokens, True
-
-    def chat(self, user_input: str, image: Optional[Image.Image]) -> Generator:
-        user_text = (user_input or "").strip()
-        if not user_text and image is None:
-            yield [], gr.update(), gr.update(), gr.update(), gr.update()
-            return
-
-        # 先展示占位，保持图片不清空；同时占位速度信息
-        yield [(user_text, "处理中…")], gr.update(value=""), gr.update(), gr.update(value="<div style='text-align: right; font-size: 13px; color: #6b7280; font-family: monospace;'>TTFT -- ms&nbsp;&nbsp;|&nbsp;&nbsp;Decode -- ms/token&nbsp;&nbsp;|&nbsp;&nbsp;Tokens --</div>"), gr.update(interactive=False)
-
-        vit_outputs = []
-        if image is not None:
-            pixel_values = load_image(image, input_size=448, max_num=1)
-            vit_output = self.vit_session.run(None, {"image": pixel_values.numpy()})[0]
-            vit_outputs.append(vit_output.copy())
-
-        prompt = self._build_single_turn_prompt(user_text, vit_outputs)
-
-        chatbot_history = [(user_text, "")]  # 将在流式过程中填充
-        for partial, ttft_ms, avg_decode_ms, decode_tokens, finished in self._stream_generate(prompt, vit_outputs):
-            chatbot_history[-1] = (user_text, partial)
-            ttft_disp = f"{ttft_ms:.0f}" if ttft_ms is not None else "--"
-            decode_disp = f"{avg_decode_ms:.1f}" if avg_decode_ms is not None else "--"
-            tok_disp = f"{decode_tokens}" if decode_tokens is not None else "--"
-            metrics_text = f"<div style='text-align: right; font-size: 13px; color: #6b7280; font-family: monospace;'>TTFT {ttft_disp} ms&nbsp;&nbsp;|&nbsp;&nbsp;Decode {decode_disp} ms/token&nbsp;&nbsp;|&nbsp;&nbsp;Tokens {tok_disp}</div>"
-            if finished:
-                yield chatbot_history, gr.update(value=""), gr.update(), gr.update(value=metrics_text), gr.update(interactive=True)
-            else:
-                yield chatbot_history, gr.update(value=""), gr.update(), gr.update(value=metrics_text), gr.update(interactive=False)
-
-    @staticmethod
-    def build_ui(demo: "InternVLGradioDemo", server_name: str = "0.0.0.0", server_port: int = 7860, share: bool = False):
-        # 自定义 JavaScript: Enter 发送, Shift+Enter 换行
-        custom_js = """
-        function() {
-            // 等待 DOM 加载完成后绑定事件
-            setTimeout(() => {
-                const textareas = document.querySelectorAll('#user-input textarea');
-                textareas.forEach(textarea => {
-                    // 移除可能存在的旧监听器
-                    textarea.removeEventListener('keydown', textarea._customKeyHandler);
-
-                    textarea._customKeyHandler = function(e) {
-                        if (e.key === 'Enter') {
-                            if (e.shiftKey) {
-                                // Shift+Enter: 插入换行符
-                                e.preventDefault();
-                                const start = this.selectionStart;
-                                const end = this.selectionEnd;
-                                const value = this.value;
-                                this.value = value.substring(0, start) + '\\n' + value.substring(end);
-                                this.selectionStart = this.selectionEnd = start + 1;
-                                // 触发 input 事件让 Gradio 感知变化
-                                this.dispatchEvent(new Event('input', { bubbles: true }));
-                            } else {
-                                // Enter: 发送消息
-                                e.preventDefault();
-                                const sendBtn = document.querySelector('#send-btn');
-                                if (sendBtn) {
-                                    sendBtn.click();
-                                }
-                            }
-                        }
-                    };
-                    textarea.addEventListener('keydown', textarea._customKeyHandler);
-                });
-            }, 500);
-        }
-        """
-
-        with gr.Blocks(title="InternVL3_5-2B_GPTQ_INT4 AX Gradio Demo", theme=gr.themes.Soft(), js=custom_js) as iface:
-            gr.HTML("""<style>
-            #image-pane img {object-fit: contain; max-height: 380px;}
-            #chat-wrap {position: relative;}
-            #metrics-display {position: absolute; right: 12px; bottom: 12px; z-index: 5; pointer-events: none; text-align: right;}
-            #metrics-display > div {display: inline-block;}
-            </style>""")
-            gr.Markdown("""### InternVL3_5-2B_GPTQ_INT4 图文对话演示\n上传一张图片 (可选)，输入问题，获取中文回答。""")
-
-            with gr.Row():
-                # 左侧：对话框和输入区域
-                with gr.Column(scale=5):
-                    with gr.Group(elem_id="chat-wrap"):
-                        chatbot = gr.Chatbot(height=500, label="对话")
-                        metrics_md = gr.Markdown("<div style='text-align: right; font-size: 13px; color: #6b7280; font-family: monospace;'>TTFT -- ms&nbsp;&nbsp;|&nbsp;&nbsp;Decode -- ms/token&nbsp;&nbsp;|&nbsp;&nbsp;Tokens --</div>", elem_id="metrics-display")
-
-                    with gr.Row():
-                        user_input = gr.Textbox(
-                            placeholder="按 Enter 发送，Shift+Enter 换行",
-                            lines=2,
-                            scale=7,
-                            max_lines=5,
-                            show_label=False,
-                            elem_id="user-input",
-                        )
-                        with gr.Column(scale=1, min_width=100):
-                            send_btn = gr.Button("发送", variant="primary", size="sm", elem_id="send-btn")
-                            clear_btn = gr.Button("清空对话", variant="secondary", size="sm")
-
-                # 右侧：图像上传和信息提示
-                with gr.Column(scale=3):
-                    image_input = gr.Image(
-                        type="pil",
-                        label="上传图片 (可选)",
-                        height=380,
-                        image_mode="RGB",
-                        show_download_button=False,
-                        elem_id="image-pane",
-                    )
-                    gr.Markdown("""- 支持单张图像理解\n- 仅当前问题与回答，不保留历史\n- 处理时间取决于硬件，请耐心等待""")
-
-            def _clear():
-                return [], gr.update(value=""), gr.update(), gr.update(value="<div style='text-align: right; font-size: 13px; color: #6b7280; font-family: monospace;'>TTFT -- ms&nbsp;&nbsp;|&nbsp;&nbsp;Decode -- ms/token&nbsp;&nbsp;|&nbsp;&nbsp;Tokens --</div>"), gr.update(interactive=True)
-
-            send_btn.click(
-                fn=demo.chat,
-                inputs=[user_input, image_input],
-                outputs=[chatbot, user_input, image_input, metrics_md, send_btn],
-                show_progress=False,
-                queue=True,
-            )
-            # 移除 user_input.submit，由自定义 JS 处理 Enter 发送，Shift+Enter 换行
-            clear_btn.click(fn=_clear, inputs=None, outputs=[chatbot, user_input, image_input, metrics_md, send_btn])
-
-        iface.queue().launch(server_name=server_name, server_port=server_port, share=share)
-
-
-def parse_args():
-    parser = argparse.ArgumentParser(description="InternVL3-5-2B AX gradio demo")
-    parser.add_argument("--hf_model", type=str, default="./InternVL3_5-2B",
-                        help="HuggingFace 模型路径")
-    parser.add_argument("--axmodel_path", type=str, default="./InternVL3_5-2B_axmodel",
-                        help="LLM axmodel 目录")
-    parser.add_argument("--vit_model", type=str, default="./vit-models/internvl_vit_model_1x3x448x448.axmodel",
-                        help="ViT axmodel 路径")
-    parser.add_argument("--port", type=int, default=7860, help="Gradio 端口")
-    parser.add_argument("--host", type=str, default="0.0.0.0", help="Gradio 监听地址")
-    parser.add_argument("--share", action="store_true", help="启用 gradio share")
-    return parser.parse_args()
-
-
-def main():
-    args = parse_args()
-    demo = InternVLGradioDemo(args.hf_model, args.axmodel_path, args.vit_model)
-    InternVLGradioDemo.build_ui(demo, server_name=args.host, server_port=args.port, share=args.share)
-
-
-if __name__ == "__main__":
-    main()

infer_axmodel.py

@@ -1,186 +0,0 @@
-from transformers import AutoProcessor, AutoModelForImageTextToText
-import torch
-import onnx
-import onnxruntime as ort
-import numpy as np
-import os
-from tqdm import tqdm
-from transformers import AutoConfig, AutoTokenizer
-from typing import List, Tuple
-from axengine import InferenceSession
-from ml_dtypes import bfloat16
-from utils.infer_func import InferManager
-import argparse
-from PIL import Image
-import torchvision.transforms as T
-from torchvision.transforms.functional import InterpolationMode
-
-
-IMAGENET_MEAN = (0.485, 0.456, 0.406)
-IMAGENET_STD = (0.229, 0.224, 0.225)
-
-def build_transform(input_size):
-    MEAN, STD = IMAGENET_MEAN, IMAGENET_STD
-    transform = T.Compose([
-        T.Lambda(lambda img: img.convert('RGB') if img.mode != 'RGB' else img),
-        T.Resize((input_size, input_size), interpolation=InterpolationMode.BICUBIC),
-        T.ToTensor(),
-        T.Normalize(mean=MEAN, std=STD)
-    ])
-    return transform
-
-def find_closest_aspect_ratio(aspect_ratio, target_ratios, width, height, image_size):
-    best_ratio_diff = float('inf')
-    best_ratio = (1, 1)
-    area = width * height
-    for ratio in target_ratios:
-        target_aspect_ratio = ratio[0] / ratio[1]
-        ratio_diff = abs(aspect_ratio - target_aspect_ratio)
-        if ratio_diff < best_ratio_diff:
-            best_ratio_diff = ratio_diff
-            best_ratio = ratio
-        elif ratio_diff == best_ratio_diff:
-            if area > 0.5 * image_size * image_size * ratio[0] * ratio[1]:
-                best_ratio = ratio
-    return best_ratio
-
-def dynamic_preprocess(image, min_num=1, max_num=12, image_size=448, use_thumbnail=False):
-    orig_width, orig_height = image.size
-    aspect_ratio = orig_width / orig_height
-
-    # calculate the existing image aspect ratio
-    target_ratios = set(
-        (i, j) for n in range(min_num, max_num + 1) for i in range(1, n + 1) for j in range(1, n + 1) if
-        i * j <= max_num and i * j >= min_num)
-    target_ratios = sorted(target_ratios, key=lambda x: x[0] * x[1])
-
-    # find the closest aspect ratio to the target
-    target_aspect_ratio = find_closest_aspect_ratio(
-        aspect_ratio, target_ratios, orig_width, orig_height, image_size)
-
-    # calculate the target width and height
-    target_width = image_size * target_aspect_ratio[0]
-    target_height = image_size * target_aspect_ratio[1]
-    blocks = target_aspect_ratio[0] * target_aspect_ratio[1]
-
-    # resize the image
-    resized_img = image.resize((target_width, target_height))
-    processed_images = []
-    for i in range(blocks):
-        box = (
-            (i % (target_width // image_size)) * image_size,
-            (i // (target_width // image_size)) * image_size,
-            ((i % (target_width // image_size)) + 1) * image_size,
-            ((i // (target_width // image_size)) + 1) * image_size
-        )
-        # split the image
-        split_img = resized_img.crop(box)
-        processed_images.append(split_img)
-    assert len(processed_images) == blocks
-    if use_thumbnail and len(processed_images) != 1:
-        thumbnail_img = image.resize((image_size, image_size))
-        processed_images.append(thumbnail_img)
-    return processed_images
-
-def load_image(image_file, input_size=448, max_num=12):
-    image = Image.open(image_file).convert('RGB')
-    transform = build_transform(input_size=input_size)
-    images = dynamic_preprocess(image, image_size=input_size, use_thumbnail=True, max_num=max_num)
-    pixel_values = [transform(image) for image in images]
-    pixel_values = torch.stack(pixel_values)
-    return pixel_values
-
-if __name__ == "__main__":
-
-    """
-    python3 infer_axmodel.py  --vit_model vit-models/internvl_vit_model_1x3x448x448.axmodel --images examples/image_0.jpg
-    """
-    prompt = None
-    parser = argparse.ArgumentParser(description="Model configuration parameters")
-    parser.add_argument("--hf_model", type=str, default="./InternVL3_5-1B",
-                        help="Path to HuggingFace model")
-    parser.add_argument("--axmodel_path", type=str, default="./InternVL3_5-1B_axmodel",
-                        help="Path to save compiled axmodel of llama model")
-    parser.add_argument("--vit_model", type=str, default=None, help="Path to save compiled axmodel of llama model")
-    parser.add_argument("-i", "--images", nargs='+', type=str, default=None,
-                        help="Path to the test image.")
-    parser.add_argument("-q", "--question", type=str, default="请你描述这幅图的内容.",
-                        help="Your question that you want to ask the model.")
-    args = parser.parse_args()
-
-    hf_model_path = args.hf_model
-    axmodel_path = args.axmodel_path
-    images = args.images
-    prompt = args.question
-
-    device = "cuda" if torch.cuda.is_available() else "cpu"
-    embeds = np.load(os.path.join(axmodel_path, "model.embed_tokens.weight.npy"))
-
-    # load the tokenizer and the model
-    tokenizer = AutoTokenizer.from_pretrained(hf_model_path)
-    config = AutoConfig.from_pretrained(hf_model_path, trust_remote_code=True)
-
-    # model = AutoModelForCausalLM.from_pretrained(
-    #     hf_model_path,
-    # ).to(device)
-
-    test_imgs_path = args.images
-    vit_axmodel_path = args.vit_model
-
-    # set the max number of tiles in `max_num`
-    pixel_values_list = []
-    if test_imgs_path is not None:
-        for img_path in test_imgs_path:
-            pixel_values = load_image(img_path, input_size=448, max_num=1)
-            pixel_values_list.append(pixel_values)
-        print(f"输入图像数: {len(pixel_values_list)}")
-        print("preprocess image done!")
-
-        # extract img feature by vit
-        vit_session = InferenceSession(vit_axmodel_path)
-        vit_output_list = []
-        for idx, pixel_values in enumerate(pixel_values_list):
-            vit_output = vit_session.run(None, {"image": pixel_values.numpy()})[0]
-            vit_output_list.append(vit_output.copy()) # 避免 vit 输出结果使用同一块内存
-
-        print(f"vit_output.shape is {vit_output_list[0].shape}, vit feature extract done!")
-
-    prompt = "<|im_start|>system\n你是由上海人工智能实验室联合商汤科技开发的书生多模态大模型, 英文名叫 InternVL3, 是一个有用无害的人工智能助手, 擅长思考和回答用户的问题. 请你在回答问题时使用简体中文.<|im_end|>\n"
-    question = args.question
-    prompt += "<|im_start|>user\n" + question
-
-    if len(pixel_values_list) > 0:
-        for idx in range(len(pixel_values_list)):
-            prompt += "\n<img>" + "<IMG_CONTEXT>" * 256 + "</img>\n"
-    prompt += "<|im_end|>\n<|im_start|>assistant\n"
-    print(f"prompt is {prompt}")
-    token_ids = tokenizer.encode(prompt)
-    # 图像理解
-    image_start_indices = np.where(np.array(token_ids) == 151669)[0].tolist() # <img> tag 151669, 151665
-    prefill_data = np.take(embeds, token_ids, axis=0)
-    prefill_data = prefill_data.astype(bfloat16)
-    token_len = len(token_ids)
-
-    for idx, image_start_index in enumerate(image_start_indices):
-        image_insert_index = image_start_index + 1
-        prefill_data[image_insert_index : image_insert_index + 256] = vit_output_list[idx][0, :, :]
-    ##################################
-
-    if hasattr(config, 'llm_config') and config.llm_config is not None: # 兼容 GPTQ INT4 模型
-        cfg = config.llm_config
-    else:
-        cfg = config
-
-    eos_token_id = None
-    if isinstance(cfg.eos_token_id, list) and len(cfg.eos_token_id) > 1:
-        eos_token_id = cfg.eos_token_id
-
-    slice_len = 128
-    prefill_max_len = 1024 - 1
-    max_seq_len = 2048 - 1  # prefill + decode max length
-
-    imer = InferManager(cfg, axmodel_path, max_seq_len=max_seq_len) # prefill + decode max length
-    # import pdb; pdb.set_trace()
-    token_ids = imer.prefill(tokenizer, token_ids, prefill_data, slice_len=slice_len)
-    imer.decode(tokenizer, token_ids, embeds, slice_len=slice_len, eos_token_id=eos_token_id)
-    print("\n")

infer_torch.py

@@ -1,212 +0,0 @@
-import math
-import numpy as np
-import torch
-import torchvision.transforms as T
-from decord import VideoReader, cpu
-from PIL import Image
-from torchvision.transforms.functional import InterpolationMode
-from transformers import AutoModel, AutoTokenizer
-
-IMAGENET_MEAN = (0.485, 0.456, 0.406)
-IMAGENET_STD = (0.229, 0.224, 0.225)
-
-def build_transform(input_size):
-    MEAN, STD = IMAGENET_MEAN, IMAGENET_STD
-    transform = T.Compose([
-        T.Lambda(lambda img: img.convert('RGB') if img.mode != 'RGB' else img),
-        T.Resize((input_size, input_size), interpolation=InterpolationMode.BICUBIC),
-        T.ToTensor(),
-        T.Normalize(mean=MEAN, std=STD)
-    ])
-    return transform
-
-def find_closest_aspect_ratio(aspect_ratio, target_ratios, width, height, image_size):
-    best_ratio_diff = float('inf')
-    best_ratio = (1, 1)
-    area = width * height
-    for ratio in target_ratios:
-        target_aspect_ratio = ratio[0] / ratio[1]
-        ratio_diff = abs(aspect_ratio - target_aspect_ratio)
-        if ratio_diff < best_ratio_diff:
-            best_ratio_diff = ratio_diff
-            best_ratio = ratio
-        elif ratio_diff == best_ratio_diff:
-            if area > 0.5 * image_size * image_size * ratio[0] * ratio[1]:
-                best_ratio = ratio
-    return best_ratio
-
-def dynamic_preprocess(image, min_num=1, max_num=12, image_size=448, use_thumbnail=False):
-    orig_width, orig_height = image.size
-    aspect_ratio = orig_width / orig_height
-
-    # calculate the existing image aspect ratio
-    target_ratios = set(
-        (i, j) for n in range(min_num, max_num + 1) for i in range(1, n + 1) for j in range(1, n + 1) if
-        i * j <= max_num and i * j >= min_num)
-    target_ratios = sorted(target_ratios, key=lambda x: x[0] * x[1])
-
-    # find the closest aspect ratio to the target
-    target_aspect_ratio = find_closest_aspect_ratio(
-        aspect_ratio, target_ratios, orig_width, orig_height, image_size)
-
-    # calculate the target width and height
-    target_width = image_size * target_aspect_ratio[0]
-    target_height = image_size * target_aspect_ratio[1]
-    blocks = target_aspect_ratio[0] * target_aspect_ratio[1]
-
-    # resize the image
-    resized_img = image.resize((target_width, target_height))
-    processed_images = []
-    for i in range(blocks):
-        box = (
-            (i % (target_width // image_size)) * image_size,
-            (i // (target_width // image_size)) * image_size,
-            ((i % (target_width // image_size)) + 1) * image_size,
-            ((i // (target_width // image_size)) + 1) * image_size
-        )
-        # split the image
-        split_img = resized_img.crop(box)
-        processed_images.append(split_img)
-    assert len(processed_images) == blocks
-    if use_thumbnail and len(processed_images) != 1:
-        thumbnail_img = image.resize((image_size, image_size))
-        processed_images.append(thumbnail_img)
-    return processed_images
-
-def load_image(image_file, input_size=448, max_num=12):
-    image = Image.open(image_file).convert('RGB')
-    transform = build_transform(input_size=input_size)
-    images = dynamic_preprocess(image, image_size=input_size, use_thumbnail=True, max_num=max_num)
-    pixel_values = [transform(image) for image in images]
-    pixel_values = torch.stack(pixel_values)
-    return pixel_values
-
-path = './InternVL3_5-1B'
-model = AutoModel.from_pretrained(
-    path,
-    torch_dtype=torch.bfloat16,
-    load_in_8bit=False,
-    low_cpu_mem_usage=True,
-    use_flash_attn=True,
-    trust_remote_code=True,
-    device_map="auto").eval()
-tokenizer = AutoTokenizer.from_pretrained(path, trust_remote_code=True, use_fast=False)
-
-# set the max number of tiles in `max_num`
-pixel_values = load_image('./examples/image_1.jpg', input_size=448, max_num=1).to(torch.bfloat16).cuda()
-generation_config = dict(max_new_tokens=1024, do_sample=True)
-
-# pure-text conversation (纯文本对话)
-question = '中国的首都'
-response, history = model.chat(tokenizer, None, question, generation_config, history=None, return_history=True)
-print(f'User: {question}\nAssistant: {response}')
-
-
-# single-image single-round conversation (单图单轮对话)
-question = '<image>\n请你描述这幅图的内容.'
-response = model.chat(tokenizer, pixel_values, question, generation_config)
-print(f'User: {question}\nAssistant: {response}')
-
-# # single-image multi-round conversation (单图多轮对话)
-# question = '<image>\nPlease describe the image in detail.'
-# response, history = model.chat(tokenizer, pixel_values, question, generation_config, history=None, return_history=True)
-# print(f'User: {question}\nAssistant: {response}')
-
-# question = 'Please write a poem according to the image.'
-# response, history = model.chat(tokenizer, pixel_values, question, generation_config, history=history, return_history=True)
-# print(f'User: {question}\nAssistant: {response}')
-
-# # multi-image multi-round conversation, combined images (多图多轮对话，拼接图像)
-# pixel_values1 = load_image('./examples/image1.jpg', max_num=12).to(torch.bfloat16).cuda()
-# pixel_values2 = load_image('./examples/image2.jpg', max_num=12).to(torch.bfloat16).cuda()
-# pixel_values = torch.cat((pixel_values1, pixel_values2), dim=0)
-
-# question = '<image>\nDescribe the two images in detail.'
-# response, history = model.chat(tokenizer, pixel_values, question, generation_config,
-#                                history=None, return_history=True)
-# print(f'User: {question}\nAssistant: {response}')
-
-# question = 'What are the similarities and differences between these two images.'
-# response, history = model.chat(tokenizer, pixel_values, question, generation_config,
-#                                history=history, return_history=True)
-# print(f'User: {question}\nAssistant: {response}')
-
-# # multi-image multi-round conversation, separate images (多图多轮对话，独立图像)
-# pixel_values1 = load_image('./examples/image1.jpg', max_num=12).to(torch.bfloat16).cuda()
-# pixel_values2 = load_image('./examples/image2.jpg', max_num=12).to(torch.bfloat16).cuda()
-# pixel_values = torch.cat((pixel_values1, pixel_values2), dim=0)
-# num_patches_list = [pixel_values1.size(0), pixel_values2.size(0)]
-
-# question = 'Image-1: <image>\nImage-2: <image>\nDescribe the two images in detail.'
-# response, history = model.chat(tokenizer, pixel_values, question, generation_config,
-#                                num_patches_list=num_patches_list,
-#                                history=None, return_history=True)
-# print(f'User: {question}\nAssistant: {response}')
-
-# question = 'What are the similarities and differences between these two images.'
-# response, history = model.chat(tokenizer, pixel_values, question, generation_config,
-#                                num_patches_list=num_patches_list,
-#                                history=history, return_history=True)
-# print(f'User: {question}\nAssistant: {response}')
-
-# # batch inference, single image per sample (单图批处理)
-# pixel_values1 = load_image('./examples/image1.jpg', max_num=12).to(torch.bfloat16).cuda()
-# pixel_values2 = load_image('./examples/image2.jpg', max_num=12).to(torch.bfloat16).cuda()
-# num_patches_list = [pixel_values1.size(0), pixel_values2.size(0)]
-# pixel_values = torch.cat((pixel_values1, pixel_values2), dim=0)
-
-# questions = ['<image>\nDescribe the image in detail.'] * len(num_patches_list)
-# responses = model.batch_chat(tokenizer, pixel_values,
-#                              num_patches_list=num_patches_list,
-#                              questions=questions,
-#                              generation_config=generation_config)
-# for question, response in zip(questions, responses):
-#     print(f'User: {question}\nAssistant: {response}')
-
-# # video multi-round conversation (视频多轮对话)
-# def get_index(bound, fps, max_frame, first_idx=0, num_segments=32):
-#     if bound:
-#         start, end = bound[0], bound[1]
-#     else:
-#         start, end = -100000, 100000
-#     start_idx = max(first_idx, round(start * fps))
-#     end_idx = min(round(end * fps), max_frame)
-#     seg_size = float(end_idx - start_idx) / num_segments
-#     frame_indices = np.array([
-#         int(start_idx + (seg_size / 2) + np.round(seg_size * idx))
-#         for idx in range(num_segments)
-#     ])
-#     return frame_indices
-
-# def load_video(video_path, bound=None, input_size=448, max_num=1, num_segments=32):
-#     vr = VideoReader(video_path, ctx=cpu(0), num_threads=1)
-#     max_frame = len(vr) - 1
-#     fps = float(vr.get_avg_fps())
-
-#     pixel_values_list, num_patches_list = [], []
-#     transform = build_transform(input_size=input_size)
-#     frame_indices = get_index(bound, fps, max_frame, first_idx=0, num_segments=num_segments)
-#     for frame_index in frame_indices:
-#         img = Image.fromarray(vr[frame_index].asnumpy()).convert('RGB')
-#         img = dynamic_preprocess(img, image_size=input_size, use_thumbnail=True, max_num=max_num)
-#         pixel_values = [transform(tile) for tile in img]
-#         pixel_values = torch.stack(pixel_values)
-#         num_patches_list.append(pixel_values.shape[0])
-#         pixel_values_list.append(pixel_values)
-#     pixel_values = torch.cat(pixel_values_list)
-#     return pixel_values, num_patches_list
-
-# video_path = './examples/red-panda.mp4'
-# pixel_values, num_patches_list = load_video(video_path, num_segments=8, max_num=1)
-# pixel_values = pixel_values.to(torch.bfloat16).cuda()
-# video_prefix = ''.join([f'Frame{i+1}: <image>\n' for i in range(len(num_patches_list))])
-# question = video_prefix + 'What is the red panda doing?'
-# # Frame1: <image>\nFrame2: <image>\n...\nFrame8: <image>\n{question}
-# response, history = model.chat(tokenizer, pixel_values, question, generation_config,
-#                                num_patches_list=num_patches_list, history=None, return_history=True)
-# print(f'User: {question}\nAssistant: {response}')
-
-# question = 'Describe this video in detail.'
-# response, history = model.chat(tokenizer, pixel_values, question, generation_config,
-#                                num_patches_list=num_patches_list, history=history, return_history=True)
-# print(f'User: {question}\nAssistant: {response}')

internvl3-5_axmodel/model.embed_tokens.weight.float32.bin

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:1604cef8ba75bc3c615e8b2853734464f54a26abea11f441e98f18fc49be24ab
-size 1244659712

internvl3-5_axmodel/model.embed_tokens.weight.npy

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:0fd10721363fc0a9e0bd780faad607e032524fb4f6bcccf78068a5f7fe5319fc
-size 1244659840

internvl3-5_tokenizer/added_tokens.json

@@ -1,37 +0,0 @@
-{
-  "</box>": 151677,
-  "</img>": 151670,
-  "</quad>": 151673,
-  "</ref>": 151675,
-  "</think>": 151668,
-  "</tool_call>": 151658,
-  "</tool_response>": 151666,
-  "<IMG_CONTEXT>": 151671,
-  "<box>": 151676,
-  "<img>": 151669,
-  "<quad>": 151672,
-  "<ref>": 151674,
-  "<think>": 151667,
-  "<tool_call>": 151657,
-  "<tool_response>": 151665,
-  "<|box_end|>": 151649,
-  "<|box_start|>": 151648,
-  "<|endoftext|>": 151643,
-  "<|file_sep|>": 151664,
-  "<|fim_middle|>": 151660,
-  "<|fim_pad|>": 151662,
-  "<|fim_prefix|>": 151659,
-  "<|fim_suffix|>": 151661,
-  "<|im_end|>": 151645,
-  "<|im_start|>": 151644,
-  "<|image_pad|>": 151655,
-  "<|object_ref_end|>": 151647,
-  "<|object_ref_start|>": 151646,
-  "<|quad_end|>": 151651,
-  "<|quad_start|>": 151650,
-  "<|repo_name|>": 151663,
-  "<|video_pad|>": 151656,
-  "<|vision_end|>": 151653,
-  "<|vision_pad|>": 151654,
-  "<|vision_start|>": 151652
-}

internvl3-5_tokenizer/config.json

@@ -1,89 +0,0 @@
-{
-  "architectures": [
-    "Qwen3ForCausalLM"
-  ],
-  "attention_bias": false,
-  "attention_dropout": 0.0,
-  "bos_token_id": 151643,
-  "debug": false,
-  "dtype": "bfloat16",
-  "eos_token_id": 151645,
-  "ep_size": 1,
-  "head_dim": 128,
-  "hidden_act": "silu",
-  "hidden_size": 2048,
-  "initializer_range": 0.02,
-  "intermediate_size": 6144,
-  "layer_types": [
-    "full_attention",
-    "full_attention",
-    "full_attention",
-    "full_attention",
-    "full_attention",
-    "full_attention",
-    "full_attention",
-    "full_attention",
-    "full_attention",
-    "full_attention",
-    "full_attention",
-    "full_attention",
-    "full_attention",
-    "full_attention",
-    "full_attention",
-    "full_attention",
-    "full_attention",
-    "full_attention",
-    "full_attention",
-    "full_attention",
-    "full_attention",
-    "full_attention",
-    "full_attention",
-    "full_attention",
-    "full_attention",
-    "full_attention",
-    "full_attention",
-    "full_attention"
-  ],
-  "max_position_embeddings": 40960,
-  "max_window_layers": 28,
-  "micro_forward": false,
-  "model_type": "qwen3",
-  "num_attention_heads": 16,
-  "num_hidden_layers": 28,
-  "num_key_value_heads": 8,
-  "quantization_config": {
-    "bits": 4,
-    "checkpoint_format": "gptq",
-    "desc_act": false,
-    "group_size": 128,
-    "lm_head": false,
-    "meta": {
-      "act_group_aware": false,
-      "damp_auto_increment": 0.01,
-      "damp_percent": 0.01,
-      "mse": 0.0,
-      "quantizer": [
-        "gptqmodel:5.0.0-dev0"
-      ],
-      "static_groups": false,
-      "true_sequential": true,
-      "uri": "https://github.com/modelcloud/gptqmodel",
-      "v2": false,
-      "v2_alpha": 0.25
-    },
-    "pack_dtype": "int32",
-    "quant_method": "gptq",
-    "sym": true
-  },
-  "rms_norm_eps": 1e-06,
-  "rope_scaling": null,
-  "rope_theta": 1000000,
-  "skip_checkpoint": false,
-  "sliding_window": null,
-  "tie_word_embeddings": false,
-  "transformers_version": "4.56.2",
-  "use_cache": false,
-  "use_deepep": false,
-  "use_sliding_window": false,
-  "vocab_size": 151936
-}

internvl3-5_tokenizer/generation_config.json

@@ -1,4 +0,0 @@
-{
-  "_from_model_config": true,
-  "transformers_version": "4.56.2"
-}

internvl3-5_tokenizer/quant_eval_results.json

@@ -1,6 +0,0 @@
-{
-  "cosine_mean": 0.96240234375,
-  "cosine_std": 0.0167236328125,
-  "cosine_min": 0.89794921875,
-  "cosine_max": 0.9912109375
-}

internvl3-5_tokenizer/quant_log.csv

@@ -1,197 +0,0 @@
-layer,module,loss,samples,damp,time
-0,self_attn.q_proj,0.0000028287,0.01000,1.122
-0,self_attn.k_proj,0.0000013575,0.01000,0.480
-0,self_attn.v_proj,0.0000011600,0.01000,0.476
-0,self_attn.o_proj,0.0000036256,0.01000,0.470
-0,mlp.gate_proj,0.0000437549,0.01000,0.482
-0,mlp.up_proj,0.0000267884,0.01000,0.481
-0,mlp.down_proj,0.0000072039,0.01000,1.513
-1,self_attn.q_proj,0.0000029302,0.01000,0.817
-1,self_attn.k_proj,0.0000015003,0.01000,0.502
-1,self_attn.v_proj,0.0000014158,0.01000,0.500
-1,self_attn.o_proj,0.0000016853,0.01000,0.489
-1,mlp.gate_proj,0.0009522146,0.01000,0.493
-1,mlp.up_proj,0.0003441322,0.01000,0.482
-1,mlp.down_proj,0.0000125897,0.01000,1.510
-2,self_attn.q_proj,0.0000063560,0.01000,0.669
-2,self_attn.k_proj,0.0000029401,0.01000,0.483
-2,self_attn.v_proj,0.0000029013,0.01000,0.483
-2,self_attn.o_proj,0.0000026976,0.01000,0.491
-2,mlp.gate_proj,0.0007206563,0.01000,0.481
-2,mlp.up_proj,0.0002565878,0.01000,0.482
-2,mlp.down_proj,0.0675307913,0.01000,1.505
-3,self_attn.q_proj,0.0000384238,0.01000,0.648
-3,self_attn.k_proj,0.0000200568,0.01000,0.485
-3,self_attn.v_proj,0.0000195524,0.01000,0.483
-3,self_attn.o_proj,0.0000032158,0.01000,0.475
-3,mlp.gate_proj,0.0008880800,0.01000,0.484
-3,mlp.up_proj,0.0003446824,0.01000,0.481
-3,mlp.down_proj,0.0000226135,0.01000,1.513
-4,self_attn.q_proj,0.0000386142,0.01000,0.654
-4,self_attn.k_proj,0.0000196975,0.01000,0.490
-4,self_attn.v_proj,0.0000199153,0.01000,0.491
-4,self_attn.o_proj,0.0000056813,0.01000,0.482
-4,mlp.gate_proj,0.0006523710,0.01000,0.485
-4,mlp.up_proj,0.0002876575,0.01000,0.491
-4,mlp.down_proj,0.0000332622,0.01000,1.537
-5,self_attn.q_proj,0.0000574806,0.01000,0.686
-5,self_attn.k_proj,0.0000248645,0.01000,0.490
-5,self_attn.v_proj,0.0000303848,0.01000,0.491
-5,self_attn.o_proj,0.0000087608,0.01000,0.482
-5,mlp.gate_proj,0.0002589391,0.01000,0.487
-5,mlp.up_proj,0.0001756437,0.01000,0.484
-5,mlp.down_proj,0.0000397888,0.01000,1.526
-6,self_attn.q_proj,0.0000415340,0.01000,0.656
-6,self_attn.k_proj,0.0000201673,0.01000,0.490
-6,self_attn.v_proj,0.0000206473,0.01000,0.488
-6,self_attn.o_proj,0.0000094276,0.01000,0.481
-6,mlp.gate_proj,0.0003516497,0.01000,0.488
-6,mlp.up_proj,0.0002418573,0.01000,0.487
-6,mlp.down_proj,0.0000589585,0.01000,1.537
-7,self_attn.q_proj,0.0000895868,0.01000,0.651
-7,self_attn.k_proj,0.0000390398,0.01000,0.490
-7,self_attn.v_proj,0.0000468701,0.01000,0.488
-7,self_attn.o_proj,0.0000154576,0.01000,0.478
-7,mlp.gate_proj,0.0004709592,0.01000,0.484
-7,mlp.up_proj,0.0003113569,0.01000,0.490
-7,mlp.down_proj,0.0000668873,0.01000,1.534
-8,self_attn.q_proj,0.0001087862,0.01000,0.669
-8,self_attn.k_proj,0.0000534895,0.01000,0.489
-8,self_attn.v_proj,0.0000521546,0.01000,0.489
-8,self_attn.o_proj,0.0000117340,0.01000,0.480
-8,mlp.gate_proj,0.0004199908,0.01000,0.477
-8,mlp.up_proj,0.0003074690,0.01000,0.480
-8,mlp.down_proj,0.0000799455,0.01000,1.522
-9,self_attn.q_proj,0.0002144293,0.01000,0.858
-9,self_attn.k_proj,0.0000993999,0.01000,0.485
-9,self_attn.v_proj,0.0000977489,0.01000,0.486
-9,self_attn.o_proj,0.0000329630,0.01000,0.474
-9,mlp.gate_proj,0.0004030658,0.01000,0.481
-9,mlp.up_proj,0.0003244922,0.01000,0.482
-9,mlp.down_proj,0.0001223796,0.01000,1.511
-10,self_attn.q_proj,0.0001646203,0.01000,0.661
-10,self_attn.k_proj,0.0000742524,0.01000,0.492
-10,self_attn.v_proj,0.0000792629,0.01000,0.491
-10,self_attn.o_proj,0.0000381727,0.01000,0.475
-10,mlp.gate_proj,0.0004072631,0.01000,0.482
-10,mlp.up_proj,0.0003358870,0.01000,0.476
-10,mlp.down_proj,0.0001950185,0.01000,1.540
-11,self_attn.q_proj,0.0003723003,0.01000,0.662
-11,self_attn.k_proj,0.0001662678,0.01000,0.494
-11,self_attn.v_proj,0.0001578873,0.01000,0.488
-11,self_attn.o_proj,0.0000987997,0.01000,0.478
-11,mlp.gate_proj,0.0003472215,0.01000,0.488
-11,mlp.up_proj,0.0003287667,0.01000,0.489
-11,mlp.down_proj,0.0002170780,0.01000,1.532
-12,self_attn.q_proj,0.0003013455,0.01000,0.655
-12,self_attn.k_proj,0.0001367587,0.01000,0.488
-12,self_attn.v_proj,0.0001409602,0.01000,0.491
-12,self_attn.o_proj,0.0000535749,0.01000,0.474
-12,mlp.gate_proj,0.0003395944,0.01000,0.482
-12,mlp.up_proj,0.0003452886,0.01000,0.484
-12,mlp.down_proj,0.0002339401,0.01000,1.532
-13,self_attn.q_proj,0.0004235713,0.01000,0.650
-13,self_attn.k_proj,0.0001706110,0.01000,0.491
-13,self_attn.v_proj,0.0001876278,0.01000,0.490
-13,self_attn.o_proj,0.0000764515,0.01000,0.480
-13,mlp.gate_proj,0.0003770797,0.01000,0.483
-13,mlp.up_proj,0.0003777360,0.01000,0.481
-13,mlp.down_proj,0.0002857019,0.01000,1.512
-14,self_attn.q_proj,0.0007051087,0.01000,0.643
-14,self_attn.k_proj,0.0003071910,0.01000,0.486
-14,self_attn.v_proj,0.0003141222,0.01000,0.481
-14,self_attn.o_proj,0.0000914036,0.01000,0.474
-14,mlp.gate_proj,0.0004245004,0.01000,0.483
-14,mlp.up_proj,0.0004339850,0.01000,0.483
-14,mlp.down_proj,0.0003975430,0.01000,1.505
-15,self_attn.q_proj,0.0015869274,0.01000,0.868
-15,self_attn.k_proj,0.0005557225,0.01000,0.479
-15,self_attn.v_proj,0.0007385406,0.01000,0.479
-15,self_attn.o_proj,0.0001717588,0.01000,0.469
-15,mlp.gate_proj,0.0005073424,0.01000,0.477
-15,mlp.up_proj,0.0005269765,0.01000,0.477
-15,mlp.down_proj,0.0005912411,0.01000,1.506
-16,self_attn.q_proj,0.0021061117,0.01000,0.868
-16,self_attn.k_proj,0.0007697947,0.01000,0.478
-16,self_attn.v_proj,0.0008052027,0.01000,0.476
-16,self_attn.o_proj,0.0002286357,0.01000,0.466
-16,mlp.gate_proj,0.0005959417,0.01000,0.479
-16,mlp.up_proj,0.0006820880,0.01000,0.488
-16,mlp.down_proj,0.0015508921,0.01000,1.524
-17,self_attn.q_proj,0.0055543600,0.01000,0.650
-17,self_attn.k_proj,0.0020179782,0.01000,0.488
-17,self_attn.v_proj,0.0025371906,0.01000,0.481
-17,self_attn.o_proj,0.0005401876,0.01000,0.471
-17,mlp.gate_proj,0.0008525012,0.01000,0.478
-17,mlp.up_proj,0.0009743239,0.01000,0.473
-17,mlp.down_proj,0.0019300004,0.01000,1.497
-18,self_attn.q_proj,0.0071615855,0.01000,0.836
-18,self_attn.k_proj,0.0024612989,0.01000,0.479
-18,self_attn.v_proj,0.0029228464,0.01000,0.477
-18,self_attn.o_proj,0.0004554841,0.01000,0.461
-18,mlp.gate_proj,0.0011518225,0.01000,0.479
-18,mlp.up_proj,0.0012972882,0.01000,0.477
-18,mlp.down_proj,0.0033720387,0.01000,1.492
-19,self_attn.q_proj,0.0129074770,0.01000,0.640
-19,self_attn.k_proj,0.0042499688,0.01000,0.478
-19,self_attn.v_proj,0.0056665521,0.01000,0.483
-19,self_attn.o_proj,0.0008474814,0.01000,0.472
-19,mlp.gate_proj,0.0015841057,0.01000,0.480
-19,mlp.up_proj,0.0018163190,0.01000,0.483
-19,mlp.down_proj,0.0073965770,0.01000,1.515
-20,self_attn.q_proj,0.0213903782,0.01000,0.638
-20,self_attn.k_proj,0.0073983278,0.01000,0.480
-20,self_attn.v_proj,0.0096556293,0.01000,0.478
-20,self_attn.o_proj,0.0011031375,0.01000,0.470
-20,mlp.gate_proj,0.0020351633,0.01000,0.478
-20,mlp.up_proj,0.0022976996,0.01000,0.474
-20,mlp.down_proj,0.0108348703,0.01000,1.502
-21,self_attn.q_proj,0.0317660292,0.01000,0.639
-21,self_attn.k_proj,0.0116202132,0.01000,0.479
-21,self_attn.v_proj,0.0158023123,0.01000,0.476
-21,self_attn.o_proj,0.0015908420,0.01000,0.474
-21,mlp.gate_proj,0.0023832206,0.01000,0.479
-21,mlp.up_proj,0.0027572736,0.01000,0.477
-21,mlp.down_proj,0.0158048804,0.01000,1.495
-22,self_attn.q_proj,0.0407548736,0.01000,0.640
-22,self_attn.k_proj,0.0150106458,0.01000,0.477
-22,self_attn.v_proj,0.0202007413,0.01000,0.477
-22,self_attn.o_proj,0.0021502327,0.01000,0.463
-22,mlp.gate_proj,0.0026785822,0.01000,0.474
-22,mlp.up_proj,0.0030653916,0.01000,0.475
-22,mlp.down_proj,0.0178709172,0.01000,1.495
-23,self_attn.q_proj,0.0446284787,0.01000,0.634
-23,self_attn.k_proj,0.0190296860,0.01000,0.480
-23,self_attn.v_proj,0.0239933381,0.01000,0.479
-23,self_attn.o_proj,0.0020754843,0.01000,0.470
-23,mlp.gate_proj,0.0031200403,0.01000,0.482
-23,mlp.up_proj,0.0035550402,0.01000,0.483
-23,mlp.down_proj,0.0204610985,0.01000,1.501
-24,self_attn.q_proj,0.0858624045,0.01000,0.842
-24,self_attn.k_proj,0.0326525819,0.01000,0.478
-24,self_attn.v_proj,0.0390621431,0.01000,0.477
-24,self_attn.o_proj,0.0024571737,0.01000,0.465
-24,mlp.gate_proj,0.0030477483,0.01000,0.478
-24,mlp.up_proj,0.0035677395,0.01000,0.477
-24,mlp.down_proj,0.0243642852,0.01000,1.499
-25,self_attn.q_proj,0.1048494028,0.01000,0.821
-25,self_attn.k_proj,0.0388391669,0.01000,0.480
-25,self_attn.v_proj,0.0599689154,0.01000,0.481
-25,self_attn.o_proj,0.0033398975,0.01000,0.469
-25,mlp.gate_proj,0.0029867548,0.01000,0.484
-25,mlp.up_proj,0.0035876434,0.01000,0.488
-25,mlp.down_proj,0.0297861800,0.01000,1.520
-26,self_attn.q_proj,0.1303999851,0.01000,0.858
-26,self_attn.k_proj,0.0390620764,0.01000,0.486
-26,self_attn.v_proj,0.0593836054,0.01000,0.496
-26,self_attn.o_proj,0.0102463794,0.01000,0.474
-26,mlp.gate_proj,0.0032039998,0.01000,0.488
-26,mlp.up_proj,0.0038089303,0.01000,0.487
-26,mlp.down_proj,0.0374598204,0.01000,1.521
-27,self_attn.q_proj,0.0429687971,0.01000,0.632
-27,self_attn.k_proj,0.0199527118,0.01000,0.472
-27,self_attn.v_proj,0.0243092128,0.01000,0.477
-27,self_attn.o_proj,0.0073438722,0.01000,0.463
-27,mlp.gate_proj,0.0060571320,0.01000,0.479
-27,mlp.up_proj,0.0061368420,0.01000,0.483
-27,mlp.down_proj,0.0483323405,0.01000,1.496

internvl3-5_tokenizer/quantize_config.json

@@ -1,24 +0,0 @@
-{
-  "bits": 4,
-  "group_size": 128,
-  "desc_act": false,
-  "sym": true,
-  "lm_head": false,
-  "quant_method": "gptq",
-  "checkpoint_format": "gptq",
-  "pack_dtype": "int32",
-  "meta": {
-    "quantizer": [
-      "gptqmodel:5.0.0-dev0"
-    ],
-    "uri": "https://github.com/modelcloud/gptqmodel",
-    "damp_percent": 0.01,
-    "damp_auto_increment": 0.01,
-    "static_groups": false,
-    "true_sequential": true,
-    "mse": 0.0,
-    "v2": false,
-    "v2_alpha": 0.25,
-    "act_group_aware": false
-  }
-}

internvl3-5_tokenizer/quantize_done.txt

@@ -1 +0,0 @@
-done

internvl3-5_tokenizer/special_tokens_map.json

@@ -1,34 +0,0 @@
-{
-  "additional_special_tokens": [
-    "<|im_start|>",
-    "<|im_end|>",
-    "<|object_ref_start|>",
-    "<|object_ref_end|>",
-    "<|box_start|>",
-    "<|box_end|>",
-    "<|quad_start|>",
-    "<|quad_end|>",
-    "<|vision_start|>",
-    "<|vision_end|>",
-    "<|vision_pad|>",
-    "<|image_pad|>",
-    "<|video_pad|>",
-    "<img>",
-    "</img>",
-    "<IMG_CONTEXT>",
-    "<quad>",
-    "</quad>",
-    "<ref>",
-    "</ref>",
-    "<box>",
-    "</box>"
-  ],
-  "eos_token": {
-    "content": "<|im_end|>",
-    "lstrip": false,
-    "normalized": false,
-    "rstrip": false,
-    "single_word": false
-  },
-  "pad_token": "<|endoftext|>"
-}

internvl3-5_tokenizer/tokenizer.json

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:6581c44164d273d4222df982905a7e0450dcf3a4a7ebe98f9ec53e4de05beffe
-size 11424300

internvl3-5_tokenizer/tokenizer_config.json

@@ -1,323 +0,0 @@
-{
-  "add_bos_token": false,
-  "add_eos_token": false,
-  "add_prefix_space": false,
-  "added_tokens_decoder": {
-    "151643": {
-      "content": "<|endoftext|>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "151644": {
-      "content": "<|im_start|>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "151645": {
-      "content": "<|im_end|>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "151646": {
-      "content": "<|object_ref_start|>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "151647": {
-      "content": "<|object_ref_end|>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "151648": {
-      "content": "<|box_start|>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "151649": {
-      "content": "<|box_end|>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "151650": {
-      "content": "<|quad_start|>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "151651": {
-      "content": "<|quad_end|>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "151652": {
-      "content": "<|vision_start|>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "151653": {
-      "content": "<|vision_end|>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "151654": {
-      "content": "<|vision_pad|>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "151655": {
-      "content": "<|image_pad|>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "151656": {
-      "content": "<|video_pad|>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "151657": {
-      "content": "<tool_call>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": false
-    },
-    "151658": {
-      "content": "</tool_call>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": false
-    },
-    "151659": {
-      "content": "<|fim_prefix|>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": false
-    },
-    "151660": {
-      "content": "<|fim_middle|>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": false
-    },
-    "151661": {
-      "content": "<|fim_suffix|>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": false
-    },
-    "151662": {
-      "content": "<|fim_pad|>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": false
-    },
-    "151663": {
-      "content": "<|repo_name|>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": false
-    },
-    "151664": {
-      "content": "<|file_sep|>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": false
-    },
-    "151665": {
-      "content": "<tool_response>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": false
-    },
-    "151666": {
-      "content": "</tool_response>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": false
-    },
-    "151667": {
-      "content": "<think>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": false
-    },
-    "151668": {
-      "content": "</think>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": false
-    },
-    "151669": {
-      "content": "<img>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "151670": {
-      "content": "</img>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "151671": {
-      "content": "<IMG_CONTEXT>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "151672": {
-      "content": "<quad>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "151673": {
-      "content": "</quad>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "151674": {
-      "content": "<ref>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "151675": {
-      "content": "</ref>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "151676": {
-      "content": "<box>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "151677": {
-      "content": "</box>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    }
-  },
-  "additional_special_tokens": [
-    "<|im_start|>",
-    "<|im_end|>",
-    "<|object_ref_start|>",
-    "<|object_ref_end|>",
-    "<|box_start|>",
-    "<|box_end|>",
-    "<|quad_start|>",
-    "<|quad_end|>",
-    "<|vision_start|>",
-    "<|vision_end|>",
-    "<|vision_pad|>",
-    "<|image_pad|>",
-    "<|video_pad|>",
-    "<img>",
-    "</img>",
-    "<IMG_CONTEXT>",
-    "<quad>",
-    "</quad>",
-    "<ref>",
-    "</ref>",
-    "<box>",
-    "</box>"
-  ],
-  "bos_token": null,
-  "clean_up_tokenization_spaces": false,
-  "eos_token": "<|im_end|>",
-  "errors": "replace",
-  "extra_special_tokens": {},
-  "model_max_length": 14588,
-  "pad_token": "<|endoftext|>",
-  "processor_class": "InternVLProcessor",
-  "split_special_tokens": false,
-  "tokenizer_class": "Qwen2TokenizerFast",
-  "unk_token": null,
-  "_commit_hash": null
-}

post_config.json

@@ -0,0 +1,14 @@
+{
+    "enable_temperature" : true,
+    "temperature" : 0.7,
+
+    "enable_repetition_penalty" : true,
+    "repetition_penalty" : 1.2,
+    "penalty_window" : 30,
+
+    "enable_top_p_sampling" : false,
+    "top_p" : 0.9,
+
+    "enable_top_k_sampling" : true,
+    "top_k" : 10
+}

utils/infer_func.py

@@ -1,283 +0,0 @@
-import torch
-import numpy as np
-from typing import List, Tuple
-from tqdm import tqdm
-from axengine import InferenceSession
-import os
-import re
-from ml_dtypes import bfloat16
-
-
-# Discover model files automatically from model_dir.
-# We expect files like: <prefix>_p128_l<idx>_together.axmodel and <prefix>_post.axmodel
-# we try to detect model prefix and layer files automatically
-def _find_axmodel_files(base_dir: str, expected_layers: int = None, expected_prefill: int = 128):
-    files = os.listdir(base_dir)
-    # match prefix, prefill size (dynamic), and layer index
-    layer_pattern = re.compile(r"^(?P<prefix>.*)_p(?P<prefill>\d+)_l(?P<idx>\d+)_together\.axmodel$")
-    post_pattern = re.compile(r"^(?P<prefix>.*)_post\.axmodel$")
-
-    # collect prefix -> [(idx, fname)]
-    prefix_map = {}
-    for fname in files:
-        m = layer_pattern.match(fname)
-        if m:
-            prefix = m.group("prefix")
-            idx = int(m.group("idx"))
-            prefix_map.setdefault(prefix, []).append((idx, fname))
-
-    if not prefix_map:
-        # fallback to hardcoded pattern if nothing detected
-        prefix = "gemma3_text"
-        layer_files = [(
-            i, f"{prefix}_p{expected_prefill}_l{i}_together.axmodel"
-        ) for i in range(expected_layers or 0)]
-    else:
-        # choose the prefix with the most layers (most likely the correct one)
-        prefix = max(prefix_map.items(), key=lambda kv: len(kv[1]))[0]
-        # debug info
-        print(f"Detected prefixes: {list(prefix_map.keys())}, chosen: {prefix}, layers: {len(prefix_map[prefix])}")
-        layer_files = sorted(prefix_map[prefix], key=lambda it: it[0])
-
-    # find post process file
-    post_file = None
-    for fname in files:
-        m = post_pattern.match(fname)
-        if m and m.group("prefix") == prefix:
-            post_file = fname
-            break
-    if post_file is None:
-        candidate = os.path.join(base_dir, f"{prefix}_post.axmodel")
-        if os.path.exists(candidate):
-            post_file = f"{prefix}_post.axmodel"
-        else:
-            for fname in files:
-                if fname.endswith("_post.axmodel"):
-                    post_file = fname
-                    break
-
-    return layer_files, post_file, prefix
-
-class InferManager:
-    def __init__(self, config, model_dir, max_seq_len=2047):
-
-        self.config = config
-        self.max_seq_len = max_seq_len
-
-        self.sub_dim = config.hidden_size // config.num_attention_heads if not config.head_dim else config.head_dim
-        self.kv_dim = self.sub_dim * config.num_key_value_heads
-
-        self.k_caches = [
-            np.zeros((1, self.max_seq_len, self.kv_dim), dtype=bfloat16)
-            for _ in range(config.num_hidden_layers)
-        ]
-        self.v_caches = [
-            np.zeros((1, self.max_seq_len, self.kv_dim), dtype=bfloat16)
-            for _ in range(config.num_hidden_layers)
-        ]
-
-        layer_files, post_file, prefix = _find_axmodel_files(model_dir, config.num_hidden_layers)
-
-        self.decoder_sessions = []
-        for _, fname in tqdm(layer_files, desc="Init InferenceSession"):
-            session = InferenceSession(os.path.join(model_dir, fname))
-            self.decoder_sessions.append(session)
-
-        # post_file was returned by _find_axmodel_files; ensure it was found
-        if post_file is None:
-            raise FileNotFoundError("Cannot find post process .axmodel file in model_dir")
-        self.post_process_session = InferenceSession(os.path.join(model_dir, post_file))
-        print("Model loaded successfully!")
-
-    @staticmethod
-    def _top_p(probs: np.ndarray, p: float) -> np.ndarray:
-        sorted_indices = np.argsort(probs)
-        filtered = probs.copy()
-        cumulative = 0
-        for idx in sorted_indices[::-1]:
-            if cumulative >= p:
-                filtered[idx] = 0
-            cumulative += filtered[idx]
-        return filtered / cumulative
-
-    @staticmethod
-    def _softmax(logits: np.ndarray) -> np.ndarray:
-        logits = logits - logits.max()
-        exp_logits = np.exp(logits)
-        return (exp_logits / np.sum(exp_logits)).astype(np.float64)
-
-    def post_process(self, logits, top_k=1, top_p=0.9, temperature=0.6):
-        logits = logits.astype(np.float32).flatten()
-        candidate_indices = np.argpartition(logits, -top_k)[-top_k:]
-        candidate_logits = logits[candidate_indices] / temperature
-        candidate_probs = self._softmax(candidate_logits)
-        candidate_probs = self._top_p(candidate_probs, top_p)
-        candidate_probs = candidate_probs.astype(np.float64) / candidate_probs.sum()
-        chosen_idx = np.random.multinomial(1, candidate_probs).argmax()
-        next_token = candidate_indices[chosen_idx]
-        return next_token, candidate_indices, candidate_probs
-
-    def gen_slice_indices(self, token_len, prefill=128, expand=128):
-        remaining = max(0, token_len - prefill)
-        extra_blocks = (remaining + expand - 1) // expand
-        return list(range(extra_blocks + 1))
-
-    def prefill(
-        self,
-        tokenizer,
-        token_ids,
-        embed_data,
-        slice_len=128,
-    ):
-        """
-        Prefill step for chunked inference.
-        """
-        seq_len = len(token_ids)
-        slice_indices = [i for i in range(seq_len // slice_len + 1)]
-        print(f"slice_indices: {slice_indices}")
-        # total_prefill_len = (
-        #     slice_len * slice_indices[-1]
-        #     if slice_indices[-1] != 0
-        #     else slice_len
-        # )
-        total_prefill_len = slice_len * (slice_indices[-1] + 1)
-        # slice_indices = self.gen_slice_indices(seq_len)
-
-        if total_prefill_len > 0:
-            for slice_idx in slice_indices:
-                indices = np.arange(
-                    slice_idx * slice_len,
-                    (slice_idx + 1) * slice_len,
-                    dtype=np.uint32
-                ).reshape((1, slice_len))
-
-                mask = (
-                    np.zeros((1, slice_len, slice_len * (slice_idx + 1)))
-                    - 65536
-                )
-                data = np.zeros((1, slice_len, self.config.hidden_size)).astype(bfloat16)
-                for i, t in enumerate(
-                    range(
-                        slice_idx * slice_len,
-                        (slice_idx + 1) * slice_len,
-                    )
-                ):
-                    if t < len(token_ids):
-                        mask[:, i, : slice_idx * slice_len + i + 1] = 0
-                        data[:, i : i + 1, :] = (
-                            embed_data[t]
-                            .reshape((1, 1, self.config.hidden_size))
-                            .astype(bfloat16)
-                        )
-
-                remain_len = (
-                    seq_len - slice_idx * slice_len
-                    if slice_idx == slice_indices[-1]
-                    else slice_len
-                )
-                mask = mask.astype(bfloat16)
-                for layer_idx in range(self.config.num_hidden_layers):
-                    input_feed = {
-                        "K_cache": (
-                            self.k_caches[layer_idx][:, 0 : slice_len * slice_idx, :]
-                            if slice_idx
-                            else np.zeros((1, 1, self.config.hidden_size), dtype=bfloat16)
-                        ),
-                        "V_cache": (
-                            self.v_caches[layer_idx][:, 0 : slice_len * slice_idx, :]
-                            if slice_idx
-                            else np.zeros((1, 1, self.config.hidden_size), dtype=bfloat16)
-                        ),
-                        "indices": indices,
-                        "input": data,
-                        "mask": mask,
-                    }
-                    outputs = self.decoder_sessions[layer_idx].run(None, input_feed, shape_group=slice_idx + 1)
-                    self.k_caches[layer_idx][
-                        :,
-                        slice_idx * slice_len : slice_idx * slice_len + remain_len,
-                        :,
-                    ] = outputs[0][:, :remain_len, :]
-                    self.v_caches[layer_idx][
-                        :,
-                        slice_idx * slice_len : slice_idx * slice_len + remain_len,
-                        :,
-                    ] = outputs[1][:, :remain_len, :]
-                    data = outputs[2]
-
-                print("Slice prefill done:", slice_idx)
-            
-            # return data[:, :remain_len, :]
-            post_out = self.post_process_session.run(
-                None,
-                {
-                    "input": data[
-                        :, seq_len - (len(slice_indices) - 1) * slice_len - 1, None, :
-                    ]
-                }
-            )[0]
-            next_token, possible_tokens, possible_probs = self.post_process(post_out)
-            possible_decoded = [tokenizer.decode([t]) for t in possible_tokens]
-            possible_probs_str = [str((t, p)) for t, p in zip(possible_decoded, possible_probs)]
-            token_ids.append(next_token)
-            return token_ids
-
-    def decode(
-        self,
-        tokenizer,
-        token_ids,
-        embed_matrix,
-        prefill_len=128,
-        slice_len=128,
-        eos_token_id=None, # 某些模型有多个 eos_token_id
-        stream=True,
-    ):
-        """Autoregressive decode; optionally stream tokens or collect silently."""
-
-        decoded_text = tokenizer.decode(token_ids[-1], skip_special_tokens=True)
-        if stream:
-            print("answer >>", decoded_text, end='', flush=True)
-
-        mask = np.zeros((1, 1, self.max_seq_len + 1), dtype=np.float32).astype(bfloat16)
-        mask[:, :, :self.max_seq_len] -= 65536
-        seq_len = len(token_ids) - 1
-        if prefill_len > 0:
-            mask[:, :, :seq_len] = 0
-
-        for step_idx in range(self.max_seq_len):
-            if prefill_len > 0 and step_idx < seq_len:
-                continue
-            cur_token = token_ids[step_idx]
-            indices = np.array([step_idx], np.uint32).reshape((1, 1))
-            data = embed_matrix[cur_token, :].reshape((1, 1, self.config.hidden_size)).astype(bfloat16)
-            for layer_idx in range(self.config.num_hidden_layers):
-                input_feed = {
-                    "K_cache": self.k_caches[layer_idx],
-                    "V_cache": self.v_caches[layer_idx],
-                    "indices": indices,
-                    "input": data,
-                    "mask": mask,
-                }
-                outputs = self.decoder_sessions[layer_idx].run(None, input_feed, shape_group=0)
-                self.k_caches[layer_idx][:, step_idx, :] = outputs[0][:, :, :]
-                self.v_caches[layer_idx][:, step_idx, :] = outputs[1][:, :, :]
-                data = outputs[2]
-            mask[..., step_idx] = 0
-            if step_idx < seq_len - 1:
-                continue
-            else:
-                post_out = self.post_process_session.run(None, {"input": data})[0]
-                next_token, possible_tokens, possible_probs = self.post_process(post_out, temperature=0.7)
-                if eos_token_id is not None and next_token in eos_token_id:
-                    break
-                elif next_token == tokenizer.eos_token_id:
-                    break
-                token_ids.append(next_token)
-
-            decoded_piece = tokenizer.decode(next_token, skip_special_tokens=True)
-            decoded_text += decoded_piece
-            if stream:
-                print(decoded_piece, end='', flush=True)
-
-        return decoded_text
-