--- license: mit language: - en - zh base_model: - Qwen/Qwen3.5-4B pipeline_tag: image-text-to-text library_name: transformers tags: - Qwen3.5 - Qwen3.5-4B - VLM - GPTQ - Int4 --- # Qwen3.5-4B This version of Qwen3.5-4B has been converted to run on the Axera NPU using **w4a16** quantization. Compatible with Pulsar2 version: 5.0 ## Convert tools links: For those who are interested in model conversion, you can try to export axmodel through the original repo : - https://huggingface.co/Qwen/Qwen3.5-4B [Pulsar2 Link, How to Convert LLM from Huggingface to axmodel](https://pulsar2-docs.readthedocs.io/en/latest/appendix/build_llm.html) [AXera NPU HOST LLM Runtime](https://github.com/AXERA-TECH/ax-llm.git) ## Support Platform - AX650 - AX650N DEMO Board - [M4N-Dock(爱芯派Pro)](https://wiki.sipeed.com/hardware/zh/maixIV/m4ndock/m4ndock.html) - [M.2 Accelerator card](https://docs.m5stack.com/zh_CN/ai_hardware/LLM-8850_Card) **Image Process** |Chips| input size | image num | ttft(168 tokens) | w4a16 | CMM | Flash | |--|--|--|--|--|--|--| |AX650| 384*384 | 1 | 845 ms | 5.7 tokens/sec| 4.91GiB | 5.59GiB | **Video Process** |Chips| input size | image num | ttft(600 tokens) | w4a16 | CMM | Flash | |--|--|--|--|--|--|--| |AX650| 384*384 | 8 | 2111 ms | 5.7 tokens/sec| 4.91GiB | 5.59GiB | The DDR capacity refers to the CMM memory that needs to be consumed. Ensure that the CMM memory allocation on the development board is greater than this value. ## How to use ## 安装 axllm 方式一：克隆仓库后执行安装脚本： ```shell git clone -b axllm https://github.com/AXERA-TECH/ax-llm.git cd ax-llm ./install.sh ``` 方式二：一行命令安装（默认分支 `axllm`）： ```shell curl -fsSL https://raw.githubusercontent.com/AXERA-TECH/ax-llm/axllm/install.sh | bash ``` 方式三：下载Github Actions CI 导出的可执行程序（适合没有编译环境的用户）： 如果没有编译环境，请到： `https://github.com/AXERA-TECH/ax-llm/actions?query=branch%3Aaxllm` 下载 **最新 CI 导出的可执行程序**（`axllm`），然后： ```shell chmod +x axllm sudo mv axllm /usr/bin/axllm ``` ## 模型下载（Hugging Face） 先创建模型目录并进入，然后下载到该目录： ```shell mkdir -p AXERA-TECH/Qwen3.5-4B-AX650-GPTQ-Int4-C256-P10K-CTX12K cd AXERA-TECH/Qwen3.5-4B-AX650-GPTQ-Int4-C256-P10K-CTX12K hf download AXERA-TECH/Qwen3.5-4B-AX650-GPTQ-Int4-C256-P10K-CTX12K --local-dir . # structure of the downloaded files tree -L 3 `-- AXERA-TECH `-- Qwen3.5-4B-AX650-GPTQ-Int4-C256-P10K-CTX12K |-- qwen3_5_vision.axmodel |-- README.md |-- config.json |-- image.png |-- model.embed_tokens.weight.bfloat16.bin |-- post_config.json |-- qwen3_5_tokenizer.txt |-- qwen3_5_text_p128_l0_together.axmodel ... |-- qwen3_5_text_p128_l23_together.axmodel |-- qwen3_5_text_post.axmodel `-- vision_cache 3 directories, 39 files ``` ## Inference with AX650 Host, such as M4N-Dock(爱芯派Pro) or AX650N DEMO Board ### 运行（CLI） ```shell root@ax650 ~/yongqiang/lhj/Qwen3_5.AXERA/ax-llm # axllm run Qwen3.5-4B-AX650-GPTQ-Int4-C256-P10K-CTX12K/ 11:40:21.412 INF Init:218 | LLM init start 11:40:21.412 INF Init:226 | mixed attention enabled: full_attention_interval=4 ref_full_layer_idx=3 tokenizer_type = 3 96% | ############################## | 26 / 27 [3.78s<3.93s, 6.87 count/s] init post axmodel ok,remain_cmm(5589 MB) 11:40:25.195 INF Init:368 | max_token_len : 2047 11:40:25.195 INF Init:371 | kv_cache_size : 512, kv_cache_num: 2047 11:40:25.195 INF Init:374 | prefill_token_num : 128 11:40:25.195 INF Init:379 | grp: 1, prefill_max_kv_cache_num : 1 11:40:25.195 INF Init:379 | grp: 2, prefill_max_kv_cache_num : 128 11:40:25.195 INF Init:379 | grp: 3, prefill_max_kv_cache_num : 256 11:40:25.195 INF Init:379 | grp: 4, prefill_max_kv_cache_num : 384 11:40:25.195 INF Init:379 | grp: 5, prefill_max_kv_cache_num : 512 11:40:25.195 INF Init:379 | grp: 6, prefill_max_kv_cache_num : 768 11:40:25.195 INF Init:379 | grp: 7, prefill_max_kv_cache_num : 1024 11:40:25.195 INF Init:379 | grp: 8, prefill_max_kv_cache_num : 1152 11:40:25.195 INF Init:384 | prefill_max_token_num : 1152 11:40:25.195 INF Init:27 | LLaMaEmbedSelector use mmap 100% | ################################ | 27 / 27 [3.79s<3.79s, 7.13 count/s] embed_selector init ok 11:40:25.478 INF Init:643 | Qwen-VL token ids: vision_start=248053 image_pad=248056 video_pad=248057 11:40:25.478 INF Init:668 | VisionModule init ok: type=Qwen3VL, tokens_per_block=144, embed_size=2048, out_dtype=fp32 11:40:25.478 WRN Init:677 | Vision preprocess backend: SimpleCV (OpenCV not found at build time; minor differences vs OpenCV are possible) 11:40:25.480 INF load_config:282 | load config: 11:40:25.480 INF load_config:282 | { 11:40:25.480 INF load_config:282 | "enable_repetition_penalty": false, 11:40:25.480 INF load_config:282 | "enable_temperature": false, 11:40:25.480 INF load_config:282 | "enable_top_k_sampling": true, 11:40:25.480 INF load_config:282 | "enable_top_p_sampling": false, 11:40:25.480 INF load_config:282 | "penalty_window": 20, 11:40:25.480 INF load_config:282 | "repetition_penalty": 1.2, 11:40:25.480 INF load_config:282 | "temperature": 0.9, 11:40:25.480 INF load_config:282 | "top_k": 10, 11:40:25.480 INF load_config:282 | "top_p": 0.8 11:40:25.480 INF load_config:282 | } 11:40:25.480 INF Init:448 | LLM init ok Commands: /q, /exit 退出 /reset 重置 kvcache /dd 删除一轮对话 /pp 打印历史对话 Ctrl+C: 停止当前生成 VLM enabled: after each prompt, input image path (empty = text-only). Use "video:" for video. ---------------------------------------- prompt >> 描述视频内容 media >> video:assets/football.mp4:1.5 17:54:19.424 INF extract_video_frames_ffmpeg:299 | Extracting raw video container to frames: assets/football.mp4 -> /tmp/axllm_video_frames/video_77305085315876_0 fps=1.4985014985014984 17:54:40.848 INF collect_video_frame_paths:379 | Video fps sampling: path=assets/football.mp4 fps=1.5 duration=60.060s target_frames=90 selected=90 17:55:01.670 INF SetKVCache:2543 | decode_grpid:3 prefill_grpid:6 history_cap:0 total_cap:256 symbolic_cap:1 precompute_len:0 input_num_token:6504 prefer_symbolic_group:0 17:55:01.670 INF SetKVCache:2565 | current prefill_max_token_num:10496 17:55:02.375 INF SetKVCache:2581 | first run 17:55:02.481 INF Run:2738 | input token num : 6504, prefill_split_num : 26 17:55:02.481 INF Run:2818 | prefill chunk p=0 history_len=0 grpid=6 kv_cache_num=0 input_tokens=256 17:55:03.405 INF Run:2818 | prefill chunk p=1 history_len=256 grpid=8 kv_cache_num=512 input_tokens=256 17:55:04.400 INF Run:2818 | prefill chunk p=2 history_len=512 grpid=9 kv_cache_num=768 input_tokens=256 17:55:05.385 INF Run:2818 | prefill chunk p=3 history_len=768 grpid=10 kv_cache_num=1024 input_tokens=256 17:55:06.362 INF Run:2818 | prefill chunk p=4 history_len=1024 grpid=11 kv_cache_num=1280 input_tokens=256 17:55:07.365 INF Run:2818 | prefill chunk p=5 history_len=1280 grpid=12 kv_cache_num=1536 input_tokens=256 17:55:08.399 INF Run:2818 | prefill chunk p=6 history_len=1536 grpid=13 kv_cache_num=1792 input_tokens=256 17:55:09.466 INF Run:2818 | prefill chunk p=7 history_len=1792 grpid=14 kv_cache_num=2048 input_tokens=256 17:55:10.592 INF Run:2818 | prefill chunk p=8 history_len=2048 grpid=15 kv_cache_num=2304 input_tokens=256 17:55:11.649 INF Run:2818 | prefill chunk p=9 history_len=2304 grpid=16 kv_cache_num=2560 input_tokens=256 17:55:12.724 INF Run:2818 | prefill chunk p=10 history_len=2560 grpid=17 kv_cache_num=2816 input_tokens=256 17:55:13.798 INF Run:2818 | prefill chunk p=11 history_len=2816 grpid=18 kv_cache_num=3072 input_tokens=256 17:55:14.886 INF Run:2818 | prefill chunk p=12 history_len=3072 grpid=19 kv_cache_num=3328 input_tokens=256 17:55:15.985 INF Run:2818 | prefill chunk p=13 history_len=3328 grpid=20 kv_cache_num=3584 input_tokens=256 17:55:17.091 INF Run:2818 | prefill chunk p=14 history_len=3584 grpid=21 kv_cache_num=3840 input_tokens=256 17:55:18.232 INF Run:2818 | prefill chunk p=15 history_len=3840 grpid=22 kv_cache_num=4096 input_tokens=256 17:55:19.386 INF Run:2818 | prefill chunk p=16 history_len=4096 grpid=23 kv_cache_num=4352 input_tokens=256 17:55:20.613 INF Run:2818 | prefill chunk p=17 history_len=4352 grpid=24 kv_cache_num=4608 input_tokens=256 17:55:21.853 INF Run:2818 | prefill chunk p=18 history_len=4608 grpid=25 kv_cache_num=4864 input_tokens=256 17:55:23.114 INF Run:2818 | prefill chunk p=19 history_len=4864 grpid=26 kv_cache_num=5120 input_tokens=256 17:55:24.395 INF Run:2818 | prefill chunk p=20 history_len=5120 grpid=27 kv_cache_num=5376 input_tokens=256 17:55:25.681 INF Run:2818 | prefill chunk p=21 history_len=5376 grpid=28 kv_cache_num=5632 input_tokens=256 17:55:26.995 INF Run:2818 | prefill chunk p=22 history_len=5632 grpid=29 kv_cache_num=5888 input_tokens=256 17:55:28.322 INF Run:2818 | prefill chunk p=23 history_len=5888 grpid=30 kv_cache_num=6144 input_tokens=256 17:55:29.667 INF Run:2818 | prefill chunk p=24 history_len=6144 grpid=31 kv_cache_num=6400 input_tokens=256 17:55:31.043 INF Run:2818 | prefill chunk p=25 history_len=6400 grpid=32 kv_cache_num=6656 input_tokens=104 17:55:32.459 INF Run:3045 | ttft: 29978.45 ms 17:55:32.459 INF Run:3076 | VLM decode positions: rope_start=6372 dense_kv_start=6504 这段视频是一个关于足球比赛的直播或录像，具体是2019年1月30日巴塞罗那队与塞维利亚队的比赛。视频展示了比赛中的精彩瞬间和进球。 - **比赛场景**：视频展示了一场足球比赛的片段，场地为标准的足球场，有清晰的白色线条标记。 - **球员和队服**： - 巴塞罗那队穿着蓝色和红色条纹的球衣，号码为10的球员是路易斯·苏亚雷斯（Luis Suárez）。 - 塞维利亚队穿着白色球衣。 - **比赛动作**： - 在多个片段中，可以看到球员们在场上积极跑动和传球。 - 一个关键片段显示，巴塞罗那队在进攻中成功突破防守，最终由苏亚雷斯打入一球。 - 进球后，苏亚雷斯与队友庆祝，场面充满激情。 - **观众和氛围**： - 观众席上坐满了观众，他们热情地为比赛加油。 - 场边的广告牌上可以看到“Nike”和“Estrella Damm”等品牌标识。 - **直播者**： - 在视频的右上角，有一个直播者正在观看和评论比赛。他穿着巴塞罗那队的球衣，表情丰富，时而惊讶，时而兴奋，显然对比赛非常投入。 - **技术细节**： - 视频底部有“FBS00MPS10"的水印，以及播放进度条，显示视频正在播放中。 - 视频标题为“Barcelona vs Sevilla 6-1 All Goals & Extended Highlights 30/01/2019 HD"，表明这是该场比赛的完整进球和扩展集锦。 这段视频通过多个镜头展示了比赛的紧张和精彩，以及球迷和直播者对比赛的热情。 17:56:46.246 NTC Run:3445 | hit eos,decode avg 5.24 token/s 17:56:46.246 INF GetKVCache:2496 | precompute_len:6892, remaining:3604 (tracked) ``` ### 启动服务（OpenAI 兼容） ```shell root@ax650:~# axllm serve AXERA-TECH/Qwen3.5-4B-AX650-GPTQ-Int4-C256-P10K-CTX12K [I][ Init][ 138]: LLM init start tokenizer_type = 1 96% | ███████████████████████████████ | 30 / 31 [4.63s<4.79s, 6.47 count/s] init post axmodel ok,remain_cmm(9563 MB) [I][ Init][ 199]: max_token_len : 2047 [I][ Init][ 202]: kv_cache_size : 1024, kv_cache_num: 2047 [I][ Init][ 205]: prefill_token_num : 128 [I][ Init][ 209]: grp: 1, prefill_max_kv_cache_num : 1 [I][ Init][ 209]: grp: 2, prefill_max_kv_cache_num : 128 [I][ Init][ 209]: grp: 3, prefill_max_kv_cache_num : 256 [I][ Init][ 209]: grp: 4, prefill_max_kv_cache_num : 384 [I][ Init][ 209]: grp: 5, prefill_max_kv_cache_num : 512 [I][ Init][ 209]: grp: 6, prefill_max_kv_cache_num : 640 [I][ Init][ 209]: grp: 7, prefill_max_kv_cache_num : 768 [I][ Init][ 209]: grp: 8, prefill_max_kv_cache_num : 896 [I][ Init][ 209]: grp: 9, prefill_max_kv_cache_num : 1024 [I][ Init][ 209]: grp: 10, prefill_max_kv_cache_num : 1152 [I][ Init][ 214]: prefill_max_token_num : 1152 [I][ Init][ 27]: LLaMaEmbedSelector use mmap 100% | ████████████████████████████████ | 31 / 31 [4.64s<4.64s, 6.69 count/s] embed_selector init ok [W][ Init][ 457]: Qwen-VL vision size override: cfg=448x448 bytes=1204224, model_input_bytes=884736 -> 384x384 (square). [I][ Init][ 641]: Qwen-VL token ids: vision_start=151652 image_pad=151655 video_pad=151656 [I][ Init][ 666]: VisionModule init ok: type=Qwen3VL, tokens_per_block=144, embed_size=2048, out_dtype=fp32 [I][ Init][ 672]: VisionModule deepstack enabled: layers=3 [I][ load_config][ 282]: load config: { "enable_repetition_penalty": false, "enable_temperature": false, "enable_top_k_sampling": false, "enable_top_p_sampling": false, "penalty_window": 20, "repetition_penalty": 1.2, "temperature": 0.9, "top_k": 10, "top_p": 0.8 } [I][ Init][ 272]: LLM init ok Starting server on port 8000 with model 'AXERA-TECH/Qwen3.5-4B-AX650-GPTQ-Int4-C256-P10K-CTX12K'... OpenAI API Server starting on http://0.0.0.0:8000 Max concurrency: 1 Models: AXERA-TECH/Qwen3.5-4B-AX650-GPTQ-Int4-C256-P10K-CTX12K ``` ### OpenAI 调用示例 ```python from openai import OpenAI API_URL = "http://127.0.0.1:8000/v1" MODEL = "AXERA-TECH/Qwen3.5-4B-AX650-GPTQ-Int4-C256-P10K-CTX12K" messages = [ {"role": "system", "content": [{"type": "text", "text": "you are a helpful assistant."}]}, {"role": "user", "content": "hello"}, ] client = OpenAI(api_key="not-needed", base_url=API_URL) completion = client.chat.completions.create( model=MODEL, messages=messages, ) print(completion.choices[0].message.content) ``` ### OpenAI 流式调用示例 ```python from openai import OpenAI API_URL = "http://127.0.0.1:8000/v1" MODEL = "AXERA-TECH/Qwen3.5-4B-AX650-GPTQ-Int4-C256-P10K-CTX12K" messages = [ {"role": "system", "content": [{"type": "text", "text": "you are a helpful assistant."}]}, {"role": "user", "content": "hello"}, ] client = OpenAI(api_key="not-needed", base_url=API_URL) stream = client.chat.completions.create( model=MODEL, messages=messages, stream=True, ) print("assistant:") for ev in stream: delta = getattr(ev.choices[0], "delta", None) if delta and getattr(delta, "content", None): print(delta.content, end="", flush=True) print(" ") ```