Add 8s audio profile for Jina nano retrieval

.gitignore

@@ -3,6 +3,7 @@ __pycache__/
 vision_cache/
 *.tmp
 tmp/
+tmp_baiyongqiang/
 WORKFLOW.md
 model.embed_tokens.weight.npy
 serve_*.log

README.md

@@ -26,22 +26,31 @@ It was rebuilt with the correct text RoPE base and passes the packaged text, ima
 - Current service model id: `AXERA-TECH/jina-embeddings-v5-omni-nano-retrieval-AX650-P128-CTX2047`
 - Vision encoder static shape: `256x256`
 - Vision soft tokens per frame: `64`
-- Audio encoder static profile: `30s`, `16kHz`, mono, `750` soft tokens
+- Audio encoder static profiles:
+  - short audio: `8s`, `16kHz`, mono, `800` mel frames, `200` soft tokens
+  - long audio: `30s`, `16kHz`, mono, `3000` mel frames, `750` soft tokens
+- Audio request sequence length with the packaged `query` prompt:
+  - `8s` audio: `200` soft tokens, about `220` total LLM prefill tokens
+  - `30s` audio: `750` soft tokens, about `772` total LLM prefill tokens
 - Current LLM prefill build:
   - `prefill_len = 128`
   - warm-prefill groups = `128 / 256 / 384 / 512 / 640 / 768 / 896`
   - effective `prefill_max_token_num = 1024`
 
-Board cold-start measurements from this package:
+Board cold-start measurements from this package with both `8s` and `30s` audio profiles loaded:
 
 | Item | Value |
 |---|---:|
-| `/health` ready time | `9.11 s` |
-| CMM remain before startup | `3919300 KB` |
-| CMM remain after startup | `2426580 KB` |
-| Cold-start CMM consumption | `1492720 KB` |
-| Process RSS after startup | `1324308 KB` |
-| Process VmSize after startup | `2827552 KB` |
+| `/v1/models` ready time | `25 s` |
+| CMM used before startup | `275004 KB` |
+| CMM used after startup | `2455548 KB` |
+| Cold-start CMM consumption | `2180544 KB` |
+| Process VmRSS after startup | `2011016 KB` |
+| Process PSS after startup | `2010169 KB` |
+| Process anonymous memory | `2006272 KB` |
+| Process VmSize after startup | `4191272 KB` |
+
+`CMM` means AXERA contiguous multimedia memory. `RSS` means resident set size in Linux process memory. `PSS` means proportional set size.
 
 ## Functional Status
 
@@ -58,6 +67,7 @@ The packaged runtime files are:
 - `llama_p128_l0_together.axmodel` ... `llama_p128_l11_together.axmodel`
 - `llama_post.axmodel`
 - `jina_v5_omni_nano_vision_256x256.axmodel`
+- `jina_v5_omni_nano_audio_8s.axmodel`
 - `jina_v5_omni_nano_audio_30s.axmodel`
 - `model.embed_tokens.weight.bfloat16.bin`
 - `jina_v5_omni_tokenizer.txt`
@@ -103,8 +113,9 @@ Current board-side `axllm serve` precision against the packaged Hugging Face ref
 | Text | `embedding_doc` | `[1, 768]` | `0.999655` |
 | Text | `red_planet_query` | `[1, 768]` | `0.999539` |
 | Image | `vision_sample` | `[1, 768]` | `0.993549` |
+| Audio | `audio_test_chunk0_8s_wav` | `[1, 768]` | `0.989344` |
 | Audio | `audio_test_chunk0_30s_wav` | `[1, 768]` | `0.994962` |
-| Video | `video_visual_red_panda_openai_mp4` | `[1, 768]` | `0.997040` |
+| Video | `video_visual_red_panda_openai_mp4` | `[1, 768]` | `0.997008` |
 
 Key observations:
 
@@ -112,7 +123,9 @@ Key observations:
 - The previous `embedding_doc = 0.9604` result was traced to an LLM build config issue, not to the OpenAI-compatible service wrapper.
 - `jina-embeddings-v5-omni-nano` uses `text_config.rope_parameters.rope_theta = 1000000.0`; the earlier `llama` build path silently fell back to `10000.0`.
 - The current LLM AXModels were rebuilt after flattening `rope_theta` into `text_config.rope_theta` for `pulsar2 llm_build`.
-- With the corrected RoPE base, all packaged precision cases are in the `0.9935 ~ 0.9997` cosine range.
+- With the corrected RoPE base, all non-audio-short packaged precision cases are in the `0.9935 ~ 0.9997` cosine range.
+- The `8s` audio profile is usable but lower than the `30s` profile in this P128 build: `0.989344` cosine for the packaged `audio_test_chunk0_8s_wav` case.
+  The previous `5s` audio profile produced only about `0.94` cosine after LLM prefill, so it is not used as a default packaged precision case.
 
 ## Board Precision Validation Flow
 
@@ -188,6 +201,7 @@ Current packaged cases:
 - `embedding_doc`
 - `red_planet_query`
 - `vision_sample`
+- `audio_test_chunk0_8s_wav`
 - `audio_test_chunk0_30s_wav`
 - `video_visual_red_panda_openai_mp4`
 
@@ -197,3 +211,5 @@ The packaged `meta.json` files define the exact text, prompt name, media asset p
 ## Known Limitation
 
 - Re-measure direct audio split latency if a low-memory benchmark implementation is needed; the packaged service audio path is already validated.
+- Audio encoder loading is controlled by `filename_audio_encoder_axmodel_short` and `filename_audio_encoder_axmodel_long` in `config.json`.
+  If only one field is configured, only that audio profile is loaded; if both are configured, the runtime selects the short profile for short audio and falls back to the long profile for longer audio.

assets/audio_test_chunk0_8s.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cc5f3d6b74a04a49c0fe151b2c3048cf5a519bfbdec78343617b6be1fb67e49d
+size 1411244

bin/axllm

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:46c3777b058b107403b00175bf3f76bdd0826a562c90dfd61d59f81cc3be21be
-size 2393432
+oid sha256:f44e9785c1ae345fd6d77bee694fdfe4e541f719a7a68eab15ed6d94bef96c39
+size 2397928

config.json

@@ -16,7 +16,8 @@
   "b_use_mmap_load_layer": true,
   "vlm_type": "Qwen3Omni",
   "filename_image_encoder_axmodel": "jina_v5_omni_nano_vision_256x256.axmodel",
-  "filename_audio_encoder_axmodel_30s": "jina_v5_omni_nano_audio_30s.axmodel",
+  "filename_audio_encoder_axmodel_short": "jina_v5_omni_nano_audio_8s.axmodel",
+  "filename_audio_encoder_axmodel_long": "jina_v5_omni_nano_audio_30s.axmodel",
   "vision_width": 256,
   "vision_height": 256,
   "vision_patch_size": 16,

jina_v5_omni_nano_audio_8s.axmodel

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2f00adf09f5bd58e2939ebd6855f79ac2eac0037b4b5cf2e72a1fe8d1a02d519
+size 703218785

python/compare_openai_api_vs_hf_multimodal.py

@@ -15,14 +15,16 @@ DEFAULT_CASES = (
     ("T0", "embedding_doc"),
     ("T1", "red_planet_query"),
     ("A", "vision_sample"),
-    ("B", "audio_test_chunk0_30s_wav"),
+    ("B8", "audio_test_chunk0_8s_wav"),
+    ("B30", "audio_test_chunk0_30s_wav"),
     ("C", "video_visual_red_panda_openai_mp4"),
 )
 CASE_LABEL_NAMES = {
     "T0": "text_document",
     "T1": "text_query",
     "A": "image",
-    "B": "audio",
+    "B8": "audio_8s",
+    "B30": "audio_30s",
     "C": "video",
 }
 

python/jina_omni_utils.py

@@ -370,11 +370,13 @@ def prepare_fixed_audio_inputs(
 
     audio = load_audio_array(audio_path)
     feat_ext = WhisperFeatureExtractor(feature_size=128)
+    max_samples = int(max_frames) * int(feat_ext.hop_length)
     audio_inputs = feat_ext(
         audio,
         sampling_rate=16000,
         return_tensors="pt",
         padding="max_length",
+        max_length=max_samples,
         return_attention_mask=True,
     )
     input_features = audio_inputs["input_features"]
@@ -611,6 +613,7 @@ def run_audio_embedding(
     dtype_name: str = "bfloat16",
     device: str | None = None,
     truncate_dim: int | None = None,
+    max_frames: int = 3000,
 ):
     model_dir = Path(model_dir)
     model = load_base_model(model_dir, modality="audio", task=task, dtype_name=dtype_name, device=device)
@@ -620,6 +623,7 @@ def run_audio_embedding(
         model_dir,
         audio_path=audio_path,
         prompt_name=prompt_name,
+        max_frames=max_frames,
         processor=processor,
         tokenizer=tokenizer,
         config=model.config,

python/testdata/service_cases/audio_test_chunk0_8s_wav/meta.json

@@ -0,0 +1,10 @@
+{
+  "case_name": "audio_test_chunk0_8s_wav",
+  "modality": "audio",
+  "prompt_name": "query",
+  "text_prefix": "",
+  "asset_path": "assets/audio_test_chunk0_8s.wav",
+  "soft_token_count": 200,
+  "audio_profile": "8s",
+  "feature_frames": 800
+}

python/testdata/service_cases/audio_test_chunk0_8s_wav/torch_embedding.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:29839fe4000c35de3a0463292a46e8550ab918cbf638b6834c02f574c94035b8
+size 3200