---
license: cc-by-nc-4.0
base_model:
  - jinaai/jina-embeddings-v5-omni-nano
tags:
  - axera
  - ax650
  - embeddings
  - retrieval
  - multimodal
  - image
  - audio
  - video
---

# jina-embeddings-v5-omni-nano-retrieval on AXERA NPU

This repository packages the retrieval-task AX650 deployment of `jinaai/jina-embeddings-v5-omni-nano`.

The package contains compiled AX650 runtime artifacts and runs with `axllm serve` on AX650 using the expected bidirectional text-embedding behavior.
It was rebuilt with the correct text RoPE base and passes the packaged text, image, audio, and frame-based video precision checks.

## Current Validation Status

- Target platform: AX650 / NPU3
- Current service model id: `AXERA-TECH/jina-embeddings-v5-omni-nano-retrieval-AX650-P128-CTX2047`
- Vision encoder static shape: `256x256`
- Vision soft tokens per frame: `64`
- Default audio encoder static profile:
  - short audio: `8s`, `16kHz`, mono, `800` mel frames, `200` soft tokens
- Optional packaged audio encoder:
  - long audio: `30s`, `16kHz`, mono, `3000` mel frames, `750` soft tokens
  - this file is packaged but not loaded by the default `config.json`
- Audio request sequence length with the packaged `query` prompt:
  - `8s` audio: `200` soft tokens, about `220` total LLM prefill tokens
  - `30s` audio: `750` soft tokens, about `772` total LLM prefill tokens if the optional long profile is enabled
- Current LLM prefill build:
  - `prefill_len = 128`
  - warm-prefill groups = `128 / 256 / 384 / 512 / 640 / 768 / 896`
  - effective `prefill_max_token_num = 1024`

Minimum board resources for the default package configuration:

- Default loaded audio profile: `jina_v5_omni_nano_audio_8s.axmodel`
- Optional `jina_v5_omni_nano_audio_30s.axmodel` is not loaded by default.
- Runtime CMM requirement: about `1.14 GiB` (`1191296 KB`).
- Runtime Linux private memory requirement: about `80 MiB` anonymous memory.

| Item | Value |
|---|---:|
| Additional CMM required | `1191296 KB` (`~1.14 GiB`) |
| Linux RAM by process VmRSS | `84352 KB` (`~82 MiB`) |
| Linux RAM by process PSS | `83369 KB` (`~81 MiB`) |
| Process private memory | `83048 KB` (`~81 MiB`) |
| Process anonymous RSS | `79456 KB` (`~78 MiB`) |
| Process VmSize | `1572856 KB` |
| `/v1/models` ready time | `15 s` |

If both packaged audio profiles are enabled by adding `filename_audio_encoder_axmodel_long` back to `config.json`, the board resource profile is:

- Loaded audio profiles: `jina_v5_omni_nano_audio_8s.axmodel` and `jina_v5_omni_nano_audio_30s.axmodel`
- Runtime CMM requirement: about `2.08 GiB` (`2180544 KB`).
- Runtime Linux private memory requirement remains about `80 MiB` anonymous memory because `release_axmodel_buffer_after_init` releases the host-side `.axmodel` file buffers.

| Item | Value |
|---|---:|
| Additional CMM required | `2180544 KB` (`~2.08 GiB`) |
| Linux RAM by process VmRSS | `84220 KB` (`~82 MiB`) |
| Linux RAM by process PSS | `83412 KB` (`~81 MiB`) |
| Process private anonymous memory | `79500 KB` (`~78 MiB`) |
| Process VmSize | `2264504 KB` |
| `/v1/models` ready time | `16 s` |

`CMM` means AXERA contiguous multimedia memory. `RSS` means resident set size in Linux process memory. `PSS` means proportional set size.
The AXModels primarily occupy CMM after initialization. The package enables `release_axmodel_buffer_after_init`, so the runtime releases the host-side `.axmodel` file buffers after `AX_ENGINE_CreateHandle` and context creation.
Without this release path, the AX650 runner keeps a private anonymous copy of each loaded `.axmodel`; that copy shows up as Linux `VmRSS`/`PSS` in addition to CMM and can exceed `1 GiB` for this package.
This is why the current minimum Linux RAM requirement is based on process `VmRSS`, `PSS`, and `Anonymous`, not on file cache or `.axmodel` file sizes.

## Functional Status

This package can:

- start `./bin/axllm serve .` on AX650
- expose `/v1/models` and `/v1/embeddings`
- run text, image, audio, and frame-based video embedding requests without depending on Hugging Face `safetensors` at runtime

This package is retrieval-only. The upstream model family defines other task adapters, but this AX650 release contains the retrieval route and its cached retrieval reference cases only.

The packaged runtime files are:

- `llama_p128_l0_together.axmodel` ... `llama_p128_l11_together.axmodel`
- `llama_post.axmodel`
- `jina_v5_omni_nano_vision_256x256.axmodel`
- `jina_v5_omni_nano_audio_8s.axmodel`
- `jina_v5_omni_nano_audio_30s.axmodel`
- `model.embed_tokens.weight.bfloat16.bin`
- `jina_v5_omni_tokenizer.txt`
- `jina_v5_omni_tokenizer/`
- `bin/axllm`

## Performance

This package exposes `/v1/embeddings` rather than token streaming, so chat-style `TTFT` is not the right primary metric.
For an embedding model on AX650, the useful execution metric is:

- text: `LLM prefill` only
- image / video: `encoder + LLM prefill`
- audio: `audio encoder + LLM prefill`

There is no decode stage in the normal embedding path, so a `TTFT`-style number mostly collapses into prefill completion plus service overhead.
If you measure only the HTTP response time of `axllm serve`, lightweight text and image requests can look artificially similar because fixed server overhead dominates them.

All values below were re-measured on the validated AX650 / NPU3 board.
Each number is the average of `3` repeated direct-runtime runs after model initialization.
These measurements exclude network overhead and client-side preprocessing outside the packaged runtime path.

| Scenario | Prompt | LLM input tokens | Soft tokens | Output shape | Media encoder / prepare | Standalone encoder axmodel | Prepare overhead | LLM prefill | Runtime total |
|---|---|---:|---:|---:|---:|---:|---:|---:|---:|
| Text document | `document` | `19` | `-` | `[1, 768]` | `-` | `-` | `-` | `68.88 ms` | `68.88 ms` |
| Text query | `query` | `12` | `-` | `[1, 768]` | `-` | `-` | `-` | `61.59 ms` | `61.59 ms` |
| Image | `query` | `91` | `64` | `[1, 768]` | `17.24 ms` | `16.42 ms` | `0.82 ms` | `65.66 ms` | `82.91 ms` |
| Video (`3` frames) | `query` | `219` | `192` | `[1, 768]` | `52.49 ms` | `49.25 ms` | `3.24 ms` | `148.40 ms` | `200.89 ms` |

Standalone encoder `.axmodel` latency was measured with `ax_run_model -r 50 -w 10` on AX650:

| Encoder axmodel | Output tokens | CMM size | Pulsar2 version | Min | Max | Avg |
|---|---:|---:|---|---:|---:|---:|
| `jina_v5_omni_nano_vision_256x256.axmodel` | `64` | `109390730 Bytes` | `6.0 563427a6` | `16.343 ms` | `16.494 ms` | `16.416 ms` |
| `jina_v5_omni_nano_audio_8s.axmodel` | `200` | `724467726 Bytes` | `6.0 563427a6` | `209.688 ms` | `210.452 ms` | `210.087 ms` |
| `jina_v5_omni_nano_audio_30s.axmodel` | `750` | `1030304014 Bytes` | `6.0 563427a6` | `3783.250 ms` | `3801.552 ms` | `3792.821 ms` |

`Media encoder / prepare` is the runtime media path before LLM prefill.
For the direct-runtime rows above, image/video inputs were already prepared for the benchmark path, so the measured prepare overhead is small.
`Standalone encoder axmodel` is the bare encoder model latency reported by `ax_run_model`; it excludes real media decode, preprocessing, tokenizer work, and LLM input assembly.
For video rows, `Standalone encoder axmodel` is `per-frame vision encoder latency x 3`.
`Prepare overhead` is `Media encoder / prepare - Standalone encoder axmodel`.

Audio is different on this `3 GB` board:

- the packaged `axllm serve` path is validated and returns correct audio embeddings
- standalone `8s` and `30s` audio encoder `.axmodel` latency is published in the table above
- but the current Python direct-benchmark implementation was OOM-killed during this measurement pass
- that failure is a board-side benchmark-method issue in Linux userspace memory, not a proof that the packaged AX650 runtime path cannot run audio embeddings
- because of that, this README does not publish a misleading full audio `prepare + LLM prefill` split-latency number yet

## Current Board Precision

Current board-side `axllm serve` precision against the packaged Hugging Face reference embeddings is:

| Modality | Case | Output shape | Cosine vs HF |
|---|---|---:|---:|
| Text | `embedding_doc` | `[1, 768]` | `0.999655` |
| Text | `red_planet_query` | `[1, 768]` | `0.999539` |
| Image | `vision_sample` | `[1, 768]` | `0.993549` |
| Audio | `audio_test_chunk0_8s_wav` | `[1, 768]` | `0.989693` |
| Video | `video_visual_red_panda_openai_mp4` | `[1, 768]` | `0.997040` |

Key observations:

- The previous severe service mismatch is fixed.
- The previous `embedding_doc = 0.9604` result was traced to an LLM build config issue, not to the OpenAI-compatible service wrapper.
- `jina-embeddings-v5-omni-nano` uses `text_config.rope_parameters.rope_theta = 1000000.0`; the earlier `llama` build path silently fell back to `10000.0`.
- The current LLM AXModels were rebuilt after flattening `rope_theta` into `text_config.rope_theta` for `pulsar2 llm_build`.
- With the corrected RoPE base, the default packaged precision cases are in the `0.9896 ~ 0.9997` cosine range.
- The default `8s` audio profile was rebuilt with a clean calibration set from `audio_test_chunk0_30s.wav` and `audio_test_chunk1_30s.wav`, rather than the earlier mixed historical calibration tar.
- The clean-calibration `8s` audio result is still only slightly higher than the earlier mixed-calibration result (`0.989693` vs about `0.989344` cosine), so the remaining gap is not primarily explained by the audio encoder calibration tar.
- The default `8s` audio profile is usable but lower than the optional `30s` profile measured during adaptation: `0.989693` cosine for the packaged `audio_test_chunk0_8s_wav` case.
  The previous `5s` audio profile produced only about `0.94` cosine after LLM prefill, so it is not used as a default packaged precision case.

## Board Precision Validation Flow

The board-side comparison does not run the original Hugging Face model.
The intended flow is:

- Run the original `jinaai/jina-embeddings-v5-omni-nano` model once on the server for each fixed test case.
- Save the resulting HF embedding as `torch_embedding.npy` under the matching `python/testdata/service_cases/<case>/` directory.
- Package the same case metadata and assets used by the HF run.
- On AX650, start `axllm serve .`, call `/v1/embeddings`, and compare the returned AXERA embedding with the packaged `torch_embedding.npy`.

This guarantees that AX650 validation uses the same inputs as the server-side HF reference, while the board only needs the packaged `.axmodel`, `.bin`, tokenizer, assets, and cached reference `.npy` files.
The board package does not require upstream `.safetensors` files.

Run the packaged comparison on the board with:

```bash
python3 python/compare_openai_api_vs_hf_multimodal.py
```

The script fails if any packaged case is missing its cached `torch_embedding.npy` reference.

## Conversion Note

If you rebuild the LLM AXModels from the original Hugging Face checkpoint, check the RoPE config before running `pulsar2 llm_build`.
The upstream nano config stores the text RoPE base as:

```json
"text_config": {
  "rope_parameters": {
    "rope_theta": 1000000.0
  }
}
```

The current AXERA `llama` `llm_build` route does not read `text_config.rope_parameters.rope_theta`.
It reads the flattened field `text_config.rope_theta`.
Before compilation, make sure the `llm_build_input/config.json` contains:

```json
"text_config": {
  "rope_theta": 1000000.0
}
```

The packaged conversion script handles this automatically.
If you prepare `llm_build_input/config.json` manually, copy the value yourself and confirm the build log prints `rope_theta=1000000.0`.
If the build silently uses the default `10000.0`, the generated LLM AXModels will have noticeably worse embedding precision.

## Runtime Compatibility Notes

The following compatibility details are required to reproduce the validated precision:

- `jina-embeddings-v5-omni-nano` uses a bidirectional EuroBERT / `LlamaModel` text tower, but the service embedding path was still building a decoder-style causal prefill mask.
- The embedding path in `axllm` must use bidirectional masking through `prefill_mask_mode`.
- The service tokenizer path was also missing the final end-of-text token used by the Python reference route.
- For `Qwen3Omni`, the correct end-of-text token id is not the older hard-coded `151643`; it must be resolved from the loaded tokenizer and is `128001` for this Jina nano package.
- LLM precision depends on the `llama` `llm_build` path receiving the flattened `text_config.rope_theta` value.
  The nano text model requires RoPE theta `1000000.0`, and the corrected package was rebuilt with that value.

With these settings, the service-side outputs match the direct board-side `axmodel` route.

## Packaged Test Cases

The package includes cached Hugging Face reference embeddings under:

```text
python/testdata/service_cases/
```

Current packaged cases:

- `embedding_doc`
- `red_planet_query`
- `vision_sample`
- `audio_test_chunk0_8s_wav`
- `video_visual_red_panda_openai_mp4`

The package also contains `audio_test_chunk0_30s_wav` as an optional long-audio reference case, but the default comparison script does not run it because the default `config.json` only loads the `8s` audio profile.

All cached reference embeddings use the upstream retrieval route and output shape `[1, 768]`.
The packaged `meta.json` files define the exact text, prompt name, media asset path, and soft-token count used for both the server-side HF reference and the board-side API request.

## Known Limitation

- Re-measure direct audio split latency if a separate direct-runtime benchmark is needed; the packaged service audio path is already validated.
- Audio encoder loading is controlled by `filename_audio_encoder_axmodel_short` and `filename_audio_encoder_axmodel_long` in `config.json`.
  If only one field is configured, only that audio profile is loaded; if both are configured, the runtime selects the short profile for short audio and falls back to the long profile for longer audio.
- The default `config.json` only sets `filename_audio_encoder_axmodel_short`.
  Requests longer than the loaded `8s` profile are truncated to the loaded profile; enable `filename_audio_encoder_axmodel_long` if long-audio quality is required.