---
license: apache-2.0
base_model: Qwen/Qwen3-VL-2B-Instruct
library_name: peft
tags: [robotics, navigation, objectnav, pointnav, vision-language-action, habitat, flow-matching]
---

# longnav-objectnav-flow-nopose-cotrain-2p5hz

Continuous-control ObjectNav policy at a **2.5 Hz observation rate**, trained on a
**three-way mixture** of pose-free ObjectNav, pose-injected ObjectNav and long-goal
PointNav (1 : 1 : 2), for 12,000 steps. Flow-matching action head, LoRA over Qwen3-VL-2B.

**Evaluated with no pose input at all**, and it is the strongest ObjectNav result in this
family at a matched 70 s budget, as well as the most path-efficient of any checkpoint we
have measured.

## What this is

A Qwen3-VL-2B backbone with LoRA adapters and a **flow-matching action head** that emits a
*timed trajectory* -- a chunk of 20 cumulative relative planar poses `[dx, dy, dtheta]`
spaced **0.04 s** apart -- tracked by a PID controller on a holonomic mobile base in a
Bullet-simulated Habitat scene. Most ObjectNav agents emit discrete actions executed by
kinematic teleport; this one drives real base dynamics.

**The headline is what it does *not* need.** The checkpoint declares a `<pose>` modality and
was trained with pose on half its ObjectNav data -- but at evaluation the pose feed is
switched off entirely (no `<pose>` marker in the conversation, no values supplied), and it
performs *better* than the same weights with pose enabled. On 101 paired episodes:

| arm | oracle success | oSPL (corrected) |
|---|---|---|
| pose injected | 0.614 | 0.289 |
| **no pose at all** | **0.663** | **0.348** |

So the navigation below is **purely visual**. Pose injection is carried for the PointNav
half of the mixture, not because ObjectNav needs it.

## Training mixture

| component | ratio | rows | what |
|---|---|---|---|
| `objectnav_nopose` | 1 | 39,061 | human ObjectNav demos, **no `<pose>` marker, `obs_poses` column absent** |
| `objectnav_pose` | 1 | 39,061 | the same episodes, with pose injection |
| `pointnav` | 2 | 7,971 | generated PointNav, goals sampled 2-30 m with stratified distance |

Sampled per example at a stated ratio (never concatenated -- concatenation makes the
mixture a function of corpus size). Realised split 0.249 / 0.253 / 0.497. 12,000 steps at
effective batch 4 is **0.56 epochs**.

The two ObjectNav components are the *same* episodes, so each trajectory is seen twice as
often as the ratio suggests; that overlap is intentional -- the point is that half of
ObjectNav is seen without GPS.

Warm-started from a two-component (ObjectNav + PointNav) run at its step 2,600, itself
descended from a converged pose-injected ObjectNav model.

## Files

| file | what |
|---|---|
| `adapter/` | LoRA adapters (r=128, alpha=256) over `Qwen/Qwen3-VL-2B-Instruct` |
| `turn_vector_head.pt` | flow-matching action head + pose encoder |
| `turn_vector_head_config.json` | head config, including the `<pose>` modality spec |
| `trainer_state.json` | full training curve, for provenance |
| tokenizer / preprocessor files | as trained |

Optimizer and RNG state are **not** included -- this is an inference checkpoint, not a
resume point.

## YOU NEED CODE THAT IS NOT YET PUBLIC

Two changes this checkpoint depends on are **not in any public release** at the time of
upload:

* **`--no-pose-injection`** (harness, `habitat_physical_nav`). Without it there is no way
  to run the pose-free arm documented above. The existing pose ablations
  (`--pose-mirror`, `--pose-rotate-deg`, `--pose-foreign`) *corrupt* the injected values;
  none of them removes the modality. Feeding zeros is not equivalent -- it is an
  in-distribution-looking input the model never trained on.
* **The mixed-modality collator fix** (`longnav`). A source lacking the pose column raised
  `KeyError` in the collator; absent / `None` / empty now all mean "no values", with the
  row-count check still enforced against markers found in `input_ids`. This is what makes
  a pose-free training component possible at all.

Until those are pushed, the commands below will not run as written. Everything else
(`flow_rollout` backend, the eval harness, `relative_se2`) is unchanged.

## Running / evaluating it

The simulator and the model **cannot share a Python environment** -- `habitat_sim` and
`transformers`+`flash-attn` have conflicting dependency sets. The harness runs the policy
in a second interpreter over a unix socket; that is what `--policy-python` and
`--policy-sys-path` are for.

### The pose-free arm (how this model is meant to be run)

```bash
python scripts/eval_objectnav_policy.py \
  --episodes   <hm3d objectnav val split> \
  --scene-root <scene_datasets> \
  --output-dir <out> \
  --episode-ids "$(cat sample101_ids.txt)" \
  --ckpt       <this checkpoint> \
  --policy-python   <env with transformers> \
  --policy-sys-path <longnav src> \
  --policy-backend flow_rollout --policy-seed 1000 \
  --max-steps 175 --dt 0.04 --gap 10 \
  --success-distance 1.0 --distance-to VIEW_POINTS \
  --seed 0 --navmesh dataset --fresh-sim-per-episode \
  --auto-stop --auto-stop-delay 15 \
  --no-pose-injection \
  --record-video --pid-preset baseline --keep-tracking-series
```

Drop `--no-pose-injection` to run the pose-injected arm instead; every other flag is
identical, which is what makes the two directly comparable.

Verify the manipulation took: the policy-server banner prints a `pose=<pose>` line when
injection is active and **no pose line at all** when it is not.

### The three settings that silently ruin the numbers

**`--gap` must match the training corpus.** `--dt` is 0.04 for every model in this family;
the tick rate never changes. What changes is `--gap`, the ticks executed per observation:

| trained on | observation rate | `--gap` | chunk length |
|---|---|---|---|
| this model (`obs2.5hz`) | 2.5 Hz | **10** | 20 |
| 5 Hz sibling | 5 Hz | 5 | 10 |
| 1 Hz sibling | 1 Hz | 25 | 30 |

A wrong `--gap` executes a fraction of each chunk and discards the rest. It does not error;
it presents as a slow, weak policy. Confirm from the checkpoint rather than the name:

```bash
python -c "import json; c=json.load(open('turn_vector_head_config.json')); print(c['fm_n_ticks'])"
# 20  -> 2.5 Hz -> --gap 10
```

**Budget is in policy steps, not seconds.** One policy step is `gap * dt` seconds, so the
same `--max-steps` buys different sim time at different rates. **Compare models at matched
sim time.** For this model: `--max-steps 175` = **70 s**, `350` = 140 s.

**`--navmesh dataset`, not `robot`.** On `robot` the agent is a physics body not
constrained to the mesh, so `snap_point` can resolve its proxy across an island boundary
and the metric silently starts measuring a different goal instance. Screening also differs
(101 episodes scored vs ~70), so the two are not comparable at all.

`--fresh-sim-per-episode` is likewise not optional: without it simulator state leaks
between episodes.

## Results

HM3D ObjectNav val, fixed 101-episode sample, `--navmesh dataset`, success distance 1.0 m,
**70 s of simulated time (`--max-steps 175` at `--gap 10`)**, no pose injected. All arms
paired on the same episodes.

| metric | value |
|---|---|
| oracle success | **0.663** |
| oracle SPL (as reported) | 0.379 |
| oracle SPL (corrected, see below) | **0.348** |
| NPWPL | **0.420** |
| closest approach | 2.25 m |
| path length at closest approach | 14.26 m |
| total path length | 19.83 m |

Against the other checkpoints in this family, same episodes, same 70 s budget:

| model | oracle | oSPL (corrected) | NPWPL |
|---|---|---|---|
| ObjectNav-only, pose (2.5 Hz) | 0.535 | 0.265 | 0.358 |
| ObjectNav+PointNav 1:1, pose | 0.564 | 0.316 | 0.407 |
| ObjectNav+PointNav 1:1, long-goal, pose (final) | 0.495 | 0.264 | 0.389 |
| **this model, no pose** | **0.663** | **0.348** | **0.420** |

The corrected oSPL of 0.348 is the highest of **any** checkpoint we have evaluated, at any
budget. Against the two-component long-goal run's final checkpoint the difference is
significant: oracle McNemar **p = 0.002**, oSPL delta **+0.085, 95% CI [+0.024, +0.146]**.

**Read the oracle metrics, not `success`/`SPL`.** This checkpoint has **no stop
classifier**; evaluation runs with `--auto-stop`, so termination is harness-driven and
`success` mostly measures `--auto-stop-delay`. `oracle_success` (did it ever reach the
goal) and NPWPL (progress toward the goal, normalised by start distance and weighted by
path length) are stop-independent.

**`oracle_spl` as reported is overstated.** The `OracleSPL` measure freezes path length
incorrectly and returns 1.000 on episodes where the agent travelled further than the
geodesic before first reaching the goal. Recompute it as
`start / max(start, path_at_closest_approach)`, gated on oracle success. Prefer the
corrected column.

## Limitations

* **No stop head** -- the policy cannot terminate its own episodes.
* **Chained PointNav is weak.** Given a second point goal mid-episode, the two-component
  sibling reached it in 12% of attempts even with an ample budget, and on failures it
  typically ended up *nearer the goal it had already reached* than the new one -- i.e. it
  does not reliably retrieve a goal announced once, deep in context. This model's PointNav
  component is there to teach a motion repertoire, not goal-following; do not deploy it as
  a point-goal navigator.
* **The efficiency lead is a 70 s result; the oracle lead holds at 140 s.** Doubling the
  budget (`--max-steps 350`) raises oracle success to **0.733** -- still the best of the
  flow-head family, and ahead of the 1:1 sibling's 0.703 -- but corrected oSPL falls to
  **0.325** and NPWPL to 0.366, so the 1:1 sibling (0.339 / 0.384) matches it on
  efficiency there. Within this checkpoint the 175 -> 350 gain is not significant
  (McNemar p = 0.14; oSPL delta -0.024, 95% CI [-0.079, +0.031]). Read the 70 s numbers as
  the efficiency claim and the 140 s numbers as the reachability claim.
* **0.56 epochs** over its mixture -- undertrained rather than converged.
* Trained on MP3D human demonstrations, evaluated on HM3D -- cross-dataset.
* n=101 has a minimum detectable difference of roughly 0.10 on a success rate. Differences
  smaller than that in the tables above are not resolvable.

## Related

* `Aasdfip/longnav-objectnav-flow-pose-2p5hz` -- the pose-injected ObjectNav-only sibling
  this lineage descends from (oracle 0.535 at the same budget)
* `Aasdfip/longnav-objectnav-flow-pose-1hz` -- 1 Hz sibling