---
base_model: Qwen/Qwen3.5-0.8B
library_name: peft
pipeline_tag: image-text-to-text
license: apache-2.0
tags:
- android
- android-world
- ui-automation
- accessibility
- vision-language
- peft
- lora
- reinforcement-learning
---

# Cerebellum Qwen3.5 AndroidWorld RL Adapter

This is a PEFT adapter for `Qwen/Qwen3.5-0.8B` trained for fast local Android UI control.

## What this is for

This model is not meant to replace a remote frontier computer-use agent. It is meant to sit
underneath one, as a small, fast, cheap execution layer between a remote frontier model and an
Android computer-use harness.

The intended split is:

- The **frontier model** does the higher-level reasoning and planning: interpreting the task,
  deciding what should happen next, recovering when something goes wrong.
- **This 0.8B model**, running locally, executes the low-level Android UI interaction the
  frontier model delegates to it -- find and tap the right element, type into the right field,
  scroll the right list. No screenshot round-trip to a remote model is needed for those steps.

The hypothesis being tested is that a large fraction of Android UI interaction does not
actually require frontier-model intelligence, so delegating it to a specialized local model
can cut inference latency and cost while reserving the frontier model for the parts where its
capability is doing real work. Whether that hypothesis holds is what the evaluation below is
trying to measure; it is not settled.

The whole project is developed and trained locally on constrained consumer hardware -- a single
RTX 3060 with 12GB of VRAM. That constraint is deliberate rather than incidental: the point is
to find out how far a small, cheap, locally trainable model gets on this job, and a model that
needs a datacenter to train is not the thing being investigated.

## Interface

It consumes a task goal, the current screenshot with randomized set-of-mark labels, the
compact accessibility tree, and up to four recent history frames/actions. It emits one
compact next-action command.

The intended runtime is the Cerebellum AndroidWorld/APK harness included in this upload.
The model is not a general chat model and should be decoded with the constrained action
grammar used by `rl_harness.policy_qwen35`.

The pre-RL supervised checkpoint is preserved in this same repository under
`sft_freeze/`. The root adapter is the RL milestone; load `sft_freeze/` directly
when a clean SFT fallback is needed.

## Status

This checkpoint is a milestone in active work, not a finished result. The 51% held-out
AndroidWorld number below is where this particular training run landed, and several whole task
families are still at zero. A second iteration is underway, built on a stronger SFT dataset and
a revised approach to the online-learning stage; the numbers here should be read as a snapshot
of the first iteration.

## Action Grammar

```text
T <label>  tap visible labelled element
P <label>  long-press visible labelled element
K <text>   type text into focused field
U/D/L/R    scroll up/down/left/right
B          Android back
H          Android home
W          wait
F          done
I          impossible
```

At inference time:

- The first token is constrained to valid action codes.
- `T` and `P` labels are constrained to labels visible on the current screen.
- `K` text is generated freely until EOS.
- Single-token actions terminate immediately.

## Current Evaluation

Latest live AndroidWorld/APK eval, on a held-out 100-task split:

- Split: `configs/android_world_mixed100_eval_infraclean_20260609.json`
- Runner: `scripts/run_android_world_parallel_eval.py`
- Harness: 5 Android emulators, collector APK state source, target app preopened
- Adapter: `checkpoints/qwen35_rl_taskcool_sft268_env5_accum4_20260605/current`
- Cases: 100 mixed controller-feasible AndroidWorld tasks
- Exclusions: `Expense*` and `SimpleSms*` were excluded because local AndroidWorld app setup currently leaves those apps in broken first-run states.

Overall result:

| Metric | Value |
| --- | ---: |
| Success | 51/100 = 51.0% |
| Infra skips | 0 |
| Average steps | 11.9 |

By family:

| Family | Result |
| --- | ---: |
| Audio record | 4/4 = 100% |
| Contacts | 11/11 = 100% |
| Clock | 9/10 = 90% |
| System | 16/21 = 76.2% |
| Recipe | 8/12 = 66.7% |
| Camera | 3/8 = 37.5% |
| Browser | 0/3 = 0% |
| Calendar | 0/8 = 0% |
| Files | 0/2 = 0% |
| Map/OsmAnd | 0/3 = 0% |
| Markor | 0/18 = 0% |

Notable task-level behavior:

- Works well: WiFi/Bluetooth toggles, stopwatch, contacts forms, audio recording, camera photo.
- Partially works: recipe duplicate deletion.
- Not solved: Markor/file workflows, calendar deletion, browser maze, OsmAnd, camera video, brightness slider.
- Slider/range labels were added to the harness after this adapter had already mostly trained, so brightness failures are expected and need targeted SFT/RL.

## SFT Curriculum

The supervised curriculum that produced the pre-RL adapter was staged rather than trained as one mixed task from scratch:

1. Tap/long-press grounding first, using randomized visible labels and constrained label decoding.
2. History-aware tap selection next, with up to four prior frames/actions.
3. Full action grammar after grounding stabilized: `T/P`, `K`, scroll, wait, and system actions.
4. Larger effective batches were used once answer-position-only loss made memory practical.
5. Training kept random screen samples, randomized labels, and held-out shard evals to catch overfit.

Important SFT choices:

- Current screenshot width `464`; history screenshot width `232`.
- Up to four history frames with images.
- Older history can be summarized as compact action text.
- Accessibility trees are compacted but capped/filtered rather than silently truncating target-bearing examples.
- `T/P` labels are generated from the current visible label set only.
- `K` is free text and is not constrained to the label set.
- The best SFT checkpoint before RL was preserved separately as a fallback milestone.

## RL Curriculum

The RL curriculum is online AndroidWorld training through the collector APK harness. It uses the same prompt/action grammar as SFT, but the state comes from live emulators rather than saved dataset rows.

Current RL setup:

- 5 Android emulators in parallel.
- Collector APK provides screenshots, accessibility tree, range metadata, and action-frame history.
- Target app is preopened for each episode.
- Maximum 20 actions per episode.
- Four rollout batches are accumulated before one optimizer update.
- Optimizer: `paged_adamw_8bit`.
- Learning rate: `1e-6`.
- Sampling temperatures: action `0.8`, label `0.7`.
- Bucket weights: `tp=1.5,k=1.0,scroll=1.2,wait=0.5,system=1.0`.
- Curriculum state is persisted in `rl_runtime_state.json` so restarts keep the baseline, cooldowns, and replay counts.

Reward/curriculum policy:

- Success is AndroidWorld task success.
- The trainer tracks a moving reward baseline and trains from advantage over that baseline.
- Tasks that repeatedly fail can enter cooldown.
- Easy tasks are downweighted but not removed, to reduce forgetting.
- Successful trajectories are kept for positive replay.
- Negative shaping is intentionally light in this checkpoint: small penalties for execution errors, invalid typing, missing tap targets, false terminal, and leaving the target app; repeat/no-change/dead-scroll penalties are currently disabled.

## Included Harness Code

This upload includes the runtime/eval harness sources under `harness_code/`:

- `rl_harness/`: policy wrapper, APK/ADB state collection, AndroidWorld action bridge, history buffer
- `scripts/run_android_world_parallel_eval.py`: 5-emulator deterministic eval
- `scripts/train_android_world_rl_qwen35_parallel.py`: online RL trainer
- `scripts/run_android_world_curriculum.py`: single-env rollout/eval runner
- `scripts/setup_android_world_device.py`: AndroidWorld app setup helper
- `android/collector_apk/`: collector accessibility service used for screenshots/tree/range metadata
- `configs/`: current AndroidWorld curriculum/eval splits
- `docs/`: Docker/AndroidWorld and RL harness notes

## Loading

```python
from peft import PeftModel
from transformers import AutoModelForImageTextToText, AutoProcessor

base = AutoModelForImageTextToText.from_pretrained(
    "Qwen/Qwen3.5-0.8B",
    torch_dtype="auto",
    device_map="auto",
)
model = PeftModel.from_pretrained(base, "dmitchelljackson/cerebellum-qwen35-history-actions-lora")
processor = AutoProcessor.from_pretrained("Qwen/Qwen3.5-0.8B")
model.eval()
```

For correct behavior, use the constrained decoding in `harness_code/rl_harness/policy_qwen35.py`.

## Limitations

This is an experimental local controller checkpoint, not a complete Android agent.
It still lacks reliable finish (`F`) behavior, robust slider control, and long-horizon
planning for file/calendar/map/browser/Markor workflows. It should be treated as a
runtime milestone for further SFT/RL rather than a production-ready model, and it assumes a
higher-level planner is supplying the goals it executes against.