---
language: en
license: apache-2.0
base_model: Qwen/Qwen3.5-0.8B
tags:
  - android
  - ui-automation
  - accessibility
  - vision-language
  - lora
  - dora
  - peft
---

# Cerebellum

Cerebellum is a local Android UI action model. It takes the current screen,
accessibility tree, task goal, and recent action history, then emits one compact
action code for the next UI step.

The current milestone is a Qwen/Qwen3.5-0.8B adapter trained for constrained
history-aware action prediction. It is intended as the supervised checkpoint
before moving into a narrow RL curriculum.

## Current Milestone

**Adapter:** `checkpoints/qwen35_rl_taskcool_sft268_env5_accum4_20260605/current`

**Base model:** `Qwen/Qwen3.5-0.8B`

**Live AndroidWorld/APK eval:** `configs/android_world_mixed100_eval_infraclean_20260609.json`, 100 mixed cases, 5 emulators, target app preopened, collector APK state source.

| Metric | Result |
| --- | ---: |
| Success | 51/100 = 51.0% |
| Infra skips | 0 |
| Average steps | 11.9 |

Strong task families: contacts (`11/11`), audio record (`4/4`), clock (`9/10`), WiFi/Bluetooth/system toggles except brightness (`16/21` system overall), camera photo (`3/3`).

Unsolved task families in this checkpoint: Markor/file workflows, calendar delete, browser maze, OsmAnd, camera video, and brightness sliders. `Expense*` and `SimpleSms*` were excluded from the current live eval because local AndroidWorld app setup currently leaves those apps in broken first-run states.

## Previous SFT Milestone

**Adapter:** `checkpoints/qwen35_history_actions_eosK_aw2_accum4_from_step140/current`

**Base model:** `Qwen/Qwen3.5-0.8B`

**Held-out eval:** shards `18,19`, `n=100`, seed `44103`

| Metric | Result |
| --- | ---: |
| Exact action string | 78/100 = 78.0% |
| Action type | 85/100 = 85.0% |
| Tap/long-press element label | 14/21 = 66.7% |

Per group:

| Group | Result |
| --- | ---: |
| Tap / long press | 14/21 = 66.7% |
| Type text | 20/25 = 80.0% |
| Scroll | 16/21 = 76.2% |
| Wait | 28/31 = 90.3% |
| System | 0/2 = 0.0% |

Scroll misses were mostly action selection errors, not direction errors:

```text
scroll_failures: {'wrong_action': 4, 'wrong_direction': 1}
scroll_confusion: {'D->T': 1, 'D->W': 1, 'U->T': 1, 'D->U': 1, 'L->W': 1}
```

Latency from the eval run was `p50=0.47s`, `avg=0.86s`, with the first warmup
sample causing a `max=25.55s`.

## Input Format

Training and eval use a chat prompt with up to four history frames and one
current frame.

```text
Task: {goal}

Step 1:
<image>
Action taken: {history action text}
{optional compact subtree for acted element}

...

Current screen:
<image>
{compact accessibility tree with randomized SoM labels}

Actions: T <label>=tap, P <label>=long-press, K {text}=type text, U/D/L/R=scroll, B=back, H=home, W=wait, F=done, I=impossible.
Next action:
```

History tap and type actions include an outlined image region plus a compact
subtree for the element acted on. Current-screen tappable elements use
randomized uppercase bigram labels. The output is constrained at inference:

- First token is constrained to valid action codes.
- `T` and `P` element labels are constrained to labels visible on the current screen.
- `K` text is generated freely until EOS.
- Single-token actions (`U/D/L/R/B/H/W/F/I`) terminate immediately.

## Output Format

| Code | Meaning | Example |
| --- | --- | --- |
| `T <label>` | Tap labelled element | `T UE` |
| `P <label>` | Long-press labelled element | `P JD` |
| `K {text}` | Type text into focused field | `K hello world` |
| `U` | Scroll up | `U` |
| `D` | Scroll down | `D` |
| `L` | Scroll left | `L` |
| `R` | Scroll right | `R` |
| `B` | Back | `B` |
| `H` | Home | `H` |
| `W` | Wait | `W` |
| `F` | Done | `F` |
| `I` | Impossible | `I` |

## Training Setup

Current trainer:

```text
scripts/train_qwen35_history_actions.py
```

Important settings for the current milestone:

```text
model_name: Qwen/Qwen3.5-0.8B
init_adapter: checkpoints/qwen35_history_actions_eosK_aw2_accum3_from_step210/current
optimizer: paged_adamw_8bit
learning_rate: 1e-4
grad_accum: 4
effective_batch: 48
quota: tp=5,k=2,scroll=2,wait=2,system=1
action_weight: 2.0
margin_weight: 0.25
margin_target: 8.0
history_len: 4
min_history: 4
history image width: 232
current image width: 464
max_raw_tree_chars: 4000
max_seq_len: 2600
```

The trainer uses focused loss computation on the answer positions rather than
materializing full-prompt vocab logits for every token. This made larger
effective batches practical on a 12 GB GPU.

## Evaluation

Current held-out eval command shape:

```powershell
docker run --rm --gpus all `
  -v "${PWD}:/workspace" `
  -v "$env:USERPROFILE\.cache\huggingface:/root/.cache/huggingface" `
  -w /workspace cerebellum:torch27-fla bash -lc `
  "python scripts/eval_qwen35_history_actions.py \
    --adapter checkpoints/qwen35_history_actions_eosK_aw2_accum4_from_step140/current \
    --n 100 \
    --seed 44103 \
    --shards 18 19 \
    --quota tp=5,k=2,scroll=2,wait=2,system=1 \
    --hist-width 232 --hist-gutter 16 \
    --curr-width 464 --curr-gutter 32 \
    --max-raw-tree-chars 4000 \
    --max-seq-len 2600 \
    --max-pix-rows 12000 \
    --max-mm-tokens 2600"
```

## Repository Layout

```text
data/
  datasets/android_control_a11y.py   AndroidControl a11y loading, tree compression, history formatting
  som.py                             randomized set-of-mark label pool and rendering
model/
  cerebellum.py                      legacy Gemma model wrapper and helpers
scripts/
  train_qwen35_grounding.py          Qwen grounding trainer
  train_qwen35_history_tap.py        Qwen tap-history trainer
  train_qwen35_history_actions.py    current Qwen all-action history trainer
  eval_qwen35_grounding.py           Qwen grounding eval
  eval_qwen35_history_tap.py         Qwen tap-history eval
  eval_qwen35_history_actions.py     current Qwen all-action eval
```

## Docker Runtime

The `cerebellum:latest` image contains the CUDA/PyTorch model stack, Android SDK,
Android emulator, and a pinned AndroidWorld checkout. Source files are mounted at
runtime, so dependency changes need a rebuild but code edits do not.

```powershell
docker compose build
docker compose run --rm shell
```

For AndroidWorld/emulator work, KVM must be available. On Windows/WSL:

```powershell
powershell -ExecutionPolicy Bypass -File scripts\ensure_wsl_kvm.ps1
docker compose up androidworld
```

See `docs/docker_androidworld.md` for the full shareable setup and smoke tests.

## Loading The Adapter

```python
from peft import PeftModel
from transformers import AutoModelForImageTextToText, AutoProcessor

base = AutoModelForImageTextToText.from_pretrained(
    "Qwen/Qwen3.5-0.8B",
    torch_dtype="auto",
    device_map="auto",
)
model = PeftModel.from_pretrained(
    base,
    "dmitchelljackson/cerebellum-qwen35-history-actions-lora",
)
processor = AutoProcessor.from_pretrained("Qwen/Qwen3.5-0.8B")
model.eval()
```

## Next Work

- Start a narrow RL curriculum from this milestone.
- Keep constrained decoding for action and visible-label selection.
- Use short tasks first so wrong taps can be recovered rather than treated as terminal failure.
- Preserve the current held-out eval as a regression check while tuning RL.