---
title: LLaDA-UI
emoji: 🖱️
colorFrom: indigo
colorTo: purple
sdk: gradio
sdk_version: 5.50.0
app_file: app.py
short_description: GUI grounding & agent with a block-diffusion VLM
python_version: "3.12"
startup_duration_timeout: 1h
pinned: false
---

# LLaDA-UI — GUI grounding & agent

Demo of [inclusionAI/LLaDA-UI](https://huggingface.co/inclusionAI/LLaDA-UI), a 16.9B MoE
vision-language GUI agent that decodes with **block-wise diffusion** instead of
autoregression.

Two modes:

- **Grounding** — give it a screenshot plus an instruction ("open the Microsoft store")
  and it returns the centre point of the matching UI element, normalised to `0–999`.
- **GUI agent** — give it a screenshot plus a high-level task and it returns
  `<think>…</think><action>…</action>` using the released Mobile / Desktop / Web action
  spaces.

## Faithfulness to the reference implementation

This Space ports the authors' `inference/inference_hf.py` and
`inference/sglang_client.py` onto ZeroGPU:

- The checkpoint's shipped `chat_template.jinja` is **overridden** with the training
  template (the shipped one injects `detailed thinking off`, which the model never saw
  and which produces garbage output).
- The image placeholder is appended **after** the instruction text, as in the reference.
- Decoding uses the model's own block-diffusion `generate()` with the reference
  hyper-parameters (`gen_length=32, steps=32, block_length=32, temperature=1.0,
  threshold=0.99` for grounding; `temperature=0.0` for the agent).
- Coordinate parsing matches the reference `parse_point` (4 numbers → bbox centre,
  2 numbers → point, `[-1,-1]` → infeasible).
- Agent system prompts are extracted verbatim from
  `inference/examples/{mobile,desktop,web}.json`.

The model code is vendored from the Hub repo with three fixes: repo-internal imports
rewritten, the missing `config.vision_pad_token_id` defaulted to `image_token_id`
(157187, the value the reference hardcodes), and the text decoder's declared attention
implementation set to `sdpa` (its `ATTENTION_CLASSES` maps every key to the SDPA class
anyway, and the block-diffusion 4-D mask requires it).

Two exact optimisations are applied on top: the vision tower is memoised across
denoising steps (only `input_ids` changes between steps), and `logits_to_keep` is set to
the active block so `lm_head` isn't run over the whole sequence each step.

Screenshots are downscaled if needed so the visual token count stays within the text
backbone's 8192-token context.

## Examples

Example screenshots are from
[OS-Copilot/ScreenSpot-v2](https://huggingface.co/datasets/OS-Copilot/ScreenSpot-v2)
(Apache-2.0) — the same benchmark the reference implementation's `--verify-ssv2` mode
evaluates on.