README.md

metadata

title: LLaDA-UI
emoji: 🖱️
colorFrom: indigo
colorTo: purple
sdk: gradio
sdk_version: 5.50.0
app_file: app.py
short_description: GUI grounding & agent with a block-diffusion VLM
python_version: '3.12'
startup_duration_timeout: 1h
pinned: false

LLaDA-UI — GUI grounding & agent

Demo of inclusionAI/LLaDA-UI, a 16.9B MoE vision-language GUI agent that decodes with block-wise diffusion instead of autoregression.

Two modes:

• Grounding — give it a screenshot plus an instruction ("open the Microsoft store") and it returns the centre point of the matching UI element, normalised to 0–999.
• GUI agent — give it a screenshot plus a high-level task and it returns <think>…</think><action>…</action> using the released Mobile / Desktop / Web action spaces.

Faithfulness to the reference implementation

This Space ports the authors' inference/inference_hf.py and inference/sglang_client.py onto ZeroGPU:

• The checkpoint's shipped chat_template.jinja is overridden with the training template (the shipped one injects detailed thinking off, which the model never saw and which produces garbage output).
• The image placeholder is appended after the instruction text, as in the reference.
• Decoding uses the model's own block-diffusion generate() with the reference hyper-parameters (gen_length=32, steps=32, block_length=32, temperature=1.0, threshold=0.99 for grounding; temperature=0.0 for the agent).
• Coordinate parsing matches the reference parse_point (4 numbers → bbox centre, 2 numbers → point, [-1,-1] → infeasible).
• Agent system prompts are extracted verbatim from inference/examples/{mobile,desktop,web}.json.

The model code is vendored from the Hub repo with three fixes: repo-internal imports rewritten, the missing config.vision_pad_token_id defaulted to image_token_id (157187, the value the reference hardcodes), and the text decoder's declared attention implementation set to sdpa (its ATTENTION_CLASSES maps every key to the SDPA class anyway, and the block-diffusion 4-D mask requires it).

Two exact optimisations are applied on top: the vision tower is memoised across denoising steps (only input_ids changes between steps), and logits_to_keep is set to the active block so lm_head isn't run over the whole sequence each step.

Screenshots are downscaled if needed so the visual token count stays within the text backbone's 8192-token context.

Examples

Example screenshots are from OS-Copilot/ScreenSpot-v2 (Apache-2.0) — the same benchmark the reference implementation's --verify-ssv2 mode evaluates on.