README.md

---
license: apache-2.0
base_model: Qwen/Qwen3.5-35B-A3B
library_name: peft
tags:
- grpo
- trl
- reinforcement-learning
- rl-environment
- openenv
- p5js
- generative-art
- lora
datasets:
- HuggingEnvs/watercolour-reference-pool
- HuggingEnvs/watercolour-rollouts-judge-led
---

# watercolour-grpo-judge-led

A LoRA adapter for `Qwen/Qwen3.5-35B-A3B`, trained with GRPO to paint watercolours by
writing [p5.brush](https://github.com/acamposuribe/p5.brush) sketches. This is the run
with the **original reward mix** from [Surya Narreddi's
write-up](https://surya.website/rling-qwen-to-paint-with-code): the pairwise judge, the
term that carries the hand-rated reference pool, holds most of the weight. Of the three
runs in this project, this one optimises the curator's taste the hardest.

## Loading it, because the obvious way fails silently

`Qwen3.5-35B-A3B` declares `Qwen3_5MoeForConditionalGeneration` and carries a vision tower,
so its layers live at `model.language_model.layers`. `AutoModelForCausalLM` resolves to the
**text-only** variant, whose layers sit at `model.layers`, and 700 of the adapter's 920
tensors then fail to match. PEFT reports that as a `UserWarning`, not an error, so you get
the base model back and nothing tells you.

```python
from transformers import Qwen3_5MoeForConditionalGeneration, AutoTokenizer
from peft import PeftModel

base = Qwen3_5MoeForConditionalGeneration.from_pretrained(
    "Qwen/Qwen3.5-35B-A3B", dtype="bfloat16", device_map="auto"
)
model = PeftModel.from_pretrained(base, "HuggingEnvs/watercolour-grpo-judge-led")
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3.5-35B-A3B")
```

If you see `Found missing adapter keys while loading the checkpoint`, the adapter did not
load and you are running the base model.

## Training

110 steps in 34h49m on one H200. Launched for 200 steps and stopped at 110, with the curve
still inching upward (+0.0013/step over the last 30). This adapter is the step-110
checkpoint.

| | |
|---|---|
| reward | `gate` 0.05 + `length` 0.05 + pairwise judge **0.60** + [HPSv3](https://huggingface.co/MizzenAI/HPSv3) **0.30** |
| learning rate | 5e-5, `constant_with_warmup`, 5 warmup steps |
| `scale_rewards` | `none` |
| LoRA | `all-linear`, r16, alpha 32. 30,431,360 trainable, 0.0866% |
| batch | 8 generations per step, `per_device_batch_size` 1, `grad_accum` 8 |
| sampling | `top_p` 0.95, `top_k` 20, `max_completion_length` 8192 |

```bash
WATERCOLOUR_JUDGE_WEIGHT=0.60 WATERCOLOUR_QUALITY_WEIGHT=0.30 \
hf jobs uv run examples/watercolour_grpo.py --flavor h200 --timeout 96h --secrets HF_TOKEN -- \
  --env-url https://YOURORG-watercolour-env.hf.space \
  --model Qwen/Qwen3.5-35B-A3B --lora --all-linear --bf16 --gradient-checkpointing \
  --subject 'a peach hibiscus' --references 4 \
  --top-p 0.95 --top-k 20 \
  --lr 5e-5 --lr-scheduler constant_with_warmup --warmup-steps 5 \
  --scale-rewards none \
  --steps 200 --num-generations 8 \
  --per-device-batch-size 1 --gradient-accumulation-steps 8 \
  --max-completion-length 8192 --probe-samples 40 --film
```

The `uv` header pins no versions (`trl`, `peft`, `transformers`, `torch`), so a run today
will resolve different ones. That is a real reproducibility gap, stated rather than hidden.

## Results

| | first third | second | third | slope t |
|---|---|---|---|---|
| reward | 0.451 | 0.647 | **0.721** | **+10.5** |
| pairwise judge term | 0.36 | | **0.70** | |
| HPSv3 term | 0.56 | | 0.77 | |
| paint coverage | 0.11 | | **0.23** | |

Absolute rewards are not comparable across reward mixes: each run optimises a different
blend. What is comparable is the shape: this run started the lowest and climbed the most,
spending its first thirty steps nearly flat before it moved.

Best group mean 0.888, at step 53, and the best single rollout of the run is 0.96. The
pairwise judge term itself climbed from 0.36 to 0.70: the model wins more comparisons
against the hand-rated pool as training advances. Paint coverage doubled, from 0.11 to
0.23, where the judge-free `hps-only` run barely moved it.

The base model's probe before training: reward 0.407, judge term 0.317, paint coverage
0.093, over 40 samples. Every training number here is recomputable from
[`watercolour-rollouts-judge-led`](https://huggingface.co/datasets/HuggingEnvs/watercolour-rollouts-judge-led).

## Siblings

| run | judge | HPSv3 |
|---|---|---|
| **`judge-led`** | 0.60 | 0.30 |
| `hps-led` | 0.30 | 0.60 |
| `hps-only` | 0.00 | 0.90 |

## Limitations

- One subject, `a peach hibiscus`, and one library. It does not generalise to other drawing tasks.
- The pairwise judge is the noisiest reward term, and its consistency (scoring the same
  image twice) has not been tested.
- Reproducing it needs an H200, an a100-large Space for HPSv3, a cpu-upgrade Space for the
  environment and inference quota for the judge. It is not cheap.

Method reproduced from [Surya Narreddi's "RL'ing Qwen to paint with
code"](https://surya.website/rling-qwen-to-paint-with-code). Trained on HF Jobs (job `6a95460d0718b0f6d8908805`).

## Where this comes from

Part of **[Paint with Code](https://huggingface.co/collections/HuggingEnvs/paint-with-code-6a955b79d63f67f1631d9be6)**, a complete recipe: the environment, the pool
that defines the reward, the trainer, the curves and every rollout.

| | |
|---|---|
| the recipe, and how to reproduce it | [`02-watercolour/`](https://github.com/adithya-s-k/HuggingEnvs/tree/main/02-watercolour) |
| the environment | [`envs/watercolour/`](https://github.com/adithya-s-k/HuggingEnvs/tree/main/02-watercolour/envs/watercolour) |
| the trainer | [`train/watercolour_grpo.py`](https://github.com/adithya-s-k/HuggingEnvs/tree/main/02-watercolour/train/watercolour_grpo.py) |
| the reference pool | [`watercolour-reference-pool`](https://huggingface.co/datasets/HuggingEnvs/watercolour-reference-pool) |
| the trained adapter | [`watercolour-grpo-judge-led`](https://huggingface.co/HuggingEnvs/watercolour-grpo-judge-led) |
| every rollout | [`watercolour-rollouts-judge-led`](https://huggingface.co/datasets/HuggingEnvs/watercolour-rollouts-judge-led) |