CLAUDE.md

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## What this is

A Gradio app deployed to a Hugging Face Space (ZeroGPU), with dependencies managed by `uv` and continuous deployment via GitHub Actions. The app (`app.py`) serves the sVLM-Council — small (~1–9B) vision-language models answering the same image+prompt side by side — as a UI, REST API, and MCP server. Space: `GF-John/svlm-council`.

## Model comparison scripts

Alongside the template `app.py`, the repo has three self-contained PEP 723 uv scripts:
`ocr_arena.py` (full-image OCR across backends), `extract_arena.py` (targeted field
extraction with N/A-abstention guardrails and `-F` format directives), and
`svlm_council.py` (small VLMs run locally via transformers; importable `query()` used by
extract_arena). Each declares its own deps inline — run with `uv run [--env-file .env] <script>`.

**When adding, replacing, or re-evaluating models, start from `MODEL_RESEARCH.md`** — it holds
the verified candidate tables (hosted + local), benchmark snapshots, provider-reliability
notes, per-model transformers quirks, prompting findings, and verified model licenses (with
per-scenario implications), plus the endpoints to re-check
since that data goes stale. Update it after any new model research.

## Commands

- Install deps: `uv sync` (creates/updates `.venv` from `pyproject.toml` + `uv.lock`)
- Add a dependency: `uv add <package>` (updates `pyproject.toml` and `uv.lock`)
- Run the app locally: `uv run app.py` (launches Gradio on `http://127.0.0.1:7860`, with Swagger docs at `/docs` and an MCP server enabled)
- `requirements.txt` is **hand-curated** (loose pins; no `gradio`/`spaces` — the Space's SDK provides those). Don't regenerate it with `uv export`; when app code gains a dependency, append it manually.

There are no tests or linters configured.

## Deployment

Push to `main` triggers `.github/workflows/deploy_to_hf_space.yaml`, which:
1. Generates `requirements.txt` from `pyproject.toml` via `uv export` **only when it doesn't already exist** (the Space itself has no `pyproject.toml`; HF Spaces read `requirements.txt`), then commits it back.
2. Pushes the repo to the HF Space — a normal push by default, or a force push if the `FORCE_PUSH` secret is set (a force push overwrites the Space's history, so use it only if you are the sole contributor).

Configuration is driven entirely by GitHub secrets (no workflow edits needed):
- `HF_TOKEN` (required) — auth token. If absent the push step is skipped (the workflow still succeeds).
- `HF_USERNAME` and `SPACE_NAME` (required) — interpolated into the push URL `.../spaces/$HF_USERNAME/$SPACE_NAME`.
- `FORCE_PUSH` (optional) — any value enables force push.

Also update the YAML frontmatter in `README.md` (`title`, `emoji`, `sdk_version`, `short_description`, etc.) — HF Spaces reads its config from there.

## Binary assets

HF Spaces rejects raw binaries in the git tree — image/video/audio/model-weight files must go through Git LFS / Xet (`git lfs track`), not be committed directly. Common binary extensions are git-ignored in `.gitignore` to prevent accidental commits; if a binary genuinely needs to ship, track it with LFS rather than removing the ignore rule.

## App architecture (`app.py`)

`app.py` is a `gr.TabbedInterface` with two tabs, each a `gr.Interface` = REST endpoint = MCP tool:
**Council** (`council_query`) serves the council from `svlm_council.py` — it imports the `COUNCIL` registry from there but has its own ZeroGPU load/infer path (the CLI's MPS single-slot eviction cache doesn't apply on the Space). Adding/changing a council member means editing `COUNCIL` in `svlm_council.py`; `app.py` picks it up automatically.
**ExtractArena** (`extract_fields`) reuses `extract_arena.py` as a library: its prompt building (`build_prompt`), reply normalization (`_normalize_value`/`_parse_json_reply`), and remote backends (`extract_arena.MODELS[name].fn` — HF Inference Providers + the moondream Space) run verbatim in the main process (unbilled); local council names instead route through the resident-model `_gpu_run_council` path (never `extract_arena`'s MPS loader). `import extract_arena` must stay **after** `_load_dotenv()` — it reads `HF_TOKEN` at import time. `HF_TOKEN` is optional: without it the hosted backends fail per-model (logged) while the rest run. Result shape mirrors `ExtractResult`: `{"model", "value", "values", "raw", "latency_s", "error"}` (`value` normalized single-field answer with `""` = abstained; `values` for multi-field). Extract knobs: `MAX_EXTRACT_MODELS` (default 3), `DEFAULT_EXTRACT_MODELS` (default `qwen3.5,muse-glimmer`).

- **Startup (module level, unbilled)**: every enabled member is loaded via `AutoProcessor`/`AutoModelForImageTextToText` and moved `.to("cuda")` — on ZeroGPU the main process runs a CUDA emulation mode, so module-level CUDA placement is the officially recommended pattern (lazy-loading inside `@spaces.GPU` is discouraged). All four members ≈ 37GB bf16, fitting the default `large` ZeroGPU slice (48GB VRAM; `xlarge` = 96GB at 2× quota). No flash-attn and no quantization; on CUDA every member is forced to sdpa attention (a spec'd `eager` — hunyuan-ocr — would materialize seq² float32 attention transients that OOM the VRAM left beside the resident council).
- **`council_query`** is the public function = REST endpoint = MCP tool. It validates (returns a structured single-element error list instead of raising, so MCP clients get something actionable), then delegates to **`_gpu_run_council`**, the only `@spaces.GPU` function — its dynamic `duration` callable scales with the number of selected models (the requested duration is ZeroGPU's quota gate, so keep it tight; tune from Space logs).
- **Result shape**: `list[dict]`, one per model: `{"model", "text", "latency_s", "error"}`; per-model failures are isolated.
- **Concurrency**: no locking in the code; Gradio's queue does it. `concurrency_limit=2` on the Council interface and `4` on ExtractArena (each concurrent `_gpu_run_council` call forks its own ZeroGPU worker and GPU, so the ceiling is the GPU pool, not VRAM; remote extract backends are I/O-bound and overlap freely). `app.queue(max_size=16)` is explicit because ZeroGPU forces `max_size=1` when unset, which 503s ("Queue is full") every caller beyond the running six plus one waiter. Within one request models run sequentially.
- **MCP docstring** is assigned to `council_query.__doc__` *after* the def (f-string — it embeds the runtime roster, max, and the MODEL_RESEARCH.md links so calling agents can pick models). It follows the `gradio-mcp-tool-docstrings` skill rules: schema + URLs in the description prose, one-line `Args:` entries, no `Returns:` section. Re-verify with the skill's `dump_mcp_schema.py` after editing it.
- **Env knobs**: `MAX_COUNCIL_MODELS` (default 3), `DEFAULT_COUNCIL_MODEL` (default `qianfan-ocr`), `COUNCIL_MEMBERS` (comma-separated subset to load; `COUNCIL_MEMBERS=hunyuan-ocr uv run app.py` is the lightweight local dev run — the full council won't fit in 32GB local RAM), `MAX_IMAGE_PIXELS` (default 3,000,000 — inputs above this are downscaled in the main process before inference; hunyuan-ocr's processor has no resolution cap of its own, and its eager attention OOMs on huge uploads without this guard).
- **Version floors**: torch is pinned `>=2.10,<2.12` because ZeroGPU only supports torch up to 2.11; gradio is 6.x (`gradio[mcp]` extra required for `mcp_server=True`).