CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

What this is

A Gradio app deployed to a Hugging Face Space (ZeroGPU), with dependencies managed by uv and continuous deployment via GitHub Actions. The app (app.py) serves the sVLM-Council — small (~1–9B) vision-language models answering the same image+prompt side by side — as a UI, REST API, and MCP server. Space: GF-John/svlm-council.

Model comparison scripts

Alongside the template app.py, the repo has three self-contained PEP 723 uv scripts: ocr_arena.py (full-image OCR across backends), extract_arena.py (targeted field extraction with N/A-abstention guardrails and -F format directives), and svlm_council.py (small VLMs run locally via transformers; importable query() used by extract_arena). Each declares its own deps inline — run with uv run [--env-file .env] <script>.

When adding, replacing, or re-evaluating models, start from MODEL_RESEARCH.md — it holds the verified candidate tables (hosted + local), benchmark snapshots, provider-reliability notes, per-model transformers quirks, prompting findings, and verified model licenses (with per-scenario implications), plus the endpoints to re-check since that data goes stale. Update it after any new model research.

Commands

• Install deps: uv sync (creates/updates .venv from pyproject.toml + uv.lock)
• Add a dependency: uv add <package> (updates pyproject.toml and uv.lock)
• Run the app locally: uv run app.py (launches Gradio on http://127.0.0.1:7860, with Swagger docs at /docs and an MCP server enabled)
• requirements.txt is hand-curated (loose pins; no gradio/spaces — the Space's SDK provides those). Don't regenerate it with uv export; when app code gains a dependency, append it manually.

There are no tests or linters configured.

Deployment

Push to main triggers .github/workflows/deploy_to_hf_space.yaml, which:

1. Generates requirements.txt from pyproject.toml via uv export only when it doesn't already exist (the Space itself has no pyproject.toml; HF Spaces read requirements.txt), then commits it back.
2. Pushes the repo to the HF Space — a normal push by default, or a force push if the FORCE_PUSH secret is set (a force push overwrites the Space's history, so use it only if you are the sole contributor).

Configuration is driven entirely by GitHub secrets (no workflow edits needed):

• HF_TOKEN (required) — auth token. If absent the push step is skipped (the workflow still succeeds).
• HF_USERNAME and SPACE_NAME (required) — interpolated into the push URL .../spaces/$HF_USERNAME/$SPACE_NAME.
• FORCE_PUSH (optional) — any value enables force push.

Also update the YAML frontmatter in README.md (title, emoji, sdk_version, short_description, etc.) — HF Spaces reads its config from there.

Binary assets

HF Spaces rejects raw binaries in the git tree — image/video/audio/model-weight files must go through Git LFS / Xet (git lfs track), not be committed directly. Common binary extensions are git-ignored in .gitignore to prevent accidental commits; if a binary genuinely needs to ship, track it with LFS rather than removing the ignore rule.

App architecture (app.py)

app.py is a gr.TabbedInterface with two tabs, each a gr.Interface = REST endpoint = MCP tool: Council (council_query) serves the council from svlm_council.py — it imports the COUNCIL registry from there but has its own ZeroGPU load/infer path (the CLI's MPS single-slot eviction cache doesn't apply on the Space). Adding/changing a council member means editing COUNCIL in svlm_council.py; app.py picks it up automatically. ExtractArena (extract_fields) reuses extract_arena.py as a library: its prompt building (build_prompt), reply normalization (_normalize_value/_parse_json_reply), and remote backends (extract_arena.MODELS[name].fn — HF Inference Providers + the moondream Space) run verbatim in the main process (unbilled); local council names instead route through the resident-model _gpu_run_council path (never extract_arena's MPS loader). import extract_arena must stay after _load_dotenv() — it reads HF_TOKEN at import time. HF_TOKEN is optional: without it the hosted backends fail per-model (logged) while the rest run. Result shape mirrors ExtractResult: {"model", "value", "values", "raw", "latency_s", "error"} (value normalized single-field answer with "" = abstained; values for multi-field). Extract knobs: MAX_EXTRACT_MODELS (default 3), DEFAULT_EXTRACT_MODELS (default qwen3.5,muse-glimmer).

• Startup (module level, unbilled): every enabled member is loaded via AutoProcessor/AutoModelForImageTextToText and moved .to("cuda") — on ZeroGPU the main process runs a CUDA emulation mode, so module-level CUDA placement is the officially recommended pattern (lazy-loading inside @spaces.GPU is discouraged). All four members ≈ 37GB bf16, fitting the default large ZeroGPU slice (48GB VRAM; xlarge = 96GB at 2× quota). No flash-attn and no quantization; on CUDA every member is forced to sdpa attention (a spec'd eager — hunyuan-ocr — would materialize seq² float32 attention transients that OOM the VRAM left beside the resident council).
• council_query is the public function = REST endpoint = MCP tool. It validates (returns a structured single-element error list instead of raising, so MCP clients get something actionable), then delegates to _gpu_run_council, the only @spaces.GPU function — its dynamic duration callable scales with the number of selected models (the requested duration is ZeroGPU's quota gate, so keep it tight; tune from Space logs).
• Result shape: list[dict], one per model: {"model", "text", "latency_s", "error"}; per-model failures are isolated.
• Concurrency: no locking in the code; Gradio's queue does it. concurrency_limit=2 on the Council interface and 4 on ExtractArena (each concurrent _gpu_run_council call forks its own ZeroGPU worker and GPU, so the ceiling is the GPU pool, not VRAM; remote extract backends are I/O-bound and overlap freely). app.queue(max_size=16) is explicit because ZeroGPU forces max_size=1 when unset, which 503s ("Queue is full") every caller beyond the running six plus one waiter. Within one request models run sequentially.
• MCP docstring is assigned to council_query.__doc__ after the def (f-string — it embeds the runtime roster, max, and the MODEL_RESEARCH.md links so calling agents can pick models). It follows the gradio-mcp-tool-docstrings skill rules: schema + URLs in the description prose, one-line Args: entries, no Returns: section. Re-verify with the skill's dump_mcp_schema.py after editing it.
• Env knobs: MAX_COUNCIL_MODELS (default 3), DEFAULT_COUNCIL_MODEL (default qianfan-ocr), COUNCIL_MEMBERS (comma-separated subset to load; COUNCIL_MEMBERS=hunyuan-ocr uv run app.py is the lightweight local dev run — the full council won't fit in 32GB local RAM), MAX_IMAGE_PIXELS (default 3,000,000 — inputs above this are downscaled in the main process before inference; hunyuan-ocr's processor has no resolution cap of its own, and its eager attention OOMs on huge uploads without this guard).
• Version floors: torch is pinned >=2.10,<2.12 because ZeroGPU only supports torch up to 2.11; gradio is 6.x (gradio[mcp] extra required for mcp_server=True).