Add ExtractArena tab to the Space app

app.py becomes a gr.TabbedInterface: the existing council tab plus an
ExtractArena tab (UI + REST + MCP tool `extract_fields`) that reuses
extract_arena.py as a library — prompt building, N/A-abstention
normalization, and JSON-reply parsing verbatim. Remote backends
(HF Inference Providers + moondream Space) run unbilled in the main
process; local council members reuse the resident _gpu_run_council path
in a single billed GPU window. HF_TOKEN is optional: hosted models fail
per-model (logged) without it while the rest run.

New env knobs: MAX_EXTRACT_MODELS (default 3) and DEFAULT_EXTRACT_MODELS
(default qwen3.5,muse-glimmer). MCP docstring follows the
gradio-mcp-tool-docstrings rules and links MODEL_RESEARCH.md for
calling agents; verified against the live MCP schema and the bib test
image (remote, local, mixed, multi-field, abstention, validation paths).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

CLAUDE.md

@@ -24,7 +24,7 @@ since that data goes stale. Update it after any new model research.
 - Install deps: `uv sync` (creates/updates `.venv` from `pyproject.toml` + `uv.lock`)
 - Add a dependency: `uv add <package>` (updates `pyproject.toml` and `uv.lock`)
 - Run the app locally: `uv run app.py` (launches Gradio on `http://127.0.0.1:7860`, with Swagger docs at `/docs` and an MCP server enabled)
-- Regenerate the deploy requirements manually: `uv export --no-hashes --format requirements-txt > requirements.txt`
+- `requirements.txt` is **hand-curated** (loose pins; no `gradio`/`spaces` — the Space's SDK provides those). Don't regenerate it with `uv export`; when app code gains a dependency, append it manually.
 
 There are no tests or linters configured.
 
@@ -47,7 +47,9 @@ HF Spaces rejects raw binaries in the git tree — image/video/audio/model-weigh
 
 ## App architecture (`app.py`)
 
-`app.py` serves the council from `svlm_council.py` — it imports the `COUNCIL` registry from there but has its own ZeroGPU load/infer path (the CLI's MPS single-slot eviction cache doesn't apply on the Space). Adding/changing a council member means editing `COUNCIL` in `svlm_council.py`; `app.py` picks it up automatically.
+`app.py` is a `gr.TabbedInterface` with two tabs, each a `gr.Interface` = REST endpoint = MCP tool:
+**Council** (`council_query`) serves the council from `svlm_council.py` — it imports the `COUNCIL` registry from there but has its own ZeroGPU load/infer path (the CLI's MPS single-slot eviction cache doesn't apply on the Space). Adding/changing a council member means editing `COUNCIL` in `svlm_council.py`; `app.py` picks it up automatically.
+**ExtractArena** (`extract_fields`) reuses `extract_arena.py` as a library: its prompt building (`build_prompt`), reply normalization (`_normalize_value`/`_parse_json_reply`), and remote backends (`extract_arena.MODELS[name].fn` — HF Inference Providers + the moondream Space) run verbatim in the main process (unbilled); local council names instead route through the resident-model `_gpu_run_council` path (never `extract_arena`'s MPS loader). `import extract_arena` must stay **after** `_load_dotenv()` — it reads `HF_TOKEN` at import time. `HF_TOKEN` is optional: without it the hosted backends fail per-model (logged) while the rest run. Result shape mirrors `ExtractResult`: `{"model", "value", "values", "raw", "latency_s", "error"}` (`value` normalized single-field answer with `""` = abstained; `values` for multi-field). Extract knobs: `MAX_EXTRACT_MODELS` (default 3), `DEFAULT_EXTRACT_MODELS` (default `qwen3.5,muse-glimmer`).
 
 - **Startup (module level, unbilled)**: every enabled member is loaded via `AutoProcessor`/`AutoModelForImageTextToText` and moved `.to("cuda")` — on ZeroGPU the main process runs a CUDA emulation mode, so module-level CUDA placement is the officially recommended pattern (lazy-loading inside `@spaces.GPU` is discouraged). All four members ≈ 37GB bf16, fitting the default `large` ZeroGPU slice (48GB VRAM; `xlarge` = 96GB at 2× quota). No flash-attn and no quantization; on CUDA every member is forced to sdpa attention (a spec'd `eager` — hunyuan-ocr — would materialize seq² float32 attention transients that OOM the VRAM left beside the resident council).
 - **`council_query`** is the public function = REST endpoint = MCP tool. It validates (returns a structured single-element error list instead of raising, so MCP clients get something actionable), then delegates to **`_gpu_run_council`**, the only `@spaces.GPU` function — its dynamic `duration` callable scales with the number of selected models (the requested duration is ZeroGPU's quota gate, so keep it tight; tune from Space logs).

README.md

@@ -7,15 +7,18 @@ sdk: gradio
 sdk_version: 6.26.0
 app_file: app.py
 pinned: false
-short_description: Compare small vision-language models on any image
+short_description: Compare small VLMs on any image — VQA and field extraction
 ---
 
 # sVLM Council
 
 Run the same image + prompt through a council of small (~1–9B) open vision-language
 models and compare their answers — targeted data extraction or vanilla VQA. `app.py`
-serves the council as a Gradio app with an MCP server and Swagger docs at `/docs`;
-the model registry is shared with [`svlm_council.py`](svlm_council.py) (the local CLI).
+serves two tabs as a Gradio app with an MCP server and Swagger docs at `/docs`:
+**Council** (the local council; registry shared with [`svlm_council.py`](svlm_council.py))
+and **ExtractArena** (named-field extraction with abstention guardrails from
+[`extract_arena.py`](extract_arena.py) — hosted models via HF Inference Providers +
+the moondream Space, plus the resident council members).
 
 The Space needs **ZeroGPU** hardware (set it in the Space settings). All council
 members are loaded to CUDA at startup (~37GB bf16 — fits the default 48GB `large`
@@ -23,7 +26,10 @@ ZeroGPU slice); only generation runs inside the billed `@spaces.GPU` window. Env
 knobs: `MAX_COUNCIL_MODELS` (default 3), `DEFAULT_COUNCIL_MODEL` (default
 `qianfan-ocr`), `COUNCIL_MEMBERS` (comma-separated subset to load, e.g.
 `COUNCIL_MEMBERS=hunyuan-ocr` for a light local run), `MAX_IMAGE_PIXELS`
-(default 3MP — larger inputs are downscaled before inference).
+(default 3MP — larger inputs are downscaled before inference),
+`MAX_EXTRACT_MODELS` (default 3), `DEFAULT_EXTRACT_MODELS` (default
+`qwen3.5,muse-glimmer`), and an optional `HF_TOKEN` secret for the hosted
+ExtractArena backends (without it they fail per-model; the rest still run).
 
 ## Deployment
 

app.py

@@ -1,17 +1,27 @@
 """sVLM-Council Space app — ask small local VLMs about an image, side by side.
 
-Gradio app + MCP server serving the council from `svlm_council.py` (the model
-registry is imported from there; the load/infer path here is ZeroGPU-specific).
+Gradio app + MCP server with two tabs:
+- Council: the council from `svlm_council.py` (the model registry is imported
+  from there; the load/infer path here is ZeroGPU-specific).
+- ExtractArena: targeted field extraction from `extract_arena.py` — its remote
+  backends (HF Inference Providers + the moondream Space) run in the main
+  process (unbilled); local council members reuse the resident GPU path.
+
 All enabled members are loaded to CUDA at module level — on ZeroGPU that runs
 under the CUDA emulation mode at startup (unbilled); only generation happens
 inside the @spaces.GPU window. The four default members total ~37GB bf16, which
 fits resident on the default `large` ZeroGPU slice (48GB).
 
 Env knobs:
-- MAX_COUNCIL_MODELS   max models per request (default 3)
+- MAX_COUNCIL_MODELS   max models per council request (default 3)
 - DEFAULT_COUNCIL_MODEL model used when none selected (default qianfan-ocr)
 - COUNCIL_MEMBERS      comma-separated subset to load at startup (default: all;
                        e.g. COUNCIL_MEMBERS=hunyuan-ocr for a light local run)
+- MAX_EXTRACT_MODELS   max models per extract request (default 3)
+- DEFAULT_EXTRACT_MODELS comma-separated models used when none selected
+                       (default qwen3.5,muse-glimmer)
+- HF_TOKEN             optional; without it the HF-Inference extract backends
+                       fail per-model (logged) while everything else runs
 """
 
 import spaces  # must be imported before any CUDA-touching torch usage on ZeroGPU
@@ -48,7 +58,11 @@ def _load_dotenv(path: Path = Path(".env")):
 
 _load_dotenv()
 
+# After _load_dotenv(): extract_arena reads HF_TOKEN at import time.
+import extract_arena  # noqa: E402
+
 MAX_COUNCIL_MODELS = int(os.environ.get("MAX_COUNCIL_MODELS", "3"))
+MAX_EXTRACT_MODELS = int(os.environ.get("MAX_EXTRACT_MODELS", "3"))
 MAX_IMAGE_PIXELS = int(os.environ.get("MAX_IMAGE_PIXELS", str(3_000_000)))
 DEFAULT_MODEL = os.environ.get("DEFAULT_COUNCIL_MODEL", "qianfan-ocr")
 MEMBERS = [
@@ -64,6 +78,23 @@ if DEFAULT_MODEL not in MEMBERS:
     )
     DEFAULT_MODEL = MEMBERS[0]
 
+# ExtractArena roster: extract_arena's remote backends + whatever council members
+# are actually loaded (local names in extract_arena.MODELS route through the
+# resident GPU path here, never through its MPS single-slot loader).
+EXTRACT_REMOTE = [n for n in extract_arena.MODELS if n not in COUNCIL]
+EXTRACT_ROSTER = EXTRACT_REMOTE + MEMBERS
+DEFAULT_EXTRACT_MODELS = [
+    name.strip()
+    for name in os.environ.get("DEFAULT_EXTRACT_MODELS", "qwen3.5,muse-glimmer").split(",")
+    if name.strip() in EXTRACT_ROSTER
+]
+if not DEFAULT_EXTRACT_MODELS:
+    logger.warning(
+        "DEFAULT_EXTRACT_MODELS matched no roster entry; falling back to {}",
+        EXTRACT_ROSTER[0],
+    )
+    DEFAULT_EXTRACT_MODELS = [EXTRACT_ROSTER[0]]
+
 DEVICE = (
     "cuda"
     if torch.cuda.is_available()  # true on ZeroGPU's main process (emulation mode)
@@ -329,7 +360,167 @@ Args:
 """
 
 
-app = gr.Interface(
+def _extract_error(message: str) -> list[dict]:
+    return [
+        {
+            "model": None,
+            "value": None,
+            "values": None,
+            "raw": "",
+            "latency_s": 0.0,
+            "error": message,
+        }
+    ]
+
+
+def _to_extract_result(
+    name: str, raw: str, latency_s: float, multi: bool, error: str | None = None
+) -> dict:
+    """Shape one reply like extract_arena.ExtractResult (normalized + raw kept)."""
+    if error:
+        return {
+            "model": name,
+            "value": None,
+            "values": None,
+            "raw": "",
+            "latency_s": latency_s,
+            "error": error,
+        }
+    raw = raw.strip()
+    if multi:
+        parsed = extract_arena._parse_json_reply(raw)
+        values = (
+            {k: extract_arena._normalize_value(v) for k, v in parsed.items()}
+            if parsed is not None
+            else None
+        )
+        value = None
+    else:
+        values = None
+        value = extract_arena._normalize_value(raw)
+    return {
+        "model": name,
+        "value": value,
+        "values": values,
+        "raw": raw,
+        "latency_s": latency_s,
+        "error": None,
+    }
+
+
+def extract_fields(
+    image: str,
+    fields: list[str] | None = None,
+    models: list[str] | None = None,
+    answer_format: str = "",
+) -> list[dict]:
+    # The real docstring (the MCP tool description) is assigned below — it needs
+    # runtime values (roster, defaults, max) a literal docstring can't interpolate.
+    if not image:
+        return _extract_error("No image provided.")
+    if isinstance(fields, str):  # tolerate a single field from API/MCP callers
+        fields = [fields]
+    fields = [f.strip() for f in (fields or []) if f and f.strip()]
+    if not fields:
+        return _extract_error("No fields provided.")
+
+    if isinstance(models, str):  # tolerate a single name from API/MCP callers
+        models = [models]
+    models = [m for m in (models or []) if m] or list(DEFAULT_EXTRACT_MODELS)
+    models = list(dict.fromkeys(models))  # results are keyed by name below
+
+    unknown = [m for m in models if m not in EXTRACT_ROSTER]
+    if unknown:
+        return _extract_error(
+            f"Unknown or unloaded model(s): {', '.join(unknown)}. "
+            f"Available: {', '.join(EXTRACT_ROSTER)}."
+        )
+    if len(models) > MAX_EXTRACT_MODELS:
+        return _extract_error(
+            f"Too many models selected ({len(models)}); "
+            f"the maximum per request is {MAX_EXTRACT_MODELS}."
+        )
+    try:
+        # One downscale for every backend: bounds remote upload size too, and
+        # keeps all models comparing the same pixels.
+        image = _prepare_image(image)
+    except Exception as exc:  # noqa: BLE001 — unreadable/corrupt upload
+        return _extract_error(
+            f"Could not read the input image: {type(exc).__name__}: {exc}"
+        )
+
+    prompt = extract_arena.build_prompt(fields, answer_format.strip() or None)
+    multi = len(fields) > 1
+    image_path = Path(image)
+
+    results: dict[str, dict] = {}
+    local = [m for m in models if m in MODELS]
+    for name in (m for m in models if m not in MODELS):  # remote — unbilled
+        logger.info("[{}] prompt: {!r}", name, prompt)
+        t0 = time.perf_counter()
+        try:
+            raw = extract_arena.MODELS[name].fn(image_path, prompt)
+            results[name] = _to_extract_result(
+                name, raw, round(time.perf_counter() - t0, 2), multi
+            )
+            logger.info("[{}] done in {}s", name, results[name]["latency_s"])
+        except Exception as exc:  # noqa: BLE001 — isolate failures per model
+            logger.exception("[{}] failed", name)  # incl. missing HF_TOKEN
+            results[name] = _to_extract_result(
+                name,
+                "",
+                round(time.perf_counter() - t0, 2),
+                multi,
+                error=f"{type(exc).__name__}: {exc}",
+            )
+    if local:  # resident members — one billed GPU window for the whole subset
+        for r in _gpu_run_council(image, prompt, local, "", 0.0, 2048):
+            results[r["model"]] = _to_extract_result(
+                r["model"], r["text"], r["latency_s"], multi, error=r["error"]
+            )
+    return [results[name] for name in models]
+
+
+_EXTRACT_ROSTER_DOCS = {
+    "moondream3": "moondream/moondream3-preview via the GF-John/moondream-pointer Space — no token needed, but its ZeroGPU quota can exhaust",
+    "qwen3.5": "Qwen/Qwen3.5-9B, hosted — most reliable remote entry; strong general VLM",
+    "muse-glimmer": "meta-models/Muse-Glimmer-30B, hosted — 86.6% Roboflow Data Extraction, the best open <=40B on the closest proxy benchmark for this task",
+    "gemma4-31b": "google/gemma-4-31B-it, hosted — fastest remote observed; best hosted availability; extraction quality unbenchmarked",
+    "gemma4-26b-a4b": "google/gemma-4-26B-A4B-it, hosted — cheapest hosted option; 3.8B-active MoE sibling of the 31B",
+    **_ROSTER_DOCS,
+}
+extract_fields.__doc__ = f"""Extract the value(s) of named fields from an image with several vision-language models (VLMs) and compare their answers side by side.
+
+Builds one extraction prompt from the field names — with an abstention guardrail instructing
+each model to answer exactly "N/A" rather than guess when a field is not visible — and runs it
+through every selected model: hosted models via HF Inference Providers or a helper Space, local
+council members on this Space's GPU. Available models:
+{"; ".join(f'"{m}" ({_EXTRACT_ROSTER_DOCS.get(m, m)})' for m in EXTRACT_ROSTER)}.
+If "models" is omitted or empty, the defaults ({", ".join(f'"{m}"' for m in DEFAULT_EXTRACT_MODELS)}) run;
+at most {MAX_EXTRACT_MODELS} models may be selected per request. Hosted models need an HF_TOKEN
+configured on the server; without it they fail per-model while the rest still run. To decide
+which models fit your task, read the maintained model research notes (benchmarks, strengths,
+quirks per member) at {MODEL_RESEARCH_URLS}.
+Inputs larger than {MAX_IMAGE_PIXELS / 1e6:.1f} megapixels are downscaled (aspect preserved)
+before inference, so very small text in very large images may need pre-cropping by the caller.
+The output is a JSON list with one object per requested model, in request order. Each object
+has: "model" (the model name, or null when the request itself was invalid); "value" (single-field
+requests: the normalized answer, "" meaning the model abstained/field not found, null on
+multi-field requests or failure); "values" (multi-field requests: an object mapping each field
+name to its normalized value with "" for absent fields, or null if the reply didn't parse as
+JSON — check "raw" then); "raw" (the model's verbatim reply, kept for debugging); "latency_s"
+(seconds, float); and "error" (null on success, otherwise a message — per-model failures don't
+abort the other models).
+
+Args:
+    image: Filepath or URL of the input image to extract from.
+    fields: List of field names to extract, e.g. ["bib number"] or ["bib number", "race name"]; single-field requests are the most reliable regime.
+    models: List of model names to run (see the description for the roster); empty or omitted runs the defaults.
+    answer_format: Optional format directive appended verbatim to the prompt, e.g. "Format the answer as a continuous sequence of digits (e.g., 12345)."; with multiple fields, phrase it per value; empty string means none.
+"""
+
+
+council_iface = gr.Interface(
     fn=council_query,
     inputs=[
         gr.Image(type="filepath", label="Input Image"),
@@ -352,7 +543,6 @@ app = gr.Interface(
         gr.Slider(label="Max Tokens", value=1024, minimum=32, maximum=4096, step=32),
     ],
     outputs=gr.JSON(label="Council Results"),
-    title="sVLM Council",
     description=(
         "Run the same image + prompt through small (~1-9B) open vision-language models and "
         "compare answers — data extraction or vanilla VQA. Model choices, benchmarks, and "
@@ -362,4 +552,47 @@ app = gr.Interface(
     api_name="council_query",
     api_visibility="public",
 )
+
+extract_iface = gr.Interface(
+    fn=extract_fields,
+    inputs=[
+        gr.Image(type="filepath", label="Input Image"),
+        gr.Dropdown(
+            label="Fields",
+            multiselect=True,
+            allow_custom_value=True,
+            choices=["bib number", "race name"],
+            value=["bib number"],
+            info="Field name(s) to extract — type your own and press Enter; one field per request is the most reliable",
+        ),
+        gr.CheckboxGroup(
+            label="Models",
+            choices=EXTRACT_ROSTER,
+            value=DEFAULT_EXTRACT_MODELS,
+            info=f"Select up to {MAX_EXTRACT_MODELS} models; see MODEL_RESEARCH.md for how to choose",
+        ),
+        gr.Textbox(
+            label="Format Directive",
+            lines=1,
+            value="",
+            info='Optional; appended verbatim to the prompt, e.g. "Format the answer as a continuous sequence of digits (e.g., 12345)."',
+        ),
+    ],
+    outputs=gr.JSON(label="Extraction Results"),
+    description=(
+        "Ask several VLMs — hosted and local — for the value of one or more named fields in an "
+        'image, with an abstention guardrail (models answer "N/A", normalized to "", instead '
+        "of guessing when a field is absent). Model choices, benchmarks, and quirks are "
+        "documented in "
+        "[MODEL_RESEARCH.md](https://huggingface.co/spaces/GF-John/svlm-council/raw/main/MODEL_RESEARCH.md)."
+    ),
+    api_name="extract_fields",
+    api_visibility="public",
+)
+
+app = gr.TabbedInterface(
+    [council_iface, extract_iface],
+    ["Council", "ExtractArena"],
+    title="sVLM Council",
+)
 app.launch(mcp_server=True, app_kwargs={"docs_url": "/docs"})

pyproject.toml

@@ -16,4 +16,6 @@ dependencies = [
     "pillow>=12.2",
     "loguru>=0.7.3",
     "typer>=0.12",
+    "huggingface-hub>=0.34",
+    "gradio-client>=1.4",
 ]

uv.lock

@@ -680,6 +680,8 @@ source = { virtual = "." }
 dependencies = [
     { name = "accelerate" },
     { name = "gradio", extra = ["mcp"] },
+    { name = "gradio-client" },
+    { name = "huggingface-hub" },
     { name = "loguru" },
     { name = "peft" },
     { name = "pillow" },
@@ -694,6 +696,8 @@ dependencies = [
 requires-dist = [
     { name = "accelerate", specifier = ">=1.0" },
     { name = "gradio", extras = ["mcp"], specifier = ">=5.38.0" },
+    { name = "gradio-client", specifier = ">=1.4" },
+    { name = "huggingface-hub", specifier = ">=0.34" },
     { name = "loguru", specifier = ">=0.7.3" },
     { name = "peft", specifier = ">=0.19.1" },
     { name = "pillow", specifier = ">=12.2" },