# Terminal-Bench Harness — Qwen3.8-27B-FP8

## 1. Launch the model server (HF Jobs)

```bash
hf jobs run \
  --flavor h200 \
  --expose 30000 \
  --secrets HF_TOKEN \
  --timeout 4h \
  --label project=terminal-bench-harness \
  --label model=qwen3_8-27b-fp8 \
  -d \
  lmsysorg/sglang:latest \
  sglang serve \
    --trust-remote-code \
    --model-path Qwen/Qwen3.8-27B-FP8 \
    --kv-cache-dtype fp8_e4m3 \
    --mem-fraction-static 0.85 \
    --attention-backend flashinfer \
    --chunked-prefill-size 32768 \
    --max-prefill-tokens 32768 \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder \
    --mamba-full-memory-ratio 3.67 \
    --host 0.0.0.0 \
    --port 30000 \
    --mamba-radix-cache-strategy extra_buffer_lazy \
    --mamba-ssm-dtype float32
```

This exposes an OpenAI-compatible endpoint at `https://<job_id>--30000.hf.jobs/v1`, reachable with an HF token that has read access to the job's namespace.

## 2. Run the benchmark (Harbor + terminal-bench 3.0)

Prerequisite: `git-lfs` must be installed (`brew install git-lfs && git lfs install`) or task binary fixtures fail to download correctly.

```bash
OPENAI_API_KEY="$(hf auth token)" harbor run \
  --agent terminus-2 \
  --model "openai/Qwen/Qwen3.8-27B-FP8" \
  --ak api_base="https://<job_id>--30000.hf.jobs/v1" \
  --repo https://huggingface.co/datasets/harborframework/terminal-bench-3.0 \
  --include-task-name atrx-vep-crispr \
  --n-tasks 1 \
  --job-name qwen3_8-27b-fp8-terminus2-smoketest-v3 \
  --n-concurrent 1 \
  -y
```

Notes:
- `OPENAI_API_KEY` must be exported into the shell running `harbor`, not passed via `--ae` (that flag only injects env vars into the sandboxed task container, not the host process making the LLM call).
- Drop `--include-task-name` and raise `--n-tasks` to run more of the dataset.

## Result: single-task smoke test (terminal-bench 3.0)

`atrx-vep-crispr__DvYyCpF`: completed with 0 exceptions, reward 0 (task itself unsolved — it's a 7h expert-time-estimate task). Runtime 3h6m, ~5.68M input / ~681K output tokens.

## 3. Full run (terminal-bench 2.1, all 89 tasks, HF Sandbox environments)

Running all 89 tasks locally via Docker is impractical (each task's environment container competes for local CPU/RAM). Instead use `--env hf-sandbox`, which runs each task's environment as its own HF Jobs sandbox instead of a local container — every task in terminal-bench 2.1 already references a prebuilt public `docker_image` in its `task.toml`, so no local builds are needed.

```bash
OPENAI_API_KEY="$(hf auth token)" harbor run \
  --agent terminus-2 \
  --model "openai/Qwen/Qwen3.8-27B-FP8" \
  --ak api_base="https://<job_id>--30000.hf.jobs/v1" \
  --ak reasoning_effort=low \
  --repo https://huggingface.co/datasets/harborframework/terminal-bench-2.1 \
  --env hf-sandbox \
  --ek flavor=cpu-upgrade \
  --agent-timeout-multiplier 4 \
  --job-name qwen3_8-27b-fp8-terminus2-tb21-full-v2 \
  --n-concurrent 3 \
  -y
```

Notes:
- `--agent-timeout-multiplier 4`: this model is a heavy reasoner (burns lots of tokens/time per turn); the terminal-bench default per-task agent timeout (often 900s) is far too tight and causes widespread `AgentTimeoutError`. 4x (60 min/task) balances fairness against runtime.
- `--ak reasoning_effort=low`: keeps the model's thinking budget in check so it doesn't blow through the timeout on easy tasks.
- `--n-concurrent`: at `8`, ~70% of trials errored out with `SandboxError`/`InternalServerError`/`ReadTimeout` (HF Sandbox infra strain, not model failures). Dropping to `3` brought the error rate down substantially. Lower is more reliable but slower.
- Mop up leftover infra errors without re-running everything:
  ```bash
  # first, if you changed --n-concurrent, also edit it in the job's config.json (and lock.json to match) before resuming
  OPENAI_API_KEY="$(hf auth token)" harbor job resume \
    -p jobs/<job-name> \
    --filter-error-type SandboxError \
    --filter-error-type InternalServerError \
    --filter-error-type ReadTimeout \
    --filter-error-type ReadError \
    --filter-error-type ConnectTimeout \
    --filter-error-type RemoteProtocolError
    # deliberately NOT AgentTimeoutError — that's a legitimate result, not infra flakiness
  ```
  Repeat as needed; each pass only touches the filtered trials, everything else is preserved.
- **Harbor's own printed end-of-run summary double-counts across resume passes** (it doesn't dedupe eval stats between attempts) — don't trust the final `Trials`/`Exceptions`/`Mean` table it prints after a multi-pass run. Get the real per-task outcome by taking, for each task name, the trial directory with the latest `finished_at` in `result.json` (reward at `verifier_result.rewards.reward`, or `exception_info.exception_type` if it errored).

### Final result (89/89 tasks, after 3 resume/mop-up passes)

| Outcome | Count | % of 89 |
|---|---|---|
| Passed | 53 | 59.6% |
| Failed (genuine, reward 0) | 16 | 18.0% |
| Errored (unscored) | 20 | 22.5% |

Pass rate among the 69 genuinely-scored tasks: **76.8%**. Of the 20 errors, 16 are legitimate `AgentTimeoutError` (task too slow even at 4x budget); only 4 are residual infra flakiness (`SandboxError` x3, `RemoteProtocolError` x1) left after diminishing returns from further retries.