Terminal-Bench Harness — Qwen3.8-27B-FP8

1. Launch the model server (HF Jobs)

hf jobs run \
  --flavor h200 \
  --expose 30000 \
  --secrets HF_TOKEN \
  --timeout 4h \
  --label project=terminal-bench-harness \
  --label model=qwen3_8-27b-fp8 \
  -d \
  lmsysorg/sglang:latest \
  sglang serve \
    --trust-remote-code \
    --model-path Qwen/Qwen3.8-27B-FP8 \
    --kv-cache-dtype fp8_e4m3 \
    --mem-fraction-static 0.85 \
    --attention-backend flashinfer \
    --chunked-prefill-size 32768 \
    --max-prefill-tokens 32768 \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder \
    --mamba-full-memory-ratio 3.67 \
    --host 0.0.0.0 \
    --port 30000 \
    --mamba-radix-cache-strategy extra_buffer_lazy \
    --mamba-ssm-dtype float32

This exposes an OpenAI-compatible endpoint at https://<job_id>--30000.hf.jobs/v1, reachable with an HF token that has read access to the job's namespace.

2. Run the benchmark (Harbor + terminal-bench 3.0)

Prerequisite: git-lfs must be installed (brew install git-lfs && git lfs install) or task binary fixtures fail to download correctly.

OPENAI_API_KEY="$(hf auth token)" harbor run \
  --agent terminus-2 \
  --model "openai/Qwen/Qwen3.8-27B-FP8" \
  --ak api_base="https://<job_id>--30000.hf.jobs/v1" \
  --repo https://huggingface.co/datasets/harborframework/terminal-bench-3.0 \
  --include-task-name atrx-vep-crispr \
  --n-tasks 1 \
  --job-name qwen3_8-27b-fp8-terminus2-smoketest-v3 \
  --n-concurrent 1 \
  -y

Notes:

• OPENAI_API_KEY must be exported into the shell running harbor, not passed via --ae (that flag only injects env vars into the sandboxed task container, not the host process making the LLM call).
• Drop --include-task-name and raise --n-tasks to run more of the dataset.

Result: single-task smoke test (terminal-bench 3.0)

atrx-vep-crispr__DvYyCpF: completed with 0 exceptions, reward 0 (task itself unsolved — it's a 7h expert-time-estimate task). Runtime 3h6m, ~5.68M input / ~681K output tokens.

3. Full run (terminal-bench 2.1, all 89 tasks, HF Sandbox environments)

Running all 89 tasks locally via Docker is impractical (each task's environment container competes for local CPU/RAM). Instead use --env hf-sandbox, which runs each task's environment as its own HF Jobs sandbox instead of a local container — every task in terminal-bench 2.1 already references a prebuilt public docker_image in its task.toml, so no local builds are needed.

OPENAI_API_KEY="$(hf auth token)" harbor run \
  --agent terminus-2 \
  --model "openai/Qwen/Qwen3.8-27B-FP8" \
  --ak api_base="https://<job_id>--30000.hf.jobs/v1" \
  --ak reasoning_effort=low \
  --repo https://huggingface.co/datasets/harborframework/terminal-bench-2.1 \
  --env hf-sandbox \
  --ek flavor=cpu-upgrade \
  --agent-timeout-multiplier 4 \
  --job-name qwen3_8-27b-fp8-terminus2-tb21-full-v2 \
  --n-concurrent 3 \
  -y

Notes:

• --agent-timeout-multiplier 4: this model is a heavy reasoner (burns lots of tokens/time per turn); the terminal-bench default per-task agent timeout (often 900s) is far too tight and causes widespread AgentTimeoutError. 4x (60 min/task) balances fairness against runtime.
• --ak reasoning_effort=low: keeps the model's thinking budget in check so it doesn't blow through the timeout on easy tasks.
• --n-concurrent: at 8, ~70% of trials errored out with SandboxError/InternalServerError/ReadTimeout (HF Sandbox infra strain, not model failures). Dropping to 3 brought the error rate down substantially. Lower is more reliable but slower.
• Mop up leftover infra errors without re-running everything:

  # first, if you changed --n-concurrent, also edit it in the job's config.json (and lock.json to match) before resuming
  OPENAI_API_KEY="$(hf auth token)" harbor job resume \
    -p jobs/<job-name> \
    --filter-error-type SandboxError \
    --filter-error-type InternalServerError \
    --filter-error-type ReadTimeout \
    --filter-error-type ReadError \
    --filter-error-type ConnectTimeout \
    --filter-error-type RemoteProtocolError
    # deliberately NOT AgentTimeoutError — that's a legitimate result, not infra flakiness
  

  Repeat as needed; each pass only touches the filtered trials, everything else is preserved.
• Harbor's own printed end-of-run summary double-counts across resume passes (it doesn't dedupe eval stats between attempts) — don't trust the final Trials/Exceptions/Mean table it prints after a multi-pass run. Get the real per-task outcome by taking, for each task name, the trial directory with the latest finished_at in result.json (reward at verifier_result.rewards.reward, or exception_info.exception_type if it errored).

Final result (89/89 tasks, after 3 resume/mop-up passes)

Pass rate among the 69 genuinely-scored tasks: 76.8%. Of the 20 errors, 16 are legitimate AgentTimeoutError (task too slow even at 4x budget); only 4 are residual infra flakiness (SandboxError x3, RemoteProtocolError x1) left after diminishing returns from further retries.