docs/local_llama_eval_evidence.md

Local llama.cpp eval evidence

Date: 2026-06-07

This note separates local llama.cpp evidence from hosted Omni evidence and from canned deterministic fallback.

Full-weight local artifact

The canonical local text artifact is the full BF16 4B model, not a quantized GGUF:

• Model repo: nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16
• Revision: dfaf35de3e30f1867dd8dbc38a7fc9fb52d3914f
• Local snapshot path: /Users/drake.thomsen/.cache/huggingface/hub/models--nvidia--NVIDIA-Nemotron-3-Nano-4B-BF16/snapshots/dfaf35de3e30f1867dd8dbc38a7fc9fb52d3914f
• Resolved snapshot size: 7.4G
• Full weight file: model.safetensors
• Weight file size: 7947142640 bytes
• Weight file SHA-256: 55d4e2519456c4a9bddf596b0748d630e3b2ce6ff6f4c2b7ed3e07e2b00dad42
• Required companion files present: config.json, tokenizer.json, tokenizer_config.json, chat_template.jinja, modeling_nemotron_h.py, configuration_nemotron_h.py

This download proves local artifact availability only. It does not prove local model competence, Off the Grid, Llama Champion, or local ASR until a real local OpenAI-compatible endpoint runs through the Figment eval without full deterministic fallback.

Endpoint checks

Checked:

lsof -nP -iTCP:8001 -sTCP:LISTEN
curl -fsS --max-time 2 http://127.0.0.1:8001/v1/models

Observed: no listener on port 8001; curl failed to connect to 127.0.0.1:8001.

Rechecked after downloading the full BF16 artifact on 2026-06-07:

curl -fsS --max-time 2 http://127.0.0.1:8001/v1/models

Observed:

• curl: (7) Failed to connect to 127.0.0.1 port 8001

Local 4B, Off the Grid, and Llama Champion claims remain proof-needed. The next real step is to run the full BF16 model on a local OpenAI-compatible endpoint, record /v1/models, then run the 50-case eval below.

Discarded quantized side check

A mistaken Q4 GGUF smoke was attempted before the full-weight requirement was clarified. It proved only that Figment could reach a local OpenAI-compatible endpoint; the model output failed navigator validation and fell back deterministically, so it does not count as route proof or model competence. The Q4 cache created by that attempt was removed.

Evidence bundle command

Prefer the bundled helper once the local endpoint is live:

PYTHON_DOTENV_DISABLED=true \
python3 scripts/run_local_4b_evidence.py \
  --base-url <local-openai-compatible-endpoint>/v1

The helper writes a timestamped evidence directory under traces/local_4b_evidence_*:

• endpoint_metadata.json: /v1/models response or connection error
• route_smoke.json: one-case configured-route smoke result
• local_4b_eval.jsonl: 50-case eval records, only after the route smoke proves configured-model validation unless --force-eval is used
• eval_summary.json: model competence, fallback, provenance, and final validation counts
• eval_evidence_manifest.json: compact evidence manifest with model/server metadata, no-cloud route flags, raw/repair/full-fallback counts, field provenance, latency summary, eval-file hash, and per-case trace hashes
• summary.json: top-level status and whether the evidence counts as route proof or 50-case local competence

Exit codes are intentionally evidence-gated: 0 for a completed eval or passed smoke-only run, 2 for endpoint unavailable, and 1 when the route smoke fails and eval is skipped. Endpoint availability alone must not be counted as model competence.

One-case route smoke

Use this only to prove the app can call the configured local route. A passed smoke is not a 50-case local competence run.

FIGMENT_MODE=local \
MODEL_STACK=local_4b_parakeet \
MODEL_BACKEND=llama_cpp \
LOCAL_MODEL_ID=nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 \
LLAMA_BASE_URL=<local-openai-compatible-endpoint>/v1 \
FIGMENT_SMOKE_ALLOW_NETWORK=true \
PYTHON_DOTENV_DISABLED=true \
python3 scripts/smoke_model_route.py

The smoke output now includes local_llm_evidence.proof_status, counts_as_no_cloud_route_proof, and counts_as_50_case_local_llm_competence. A skipped smoke must not be counted as local model competence.

Real 50-case eval command

Run this directly only after starting a real local OpenAI-compatible server and recording /v1/models metadata. The bundled helper above runs the same eval after a passing route smoke.

FIGMENT_MODE=local \
MODEL_STACK=local_4b_parakeet \
MODEL_BACKEND=llama_cpp \
LOCAL_MODEL_ID=nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 \
LLAMA_BASE_URL=<local-openai-compatible-endpoint>/v1 \
PYTHON_DOTENV_DISABLED=true \
python3 scripts/run_eval.py \
  --backend llama_cpp \
  --model-stack local_4b_parakeet \
  --cases data/eval/initial_handwritten_cases.jsonl \
  --cases data/eval/adversarial_strict_cases.jsonl \
  --cases data/eval/comprehensive_hosted_cases.jsonl \
  --output traces/local_llama_cpp_eval_$(date -u +%Y%m%dT%H%M%SZ).jsonl

A real local evidence bundle should include the eval JSONL trace, eval_evidence_manifest.json, /v1/models response or equivalent server metadata, model file/hash or server launch command, and confirmation that no hosted model credentials were required for the run.