File size: 5,161 Bytes
5dcfc5c 2d63573 5dcfc5c 2d63573 5dcfc5c 2d63573 5dcfc5c 2d63573 5dcfc5c 2d63573 5dcfc5c 2d63573 5dcfc5c 2d63573 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 | # Local llama.cpp eval evidence
Date: 2026-06-07
This note separates local llama.cpp evidence from hosted Omni evidence and from canned deterministic fallback.
## Full-weight local artifact
The canonical local text artifact is the full BF16 4B model, not a quantized GGUF:
- Model repo: `nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16`
- Revision: `dfaf35de3e30f1867dd8dbc38a7fc9fb52d3914f`
- Local snapshot path: `/Users/drake.thomsen/.cache/huggingface/hub/models--nvidia--NVIDIA-Nemotron-3-Nano-4B-BF16/snapshots/dfaf35de3e30f1867dd8dbc38a7fc9fb52d3914f`
- Resolved snapshot size: `7.4G`
- Full weight file: `model.safetensors`
- Weight file size: `7947142640` bytes
- Weight file SHA-256: `55d4e2519456c4a9bddf596b0748d630e3b2ce6ff6f4c2b7ed3e07e2b00dad42`
- Required companion files present: `config.json`, `tokenizer.json`, `tokenizer_config.json`, `chat_template.jinja`, `modeling_nemotron_h.py`, `configuration_nemotron_h.py`
This download proves local artifact availability only. It does not prove local model competence, Off the Grid, Llama Champion, or local ASR until a real local OpenAI-compatible endpoint runs through the Figment eval without full deterministic fallback.
## Endpoint checks
Checked:
```bash
lsof -nP -iTCP:8001 -sTCP:LISTEN
curl -fsS --max-time 2 http://127.0.0.1:8001/v1/models
```
Observed: no listener on port `8001`; `curl` failed to connect to `127.0.0.1:8001`.
Rechecked after downloading the full BF16 artifact on 2026-06-07:
```bash
curl -fsS --max-time 2 http://127.0.0.1:8001/v1/models
```
Observed:
- `curl: (7) Failed to connect to 127.0.0.1 port 8001`
Local 4B, Off the Grid, and Llama Champion claims remain proof-needed. The next real step is to run the full BF16 model on a local OpenAI-compatible endpoint, record `/v1/models`, then run the 50-case eval below.
## Discarded quantized side check
A mistaken Q4 GGUF smoke was attempted before the full-weight requirement was clarified. It proved only that Figment could reach a local OpenAI-compatible endpoint; the model output failed navigator validation and fell back deterministically, so it does not count as route proof or model competence. The Q4 cache created by that attempt was removed.
## Evidence bundle command
Prefer the bundled helper once the local endpoint is live:
```bash
PYTHON_DOTENV_DISABLED=true \
python3 scripts/run_local_4b_evidence.py \
--base-url <local-openai-compatible-endpoint>/v1
```
The helper writes a timestamped evidence directory under `traces/local_4b_evidence_*`:
- `endpoint_metadata.json`: `/v1/models` response or connection error
- `route_smoke.json`: one-case configured-route smoke result
- `local_4b_eval.jsonl`: 50-case eval records, only after the route smoke proves configured-model validation unless `--force-eval` is used
- `eval_summary.json`: model competence, fallback, provenance, and final validation counts
- `eval_evidence_manifest.json`: compact evidence manifest with model/server metadata, no-cloud route flags, raw/repair/full-fallback counts, field provenance, latency summary, eval-file hash, and per-case trace hashes
- `summary.json`: top-level status and whether the evidence counts as route proof or 50-case local competence
Exit codes are intentionally evidence-gated: `0` for a completed eval or passed smoke-only run, `2` for endpoint unavailable, and `1` when the route smoke fails and eval is skipped. Endpoint availability alone must not be counted as model competence.
## One-case route smoke
Use this only to prove the app can call the configured local route. A passed smoke is not a 50-case local competence run.
```bash
FIGMENT_MODE=local \
MODEL_STACK=local_4b_parakeet \
MODEL_BACKEND=llama_cpp \
LOCAL_MODEL_ID=nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 \
LLAMA_BASE_URL=<local-openai-compatible-endpoint>/v1 \
FIGMENT_SMOKE_ALLOW_NETWORK=true \
PYTHON_DOTENV_DISABLED=true \
python3 scripts/smoke_model_route.py
```
The smoke output now includes `local_llm_evidence.proof_status`, `counts_as_no_cloud_route_proof`, and `counts_as_50_case_local_llm_competence`. A skipped smoke must not be counted as local model competence.
## Real 50-case eval command
Run this directly only after starting a real local OpenAI-compatible server and recording `/v1/models` metadata. The bundled helper above runs the same eval after a passing route smoke.
```bash
FIGMENT_MODE=local \
MODEL_STACK=local_4b_parakeet \
MODEL_BACKEND=llama_cpp \
LOCAL_MODEL_ID=nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 \
LLAMA_BASE_URL=<local-openai-compatible-endpoint>/v1 \
PYTHON_DOTENV_DISABLED=true \
python3 scripts/run_eval.py \
--backend llama_cpp \
--model-stack local_4b_parakeet \
--cases data/eval/initial_handwritten_cases.jsonl \
--cases data/eval/adversarial_strict_cases.jsonl \
--cases data/eval/comprehensive_hosted_cases.jsonl \
--output traces/local_llama_cpp_eval_$(date -u +%Y%m%dT%H%M%SZ).jsonl
```
A real local evidence bundle should include the eval JSONL trace, `eval_evidence_manifest.json`, `/v1/models` response or equivalent server metadata, model file/hash or server launch command, and confirmation that no hosted model credentials were required for the run.
|