betterwithage commited on
Commit
16d817c
·
verified ·
1 Parent(s): 395f45f

R4: RESULT_R4_ENERGY_BENCH.md (audit record, additive doc)

Browse files

R4 result/audit doc: restraint->energy tie-in, MEASURED-or-SAMPLE bench, estate KPI/hologram tile, live verification. Doc only; no runtime change.

team/AUDIT/frontier/RESULT_R4_ENERGY_BENCH.md ADDED
@@ -0,0 +1,62 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # RESULT — R4: a11oy Restraint → ENERGY + estate KPI + MEASURED benchmark
2
+
3
+ **Role:** Restraint Dev R4 @ SZL Holdings
4
+ **Repo:** `szl-holdings/a11oy` · **Space:** `SZLHOLDINGS/a11oy` (live: https://szlholdings-a11oy.hf.space)
5
+ **Date:** 2026-06-14
6
+ **Doctrine:** locked=8 @ c7c0ba17 · Λ = Conjecture 1 (<1.0) · trust<100% · 0 CDN · 0 visible codenames · signed receipts · additive-only · **never fabricate benchmark/energy numbers — MEASURED only on a live on-box probe, else SAMPLE/ROADMAP with methodology shown** · Ponytail cited MIT · *"The half-state is the only unacceptable outcome."*
7
+
8
+ ---
9
+
10
+ ## What shipped (front + back, all additive)
11
+
12
+ ### 1. RESTRAINT → ENERGY tie-in
13
+ - **`szl_restraint_energy.py`** (new module). Consumes R1's `szl_restraint` (`descend_ladder`, `benchmark`, `TOKENS_PER_LOC=9.0`, `J_PER_OUTPUT_TOKEN=0.65`) + Forge's `szl_energy_sovereign.energy_fields_for_receipt()` + `szl_joules_truth`. **Edits none of them.**
14
+ - Every restraint decision estimates **lines saved → tokens saved → joules saved**. The J/token is the **LIVE on-box MEASURED** figure ONLY when the sovereign GPU probe is live (label `MEASURED`); otherwise the honest `SAMPLE` constant — decided by the **same single source of truth** (`szl_joules_truth` / `szl_energy_sovereign`) as the rest of the energy story.
15
+ - In-memory cumulative `_SavingsLedger` (resets on restart — honestly noted, not a persisted meter).
16
+ - Endpoints (inserted at router position 0): `GET/POST /api/a11oy/v1/restraint/energy`, `GET /api/a11oy/v1/restraint/bench-measured`, `GET /api/a11oy/v1/restraint/kpi`.
17
+ - **"Frugality → energy" panel on the live `/energy` page.** Because Forge's `szl_energy_sovereign.register()` owns `/energy` (front of the route table, registered before the disk-file route), a `web/energy.html` disk edit never reaches the user. Fix: `register()` now **re-renders Forge's own page byte-for-byte** (`_html(_posture())`, never edited) and **appends a self-contained, scoped panel before `</body>`**, then front-wins the route via position-0 insert. If the energy module is unavailable, it 404s and Forge's route keeps serving (no shadow, no half-state). Panel labels joules `MEASURED`-only-on-live-probe, lines as MODELED, and links to `/restraint-bench`.
18
+
19
+ ### 2. MEASURED benchmark dashboard
20
+ - **`benchmarks/restraint/run_bench.py`** — runnable two-arm harness (baseline no-skill vs a11oy-restraint) over the same 5 everyday tasks as Ponytail's promptfoo methodology. `SAMPLE` mode without `--model`; **`MEASURED`** when run against a wired OpenAI-compatible client; writes `benchmarks/restraint/results.json`.
21
+ - **`web/restraint-bench.html`** + **`/restraint-bench`** page — shows OUR reproduced numbers (% less code, % cheaper, × faster), labelled **MEASURED-only-when-run-on-our-stack** (currently `ROADMAP`/`SAMPLE` with methodology, since no results artifact + no live GPU). Ponytail's published numbers are CITED as **theirs**; ours are labelled as **ours**.
22
+ - **Exact reproduce command** (shown on page + in API):
23
+ ```
24
+ python benchmarks/restraint/run_bench.py --repeat 10 --out benchmarks/restraint/results.json
25
+ ```
26
+
27
+ ### 3. Estate KPI board + hologram
28
+ - **`szl_ecosystem_routes.py`** — `build_kpi_board` now fetches `/restraint/kpi` and adds a top-level `restraint` key (frugality rate, cumulative lines/tokens/joules saved, energy honesty label, Λ, Ponytail/MIT provenance).
29
+ - **`web/estate-hologram.html`** — added a live RESTRAINT tile (`#restraintBadge`, `#restraint`) rendered from the kpi-board handler.
30
+
31
+ ---
32
+
33
+ ## Live verification (https://szlholdings-a11oy.hf.space)
34
+
35
+ | Surface | Result |
36
+ |---|---|
37
+ | `/api/a11oy/v1/restraint/energy` | 200 · cumulative ledger · `joules_label=SAMPLE`, `energy_label_upstream=ROADMAP` (honest — no live GPU probe) |
38
+ | `/api/a11oy/v1/restraint/bench-measured` | 200 · 5 rows · `overall_label=ROADMAP` · reproduce command present · Ponytail cited |
39
+ | `/api/a11oy/v1/restraint/kpi` | 200 · tile `restraint` · `label=LIVE` · provenance Ponytail/MIT |
40
+ | `/restraint-bench` | 200 · title "a11oy Restraint — MEASURED benchmark" (real page, not SPA) |
41
+ | `/energy` | 200 · **frugality_panel injected** (19.6 KB vs 15.3 KB) · Forge sovereign content fully intact · one `</body>` |
42
+ | `/estate-hologram` | 200 · RESTRAINT tile (`#restraint`, `restraintBadge`) present |
43
+ | `/api/a11oy/v1/ecosystem/kpi-board` | 200 · `restraint` key present |
44
+ | Existing `/api/a11oy/v1/energy/{sovereign,jtoken,budget}` | 200 (no regression) |
45
+ | `/restraint` (R1) | 200, intact (not edited) |
46
+
47
+ **Honesty state:** GPU probe NOT live → J/token is `SAMPLE`, energy upstream `ROADMAP`, bench `ROADMAP`. This is the **correct, honest** state — numbers flip to `MEASURED` automatically when the sovereign GPU exporter goes live and a real bench run is committed. No fabricated numbers.
48
+
49
+ **0 CDN** on live `/energy` (only Google Fonts). **0 visible codenames**. Panel JS validated with `node --check`; module validated with `ast.parse`.
50
+
51
+ ---
52
+
53
+ ## Push trail (curl Git Data API + HF NDJSON; never `COPY . .`; ADDITIVE)
54
+ - **GitHub `main`:** `baa113321b4a5f5c903f83d0f357845ec718fd86` (final additive `/energy` wrapper). Prior R4 files at `be8db473`/`7e14171c`.
55
+ - **HF Space:** commit `395f45f07c860fec8080e222fd4a5d6e2f48b02e` → factory rebuild → **RUNNING** at sha `395f45f0`.
56
+ - Never edited: `szl_restraint.py` (R1), `a11oy_nemo_core`/`a11oy_react_core` (R2), `szl_energy_sovereign.py` (Forge). Wrapper consumes Forge's `_html(_posture())` read-only.
57
+
58
+ ---
59
+
60
+ ## Provenance / citations
61
+ - Frugality ladder adopted from **Ponytail** (https://github.com/DietrichGebert/ponytail), **MIT** licensed; their published benchmark numbers cited as theirs, ours labelled as ours.
62
+ - Energy honesty model: vLLM/SGLang J/token, NVIDIA Dynamo, LiteLLM, RouteLLM patterns (via `szl_energy_sovereign`).