docs/smaller-llm-load-bearing-action-plan.md

Smaller LLM Load-Bearing Action Plan

Date: 2026-06-07

This note captures the adversarial review items from the Build Small Hackathon review. The goal is to make Figment more honest and more robust when smaller LLMs or companion small models are load-bearing, while preserving deterministic safety gates.

Source Constraints

• Build Small requires a Gradio app hosted as a Hugging Face Space, a short demo video, and a social post.
• Total model parameters must be <= 32B.
• Backyard AI is judged on a specific real problem, real use by the target person, honest fit to small models, and Gradio polish.
• Off the Grid needs no cloud APIs at runtime.
• Well-Tuned needs a published fine-tuned model used by the app.
• Llama Champion needs a llama.cpp runtime path.

Current Evidence Snapshot

• Local tests passed before this note was written: python3 -m pytest -q reported 105 passed.
• Hosted Omni baseline eval: 28/50 whole-output competence, 22/50 full deterministic fallback, 50/50 final validation.
• Hosted Omni load-bearing follow-up eval: 31/50 whole-output competence, 8/50 full fallback, 480/650 model-retained fields, 170/650 deterministic patches, 50/50 final validation.
• Public Space API still reported runtime.stage=NO_APP_FILE with only .gitattributes and README.md.
• Local 4B and Parakeet paths are configured or labeled, but not yet proven with real no-cloud eval or ASR evidence.

Work Items

1. Make the public Hugging Face Space runnable

• Push the full Gradio app, requirements, protocol cards, demo assets, and README to build-small-hackathon/figment.
• Verify a public cold boot from the Space URL, not only local startup.
• Record Space evidence in docs/submission_checklist.md.
• Ensure no-secret mode cold-boots with typed intake, honest canned fallback, and trace labeling.

2. Add a parameter and evidence ledger

• Create a compact ledger with model ID, route, total-parameter source, active-parameter note, adapter parameter count, ASR companion count, endpoint locality, and organizer-confirmation status.
• Explicitly mark active parameters as not the compliance number.
• Include the Omni 31B body-count versus 33B sidebar ambiguity.
• Include the local 4B + Parakeet additive story and adapter headroom.

3. Prove the smaller local LLM path separately

• Run the 50-case eval through MODEL_BACKEND=llama_cpp against a real local OpenAI-compatible endpoint.
• Record model/server metadata, no-cloud evidence, raw success, repair success, full fallback, field provenance, latency, and trace hashes.
• Add a run note that distinguishes local 4B evidence from hosted Omni evidence.
• Ensure whole fallback cannot count as local model competence.

4. Publish model scorecards separate from app safety

• Treat final_validation_successes as app safety, not model competence.
• Publish raw-only, repaired, hybrid, full-fallback, and per-field scorecards.
• Update stale README and checklist language that still says no full eval has run.
• Highlight remaining weak areas, especially SBAR and latency.

5. Align live app validation with strict eval validation

• Make runtime navigation use strict schema checks, retrieved-card constraints, and retrieved-card observation grounding.
• If runtime and eval validation intentionally differ, trace the mode explicitly.
• Prevent sparse raw model output from being labeled fully model_raw when it would fail strict eval.

6. Add hybrid route labeling and field provenance UI

• Add a distinct route for outputs with deterministic patches, such as model_with_deterministic_patches.
• Derive runtime labels from trace state rather than configured backend.
• Show per-field or per-section provenance counts in Navigator Output and Trace.
• Rename config-derived labels like "Hosted Omni (live)" to "Configured backend" unless the post-run trace proves live generation.

7. Cap or batch repair calls for smaller models

• Cap focused repair attempts.
• Prefer one compact JSON-only repair call when multiple scopes fail.
• Add metrics for repair call count and repair latency.
• Keep deterministic fallback for fields that still fail after the cap.

8. Score expected eval labels directly

• Compare actual red flags to expected_red_flag_rule_ids.
• Compare actual urgency to expected_min_protocol_urgency.
• Compare source cards and candidate pathways to target or expected card IDs.
• Compare missing observations to expected missing-observation cues.
• Check case-level forbidden behavior explicitly.

9. Treat Parakeet as unproven until real local ASR exists

• Add transcript_source or equivalent provenance to audio drafts.
• Keep typed transcript heuristics labeled as typed or heuristic, not Parakeet ASR.
• Only emit Parakeet provenance from a real gated ASR adapter or smoke.
• Add a real local ASR proof note before making Parakeet demo-visible.

10. Harden audio confirmation UX

• Prevent "Apply Audio Draft" from becoming quiet bulk acceptance.
• Preserve applied draft fields as applied_unreviewed until the responder explicitly accepts, edits, or rejects each suggestion.
• Block navigation while any applied audio-derived field is still unreviewed.
• Keep manual edits winning over audio drafts.

11. Fix demo audio and hosted audio disclosure

• Derive canned/demo transcript text from one source of truth so ages and case facts stay aligned.
• Label committed Voxtral clips as synthetic demo assets, not local ASR proof.
• Add hosted-mode UI copy that audio is sent to the configured hosted endpoint and must be synthetic or de-identified.
• Add size and duration caps for hosted audio drafts.

12. Keep badge and submission copy evidence-gated

• Keep Off the Grid, Llama Champion, Well-Tuned, Backyard AI user-use, demo video, and social post claims conditional until artifacts exist.
• Replace overclaiming workback/social snippets with achieved-versus-targeted wording.
• Fill user-test notes from a real trained-responder session before claiming the target user used or tested Figment.

Parallel Work Map

Worker A: Submission Docs And Badge Honesty

Owns work items 1, 2, 4, and 12 where they are documentation-only.

Primary files:

• README.md
• docs/submission_checklist.md
• docs/hosted_omni_eval_results.md
• docs/model_load_bearing_work_items.md
• docs/figment-workback-plan.md
• New ledger note if needed under docs/

Worker B: Runtime Validation And Provenance

Owns work items 5, 6, and 7.

Primary files:

• figment/navigator.py
• figment/trace.py
• figment/validators.py
• figment/field_provenance.py
• figment/focused_repair.py
• app.py route/provenance display only
• Related tests under tests/

Worker C: Eval Scoring And Local LLM Evidence

Owns work items 3 and 8.

Primary files:

• scripts/run_eval.py
• scripts/smoke_model_route.py
• figment/eval_metrics.py
• tests/test_eval_runner.py
• tests/test_eval_metrics.py
• tests/test_model_route_smoke_script.py
• Eval docs under docs/

Worker D: Audio Provenance And Confirmation Safety

Owns work items 9, 10, and 11.

Primary files:

• figment/audio_intake.py
• figment/model_client.py
• app.py audio UI/functions only
• data/demo_audio/manifest.json
• tests/test_audio_app_flow.py
• tests/test_runtime_honesty.py

Integration Rules

• Do not weaken deterministic red-flag rules, intake confirmation, validators, or raw-audio scrubbing.
• Do not count canned fallback or deterministic patches as pure model competence.
• Do not claim Off the Grid, Well-Tuned, Llama Champion, or target-user use without traceable evidence.
• Preserve typed-only operation when audio is disabled.
• Keep hosted demo behavior honest when secrets or model endpoints are absent.