docs/adversarial-review-action-items.md

# Figment adversarial review action items

Date: 2026-06-07

This note tracks the nine outstanding items from the adversarial review against the Build Small Hackathon outline, with emphasis on making model behavior visibly load-bearing while preserving Figment's deterministic safety contract.

Important nuance: the current Omni-first architecture can technically support an Off the Grid claim if Nemotron 3 Nano Omni is self-hosted on adequate local hardware with no runtime cloud APIs. The present gap is proof and available hardware, not that the architecture is inherently cloud-only. On the hardware available in this workspace, the practical local proof remains the smaller 4B text path, optionally with Parakeet after its gate passes.

## 1. Make the Hugging Face Space actually runnable

Current risk: the Space exists, but the public Space API showed `runtime.stage=NO_APP_FILE` and only `.gitattributes` plus `README.md` in the remote Space. That fails the required hosted Gradio artifact even if the local repo is healthy.

Fix direction:
- Push the full Gradio app, requirements, protocol cards, and demo assets to `build-small-hackathon/figment`.
- Verify cold boot from the Space URL, not only locally.
- Align Space metadata and dependency versions with the local repo.

Done when:
- The public Space boots to the Figment Gradio UI.
- The Space can run typed intake with canned fallback and trace labeling when secrets are absent.
- Hosted Omni mode works when secrets are present, or failure is clearly labeled.

## 2. Make live model contribution visible

Current risk: checked-in demo traces are canned, and UI labels can imply configured hosted/local mode even after fallback.

Fix direction:
- Add trace and UI states that distinguish `live_model_generated`, `model_repaired`, `validation_fallback`, and `canned_backend`.
- Commit at least one validated non-canned trace from hosted Omni or a local OpenAI-compatible model.
- Ensure fallback output never looks like live model proof.

Done when:
- Trace JSON records raw route, final route, fallback reason, and validation status.
- The Navigator and Trace tabs show the actual post-run route, not only the configured backend.

## 3. Make the 4B or Omni model load-bearing in bounded fields

Current risk: deterministic rules, retrieval, validators, and canned fallback can make the app look useful without proving model reasoning.

Fix direction:
- Keep deterministic rules, urgency floors, retrieval, and validators as safety boundaries.
- Make the configured model responsible for bounded, visible fields: candidate pathways, missing observations, checklist, uncertainty notes, and SBAR draft.
- Preserve fallback for safety, but treat fallback as reliability support rather than model competence.

Done when:
- A non-canned trace shows model-generated values for the bounded navigator fields.
- The trace/UI makes clear which fields came from model output versus deterministic fallback.

## 4. Build the eval harness before fine-tuning

Current risk: target metrics are documented, but no model eval script or result artifact exists.

Fix direction:
- Add `scripts/run_eval.py`.
- Record raw model output, repaired output, fallback output, validation failures, fallback reason, latency, model ID, and trace hash.
- Separate raw model pass, repair pass, and canned fallback pass.

Done when:
- The initial eval cases can be run against `canned`, `hosted_omni`, and `llama_cpp` routes.
- Eval output cannot count fallback as small-model success.

## 5. Expand eval data from seed fixtures to proof

Current risk: 10 handwritten cases cover card shape and references but are not enough to prove small-model protocol navigation.

Fix direction:
- Expand toward at least 50 cases.
- Include negatives, negations, paraphrases, multi-card cases, no-relevant-card cases, noisy notes, prompt injection, routine/monitor cases, and ASR-like transcription errors.

Done when:
- The eval set exercises both safety boundaries and usefulness.
- The README can report measured results rather than targets only.

## 6. Tighten validators for small-model failure modes

Current risk: validators check important basics but miss some subtle failures: wrong known-but-not-retrieved cards, generic checklists, shallow SBAR grounding, unsafe synonyms, and fallback masking.

Fix direction:
- Validate source cards against retrieved cards, not all known cards.
- Require fired rule cards to be included.
- Enforce full schema shape.
- Compare missing observations to retrieved card requirements where possible.
- Broaden forbidden action detection.
- Track fallback separately in eval metrics.

Done when:
- A valid-looking but wrong, generic, or unsafe model output fails deterministically.
- Tests cover the new validator behavior.

## 7. Stop UI and trace overclaiming

Current risk: audio and model labels can imply Omni, local 4B, or SQLite FTS even when the runtime path was typed transcript, canned fallback, or JSON search.

Fix direction:
- Derive post-run badges from trace state.
- Conditionally relabel or disable audio UI when audio intake is off.
- Label typed transcript heuristics separately from real Omni audio or Parakeet ASR.
- Report retrieval source as `sqlite_fts` or `json_fallback`.

Done when:
- The UI remains honest under no-secret, hosted, local, audio-disabled, and fallback modes.

## 8. Prove or demote local/off-grid claims

Current risk: local Parakeet is metadata-only today, and local 4B is currently just an OpenAI-compatible client route. Omni can be off-grid in principle on adequate hardware, but this repo still needs evidence for any claimed off-grid runtime.

Fix direction:
- Add gated smoke tests for local OpenAI-compatible text navigation.
- Add optional Parakeet/ASR proof only behind its explicit gate.
- Add a hardware/evidence note that distinguishes theoretical Omni self-hosting from the currently verified local route.
- Claim Off the Grid only for a recorded no-cloud run, whether that run uses self-hosted Omni on suitable hardware or the smaller local stack.

Done when:
- A trace and short run note prove a no-cloud model path, or the claim is clearly marked unproven.

## 9. Complete submission evidence and docs

Current risk: the repo lacks final submission evidence such as user test notes, safety statement, license file, and demo/social links.

Fix direction:
- Add `docs/user_test_notes.md`.
- Add `docs/safety_statement.md`.
- Add `LICENSE`.
- Add demo video and social post placeholders or final links.
- Replace broad badge language with achieved-versus-targeted status.

Done when:
- Backyard AI evidence shows a real trained responder used it on synthetic or de-identified scenarios.
- The README/submission materials only claim badges backed by artifacts.