docs/local_4b_v4_scaffolding_eval_shape_plan.md

Local 4B V4 Scaffolding And Eval Shape Plan

Date: 2026-06-10

Purpose

This plan defines the scaffolding and evaluation-shape fixes to make before any v4 LoRA training run.

The v3 model is not generally broken. The latest field-workflow holdout shows strong safety and protocol-navigation behavior, but weak handoff usefulness and weak observation-cue scoring. Some of the weakness is model-owned, especially REFERRAL-SBAR-v1 and radio handoff behavior. Some is harness-owned: the evaluator currently expects metadata cues that the app can deterministically know, such as retrieved card IDs and validation status, to appear inside model-authored missing-observation fields.

The goal is to stop asking the model to memorize app metadata, then measure the remaining model-owned gap clearly.

Current Evidence

Primary trace:

• traces/local_4b_finetuned_v3_field_holdout_sequential_20260610T102450Z/local_4b_eval.jsonl
• traces/local_4b_finetuned_v3_field_holdout_sequential_20260610T102450Z/eval_summary.json
• traces/local_4b_finetuned_v3_field_holdout_sequential_20260610T102450Z/eval_evidence_manifest.json

V3 field-workflow holdout result:

• total_cases: 150
• competence_successes: 107
• raw_configured_model_successes: 93
• repair_successes: 14
• fallback_uses: 2
• final_validation_successes: 148
• model_visible_fields_retained: 0.9415

Strong areas:

• min_urgency_met: 150/150
• red_flags_match: 150/150
• forbidden_behavior_absent: 150/150
• target_card_in_candidate_pathways: 149/150
• target_card_in_source_cards: 149/150
• expected_source_cards_present: 144/150

Weak areas:

• missing_observation_cues_present: 0/150
• REFERRAL-SBAR-v1: 0/27 competence
• radio_handoff: 0/16 competence
• sbar_handoff_usefulness: 0/10 competence
• source_card_discipline: 2/6 competence

Top missing observation cues among failed competence cases:

• navigator validation result: 40
• manual correction status for audio-derived fields: 40
• retrieved protocol card IDs: 37
• deterministic rule results: 23
• objective observations only: 18
• relevant background and timeline: 17
• specific request or receiving pathway: 17
• situation or reason for handoff: 15
• red flags already fired: 15
• confirmed intake status: 12
• source protocol card IDs: 12

Diagnosis

The current eval mixes three different things in expected_missing_observations:

1. Clinical or workflow observations the medic may need to collect.
2. Handoff content cues the model should include when the target is SBAR or radio handoff.
3. App/harness metadata the model should not need to invent, such as validation status, retrieved card IDs, deterministic rule results, and manual correction status.

This makes the score hard to interpret. A model can be safe, cite the right cards, preserve red flags, and produce useful protocol navigation while still failing every expected-label row because it did not phrase app metadata as a missing observation.

The fix is not to train the 4B model to recite every metadata cue. The fix is to move deterministic metadata into deterministic output surfaces and reserve model training for bounded, model-owned text.

Fix 1: Split Observation Cues By Ownership

Add an ownership split to generated eval cases and scoring:

• expected_model_observation_cues: facts or observations the responder may need to collect or confirm.
• expected_handoff_cues: SBAR/radio facts that should appear in the handoff fields.
• expected_harness_evidence_cues: deterministic app metadata that should be visible in trace or UI, not authored as missing observations.

Initial harness-owned cues:

• navigator validation result
• manual correction status for audio-derived fields
• retrieved protocol card IDs
• deterministic rule results
• confirmed intake status
• source protocol card IDs

Implementation targets:

• scripts/generate_field_workflow_holdout.py
• scripts/generate_finetune_data.py
• figment/eval_metrics.py
• scripts/run_eval.py

Do not rewrite the frozen field_workflow_holdout_v1 cases in place. Preserve them and add a derived scoring view or v1.1 manifest that maps existing expected_missing_observations into ownership buckets.

Fix 2: Add Deterministic Evidence Badges

Add deterministic evidence fields to the trace and app-facing navigator output so the UI can show metadata without asking the model to write it:

• intake confirmation status,
• retrieved protocol card IDs,
• fired deterministic rule IDs,
• urgency floor,
• validator status,
• audio/manual correction status,
• source-card set used by final output,
• fallback or repair tier.

Preferred shape:

{
  "harness_evidence": {
    "confirmed_intake": true,
    "retrieved_card_ids": ["..."],
    "deterministic_rule_ids": ["..."],
    "urgency_floor": "emergency",
    "validator_status": "passed",
    "audio_correction_status": "not_applicable",
    "final_route": "configured"
  }
}

This object should be deterministic and excluded from model-retained-field credit unless the model actually authored it. The UI can render it as compact badges beside the handoff rather than burying it in missing_info_to_collect.

Fix 3: Make Handoff A First-Class Eval Surface

Today the SBAR/radio misses are visible mostly through target card and observation-cue failures. Add explicit handoff metrics:

• sbar_situation_present
• sbar_background_present
• sbar_assessment_observation_only
• sbar_request_present
• sbar_source_card_cited
• sbar_red_flags_visible
• handoff_brevity_ok
• handoff_unsupported_fact_count
• handoff_readiness_passed

For radio_handoff and sbar_handoff_usefulness, these metrics should drive competence more than generic missing-observation cue coverage.

Acceptance target after scaffolding, before v4 training:

• radio_handoff and sbar_handoff_usefulness should no longer fail only because of harness-owned metadata cues.
• SBAR failures should report a specific missing handoff slot or unsupported fact, not a generic observation-cue miss.

Fix 4: Add A Deterministic SBAR Draft Scaffold

Build a deterministic SBAR draft before the model writes final text.

Inputs:

• confirmed intake,
• deterministic red flags,
• urgency floor,
• retrieved card IDs,
• target protocol card,
• high-value observation cues,
• source-card titles.

The scaffold should produce slot-limited draft facts:

• situation: patient, chief concern, target pathway or reason for handoff.
• background: only confirmed context and timeline.
• assessment: observation-only summary plus fired rule IDs, no diagnosis.
• request: the specific review, transport, callback, or protocol-navigation ask.

Then ask the model to rewrite only within those facts. If the model fails SBAR grounding, repair only the SBAR fields rather than falling back the whole navigator output.

Implementation targets:

• figment/prompt_builder.py
• figment/navigator.py
• figment/focused_repair.py
• scripts/run_eval.py

Fix 5: Trigger Competence Repair For Safe But Weak Outputs

The current repair path is mostly validation-driven. Many v3 SBAR misses are safe and schema-valid, so repair never fires.

Add an optional eval and app repair mode for model-owned competence failures:

• If final validation passes but handoff_readiness_passed fails, run a focused handoff_note_sbar repair.
• If source-card discipline fails but validation passes, run a focused source-card/candidate-pathway repair.
• If model-owned observation cue coverage fails, run a focused missing-observation repair.

This should not hide safety failures. Safety validation still wins. The competence repair should be reported separately:

• validation_repair_attempted
• competence_repair_attempted
• competence_repair_success
• competence_repair_scope

Fix 6: Normalize Cue Matching

For model-owned cues, add alias-aware matching so the eval rewards equivalent field phrasing:

• red flags already fired can match fired deterministic red flags, rule ids triggered, or a direct rule ID mention.
• source protocol card IDs can match a cited source-card ID in SBAR or source fields.
• specific request or receiving pathway can match a direct callback/transport/receiving-clinician request.
• objective observations only can match absence of diagnosis plus observation-only assessment language.

Do not relax safety, urgency, red-flag, source-card, or forbidden-behavior checks. Only relax brittle cue phrase matching for useful equivalent language.

Fix 7: Add Runtime Evidence To Prevent Bad Parallel Evals

The attempted parallel v3 holdout run produced invalid records because llama-server split the context across parallel slots and hit KV/cache overflow. Keep those records quarantined.

Add run metadata to every local eval bundle:

• server command,
• GGUF path and SHA-256,
• n_ctx,
• n_parallel,
• prompt cache settings,
• endpoint /v1/models payload,
• whether any server HTTP 500s occurred,
• whether the run is eligible for scored reporting.

Eval runner should refuse to mark a run clean if backend errors include Context size has been exceeded or failed to find free space in the KV cache.

Implementation Order

1. Add ownership bucketing for expected cues and update summary metrics.
2. Add deterministic harness_evidence to traces and UI-facing output.
3. Add explicit handoff-readiness metrics.
4. Add SBAR scaffold and focused SBAR repair.
5. Add competence repair for safe but weak outputs.
6. Add alias-aware cue matching for model-owned cues.
7. Add local runtime clean-run metadata and invalid-run detection.
8. Rerun v3 on the 150-case holdout and the locked 50-case regression before training v4.

Acceptance Gates

Before starting v4 training:

• The new eval report separates:
  • validation success,
  • model-owned observation cue coverage,
  • harness-owned evidence visibility,
  • handoff readiness,
  • competence repair success.
• The v3 holdout rerun no longer has expected_label_successes = 0/150 purely because of harness-owned metadata cues.
• REFERRAL-SBAR-v1 failures identify concrete handoff defects instead of only missing metadata cues.
• No regression on:
  • urgency floor,
  • red-flag match,
  • forbidden behavior,
  • final validation,
  • source-card validity.

Expected scaffolding-only improvement:

• Better interpretability immediately.
• Some increase in competence from deterministic handoff scaffolding and competence repair.
• Remaining SBAR/radio gaps become cleaner targets for v4 training.

Do not train v4 until this pass is done. Otherwise the v4 dataset will teach the model to satisfy a muddled scorer rather than to help field medics.