docs/local_4b_prompting_scaffolding_fixes.md

Local 4B Prompting And Scaffolding Fixes

Date: 2026-06-07

This note captures the prompt and controller changes I would make before reaching for a larger model. The goal is to make nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 more load-bearing on the local llama.cpp route while keeping Figment's deterministic validators and safety boundaries intact.

Evidence Snapshot

Primary trace:

• traces/local_4b_evidence_20260607T231248Z/
• traces/local_4b_evidence_20260607T231248Z/local_4b_eval.jsonl
• traces/local_4b_evidence_20260607T231248Z/eval_summary.json

Observed local 4B behavior:

• 50/50 final validation successes.
• 18/50 competence successes.
• 13/50 raw configured-model successes.
• 5/50 repair successes.
• 9/50 full deterministic fallbacks.
• 499/650 visible fields retained from model output, or 76.8%.
• Expected-label full success was 2/50.

The failure pattern looks fixable with better scaffolding and fine-tuning. The model is not generally missing urgency: min_urgency_met passed 50/50. It is mostly missing exact required-observation cues, candidate pathway/source-card rubric details, negation discipline, and grounded SBAR phrasing.

Expected-label failure counts:

• missing_observation_cues_present: 47 failures.
• target_card_in_candidate_pathways / expected_candidate_pathways_present: 15 failures.
• expected_source_cards_present: 12 failures.
• red_flags_match: 7 failures.
• forbidden_behavior_absent: 4 failures.
• target_card_in_source_cards: 1 failure.
• min_urgency_met: 0 failures.

Current Anchors

Current prompt scaffolding already exists but is too advisory:

• figment/prompt_builder.py builds allowed_facts_inventory, required_observations_inventory, routine_or_negated_case_guidance, and a required JSON skeleton.
• figment/validators.py enforces source-card constraints, urgency floors, missing-observation grounding, SBAR grounding, and forbidden clinical language.
• figment/focused_repair.py already has a missing_observations repair scope, but the repair prompt still asks the model to infer the exact observation language.
• figment/eval_metrics.py scores expected labels separately from safety validation, which is the right separation.

Fix 1: Promote Required Observations From Context To Targets

Problem: required_observations_inventory is present, but the 4B model treats it like optional supporting context.

Change:

• Add a compact required_observation_targets payload to the prompt context.
• Give each target a stable id, card id, normalized cue tokens, and display text.
• Tell the model that missing_info_to_collect and next_observations_to_collect must include at least one target cue for every cited non-exempt card that has required observations.
• After model output, deterministically patch missing target cues into those two fields before falling back.
• Trace each patch as deterministic_required_observation_fill, not model competence.

The key distinction is that the LLM can phrase the responder-facing sentence, but the app owns the checklist target. This should attack the largest failure class directly without weakening the validator.

Done when:

• missing_observation_cues_present improves from 3/50 passing to at least 45/50 passing.
• The trace shows whether each required-observation cue was model-written, repaired, or deterministically filled.
• Whole-output competence and field-retention metrics do not count deterministic fills as raw model success.

Fix 2: Pre-Fill Non-Creative Control Fields

Problem: the model is being asked to regenerate facts the deterministic system already knows.

Change:

• Pre-fill protocol_urgency from the deterministic urgency floor.
• Pre-fill fired red_flags from deterministic rule results.
• Pre-fill mandatory source_cards from fired rule card ids plus the retrieved/selected card ids.
• Pre-fill candidate pathway options from retrieval before asking the model to write reasons.
• Ask the model to write bounded text for reason_relevant, responder_checklist, do_not_do, SBAR slots, plain language, and uncertainty handling.

This narrows the 4B model's job from "reconstruct the whole navigation state" to "explain and operationalize already-bounded state." That is a better match for a small model.

Done when:

• Source-card and candidate-pathway expected-label failures drop sharply.
• Fallbacks caused by sparse or malformed source-card fields become rare.
• The model is still visibly load-bearing on prose, checklists, and pathway reasons.

Fix 3: Add A Negation Ledger

Problem: routine or denied-symptom cases still sometimes inherit nearby emergency-card language.

Change:

• Add a case_fact_ledger to the prompt with three explicit buckets: present, absent_or_denied, and unclear.
• Add must_not_fire_rule_ids when a rule's trigger terms are absent or denied.
• Put the ledger before the protocol cards in the prompt.
• Require any red flag to cite a present fact or deterministic rule result.
• For SBAR and checklist text, forbid copying high-risk card language unless supported by present facts or deterministic red flags.

Done when:

• red_flags_match failures drop from 7/50 to 1/50 or less.
• Routine negated cases stay routine when the deterministic urgency floor is routine.

Fix 4: Make SBAR A Filled Template

Problem: SBAR failures are not mainly creative-writing failures. They are slot-grounding failures.

Change:

• Build an SBAR draft template with fixed slots:
  • situation: chief concern plus setting, if confirmed.
  • background: confirmed age, pregnancy status, and relevant context only.
  • assessment_observations_only: confirmed symptoms, vitals, and deterministic red flags only.
  • handoff_request: local protocol, supervisor, clinician, or emergency pathway request only.
• Ask the model to lightly rewrite the template, not invent SBAR content.
• Keep unsupported high-risk facts out of the template before the model sees it.

Done when:

• SBAR grounding failures do not force full fallback when the rest of the model output is usable.
• Unsupported high-risk SBAR terms such as pregnancy, oxygen, or pressure only appear when present in confirmed intake, deterministic rules, or card text that is allowed for that slot.

Fix 5: Separate Observations From Clinical Actions

Problem: the expected-label scorer and validators need sharper language around terms that can be either safe observations or unsafe instructions.

Change:

• Split observation cues from intervention cues.
• Treat oxygen saturation, SpO2, and room-air saturation as observation language.
• Treat administer oxygen, start oxygen, oxygen-flow settings, dosing, and medication instructions as intervention language.
• Update expected-label forbidden checks so observation requests do not get penalized as unsafe oxygen instructions.
• Keep forbidden clinical action patterns strict.

Done when:

• The model can ask for oxygen saturation as missing information without being pushed toward oxygen administration language.
• forbidden_behavior_absent failures are true safety failures, not observation/action ambiguity.

Fix 6: Make Focused Repair Deterministic-Target-Aware

Problem: the current focused repair scope for missing observations asks the model to repair only the two observation arrays, but it does not force exact target coverage.

Change:

• For the missing_observations repair scope, include only:
  • the failed card ids,
  • the missing target cue ids,
  • the allowed display text,
  • the previous two observation arrays.
• Require the repair output to return exactly missing_info_to_collect and next_observations_to_collect.
• Reject repairs that omit target cue ids.
• If repair still omits a cue, patch deterministically rather than asking for another broad repair.

Done when:

• Missing-observation repairs become short, low-latency, and predictable.
• Repaired observation fields count as model_repaired; deterministic fills count separately.

Fix 7: Add A Structured Output Contract For Target Coverage

Problem: natural-language arrays are hard to audit for exact expected-label coverage.

Change:

• Add an internal-only field during model generation, such as selected_required_observation_ids.
• Strip it from the user-facing navigator output after validation.
• Validate that every cited required-observation card has at least one selected observation id.
• Use the selected ids to prove why a natural-language observation sentence satisfies the target.

Done when:

• Expected-label scoring can distinguish "model selected the right cue but phrased it differently" from "model missed the cue."
• The visible output remains clean while trace evidence becomes more exact.

Suggested Implementation Order

1. Add required_observation_targets and case_fact_ledger in figment/prompt_builder.py.
2. Add deterministic target-fill helpers shared by navigator finalization and tests.
3. Update focused repair for the missing_observations scope to use target ids.
4. Pre-fill or lock control fields before the LLM call.
5. Convert SBAR generation to a slot template plus bounded rewrite.
6. Split observation/action forbidden-language scoring for oxygen-like terms.
7. Rerun scripts/run_local_4b_evidence.py and compare against the 2026-06-07 trace.

Non-Negotiables

• Do not weaken deterministic red-flag rules.
• Do not loosen validators to inflate model competence.
• Do not count deterministic target fills as raw model output.
• Do not claim Parakeet ASR proof from typed transcripts or local artifact presence.
• Keep the final local artifact full-weight/BF16 for the local route; no quantized local model claim.