docs/local_4b_v3_training_plan.md

# Local 4B V3 Training Plan

Date: 2026-06-09

## Purpose

The v3 goal is not to make the local 4B model a broadly useful assistant. The goal is to make it better at the specific job Figment exists to support: helping rural clinic medics and disaster first-response medics move faster and more safely through patient intake, red-flag escalation, protocol navigation, missing-observation collection, and handoff drafting.

The v2 LoRA is a real improvement, but it was trained from the current evaluator's pressure points. V3 should keep those gains while reducing the risk that the model has learned "pass the 50-case exam" rather than "make the medic's workflow easier inside this harness."

## Current Evidence

Latest v2 eval trace:

- Trace: `traces/local_4b_finetuned_v2_evidence_20260609T103344Z/`
- Dataset: `data/finetune/figment_sft_v2.jsonl`
- Dataset manifest: `data/finetune/figment_sft_v2_manifest.json`
- Exact overlap between v2 training case ids and the 50 eval case ids: `0/50`

V2 eval results:

- `competence_successes`: `33/50`
- `raw_configured_model_successes`: `33/50`
- `fallback_uses`: `0`
- `repair_successes`: `0`
- `final_validation_successes`: `50/50`
- `model_retained_field_count`: `627/650`
- `model_field_pass_rate`: `0.9646`
- `expected_label_successes`: `15/50`
- Remaining expected-label failures: mostly missing-observation cue coverage (`29`), source-card coverage (`9`), red-flag match (`7`), and candidate pathway coverage (`4`).

V2 data shape:

- `1500` accepted rows.
- `1000` full navigator rows.
- `500` focused repair rows.
- Dominant categories were current-eval failure classes:
  - `missing_observation_cues`: `483`
  - `negation_safety_boundary`: `236`
  - `source_card_candidate_pathway`: `227`
  - `focused_repair:handoff_note_sbar`: `125`
  - `focused_repair:missing_observations`: `125`

Interpretation:

- V2 is not merely memorizing the named 50 eval cases.
- V2 is still at risk of semantic over-rotation because its category mix is tightly coupled to the current evaluator's visible failures.
- V3 should introduce a separate field-workflow target and a new held-out workflow suite before adding more training rows.

## V3 North Star

Train and evaluate the local 4B route as a bounded field-workflow model.

The model should be good at:

- Turning messy confirmed intake into a concise, card-cited navigator output.
- Preserving deterministic red flags and urgency floors.
- Avoiding diagnosis, treatment orders, dosing, discharge advice, or autonomous triage.
- Asking for the next observations that will actually help a responder move the case forward.
- Producing compact SBAR handoff language grounded in confirmed intake, deterministic rules, and retrieved protocol cards.
- Handling denied symptoms, uncertain reports, ASR-like noise, missing vitals, and low-resource context without hallucinating.
- Reducing responder cognitive load: fewer irrelevant fields, fewer generic checklists, clearer next steps, and faster handoff readiness.

The model does not need to be good at:

- General chat.
- Medical reasoning outside retrieved protocol cards.
- Open-ended clinical advice.
- Audio transcription itself.
- Replacing deterministic red-flag rules, validators, or local protocol.

## Anti-Overfitting Policy

V3 must use three distinct evaluation surfaces:

1. Locked 50-case regression eval.
   - Existing eval files stay locked.
   - Never train on these cases or close paraphrases.
   - Use this to ensure v3 does not regress from v2.

2. New field-workflow holdout eval.
   - Create before v3 training data generation.
   - Freeze case ids, row hashes, expected outcomes, and prompt hashes.
   - Never train on it.
   - Use it as the primary v3 success metric.

3. Synthetic development eval.
   - Regenerable and expandable.
   - Used for iteration, debugging, and per-category acceptance.
   - Safe to use for failure analysis, but not copied into training rows.

Every generated training row should pass near-neighbor rejection against both locked eval surfaces. Exact id exclusion is not enough. Reject rows with high similarity in:

- normalized confirmed intake text,
- target card,
- retrieved card set,
- red-flag set,
- missing-observation target set,
- SBAR situation/background wording,
- scenario template and patient presentation.

## New Field-Workflow Holdout

Create `data/eval/field_workflow_holdout_v1.jsonl` with 150 to 250 cases.

This holdout should measure whether Figment makes field work easier, not only whether it satisfies current evaluator slots.

Recommended categories:

- Rural clinic intake: limited equipment, missing vitals, one medic, delayed clinician callback.
- Disaster triage desk: noisy notes, multiple patients, scarce transport, incomplete identity details.
- Radio or runner handoff: fragmented observations, corrections, repeated facts, ambiguous timing.
- ASR-like confirmed text: homophones, dropped negations, punctuation-free fragments, but still confirmed by the responder before navigation.
- Escalation precision: clear red flags, near misses, denied symptoms, historical symptoms, contradictory witness reports.
- Missing-observation prioritization: ask for the few observations that change escalation or handoff quality first.
- SBAR usefulness: compact situation/background/assessment/request that a receiving clinician or transport coordinator can act on.
- Source-card discipline: relevant card, distractor card, missing card, and safety-boundary fallback cases.
- Low-resource constraints: no pulse ox, no BP cuff, no transport yet, paper protocol only, intermittent radio.
- Workflow recovery: previous output weak or overlong; focused repair should improve only the bad fields.

Holdout scoring should include both current harness metrics and workflow metrics:

- current schema and deterministic validation,
- red-flag match,
- urgency floor preservation,
- source-card and candidate-pathway correctness,
- observation cue coverage,
- forbidden clinical behavior absence,
- SBAR grounding,
- number of high-value next observations in the first five suggestions,
- generic/low-value checklist ratio,
- unsupported-fact count,
- output brevity and scanability,
- "handoff readiness" binary score,
- estimated responder time saved proxy.

For v3, the primary success claim should come from this holdout, not from the existing 50-case eval alone.

## V3 Dataset Target

Create `data/finetune/figment_sft_v3.jsonl` with 2500 to 3500 accepted rows after validation.

Recommended accepted-row mix:

- 900 to 1100 full navigator rows from rural clinic and disaster workflow scenarios.
- 400 to 600 escalation precision rows covering red flags, denied red flags, ambiguous reports, and contradiction handling.
- 350 to 500 missing-observation prioritization rows where the target is not "include every cue" but "surface the next useful observations in priority order."
- 300 to 450 SBAR usefulness rows focused on compact, grounded handoff language.
- 200 to 300 source-card discipline rows with distractors, missing relevant cards, and safety-boundary fallbacks.
- 150 to 250 low-resource workflow rows where missing equipment changes what the model should ask for.
- 300 to 500 focused-repair rows sampled from actual v2 failures and new workflow dev failures.

Do not simply add more `missing_observation_cues` rows in the v2 style. V3 should include required observations, but the target behavior is field usefulness: ask for observations that reduce uncertainty, support escalation, or improve handoff.

## Teacher Generation Strategy

Continue using `nvidia/nemotron-3-ultra-550b-a55b` as the teacher through the existing OpenAI-compatible hosted endpoint and existing secret wiring.

Teacher calls should generate three artifacts per candidate scenario:

1. Scenario spec.
   - Synthetic and de-identified.
   - Includes setting, responder constraints, confirmed intake, available supplies, missing equipment, and communication channel.

2. Workflow rubric.
   - Expected urgency floor.
   - Expected red flags.
   - Relevant and distractor cards.
   - High-value observations in priority order.
   - SBAR facts allowed and disallowed.
   - Safety-boundary constraints.

3. Gold assistant output.
   - The exact harness prompt shape as input.
   - Final assistant output as JSON only.
   - No visible reasoning, markdown, or teacher critique.

Teacher output is not trusted directly. Accept a row only after deterministic validators and workflow validators pass.

## V3 Validators

Keep all v2 validators:

- schema validation,
- harness prompt alignment,
- card-id validation,
- deterministic urgency floor validation,
- red-flag validation,
- source-card validation,
- candidate-pathway validation,
- forbidden behavior scanner,
- no teacher notes or reasoning leakage,
- synthetic/de-identified metadata,
- no locked-eval copy or near paraphrase.

Add v3 workflow validators:

- Priority validator: the first 3 to 5 `next_observations_to_collect` must include observations that would materially help escalation, monitoring, or handoff.
- Generic-output validator: reject rows dominated by vague suggestions such as "monitor closely", "repeat vitals", or "follow protocol" without case-specific observation targets.
- Low-resource validator: if equipment is unavailable, the output should not ask for that measurement as if it were immediately available; it can ask for alternatives or state unavailable status.
- Handoff-readiness validator: SBAR must include a concise situation, grounded background, observation-only assessment, and a specific request/pathway.
- Unsupported-fact validator: SBAR and checklist must not add facts absent from confirmed intake, deterministic rules, or retrieved cards.
- Cognitive-load validator: reject overlong lists unless the case has genuine multi-card complexity.
- Similarity validator: reject near-neighbors of locked 50-case eval and field-workflow holdout cases.

## Training Technique

Use the v2 LoRA as a baseline, but do not assume v3 should continue from it blindly.

Run two training variants on Modal:

1. Fresh v3 LoRA from the full BF16 base.
   - Best for checking whether v2 overfit is baked into the adapter.
   - Train on v3 only, with task-balanced sampling.

2. Continued LoRA from v2.
   - Best for preserving v2 schema and raw-output gains.
   - Train at lower learning rate with replay rows from v2.

Compare both variants on the locked 50-case eval and the new field-workflow holdout.

Recommended training recipe:

- Base model: `nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16`.
- Method: LoRA SFT, full-weight BF16 base, merged back to BF16 before GGUF conversion.
- Context: keep `16384` unless memory requires an explicitly labeled short-context ablation.
- Start with LoRA rank `16` or `32`; use rank `32` only if v3 underfits workflow diversity.
- Keep dropout around `0.05`.
- Use task-balanced sampling rather than raw row order.
- Use replay mixing: 15% to 25% high-quality v2 rows so schema discipline does not regress.
- Do not optimize checkpoint selection on training loss alone.

If SFT improves format but workflow usefulness remains brittle, add a small preference-tuning stage using deterministic pairs:

- preferred output: concise, grounded, high-value next observations, correct cards, safe SBAR;
- rejected output: schema-valid but generic, overlong, eval-cue-stuffed, unsupported, or not useful to a field medic.

Use preference tuning only after the holdout suite exists.

## Modal Job Shape

Reuse the existing Modal training structure:

- Stage `data/finetune/figment_sft_v3.jsonl` and manifests into the Modal data volume.
- Train adapter under a new dataset version such as `figment_sft_v3`.
- Save adapter and training manifest under a new output name, for example `figment-sft-v3-lora`.
- Run a merge-only job to produce merged BF16 Hugging Face weights.
- Pull merged weights locally.
- Convert to BF16 GGUF with the repo-local `tools/llama.cpp/convert_hf_to_gguf.py`.
- Serve with `llama-server` under the same local OpenAI-compatible route.
- Run locked 50-case eval and field-workflow holdout eval.

Do not publish or treat the checkpoint as accepted until both eval surfaces are complete and artifact-linked.

## Acceptance Targets

V3 should be judged against v2, not just against the original baseline.

Must not regress on the locked 50-case eval:

- `competence_successes`: at least `33/50`.
- `raw_configured_model_successes`: at least `33/50`.
- `fallback_uses`: `0`.
- `final_validation_successes`: `50/50`.
- `model_field_pass_rate`: at least `0.96`.
- `deterministic_patch_count`: at most `23`, or a documented reason if workflow improvements trade off with cue-stuffing.
- `forbidden_behavior_absent`: `50/50`.

Must improve on field-workflow holdout:

- Handoff readiness: at least `80%`.
- High-value first-five observation coverage: at least `80%`.
- Unsupported-fact rate: at most `5%`.
- Generic-output failure rate: at most `10%`.
- Low-resource mismatch rate: at most `5%`.
- Red-flag/urgency safety: no critical misses.
- No increase in forbidden clinical behavior.

Stretch target:

- Improve locked expected-label success above v2's `15/50`, but do not optimize v3 primarily for that number if it conflicts with field-workflow usefulness.

## Immediate Implementation Steps

1. Add `data/eval/field_workflow_holdout_v1.jsonl` and a manifest with frozen hashes.
2. Add a field-workflow eval runner or extend `scripts/run_eval.py` with workflow metrics that do not train on the holdout.
3. Add v3 scenario generators for rural clinic, disaster triage, radio handoff, ASR-like confirmed text, low-resource constraints, and workflow repair.
4. Add workflow validators for prioritization, generic output, low-resource mismatch, handoff readiness, unsupported facts, cognitive load, and near-neighbor rejection.
5. Generate a 100-row v3 smoke dataset and inspect category diversity.
6. Generate the full v3 dataset with the Ultra teacher.
7. Run harness alignment verification on v3.
8. Train fresh-v3 and continued-from-v2 LoRA variants on Modal.
9. Merge, convert, serve, and evaluate both variants locally.
10. Select the checkpoint that best improves field-workflow holdout without regressing locked 50-case safety and competence.

## Decision Rule

Accept v3 only if it improves the real product job: making rural clinic and disaster response intake/escalation/handoff faster, more grounded, and easier to act on.

If v3 only improves the locked 50-case eval while failing the field-workflow holdout, reject it as overfit.

If v3 improves the field-workflow holdout but slightly underperforms one non-safety cue-count metric from the locked eval, prefer the field-workflow result and update the next training/eval plan accordingly.