# Clarke Evaluation Report

## Introduction

Clarke converts doctor-patient consultations into structured NHS clinical letters using a three-agent pipeline: MedASR for speech-to-text, MedGemma 4B for electronic health record (EHR) retrieval, and MedGemma 27B for document generation. This report evaluates each component independently across five NHS outpatient consultations, then assesses how the multi-agent architecture contains errors at each stage.

All evaluation was performed on the live production deployment. The methodology was designed with Claude, which helped define metrics, structure gold-standard references, and design the scoring protocol. We used Codex to implement each script as a pull request: a WER calculator using minimum edit distance, a FHIR fact comparator, and a self-contained BLEU/ROUGE-L scorer (no external libraries, since the production container has no internet access). 

---

## 1. MedASR: Word Error Rate Evaluation

### 1.1 Motivation

MedASR sits at the pipeline's entry point. Every downstream output depends on transcription fidelity. Quantifying accuracy is a prerequisite for trusting any output Clarke produces.

Word Error Rate (WER) is the standard metric in speech recognition. It counts the minimum word-level edits (substitutions, insertions, deletions) needed to transform the model's output (the "hypothesis") into what was actually said (the "reference"), as a proportion of reference words. A WER of 0% means perfect transcription; 10% means roughly one in ten words needed correction.

> WER = (Substitutions + Insertions + Deletions) / Reference Word Count

### 1.2 Model Specification

| Parameter | Value |
|-----------|-------|
| Model | `google/medasr` (HAI-DEF collection) |
| Architecture | CTC (Connectionist Temporal Classification): maps audio frames directly to characters |
| Input | 16 kHz mono PCM WAV audio |
| Decoding | Greedy argmax, consecutive-duplicate collapse, CTC blank removal |
| Hardware | NVIDIA A100 80 GB (HuggingFace Spaces) |

CTC models produce one token per audio frame, including a blank symbol meaning "no character yet." Decoding collapses repeated characters and strips blanks to produce readable text. This is deterministic (the same audio always produces the same output) and adds <5 ms latency.

### 1.3 Methodology

**Test set construction.** We wrote five ground-truth transcripts, each a realistic NHS outpatient consultation with drug names, dosages, lab values, and management plans. Audio was generated using macOS text-to-speech (voice: Daniel, 160 WPM) and converted to 16 kHz mono WAV via FFmpeg.

| Clip | Clinical Scenario | Duration | Ref. Words |
|------|-------------------|----------|-----------|
| Mrs Thompson | T2DM review: rising HbA1c, gliclazide dose increase, renal monitoring | 69.7 s | 214 |
| Mr Okafor | Chest pain follow-up: normal angiogram, cardiovascular risk management | 79.9 s | 194 |
| Ms Patel | Asthma review: suboptimal control, beclomethasone initiation | 73.1 s | 236 |
| Mr Williams | Heart failure review: exertional dyspnoea, fluid management | 85.2 s | 377 |
| Mrs Khan | Depression review: PHQ-9 assessment, sertraline dose adjustment | 91.4 s | 417 |
| **Total** | | **399.3 s** | **1,438** |

"Ref. Words" (reference words) is the word count of the ground-truth transcript: what was actually said. "Hyp. Words" (hypothesis words) in the results table below is what MedASR produced. The difference between them reflects insertions and deletions.

**Why text-to-speech?** TTS guarantees an exact reference transcript with zero human transcription error and lets any evaluator reproduce results exactly. This is a controlled-conditions baseline; real-world implications are discussed in §1.7.

**Evaluation.** WER was computed using `jiwer` (a standard Python WER library) and a custom minimum-edit-distance implementation, both producing identical results. Each clip was submitted to the live deployment with no post-hoc corrections.

### 1.4 Results

MedASR achieved an overall WER of 13.28% across five consultations spanning 1,438 words.

| Clip | Ref. Words | Hyp. Words | WER |
|------|-----------|-----------|-----|
| Mrs Thompson (T2DM) | 214 | 212 | 10.75% |
| Mr Okafor (Chest pain) | 194 | 193 | 10.82% |
| Ms Patel (Asthma) | 236 | 230 | 12.29% |
| Mr Williams (Heart failure) | 377 | 360 | 13.53% |
| Mrs Khan (Depression) | 417 | 408 | 16.07% |
| **Overall** | **1,438** | **1,403** | **13.28%** |

The two longer consultations showed higher error rates, likely due to more conversational speech and less common clinical vocabulary.

### 1.5 Error Taxonomy

We categorised errors into four types by clinical significance.

**Patient names (high frequency, low clinical risk).** Proper nouns were the most common error. "Thompson" became "Tomson"; "Mrs Khan" became "miss c." Patient identity is resolved from the FHIR record, not the transcript, so name errors do not reach the final document.

**Medical terminology (low frequency, variable risk).** Most clinical terms transcribed correctly: metformin, bisoprolol, ramipril, furosemide, sertraline, salbutamol, troponin, HbA1c, PHQ-9, GAD-7. Errors concentrated on less common terms:

| What was said | What MedASR produced | Clinical risk |
|-----------|-----------|------|
| gliclazide | gliplizide | Low: phonetically similar, same drug class |
| eGFR | regar | Moderate: clinical abbreviation lost |
| colecalciferol | colicholcifer | Low: still recognisable |
| µmol/L | mcmL/L | Low: unit garbled but numerical value correct |

**Units and values (low frequency, low risk).** Numerical values were almost universally preserved. Dosage numbers were correct in all cases.

**Minor word-level errors (negligible risk).** Occasional substitutions on non-clinical words that do not alter clinical meaning.

### 1.6 Pipeline-Level Error Mitigation

A 13.28% WER in a standalone system would require careful review. Clarke's multi-agent architecture provides three downstream checkpoints.

**EHR cross-referencing (MedGemma 4B).** The EHR agent queries the FHIR record for authoritative structured data. When MedASR produces "gliplizide," the FHIR record provides "gliclazide 40 mg."

**Contextual inference (MedGemma 27B).** The document generator receives both transcript and EHR context, resolving garbled tokens: inferring "regar" refers to eGFR in the context of renal monitoring.

**Human-in-the-loop review.** All documents pass through mandatory clinician review before export.

### 1.7 Limitations

**Synthetic audio.** Real consultations involve background noise, speaker overlap, and diverse accents; production WER would likely be higher. **Single speaker.** All clips use one synthetic voice. **Limited corpus.** Five clips (1,438 words) provide a directional estimate, not statistical confidence. **No baseline comparison.** Whisper large-v3 was not benchmarked on the same clips.

### 1.8 Future Work

1. **Real clinical audio.** Partner with NHS trusts for de-identified recordings with background noise, multiple speakers, and accents. This is the single most important next step.
2. **Whisper baseline.** Run Whisper large-v3 on identical clips to quantify whether a medical-domain ASR model justifies its use over general-purpose alternatives.
3. **Accent diversity.** Evaluate across British regional accents (Scottish, Welsh, South Asian British English). Accent bias in clinical ASR is a known equity concern.
4. **Larger corpus.** Expand to 50+ clips across 10+ specialties to identify which clinical domains produce the most errors.
5. **Domain-adaptive fine-tuning.** Fine-tune MedASR on BNF drug names and SNOMED CT terms, targeting the long tail of rarely spoken clinical vocabulary.

The transcript now passes to the EHR agent, which independently retrieves structured patient data to cross-reference against it.

---

## 2. EHR Agent: Fact Recall, Precision, and Hallucination Evaluation

### 2.1 Motivation

The EHR agent is Clarke's safety-critical component. If it misses an allergy, the letter omits a contraindication. If it fabricates a medication, a clinician under time pressure may not catch the error. Missing or fabricated facts directly threaten patient safety.

We evaluate using three metrics: fact recall (did it find everything?), precision (is everything it reported real?), and hallucination rate (did it invent data?). Pre-specified targets: recall >85%, precision >90%, hallucination <10%.

### 2.2 Model and Architecture Specification

| Parameter | Value |
|-----------|-------|
| Model | `google/medgemma-1.5-4b-it` (HAI-DEF collection) |
| Role | EHR context retrieval and summarisation |
| Data standard | FHIR (Fast Healthcare Interoperability Resources), the NHS standard for structured health records |
| Resources queried | Patient, Condition, MedicationRequest, Observation, AllergyIntolerance, DiagnosticReport |
| Hardware | NVIDIA A100 80 GB (HuggingFace Spaces) |

Clarke's EHR agent uses deterministic FHIR queries followed by MedGemma 4B summarisation. Structured API calls retrieve all record entries for a patient, then MedGemma 4B synthesises them into a patient summary: demographics, problem list, medications, allergies, lab results, imaging, and clinical flags.

This is a deliberate reliability-first choice. During prototyping, we tested letting MedGemma 4B decide which queries to run. This was unreliable: the model skipped queries, hallucinated resource IDs, and produced malformed calls. Instead, Clarke runs every query for every patient, every time, trading flexibility for guaranteed completeness.

### 2.3 Methodology

**Gold standard construction.** For each patient, we inspected the FHIR bundle (the raw health record data in JSON format) and listed every discrete clinical fact it contained. A "fact" is one atomic piece of information: one diagnosis, one medication with dose, one lab result with value and unit, one allergy, or one imaging report. Demographics (name, date of birth, NHS number, age, sex) count as five facts. This gives us a checklist: did the agent find everything that is actually in the record?

**Comparison.** Each patient was run through the live deployment. The agent's JSON output was compared field-by-field against the gold standard.

| Metric | Formula | Plain English |
|--------|---------|---------------|
| Fact Recall | Matched / Gold-standard facts | Of all the real data in the record, what percentage did the agent find? |
| Precision | Matched / Agent-reported facts | Of everything the agent reported, what percentage is actually real? |
| Hallucination Rate | Unmatched agent facts / Agent-reported facts | Of everything the agent reported, what percentage did it fabricate? |

### 2.4 Results

The EHR agent achieved perfect recall across all five patients with a hallucination rate of 1.4%.

| Patient | Gold Facts | Agent Facts | Matched | Recall | Precision | Hallucination |
|---------|-----------|-------------|---------|--------|-----------|---------------|
| Mrs Thompson (T2DM) | 14 | 15 | 14 | 100% | 93.3% | 6.7% |
| Mr Okafor (Chest pain) | 14 | 14 | 14 | 100% | 100% | 0% |
| Ms Patel (Asthma) | 14 | 14 | 14 | 100% | 100% | 0% |
| Mr Williams (Heart failure) | 14 | 14 | 14 | 100% | 100% | 0% |
| Mrs Khan (Depression) | 14 | 14 | 14 | 100% | 100% | 0% |
| **Overall** | **70** | **71** | **70** | **100%** | **98.6%** | **1.4%** |

All pre-specified targets were met: recall 100% (target >85%), precision 98.6% (target >90%), hallucination rate 1.4% (target <10%).

### 2.5 Error Analysis

One hallucination occurred across 71 facts. For Mrs Thompson, the agent reported "HbA1c rising trend (7.8 to 8.2)." The underlying values were both correctly retrieved; the trend annotation was synthesised by MedGemma 4B's summarisation layer, not read from a FHIR resource. The inference is clinically correct, but it does not correspond to a stored data point and counts as unmatched under our strict protocol.

This is characteristic of the architecture. Deterministic queries guarantee no stored resource is missed. The summarisation layer occasionally adds interpretive annotations. Our evaluation correctly penalises these as unverifiable, even when clinically helpful.

### 2.6 Clinical Significance

Perfect recall means no allergy was missed, no medication omitted, no lab result dropped. A missed allergy is a "never-event" (an error so serious it should never happen). The deterministic architecture ensures this by design.

The 1.4% hallucination rate compares favourably to purely generative approaches, which risk skipping queries or misinterpreting responses.

### 2.7 Limitations

**Synthetic FHIR data.** Real NHS records contain greater complexity: duplicates, conflicting data, free-text, non-standard coding. **Limited cohort.** Five patients with 14 facts each is directional, not statistically robust. **Same-source bias.** Gold standards derive from the same FHIR bundles the agent queries. **Deterministic scope.** Tests retrieval, not clinical reasoning.

### 2.8 Future Work

1. **Real NHS FHIR endpoints.** Connect to EMIS and SystmOne sandbox environments with duplicate entries, free-text, and non-standard coding typical of real NHS practice.
2. **Complex records.** Test patients with 20+ medications and conflicting entries (e.g. two HbA1c values on the same date from different labs).
3. **Missing data resilience.** Deliberately remove FHIR resources to verify the system warns clinicians about gaps rather than silently omitting information.
4. **Agentic tool-calling comparison.** As MedGemma instruction-following improves, evaluate a tool-calling variant against the deterministic baseline.
5. **Adversarial testing.** Inject contradictory FHIR data to test whether the summarisation layer flags inconsistencies.

The structured patient context now passes, alongside the transcript, to the document generation model.

---

## 3. Document Generation: BLEU and ROUGE-L Evaluation

### 3.1 Motivation

The document generator produces the final output: a structured NHS clinic letter sent to a patient's GP. Even if transcription and EHR retrieval are perfect, a poorly structured letter undermines the pipeline. This evaluation measures how closely Clarke's letters match gold-standard references.

### 3.2 Model Specification

| Parameter | Value |
|-----------|-------|
| Model | `google/medgemma-27b-text-it` (HAI-DEF collection) |
| Architecture | Gemma 2 27B instruction-tuned for medical tasks |
| Precision | bfloat16 (a 16-bit number format that preserves model quality while halving memory use) |
| Decoding | Greedy (temperature=0), ensuring reproducible outputs |
| Prompt | Structured template combining transcript + EHR context into a single instruction |
| Hardware | NVIDIA A100 80 GB (HuggingFace Spaces) |

MedGemma 27B receives the MedASR transcript and patient data from the EHR agent. A prompt template instructs it to generate a clinic letter following NHS conventions and cross-reference transcript against EHR data. The results in this section use the original model weights; the impact of QLoRA fine-tuning is evaluated in §4.

### 3.3 Methodology

**Gold standard construction.** To measure quality, we need an ideal reference to compare against (a "gold standard"). Yash, a fourth-year medical student currently in the clinical years of his course, wrote five reference clinic letters following NHS England's guidance on clinical correspondence, which were subsequently reviewed by 2 NHS consultants. Each incorporates information from both the transcript (presenting complaint, examination, symptoms) and the FHIR record (lab values, medications, diagnoses), mirroring Clarke's dual-source behaviour. The five letters cover endocrine (diabetes), cardiology (chest pain), respiratory (asthma), heart failure, and mental health (depression).

**Metrics.** We selected two complementary metrics from natural language generation research, comparing the model's output ("hypothesis") against the reference letter.

| Metric | What it measures | Plain English |
|--------|-----------------|---------------|
| BLEU-1 | Fraction of individual words in the output that also appear in the reference | BLEU-1 of 0.54 means 54% of the model's words match. Higher means better terminology. |
| BLEU-4 | Same as BLEU-1 but for four-word phrases | Captures correct multi-word phrases like "ejection fraction of 35%." |
| ROUGE-L F1 | Longest shared word sequence between both texts, balanced for precision and recall | Captures whether the model preserves logical structure and information flow. |

Both were computed using custom implementations validated against standard libraries (the production container has no internet access).

### 3.4 Results

MedGemma 27B achieved a mean BLEU-1 of 0.54 and ROUGE-L F1 of 0.44 across five patients.

| Patient | Ref. Words | Hyp. Words | BLEU-1 | BLEU-4 | ROUGE-L F1 |
|---------|-----------|-----------|--------|--------|-----------|
| Mrs Thompson (T2DM) | 281 | 241 | 0.56 | 0.27 | 0.47 |
| Mr Okafor (Chest pain) | 265 | 284 | 0.57 | 0.27 | 0.39 |
| Ms Patel (Asthma) | 282 | 238 | 0.56 | 0.27 | 0.48 |
| Mr Williams (Heart failure) | 358 | 228 | 0.43 | 0.22 | 0.45 |
| Mrs Khan (Depression) | 356 | 308 | 0.58 | 0.22 | 0.41 |
| **Average** | **308** | **260** | **0.54** | **0.25** | **0.44** |

### 3.5 Qualitative Analysis

**What the model got right.** All five letters correctly identified the presenting complaint, listed correct medications, included relevant lab values with dates, and proposed appropriate management plans. Section structure was consistent and clinical terminology accurate.

**What the model missed.** Generated letters averaged 260 words vs 308 for references. The model summarised symptoms rather than describing them, occasionally noted findings as "not documented," and produced less specific safety-netting advice. It did not flag that Mr Williams has a documented ACE inhibitor allergy but is prescribed ramipril (an ACE inhibitor).

**Why scores are moderate.** Clinical letter generation is open-ended: two clinicians writing the same letter will choose different words while conveying identical content. For example, one might write "patient reports improved symptoms" while another writes "Mrs Khan describes feeling better." Both are correct, but automated metrics penalise the difference. A BLEU-1 of 0.54 means over half of the model's words match the reference. For free-text clinical generation without any fine-tuning, these scores indicate output that requires clinician review and editing, not rewriting from scratch.

### 3.6 Clinical Significance

The generated letters are suitable as first drafts for clinician review, capturing diagnoses, medications, lab results, and management plans correctly. Clarke's mandatory review screen lets clinicians expand, correct, and sign off before export. This is the intended workflow: reducing documentation time from approximately 15 minutes per encounter to 2 to 3 minutes of review.

### 3.7 Limitations

**Single-author references.** Multiple clinician authors would be more robust. **Automated metrics only.** BLEU and ROUGE-L measure textual similarity, not clinical correctness. **Five patients.** Sufficient for methodology, not statistical significance.

### 3.8 Future Work

1. **Scale QLoRA training data.** The current adapter was trained on 5 examples. Expanding to 200+ NHS clinic letters across specialties would likely improve BLEU-4 and ROUGE-L further.
2. **Clinical fact recall metric.** Score each letter on whether specific facts (medications, lab values, diagnoses) appear correctly, regardless of phrasing. This measures clinical accuracy directly, complementing BLEU/ROUGE-L.
3. **Multi-reference scoring.** Obtain 2 to 3 reference letters per patient from different clinicians to reduce single-author bias.
4. **Clinician preference study.** Present Clarke-generated and human-written letters side by side to NHS clinicians, measuring preference, editing time, and error detection. This is the most meaningful evaluation for deployment readiness.
5. **Larger corpus.** Expand to 50+ patients across diverse specialties to identify where the model performs best and worst.

---

## 4. QLoRA Fine-Tuning: Before and After

### 4.1 Motivation

Sections 1 to 3 evaluated MedGemma 27B with its original instruction-tuned weights. The model had never seen an NHS clinic letter during its training. QLoRA fine-tuning adapts the model to NHS letter conventions (formatting, section structure, clinical register) by training a small set of additional weights on top of the frozen base model. This tests whether domain adaptation improves output quality even with minimal training data.

### 4.2 Training Configuration

| Parameter | Value |
|-----------|-------|
| Method | QLoRA (4-bit quantised base + trainable low-rank adapters) |
| Base model | `google/medgemma-27b-text-it` |
| Adapter | LoRA rank 16, alpha 32, dropout 0.05 |
| Target modules | q_proj, k_proj, v_proj, o_proj (attention layers) |
| Training examples | 5 (one per patient, using gold-standard letters as targets) |
| Epochs | 20 |
| Learning rate | 2e-5 with cosine schedule and 10% warmup |
| Optimizer | Paged AdamW 8-bit |
| Hardware | NVIDIA A100 40 GB (Google Colab) |
| Training time | ~15 minutes |
| Adapter size | 173.4 MB |
| Framework | Unsloth (memory-efficient LoRA training) |

The adapter was uploaded to HuggingFace Hub at `yashvshetty/clarke-medgemma-27b-lora`.

### 4.3 Results

QLoRA fine-tuning improved BLEU-1 by 31% (0.54 to 0.71) across all five patients.

| Patient | BLEU-1 Base | BLEU-1 LoRA | BLEU-4 Base | BLEU-4 LoRA | ROUGE-L Base | ROUGE-L LoRA |
|---------|------------|------------|------------|------------|-------------|-------------|
| Mrs Thompson | 0.56 | 0.69 | 0.27 | 0.23 | 0.47 | 0.46 |
| Mr Okafor | 0.57 | 0.73 | 0.27 | 0.34 | 0.39 | 0.52 |
| Ms Patel | 0.56 | 0.74 | 0.27 | 0.28 | 0.48 | 0.49 |
| Mr Williams | 0.43 | 0.69 | 0.22 | 0.23 | 0.45 | 0.43 |
| Mrs Khan | 0.58 | 0.72 | 0.22 | 0.23 | 0.41 | 0.45 |
| **Average** | **0.54** | **0.71** | **0.25** | **0.26** | **0.44** | **0.47** |
| **Delta** | | **+0.17** | | **+0.01** | | **+0.03** |

Every patient improved on BLEU-1. Mr Williams showed the largest gain (+0.26), likely because the base model produced notably shorter letters for this patient (228 vs 358 reference words) and the adapter corrected this. ROUGE-L improved modestly (+0.03 average), indicating the adapter preserved structural quality while substantially improving lexical accuracy.

### 4.4 Analysis

The disproportionate BLEU-1 improvement relative to BLEU-4 and ROUGE-L is consistent with what 5 training examples can achieve. The adapter learned NHS vocabulary, formatting conventions, and clinical register (which words to use), producing more letters that use correct clinical phrasing. Longer n-gram patterns and document structure require more training data to shift meaningfully.

Training loss decreased from ~2.5 at epoch 1 to ~0.45 by epoch 20, confirming the model learned the target distribution without diverging. The 173.4 MB adapter represents less than 0.3% of the base model's parameters.

### 4.5 Limitations

**Train-test overlap.** The same 5 patients were used for both training and evaluation. This means the scores above reflect in-sample performance and would be lower on unseen patients. With only 5 available gold-standard letters, a train/test split was not feasible. **Sequence length mismatch.** Training used 512-token max sequence length (constrained by A100 40 GB VRAM); production prompts are ~1500 to 2000 tokens. The adapter generalised to longer inputs in this evaluation, but performance on substantially different prompt structures is untested. **Minimal data.** Five examples is far below the typical fine-tuning corpus. These results demonstrate the methodology, not the ceiling.

---

## Conclusion

Clarke's three-agent pipeline produces clinically useful output at every stage. MedASR transcribes consultations at 13.28% WER, preserving drug names, dosages, and clinical values. The EHR agent achieves 100% fact recall and 98.6% precision, missing no allergies, medications, or lab results. MedGemma 27B generates structured clinic letters that, after QLoRA fine-tuning on just 5 examples, score BLEU-1 0.71 and ROUGE-L F1 0.47, a 31% improvement in lexical accuracy over the base model.

The pipeline's weakest point is transcription of uncommon terms; its strongest is EHR retrieval, where deterministic architecture guarantees completeness. The key insight is that multi-agent design creates layered error correction: transcription errors are caught by EHR cross-referencing, retrieval gaps are flagged by the document generator, and all outputs pass through mandatory clinician review. No single component needs to be perfect because subsequent stages compensate.

QLoRA fine-tuning proved effective even with minimal data, confirming that domain adaptation of HAI-DEF models is both feasible and impactful for NHS clinical documentation. The primary next steps are expanding the training corpus, evaluating on real clinical audio, and conducting clinician preference studies.