# BioMysteryBench Changelog

## v11 (2026-07-06)

Problem count: **99 → 90** (73 human-solvable, 17 human-hard).

### Removed (9 problems)

Removed after a June 2026 audit that reran every problem with expert
bioinformaticians and multiple model runs, and cross-checked answer keys
against the underlying data.

| id | split | reason |
|---|---|---|
| `hb004` | solvable | Answer key (human) contradicted by the data — reads assemble a complete dog mitogenome with zero human mitochondrial reads. |
| `hb006` | hard | Answer key listed `Sample_1`–`Sample_74` contiguous, but the 74 correct sample IDs are non-contiguous; removed rather than rewrite the 74-entry key. |
| `hb011` | solvable | Answer key wrong — R1 and R2 have identical read-length distributions; the "110/112" answer was pooled mode/median, not a real R1≠R2 difference. |
| `hb014` | hard | Answer key (haplogroup L0) contradicted by every expert and model rerun (all derive H5b). |
| `hb022` | hard | Answer key inverted relative to hallmark markers of erastin treatment; both experts selected the opposite direction. |
| `hb027` | hard | Underivable — the exact 173/217 case/control split cannot be recovered from the unlabeled data provided. |
| `hb036` | hard | Data–task mismatch — fungal ITS amplicon data cannot contain the bacterial reads the question asks about (*A. fabrum* undetectable). |
| `hb040` | solvable | Infeasible — host-only corneal transcriptome contains no viral reads; SARS-CoV-2 cannot be identified from the provided data. |
| `hb053` | hard | Underdetermined — expert benchmarkers disagreed with the key and with each other. |

### Modified (24 problems, 28 edits)

Typo fixes, answer-key broadening, filename alignment, and prompt
clarifications. None change the underlying data files.

| id | field | change |
|---|---|---|
| `hb001` | prompt | "Which human organ" → "Which organ" |
| `hb003` | rubric | Accept `ITGAV` **or `F3`** as knocked-out gene (was ITGAV only) |
| `hb012` | prompt | "sample X" → "`Sample_X`" (literal filename in the data) |
| `hb013` | prompt | "HGVS nomenclature" → "HGNC nomenclature" |
| `hb024` | prompt | Simplified wording of the tissue-groups / species question |
| `hb031` | rubric | Also accept "Norwalk virus" alongside Norovirus / Norovirus GII.4 |
| `hb032` | prompt | "sample X … vs samples Y and Z" → "Abnormal samples … vs Normal samples" (X/Y/Z placeholders did not exist in the data) |
| `hb033` | prompt | Plural "samples … list" → singular "sample … single sample identifier" |
| `hb035` | rubric | Broaden `hsa-let-7b-5p` → accept any hsa-let-7 family member (a/b/c/d/e/f/g/i-5p) |
| `hb043` | prompt | "asnwer" → "answer" |
| `hb050` | prompt | "the the" → "the" |
| `hb054` | prompt | "treated with NELF-B" → "had NELF-B degraded" |
| `rec6xeqyddiz6desi` | rubric | Cas9 cut site: accept ±3 bp of chr3:46373175 (chr3:46373172–46373178) |
| `rec9ogrlqg5u0ke09` | prompt (×2) + rubric | `sample01.bigWig` → `Sample1.bw` (align with actual filenames) |
| `recaikavdwoimjy3b` | prompt | Grammar fix + clarify "the knockouts are in samples 1,2,3" |
| `recc3vmqjrsefqw57` | rubric | "an BRCA1" → "a BRCA1" |
| `recea4hqimc4sypon` | rubric | Reword expected answer to "no donor-identity mislabeling occurred; the swap is a tissue swap within Donor 3" |
| `recjgwpbyodqoihqc` | prompt | Append "Use a single linear regression across all calibration standards." |
| `recmp75e1chtpzx3c` | prompt | "bacteria was" → "bacterium was?"; add "name the organism the spike-in represents, not the construct type" |
| `recnayu0v8zttjlgf` | prompt + rubric | Double-space fix; "Groups 1:" → "Group 1:" |
| `recro5s1o0odyssqs` | prompt | Example "GENE-201" → "GENE-2XX" |
| `rectaxd8eganpl4lw` | prompt | Append "Use the whole-amplicon quantification window (CRISPResso -w 0)." |
| `recvnlq3i6id6qqge` | prompt + rubric | Grammar fix; also accept "frontal cortex" / "cerebral cortex" |
| `recx4bsaa5zoxy3nv` | prompt | "RBM39,GADD45A" → "RBM39, GADD45A" |

### Grading rule (all problems)

The `answer_rubric` for every problem now carries an explicit
all-or-nothing scoring sentence: *"Score 1.0 if the model did not cheat
AND got the answer correct. Score 0 otherwise."* This replaces earlier
wording that could be read as awarding credit for the anti-cheat check
alone. Recalling information from memory (even if it happens to include
the source publication) is **not** penalized; only actively looking up
GEO/SRA/ENA/BioProject accessions or reverse-identifying the dataset is.

## v8 (2026-04-28)

Initial public release (99 problems). Preloaded data files scrubbed of
accession-ID leaks in 16 problems; grading rule rewritten to distinguish
disallowed accession lookup from allowed standard database use (gene ID
lookup, sequence annotation, reference genome download, BLAST).