ORACLE-LIMITS.md

# The oracle's real limits

The verifier is the reason to use PyBytecode at all, so its limits belong in front of a user, not
in an appendix. Everything here is measured; sources are named per section.

---

## 1. The pre-flight 100% proves almost nothing. Read this before quoting it.

Every grading command prints `PRE-FLIGHT 600/600 = 100%` before it scores. **That number is
trivial by construction and is not evidence of soundness.**

Pre-flight grades each reference label against itself. The oracle asks whether
`compile(prediction)` and `compile(reference)` produce the same code object — so at pre-flight it
is comparing `compile(x)` with `compile(x)`. It would return 100% for *any* deterministic
function of the source, including a stub that hashes the input string and ignores the bytecode
entirely.

What pre-flight actually detects is a **broken harness**: a benchmark whose `.pyc` files do not
match their sources, a Python version mismatch (3.11 or 3.13 against a 3.12 benchmark), a corrupt
row. Those are real failure modes and worth catching, which is why it runs. But a passing
pre-flight says the instrument is plugged in, not that it measures anything.

**Soundness evidence comes from the mutation test and the blind-spot probes, not from
pre-flight**: corrupt a label and require the oracle to reject it. Measured
(`evidence/ORACLE-MUTATION.md`): **0 true survivors in 1,239 mutants**, and
**18/18 targeted blind-spot probes** behave as required — including the historical failure where
a `try:` body and a `try/else:` body were indistinguishable, docstring changes, docstring
removal, float-vs-int, bool-vs-int and `-0.0` vs `0.0`.

Even the mutation kill rate is weak evidence on its own: for a byte-identical oracle a kill is
close to tautological, since a mutant survives only if it compiles to a structurally identical
code object. The probes are the load-bearing test, because they ask the question that actually bit
us once — *is a behaviourally load-bearing field missing from the fingerprint?*

**Caveat on mutation supply, stated rather than hidden:** 188 of 600 wild rows (31%) produced no
effective mutant within 30 tries, and 323 void attempts were discarded. The wild kill rate is
measured on the 412 rows that did produce one.

## 2. The 0.33% wild false-reject floor

Against `.pyc` files built by someone else, the oracle refuses a small fraction of correct
answers. Measured on 600 wild install-time `.pyc` from installed site-packages
(`evidence/GATE-RESULT.md`):

| | certified | false reject | false accepts |
|---|---|---|---|
| L0 (old constant encoding) | 585/600 = 97.5% | 15 = 2.5% | 0 / 1,274 |
| **L1 (shipping)** | **598/600 = 99.67%** | **2 = 0.33%** | 0 / 1,274 |

13 of the 15 L0 failures were our own defect — `repr()` of a `set`/`frozenset`/`dict` follows the
compiling process's hash seed, which also made the L0 verdict **non-deterministic** (585 / 592 /
584 / 585 / 589 under `PYTHONHASHSEED` 0–4). L1 fixes it and returns 598 under all five seeds.

The remaining **0.33% is a real floor and is not fixable.** One distinct module
(`pandas/_testing/__init__.py`) compiles differently under CPython 3.12.3 than under 3.12.13 —
`co_code` 2,692 vs 2,696 bytes, and a differing `co_exceptiontable`. The source is correct; the
*compiler patch release* differs. No normalisation removes this without abandoning the
byte-identical guarantee.

**It degrades to a false REJECT, never a false accept.** You are told "unknown" about a correct
answer; you are never told "verified" about a wrong one. That is the safe direction, and it is the
direction the design chose deliberately.

## 3. Optimization level must match the producer's, or verification collapses

A `.pyc` built with `-O` or `-OO` is a different code object. Measured on 679 sources compiled by
a foreign interpreter at each level and graded at each level:

| producer ↓ / grader → | 0 | 1 | 2 |
|---|---|---|---|
| **0** | **100.0%** | 95.43% | 23.86% |
| **1** | 95.43% | **100.0%** | 24.15% |
| **2** | 23.86% | 24.15% | **100.0%** |

The diagonal is 679/679 at every level. Guessing wrong is **not** a graceful degradation — it
collapses to ~24%. There are only three levels and trying all three costs three compiles, so this
is *"needs normalisation to survive"*, not *"breaks"*. **The harness does not currently search the
three levels automatically; a user verifying a foreign `.pyc` must do it.**

**What trying all three costs you, and it is not nothing.** The certificate changes meaning from
*"byte-identical to the code object the original source compiles to"* to *"byte-identical to the
code object that was actually shipped"*. At `optimize>=1` docstrings are **absent from the `.pyc`**,
and at `>=2` asserts are gone too. So against an `-O` artifact, **docstring recovery cannot be
certified at all** — the information is not in the file. That bears directly on our headline
differentiator: 115 of 679 benchmark rows carry a real docstring, and none of that could be proven
against an `-O` `.pyc`. It is a limit of the artifact, not unsoundness in the oracle.

## 4. What the oracle deliberately ignores

Excluded from the fingerprint, with how often each would have caused a false reject on the 600
wild rows had it been included:

| Field | In fingerprint? | Rows differing / 600 |
|---|---|---|
| `co_filename` | excluded | **600 (100%)** — every wild `.pyc` carries its builder's absolute path |
| `co_linetable` | excluded | **122 (20.3%)** — varies across builds with no semantic content |
| `co_firstlineno` | excluded | 0 |

Docstrings, `co_consts`, `co_names` and `co_exceptiontable` are **included** — the first because
docstring fidelity is a claim we make, the last because omitting it once produced a false proof.

## 5. Unverified means unknown, not wrong

The oracle is **sound but incomplete**:

```
verified   = PROVABLY correct. Identical code object => identical behaviour. No false positives.
unverified = UNKNOWN. A correct decompilation that compiles differently — a `while` where the
             original had a `for`, a differently-ordered but equivalent boolean — does not verify.
```

Reported accuracy is therefore a **lower bound on correctness**, not an estimate of it. Treating
the unverified remainder as errors understates the model; treating it as correct is unsafe.

## 6. Not tested — unknown, not claimed

- **Cross-minor (3.13).** No 3.13 interpreter on the measurement box; nothing was downloaded.
  The benchmark and the model are 3.12 only.
- **PyInstaller / Nuitka containers.** `import PyInstaller` → `ModuleNotFoundError`. Not measured.
- **`.pyc` from non-CPython or patched builds.** Not measured. Given that a *patch release*
  already produces the 0.33% floor, a patched build is a live risk, not a theoretical one.
- **Obfuscated or deliberately adversarial bytecode.** Not measured. No malware was fetched.

## 7. Where these limits are stated to users

| Limit | Stated in |
|---|---|
| Pre-flight is trivial | this file; `EVAL.md`; `harness/README.md`; both benchmark data cards |
| 0.33% wild false-reject floor | this file; `EVAL.md`; `weights/MODEL-CARD.md` |
| `-O` mismatch collapse, docstrings unprovable | this file; `EVAL.md`; `weights/MODEL-CARD.md` |
| unverified ≠ wrong | this file; `EVAL.md`; `weights/MODEL-CARD.md`; `harness/README.md` |
| 3.13 / PyInstaller untested | this file; `EVAL.md`; `weights/MODEL-CARD.md` |
| **19.4% formatting ceiling (§8)** | **this file** |

## 8. The formatting ceiling: 19.4% of real modules cannot certify a re-formatted answer

This section was added on 2026-09-11. It discloses a ceiling that was measured earlier and was
not previously written down here, which left §7's promise of a complete accounting unmet.

**`ast.unparse` is not bytecode-preserving.** Compiling a source file, and compiling that same
file after an AST round trip, do not always produce the same code object:

```python
compile(ast.unparse(ast.parse(src)))  !=  compile(src)     # for 100 of 516 stdlib modules
```

Measured on all 516 CPython 3.12 stdlib modules: **100 fail the round trip — an 80.6% pass rate,
so 19.4% of real modules contain at least one affected construct.**

**The cause**, reduced to a repro verified on the measurement box:

```python
a = {name for name, value in ns.items() if getattr(value, 'x', False)}   # POP_JUMP_IF_FALSE
a = {name                                                               # POP_JUMP_IF_TRUE
     for name, value in ns.items()                                      # + JUMP_BACKWARD
     if getattr(value, 'x', False)}
```

Identical AST, different bytecode. CPython 3.12's CFG optimiser lays out basic blocks according to
the **physical line layout** of a comprehension. An `if` *statement* wrapped the same way does not
change; comprehensions do.

**What this means for anyone reading a score from this oracle.**

- It is a ceiling on the **oracle**, not a defect in any model. A decompilation that is
  semantically perfect but wraps a comprehension across lines differently from the original will
  **not certify**. It is scored as unverified, which per §5 means *unknown*, not *wrong*.
- It applies to **every system measured against this oracle** — ours, PyLingual's, and any other.
  It is not a differential advantage or disadvantage to anybody.
- It is **whole-module scale**: 19.4% is the fraction of *modules* containing at least one
  affected construct, not the fraction of constructs affected. Function- and class-level units are
  affected at a lower rate, because the chance of containing a comprehension is lower.
- **It compounds with §2's 0.33% floor and §3's `-O` collapse.** These are independent sources of
  false rejection; none of them ever produces a false accept.

**Why the numbers already published are not invalidated by this disclosure.** Every published
score was produced by grading a model's output against a reference *source*, never against an
`ast.unparse` round trip, so no published figure was computed through the lossy path. This section
discloses a bound on how high any such score could ever go; it does not move one. It also rules
out a class of pipeline we did not build: an AST-splice reassembler would have discarded roughly
one module in five before asking the model a single question.

**Measured, and stated as such.** 100/516 is a direct count over the CPython 3.12 stdlib on the
measurement box. The corresponding ceiling on the internal 522-row module-shaped eval set — 92.5%
overall, 88.3% on rows of ≥400 representation lines — is measured on a different set and is
reported with those internal results rather than here.