# The oracle's real limits The verifier is the reason to use PyBytecode at all, so its limits belong in front of a user, not in an appendix. Everything here is measured; sources are named per section. --- ## 1. The pre-flight 100% proves almost nothing. Read this before quoting it. Every grading command prints `PRE-FLIGHT 600/600 = 100%` before it scores. **That number is trivial by construction and is not evidence of soundness.** Pre-flight grades each reference label against itself. The oracle asks whether `compile(prediction)` and `compile(reference)` produce the same code object — so at pre-flight it is comparing `compile(x)` with `compile(x)`. It would return 100% for *any* deterministic function of the source, including a stub that hashes the input string and ignores the bytecode entirely. What pre-flight actually detects is a **broken harness**: a benchmark whose `.pyc` files do not match their sources, a Python version mismatch (3.11 or 3.13 against a 3.12 benchmark), a corrupt row. Those are real failure modes and worth catching, which is why it runs. But a passing pre-flight says the instrument is plugged in, not that it measures anything. **Soundness evidence comes from the mutation test and the blind-spot probes, not from pre-flight**: corrupt a label and require the oracle to reject it. Measured (`evidence/ORACLE-MUTATION.md`): **0 true survivors in 1,239 mutants**, and **18/18 targeted blind-spot probes** behave as required — including the historical failure where a `try:` body and a `try/else:` body were indistinguishable, docstring changes, docstring removal, float-vs-int, bool-vs-int and `-0.0` vs `0.0`. Even the mutation kill rate is weak evidence on its own: for a byte-identical oracle a kill is close to tautological, since a mutant survives only if it compiles to a structurally identical code object. The probes are the load-bearing test, because they ask the question that actually bit us once — *is a behaviourally load-bearing field missing from the fingerprint?* **Caveat on mutation supply, stated rather than hidden:** 188 of 600 wild rows (31%) produced no effective mutant within 30 tries, and 323 void attempts were discarded. The wild kill rate is measured on the 412 rows that did produce one. ## 2. The 0.33% wild false-reject floor Against `.pyc` files built by someone else, the oracle refuses a small fraction of correct answers. Measured on 600 wild install-time `.pyc` from installed site-packages (`evidence/GATE-RESULT.md`): | | certified | false reject | false accepts | |---|---|---|---| | L0 (old constant encoding) | 585/600 = 97.5% | 15 = 2.5% | 0 / 1,274 | | **L1 (shipping)** | **598/600 = 99.67%** | **2 = 0.33%** | 0 / 1,274 | 13 of the 15 L0 failures were our own defect — `repr()` of a `set`/`frozenset`/`dict` follows the compiling process's hash seed, which also made the L0 verdict **non-deterministic** (585 / 592 / 584 / 585 / 589 under `PYTHONHASHSEED` 0–4). L1 fixes it and returns 598 under all five seeds. The remaining **0.33% is a real floor and is not fixable.** One distinct module (`pandas/_testing/__init__.py`) compiles differently under CPython 3.12.3 than under 3.12.13 — `co_code` 2,692 vs 2,696 bytes, and a differing `co_exceptiontable`. The source is correct; the *compiler patch release* differs. No normalisation removes this without abandoning the byte-identical guarantee. **It degrades to a false REJECT, never a false accept.** You are told "unknown" about a correct answer; you are never told "verified" about a wrong one. That is the safe direction, and it is the direction the design chose deliberately. ## 3. Optimization level must match the producer's, or verification collapses A `.pyc` built with `-O` or `-OO` is a different code object. Measured on 679 sources compiled by a foreign interpreter at each level and graded at each level: | producer ↓ / grader → | 0 | 1 | 2 | |---|---|---|---| | **0** | **100.0%** | 95.43% | 23.86% | | **1** | 95.43% | **100.0%** | 24.15% | | **2** | 23.86% | 24.15% | **100.0%** | The diagonal is 679/679 at every level. Guessing wrong is **not** a graceful degradation — it collapses to ~24%. There are only three levels and trying all three costs three compiles, so this is *"needs normalisation to survive"*, not *"breaks"*. **The harness does not currently search the three levels automatically; a user verifying a foreign `.pyc` must do it.** **What trying all three costs you, and it is not nothing.** The certificate changes meaning from *"byte-identical to the code object the original source compiles to"* to *"byte-identical to the code object that was actually shipped"*. At `optimize>=1` docstrings are **absent from the `.pyc`**, and at `>=2` asserts are gone too. So against an `-O` artifact, **docstring recovery cannot be certified at all** — the information is not in the file. That bears directly on our headline differentiator: 115 of 679 benchmark rows carry a real docstring, and none of that could be proven against an `-O` `.pyc`. It is a limit of the artifact, not unsoundness in the oracle. ## 4. What the oracle deliberately ignores Excluded from the fingerprint, with how often each would have caused a false reject on the 600 wild rows had it been included: | Field | In fingerprint? | Rows differing / 600 | |---|---|---| | `co_filename` | excluded | **600 (100%)** — every wild `.pyc` carries its builder's absolute path | | `co_linetable` | excluded | **122 (20.3%)** — varies across builds with no semantic content | | `co_firstlineno` | excluded | 0 | Docstrings, `co_consts`, `co_names` and `co_exceptiontable` are **included** — the first because docstring fidelity is a claim we make, the last because omitting it once produced a false proof. ## 5. Unverified means unknown, not wrong The oracle is **sound but incomplete**: ``` verified = PROVABLY correct. Identical code object => identical behaviour. No false positives. unverified = UNKNOWN. A correct decompilation that compiles differently — a `while` where the original had a `for`, a differently-ordered but equivalent boolean — does not verify. ``` Reported accuracy is therefore a **lower bound on correctness**, not an estimate of it. Treating the unverified remainder as errors understates the model; treating it as correct is unsafe. ## 6. Not tested — unknown, not claimed - **Cross-minor (3.13).** No 3.13 interpreter on the measurement box; nothing was downloaded. The benchmark and the model are 3.12 only. - **PyInstaller / Nuitka containers.** `import PyInstaller` → `ModuleNotFoundError`. Not measured. - **`.pyc` from non-CPython or patched builds.** Not measured. Given that a *patch release* already produces the 0.33% floor, a patched build is a live risk, not a theoretical one. - **Obfuscated or deliberately adversarial bytecode.** Not measured. No malware was fetched. ## 7. Where these limits are stated to users | Limit | Stated in | |---|---| | Pre-flight is trivial | this file; `EVAL.md`; `harness/README.md`; both benchmark data cards | | 0.33% wild false-reject floor | this file; `EVAL.md`; `weights/MODEL-CARD.md` | | `-O` mismatch collapse, docstrings unprovable | this file; `EVAL.md`; `weights/MODEL-CARD.md` | | unverified ≠ wrong | this file; `EVAL.md`; `weights/MODEL-CARD.md`; `harness/README.md` | | 3.13 / PyInstaller untested | this file; `EVAL.md`; `weights/MODEL-CARD.md` | | **19.4% formatting ceiling (§8)** | **this file** | ## 8. The formatting ceiling: 19.4% of real modules cannot certify a re-formatted answer This section was added on 2026-09-11. It discloses a ceiling that was measured earlier and was not previously written down here, which left §7's promise of a complete accounting unmet. **`ast.unparse` is not bytecode-preserving.** Compiling a source file, and compiling that same file after an AST round trip, do not always produce the same code object: ```python compile(ast.unparse(ast.parse(src))) != compile(src) # for 100 of 516 stdlib modules ``` Measured on all 516 CPython 3.12 stdlib modules: **100 fail the round trip — an 80.6% pass rate, so 19.4% of real modules contain at least one affected construct.** **The cause**, reduced to a repro verified on the measurement box: ```python a = {name for name, value in ns.items() if getattr(value, 'x', False)} # POP_JUMP_IF_FALSE a = {name # POP_JUMP_IF_TRUE for name, value in ns.items() # + JUMP_BACKWARD if getattr(value, 'x', False)} ``` Identical AST, different bytecode. CPython 3.12's CFG optimiser lays out basic blocks according to the **physical line layout** of a comprehension. An `if` *statement* wrapped the same way does not change; comprehensions do. **What this means for anyone reading a score from this oracle.** - It is a ceiling on the **oracle**, not a defect in any model. A decompilation that is semantically perfect but wraps a comprehension across lines differently from the original will **not certify**. It is scored as unverified, which per §5 means *unknown*, not *wrong*. - It applies to **every system measured against this oracle** — ours, PyLingual's, and any other. It is not a differential advantage or disadvantage to anybody. - It is **whole-module scale**: 19.4% is the fraction of *modules* containing at least one affected construct, not the fraction of constructs affected. Function- and class-level units are affected at a lower rate, because the chance of containing a comprehension is lower. - **It compounds with §2's 0.33% floor and §3's `-O` collapse.** These are independent sources of false rejection; none of them ever produces a false accept. **Why the numbers already published are not invalidated by this disclosure.** Every published score was produced by grading a model's output against a reference *source*, never against an `ast.unparse` round trip, so no published figure was computed through the lossy path. This section discloses a bound on how high any such score could ever go; it does not move one. It also rules out a class of pipeline we did not build: an AST-splice reassembler would have discarded roughly one module in five before asking the model a single question. **Measured, and stated as such.** 100/516 is a direct count over the CPython 3.12 stdlib on the measurement box. The corresponding ceiling on the internal 522-row module-shaped eval set — 92.5% overall, 88.3% on rows of ≥400 representation lines — is measured on a different set and is reported with those internal results rather than here.