The oracle's real limits
The verifier is the reason to use PyBytecode at all, so its limits belong in front of a user, not in an appendix. Everything here is measured; sources are named per section.
1. The pre-flight 100% proves almost nothing. Read this before quoting it.
Every grading command prints PRE-FLIGHT 600/600 = 100% before it scores. That number is
trivial by construction and is not evidence of soundness.
Pre-flight grades each reference label against itself. The oracle asks whether
compile(prediction) and compile(reference) produce the same code object — so at pre-flight it
is comparing compile(x) with compile(x). It would return 100% for any deterministic
function of the source, including a stub that hashes the input string and ignores the bytecode
entirely.
What pre-flight actually detects is a broken harness: a benchmark whose .pyc files do not
match their sources, a Python version mismatch (3.11 or 3.13 against a 3.12 benchmark), a corrupt
row. Those are real failure modes and worth catching, which is why it runs. But a passing
pre-flight says the instrument is plugged in, not that it measures anything.
Soundness evidence comes from the mutation test and the blind-spot probes, not from
pre-flight: corrupt a label and require the oracle to reject it. Measured
(evidence/ORACLE-MUTATION.md): 0 true survivors in 1,239 mutants, and
18/18 targeted blind-spot probes behave as required — including the historical failure where
a try: body and a try/else: body were indistinguishable, docstring changes, docstring
removal, float-vs-int, bool-vs-int and -0.0 vs 0.0.
Even the mutation kill rate is weak evidence on its own: for a byte-identical oracle a kill is close to tautological, since a mutant survives only if it compiles to a structurally identical code object. The probes are the load-bearing test, because they ask the question that actually bit us once — is a behaviourally load-bearing field missing from the fingerprint?
Caveat on mutation supply, stated rather than hidden: 188 of 600 wild rows (31%) produced no effective mutant within 30 tries, and 323 void attempts were discarded. The wild kill rate is measured on the 412 rows that did produce one.
2. The 0.33% wild false-reject floor
Against .pyc files built by someone else, the oracle refuses a small fraction of correct
answers. Measured on 600 wild install-time .pyc from installed site-packages
(evidence/GATE-RESULT.md):
| certified | false reject | false accepts | |
|---|---|---|---|
| L0 (old constant encoding) | 585/600 = 97.5% | 15 = 2.5% | 0 / 1,274 |
| L1 (shipping) | 598/600 = 99.67% | 2 = 0.33% | 0 / 1,274 |
13 of the 15 L0 failures were our own defect — repr() of a set/frozenset/dict follows the
compiling process's hash seed, which also made the L0 verdict non-deterministic (585 / 592 /
584 / 585 / 589 under PYTHONHASHSEED 0–4). L1 fixes it and returns 598 under all five seeds.
The remaining 0.33% is a real floor and is not fixable. One distinct module
(pandas/_testing/__init__.py) compiles differently under CPython 3.12.3 than under 3.12.13 —
co_code 2,692 vs 2,696 bytes, and a differing co_exceptiontable. The source is correct; the
compiler patch release differs. No normalisation removes this without abandoning the
byte-identical guarantee.
It degrades to a false REJECT, never a false accept. You are told "unknown" about a correct answer; you are never told "verified" about a wrong one. That is the safe direction, and it is the direction the design chose deliberately.
3. Optimization level must match the producer's, or verification collapses
A .pyc built with -O or -OO is a different code object. Measured on 679 sources compiled by
a foreign interpreter at each level and graded at each level:
| producer ↓ / grader → | 0 | 1 | 2 |
|---|---|---|---|
| 0 | 100.0% | 95.43% | 23.86% |
| 1 | 95.43% | 100.0% | 24.15% |
| 2 | 23.86% | 24.15% | 100.0% |
The diagonal is 679/679 at every level. Guessing wrong is not a graceful degradation — it
collapses to ~24%. There are only three levels and trying all three costs three compiles, so this
is "needs normalisation to survive", not "breaks". The harness does not currently search the
three levels automatically; a user verifying a foreign .pyc must do it.
What trying all three costs you, and it is not nothing. The certificate changes meaning from
"byte-identical to the code object the original source compiles to" to "byte-identical to the
code object that was actually shipped". At optimize>=1 docstrings are absent from the .pyc,
and at >=2 asserts are gone too. So against an -O artifact, docstring recovery cannot be
certified at all — the information is not in the file. That bears directly on our headline
differentiator: 115 of 679 benchmark rows carry a real docstring, and none of that could be proven
against an -O .pyc. It is a limit of the artifact, not unsoundness in the oracle.
4. What the oracle deliberately ignores
Excluded from the fingerprint, with how often each would have caused a false reject on the 600 wild rows had it been included:
| Field | In fingerprint? | Rows differing / 600 |
|---|---|---|
co_filename |
excluded | 600 (100%) — every wild .pyc carries its builder's absolute path |
co_linetable |
excluded | 122 (20.3%) — varies across builds with no semantic content |
co_firstlineno |
excluded | 0 |
Docstrings, co_consts, co_names and co_exceptiontable are included — the first because
docstring fidelity is a claim we make, the last because omitting it once produced a false proof.
5. Unverified means unknown, not wrong
The oracle is sound but incomplete:
verified = PROVABLY correct. Identical code object => identical behaviour. No false positives.
unverified = UNKNOWN. A correct decompilation that compiles differently — a `while` where the
original had a `for`, a differently-ordered but equivalent boolean — does not verify.
Reported accuracy is therefore a lower bound on correctness, not an estimate of it. Treating the unverified remainder as errors understates the model; treating it as correct is unsafe.
6. Not tested — unknown, not claimed
- Cross-minor (3.13). No 3.13 interpreter on the measurement box; nothing was downloaded. The benchmark and the model are 3.12 only.
- PyInstaller / Nuitka containers.
import PyInstaller→ModuleNotFoundError. Not measured. .pycfrom non-CPython or patched builds. Not measured. Given that a patch release already produces the 0.33% floor, a patched build is a live risk, not a theoretical one.- Obfuscated or deliberately adversarial bytecode. Not measured. No malware was fetched.
7. Where these limits are stated to users
| Limit | Stated in |
|---|---|
| Pre-flight is trivial | this file; EVAL.md; harness/README.md; both benchmark data cards |
| 0.33% wild false-reject floor | this file; EVAL.md; weights/MODEL-CARD.md |
-O mismatch collapse, docstrings unprovable |
this file; EVAL.md; weights/MODEL-CARD.md |
| unverified ≠ wrong | this file; EVAL.md; weights/MODEL-CARD.md; harness/README.md |
| 3.13 / PyInstaller untested | this file; EVAL.md; weights/MODEL-CARD.md |
| 19.4% formatting ceiling (§8) | this file |
8. The formatting ceiling: 19.4% of real modules cannot certify a re-formatted answer
This section was added on 2026-09-11. It discloses a ceiling that was measured earlier and was not previously written down here, which left §7's promise of a complete accounting unmet.
ast.unparse is not bytecode-preserving. Compiling a source file, and compiling that same
file after an AST round trip, do not always produce the same code object:
compile(ast.unparse(ast.parse(src))) != compile(src) # for 100 of 516 stdlib modules
Measured on all 516 CPython 3.12 stdlib modules: 100 fail the round trip — an 80.6% pass rate, so 19.4% of real modules contain at least one affected construct.
The cause, reduced to a repro verified on the measurement box:
a = {name for name, value in ns.items() if getattr(value, 'x', False)} # POP_JUMP_IF_FALSE
a = {name # POP_JUMP_IF_TRUE
for name, value in ns.items() # + JUMP_BACKWARD
if getattr(value, 'x', False)}
Identical AST, different bytecode. CPython 3.12's CFG optimiser lays out basic blocks according to
the physical line layout of a comprehension. An if statement wrapped the same way does not
change; comprehensions do.
What this means for anyone reading a score from this oracle.
- It is a ceiling on the oracle, not a defect in any model. A decompilation that is semantically perfect but wraps a comprehension across lines differently from the original will not certify. It is scored as unverified, which per §5 means unknown, not wrong.
- It applies to every system measured against this oracle — ours, PyLingual's, and any other. It is not a differential advantage or disadvantage to anybody.
- It is whole-module scale: 19.4% is the fraction of modules containing at least one affected construct, not the fraction of constructs affected. Function- and class-level units are affected at a lower rate, because the chance of containing a comprehension is lower.
- It compounds with §2's 0.33% floor and §3's
-Ocollapse. These are independent sources of false rejection; none of them ever produces a false accept.
Why the numbers already published are not invalidated by this disclosure. Every published
score was produced by grading a model's output against a reference source, never against an
ast.unparse round trip, so no published figure was computed through the lossy path. This section
discloses a bound on how high any such score could ever go; it does not move one. It also rules
out a class of pipeline we did not build: an AST-splice reassembler would have discarded roughly
one module in five before asking the model a single question.
Measured, and stated as such. 100/516 is a direct count over the CPython 3.12 stdlib on the measurement box. The corresponding ceiling on the internal 522-row module-shaped eval set — 92.5% overall, 88.3% on rows of ≥400 representation lines — is measured on a different set and is reported with those internal results rather than here.