Upload README.md with huggingface_hub

README.md

@@ -17,97 +17,16 @@ tags:
 
 # PyBytecode v3 — 1.5B
 
-**A 1.5B open model that decompiles Python 3.12 bytecode back to source, and ships with a sound
-verifier that proves when an output is correct.**
+Turns Python 3.12 bytecode back into Python source. Hand it a disassembled code object, get source
+code back.
 
-Hand it a disassembled code object, get Python back. Then run the verifier: it recompiles the
-prediction and compares the resulting code object against the target, byte for byte. If they
-match, the decompilation is **proven** correct — not scored, not ranked, proven.
-
-That verifier is the distinguishing property of this release. Most generative models ask you to
-trust an aggregate accuracy. This one lets you check the single file in front of you.
-
-```
-verified   = recompile(prediction) yields a code object byte-identical to the target's
-           = PROVABLY correct. Sound: there is no such thing as a false "verified".
-unverified = UNKNOWN, not wrong. A correct decompilation that compiles differently
-             (a `while` where the original had a `for`) does not certify.
-```
-
-Three consequences worth being explicit about:
-
-- **Reported accuracy is a lower bound on correctness.** Everything counted here is proven; some
-  of what is not counted is right too.
-- **Best-of-N buys real accuracy, not a better guess.** Sample until something certifies, then
-  stop. The verifier decides when to stop, so extra budget cannot mislead you.
-- **The check is exact.** Comparison runs over the real code object, recursively, including
-  docstrings and `co_exceptiontable` — not over a textual disassembly. That is not decoration: an
-  earlier representation omitted the exception table's `end`, which made a `try:` body and a
-  `try/else:` body indistinguishable, and it once certified a wrong prediction.
+The unusual part: **you can check every answer.** Recompile what the model wrote and compare it
+against the bytecode you started with — if they match, that file is exactly right, and you know it
+without trusting an accuracy number.
 
 Weights are Apache-2.0. A GGUF build ships alongside for llama.cpp / LM Studio / Ollama.
 
----
-
-## How well it works
-
-Strict L1 oracle throughout. Every figure carries a 95% interval and names the set it was measured
-on.
-
-### On the published benchmark — `csn-3.12-licensed`, n=600 across 117 repositories
-
-| | certified | 95% CI |
-|---|---|---|
-| **PyBytecode v3, greedy** | **506 / 600 = 84.33%** | [80.48, 87.94] |
-| **PyBytecode v3, verified best-of-32** | **562 / 600 = 93.67%** | [90.86, 96.08] |
-| Untuned `Qwen2.5-Coder-1.5B-Instruct`, greedy | **4 / 600 = 0.67%** | [0.16, 1.35] |
-
-Intervals are **repo-clustered** (bootstrap over repositories, 10,000 resamples), because rows
-from one repository share an author and a house style and are not independent. Design effects
-1.65 / 1.79 / 0.84.
-
-The benchmark ships with the model, licence-resolved per row at the exact commit, so these numbers
-are recomputable from files in this repository rather than taken on faith.
-
-### The untuned base scores ~0, which is what makes the benchmark trustworthy
-
-The same base model, given the same prompt, the same decoding budget and the same oracle, certifies
-**4 of 600** here and **0 of 400** on our earlier benchmark. Under looser oracles it is clearer
-still: the base produces **syntactically valid Python on 52% of rows** and reaches AST-identity on
-**0%**. It paraphrases the disassembly into plausible-looking code with invented helpers rather
-than inverting it.
-
-A strong general code model cannot guess its way to a single point on this task. The benchmark is
-not solvable by pattern-matching, and everything v3 scores was learned from the fine-tune.
-
-### The benchmark rebuild reproduced the result
-
-The published 600-row set was built from scratch: different rows (only 85 of 600 appear in the
-earlier 400-row set), 117 repositories instead of 24, and a 1% per-repo cap replacing a 15%
-concentration. It lands in the same place — certified@32 **93.67%** against **93.25%**, greedy
-**84.33%** against **83.75%**. The difference of **+0.42pp** carries a 95% interval of
-[−4.38, +5.22], so it is well inside noise, and the two size profiles are near-identical (median
-59 vs 62 rep lines; 93.3% vs 94.0% of rows under 200), which is the composition factor that would
-otherwise move the number. Standardised to the older set's size mix the new set reads 94.48%.
-A result that survives a rebuild of the set it was measured on is worth more than the same result
-quoted twice.
-
-### Which numbers came from where
-
-| Set | Published? | Why |
-|---|---|---|
-| `csn-3.12-licensed` (600) | **yes** | every row licence-resolved at its commit |
-| `pybytecode-mbpp-3.12` (383) | **yes** | MBPP, CC-BY-4.0 |
-| CSN-3.12 (400) | no | 45% of its rows are not redistributable |
-| Held-out (279) | no | per-row attribution was not retained |
-
-Our own results are measured on both the published set and the earlier pair. The figures in
-*Comparison with other systems* exist only on the earlier pair, because that is where those runs
-were done; they are labelled there because you cannot re-run them from what we shipped.
-
----
-
-## How to use it well
+## Quickstart
 
 ```python
 from transformers import AutoModelForCausalLM, AutoTokenizer
@@ -128,7 +47,13 @@ prediction = tok.decode(model.generate(**batch, max_new_tokens=2048,
                         skip_special_tokens=True)
 ```
 
-**Then certify it** — this is the step that matters:
+Greedy decoding (temperature 0) for a single shot; temperature ~0.8 when you sample several
+candidates.
+
+## Checking the answer
+
+Compile the model's output and compare the resulting code object to the one you were decompiling.
+Same code object means same behaviour, so a match tells you *this* answer is correct:
 
 ```python
 from harness.pybytecode_core.verify import code_fingerprint
@@ -138,233 +63,82 @@ def certified(prediction: str, reference_code_object) -> bool:
     return code_fingerprint(got) == code_fingerprint(reference_code_object)
 ```
 
-Greedy decoding (temperature 0) for single-shot; temperature ~0.8 when sampling for best-of-N.
-
-### Measure your input size first
+Two things follow. A failed check means "not confirmed", not "wrong" — a correct rewrite that
+compiles differently (a `while` where the original had a `for`) won't match, so the accuracy
+figures below are a floor, not an estimate. And because the check is cheap and reliable, sampling
+several answers and keeping the first one that passes is a real gain rather than a nicer guess.
 
-**This model works on function-sized units and degrades sharply past ~200 representation lines.**
-That is the single most useful thing to know before you start, and it costs one line to check:
+## Results
 
-```python
-from harness.pybytecode_core.rep import disassemble_v2
-rep_lines = disassemble_v2(code_object).count("\n")   # the model's actual input length
-```
+On the benchmark published with this model:
 
-| your input | what to expect |
+| | certified |
 |---|---|
-| **under 100 rep lines** | the model's home ground — ~89–96% greedy, ~97–99% at best-of-32 |
-| **100–200** | still strong, but sampling starts to earn its cost |
-| **200–300** | roughly a coin flip greedy; best-of-32 recovers much of it |
-| **over ~300** | greedy rarely certifies; best-of-32 recovers some. Expect misses |
-| **over ~400** | we certified nothing here, greedy **or** at 32 samples |
-
-Above the knee a symbolic decompiler is the better tool, and the measurements are in *Comparison
-with other systems* below.
-
-The full curve, pooled over both earlier benchmarks (n=679) so the thin upper buckets carry as
-many rows as we can give them:
-
-| rep lines | rows | v3 greedy | v3 best-of-32 | untuned base | PyLingual k=32 |
-|---|---|---|---|---|---|
-| 0–49 | 311 | 94.21% | 98.71% | 0.96% | 99.04% |
-| 50–99 | 229 | 88.65% | 96.94% | 0.00% | 95.63% |
-| 100–199 | 105 | 78.10% | 88.57% | 0.00% | 84.76% |
-| 200–299 | 19 | 47.37% | 84.21% | 0.00% | 84.21% |
-| 300–399 | 11 | 18.18% | 54.55% | 0.00% | 81.82% |
-| 400–599 | 2 | 0.00% | 50.00% | 0.00% | 100.00% |
-| 600+ | 2 | 0.00% | 0.00% | 0.00% | 50.00% |
-
-- **The greedy knee is at ~200 rep lines**: 78.1% → 47.4% → 18.2% → 0% across four consecutive
-  buckets.
-- **Best-of-32 postpones the knee to ~300; it does not remove it.** Sampling 32 times buys roughly
-  one bucket of headroom.
-- Almost every point we score comes from small units. On the earlier CSN-3.12 set (n=400),
-  **97.61%** of greedy certifications and **95.98%** of best-of-32 certifications are units under
-  200 rep lines; on the published 600-row set it is **96.64%** of greedy certifications. The
-  headline accuracy is a statement about small units.
-- The top two buckets are n=2 each and carry nothing on their own. What carries the conclusion is
-  the monotone decline through n=311/229/105/19/11 below them.
-
-The same stratification on the **published** 600-row benchmark — recomputable from the files in
-this repo with `harness/size_curve.py` — reproduces the shape:
-
-| rep lines | rows | v3 greedy | 95% CI | v3 best-of-32 | 95% CI |
-|---|---|---|---|---|---|
-| 0–49 | 224 | 95.98% | [93.01, 98.51] | 98.21% | [96.31, 99.58] |
-| 50–99 | 224 | 89.73% | [85.17, 93.93] | 98.21% | [96.26, 99.57] |
-| 100–199 | 112 | 65.18% | [55.36, 74.14] | 85.71% | [77.57, 92.98] |
-| 200–299 | 27 | 51.85% | — | 77.78% | — |
-| 300–399 | 5 | 60.00% | — | 100.00% | — |
-| 400–599 | 5 | 0.00% | — | 0.00% | — |
-| 600+ | 3 | 0.00% | — | 0.00% | — |
-
-Intervals are repo-clustered, and omitted below 30 rows / 10 repositories rather than printed at a
-width that would imply precision we do not have. The 300–399 bucket reading above the one below it
-is n=5 noise, not a recovery. Median input in this benchmark is **59** rep lines; p90 is 167, max
-1,622.
-
-On this set the top two buckets stay at **0% even with 32 samples** — above ~400 rep lines,
-sampling did not rescue a single one of the 8 rows. **96.64%** of greedy certifications and
-**95.37%** of best-of-32 certifications come from units under 200 rep lines.
-
-### Decide how much budget to spend
-
-The best-of-32 budget is **1 greedy decode + 31 sampled candidates at temperature 0.8**, with
-verified early stop — sampling halts on the first candidate that certifies, so the mean cost is
-far below 32. Across the 94 greedy failures it took a mean of 15.16 samples and recovered 56:
-
-| budget | certified | |
-|---|---|---|
-| @1 (greedy alone) | 506 / 600 | 84.33% |
-| @2 | 525 / 600 | 87.50% |
-| @4 | 546 / 600 | 91.00% |
-| @8 | 555 / 600 | 92.50% |
-| @16 | 558 / 600 | 93.00% |
-| @32 | 562 / 600 | 93.67% |
-
-Most of the recovery arrives in the first few samples: @4 already captures two thirds of what @32
-gets. Because early stop depends only on the index of the first passing sample, this curve is
-exact rather than an estimate — the same cached generations give every point.
-
----
-
-## What this model does not do
-
-- **Large units — this is the measured limit.** Certification falls from 94% under 50
-  representation lines to 18% at 300–399 and 0% above 400. See *Measure your input size*: the
-  curve is the honest specification of what this model does. It was trained on functions; a whole
-  module is a different regime in both size and shape, and we have not evaluated one.
-- **Python 3.12 only.** A scope statement, not the explanation for the size behaviour above:
-  trained and measured on 3.12, and the oracle refuses cross-minor input by design. Every number
-  on this card is in-version, so nothing here is contaminated by a version mismatch.
-- **It did not decompile real malware.** On the one wild PyInstaller-packed sample in scope, the
-  entry-point module produced **nothing certifiable** — the prediction did not even compile. 3 of
-  9 units in that sample certified, and those three were PyInstaller's own bootstrap plus a
-  bundled stdlib module. Extraction and representation work (1,510/1,510 code objects recovered);
-  end-to-end decompilation of real malware logic is **not demonstrated**.
-
-  That entry point was both 3.13 *and* 491 representation lines, so **size and version are
-  confounded in it and it is evidence for neither** on its own. It is reported because you should
-  know the result, not because it diagnoses anything. The benign in-version curve above is the
-  instrument, and it already shows the model failing at that size on clean 3.12 input.
-- **Annotated functions** are handicapped: the training corpus was compiled with PEP-563
-  stringised annotations inherited, real `.pyc` files are not, and the benchmarks deliberately do
-  not hand the model its training-time distribution back.
-
-## The oracle's limits, where you will meet them
-
-Full detail in [`ORACLE-LIMITS.md`](ORACLE-LIMITS.md).
-
-- **0.33% false-reject floor on foreign `.pyc`.** On 600 wild install-time `.pyc`, 598 certify;
-  2 do not, because CPython 3.12.3 and 3.12.13 generate different code for the same source. No
-  normalisation fixes that without abandoning the byte-identical guarantee. It fails to a **false
-  reject** — you are told "unknown" about a correct answer, never "verified" about a wrong one.
-- **Optimization level must match the producer's.** A `.pyc` built with `-O` is a different code
-  object. The wrong level does not degrade gracefully, it collapses to ~24%. Try all three; it
-  costs three compiles. But note that at `optimize>=1` docstrings are absent from the `.pyc`, so
-  **docstring recovery becomes unprovable** against such a file — a limit of the artifact, not of
-  the oracle.
-- **The 100% pre-flight the harness prints proves nothing about soundness.** It grades each
-  reference against itself, i.e. compares `compile(x)` with `compile(x)`, so any deterministic
-  function of the source scores 100% — including a stub that ignores the bytecode entirely. It
-  detects a broken harness and nothing more. The soundness evidence is the mutation test (0 true
-  survivors in 1,239 mutants) and 18 targeted blind-spot probes (18/18).
-- **Untested:** cross-minor 3.13, Nuitka, non-CPython builds, obfuscated bytecode.
-
-## Reproduce every number on this card
-
-No model, no GPU, no network, no third-party decompiler — CPython 3.12 standard library only:
-
-```bash
-cd harness
-python3 grade.py --bench ../benchmarks/csn-3.12-licensed/bench.jsonl --self-test-only --out /tmp/st.json
-python3 analyze_scores.py --bench ../benchmarks/csn-3.12-licensed/bench.jsonl \
-    --greedy ../generations/gen_v3_csn600.jsonl \
-    --base   ../generations/gen_base_csn600.jsonl \
-    --out /tmp/scores.json --rows-out /tmp/rows.jsonl
-python3 size_curve.py --bench ../benchmarks/csn-3.12-licensed/bench.jsonl \
-    --greedy ../generations/gen_v3_csn600.jsonl \
-    --base   ../generations/gen_base_csn600.jsonl \
-    --out /tmp/size_curve.json
-```
-
-Per-row verdicts for all 600 rows — repo, function, commit SHA, SPDX, tuned verdict, base verdict
-— land in `rows.jsonl`. Every aggregate above is recomputable from that file.
-
-## Comparison with other systems
+| **PyBytecode v3, one attempt** | **506 / 600 = 84.33%** |
+| **PyBytecode v3, up to 32 tries** | **562 / 600 = 93.67%** |
+| `Qwen2.5-Coder-1.5B-Instruct` before fine-tuning, one attempt | 4 / 600 = 0.67% |
 
-[PyLingual](https://github.com/syssec-utd/pylingual) is another system that performs this task, by
-symbolic reconstruction rather than generation. We ran it on the same sets, under the same oracle,
-at a matched budget.
+The benchmark is `csn-3.12-licensed`: 600 real functions from 117 GitHub repositories, compiled to
+3.12 bytecode, shipped with the model. The base model before fine-tuning gets essentially none of
+them, so this is not something a general code model can guess its way through.
 
-| CSN-3.12, n=400, 24 repos | certified | 95% CI (repo-clustered) |
-|---|---|---|
-| untuned base, greedy | 0 / 400 = 0.00% | — |
-| v3 greedy | 335 / 400 = 83.75% | [77.84, 88.89] |
-| v3 certified@32 | 373 / 400 = 93.25% | [88.82, 96.89] |
-| PyLingual k=32 | 377 / 400 = 94.25% | [91.06, 96.80] |
-| **union** | **393 / 400 = 98.25%** | [96.81, 99.51] |
+Full numbers, confidence intervals, method, per-budget curve and the comparison with other
+systems: [`EVAL.md`](EVAL.md).
 
-| Held-out, n=279 | certified | 95% CI (Wilson) |
-|---|---|---|
-| untuned base, greedy | 3 / 279 = 1.08% | [0.37, 3.11] |
-| v3 greedy | 254 / 279 = 91.04% | [87.11, 93.86] |
-| v3 certified@32 | 272 / 279 = 97.49% | [94.91, 98.78] |
-| PyLingual k=32 | 267 / 279 = 95.70% | [92.63, 97.52] |
-| **union** | **275 / 279 = 98.57%** | [96.37, 99.44] |
+[PyLingual](https://github.com/syssec-utd/pylingual) is another system that does this task, by
+symbolic reconstruction rather than generation. On our earlier benchmarks it scores about the same
+as we do, and the two miss on different inputs — so running both and keeping whichever answer
+passes the check gets you more than either alone. Numbers in [`EVAL.md`](EVAL.md).
 
-**Neither difference is statistically significant** (exact paired McNemar, p = 0.6177 on CSN and
-p = 0.2266 on held-out). We claim no accuracy advantage, and none is claimed against us. On cost
-the comparison runs against us: ~1.5B parameters against their ~320M, and slower wall-clock.
+## When it works well, and when it doesn't
 
-Docstring recovery on the 115 held-out rows carrying a real docstring: **115 / 115 = 100%**
-(PyLingual: 113/115 = 98.26%, under an oracle that does not score docstrings at all).
+**It is good on individual functions and gets much worse on long ones.** Size is measured in
+*disassembly lines* — how long the input you hand the model is. One line tells you:
 
-The held-out interval is **Wilson, not clustered**: every row of that set carries the same
-placeholder repository value, so its clustering cannot be assessed. The CSN design effects of
-2.34–2.69 show why that matters — on a set where one repository supplied 15% of rows, a naive
-interval would have been far too narrow.
-
-These figures were measured on our earlier benchmark pair, which is **not** published. PyLingual
-has not been run on the 600-row licensed benchmark released alongside this model.
-
-### The two approaches fail on different inputs
-
-The more interesting measured result is not which number is larger, but that the two systems lose
-on different inputs, along an axis that turns out to be **unit size**:
-
-| rep lines | rows | PyBytecode greedy | PyBytecode @32 | PyLingual k=32 |
-|---|---|---|---|---|
-| 100–199 | 105 | 78.10% | 88.57% | 84.76% |
-| 200–299 | 19 | 47.37% | 84.21% | 84.21% |
-| 300–399 | 11 | **18.18%** | 54.55% | **81.82%** |
-| 400–599 | 2 | **0.00%** | 50.00% | **100.0%** |
-
-A generative model degrades as its input grows; a symbolic reconstruction does not. Two things
-follow, both useful:
-
-- **Large units are not intrinsically unsolvable.** A symbolic system handles them at 81.82% where
-  we score 18.18%. What our curve measures is a capability limit of a 1.5B model, not a property
-  of the task — which is a statement about where to spend the next round of scale.
-- **The union exceeds either system**, 98.25% and 98.57% above. Because verification is sound, the
-  union costs nothing to take: run either, keep the answer that certifies, fall through to the
-  other. Neither system can certify a wrong answer, so combining them cannot mislead you.
-
-## Training
+```python
+from harness.pybytecode_core.rep import disassemble_v2
+rep_lines = disassemble_v2(code_object).count("\n")
+```
 
-| | |
-|---|---|
-| Base | [`Qwen/Qwen2.5-Coder-1.5B-Instruct`](https://huggingface.co/Qwen/Qwen2.5-Coder-1.5B-Instruct) (Apache-2.0) |
-| Method | LoRA r=16, α=32, all attention + MLP projections, 1 epoch, lr 2e-4, merged into the base |
-| Rows | 48,196 pairs of Python 3.12 disassembly → source |
-| Corpus | `codeparrot/github-code-clean`, per-row filtered to 7 permissive licences; all GPL/LGPL/AGPL/MPL/EPL dropped before extraction |
-| Decontamination | shard-disjoint, repo-disjoint, and identifier-blind fingerprint-disjoint |
-
-**The training corpus is not distributed.** Per-row attribution was not retained during
-extraction, so the corpus cannot be redistributed without dropping required MIT/BSD/Apache
-notices. That is a property of the artifact, not of the licences — nothing in it is unlicensed or
-reciprocally licensed. Full lineage, including what was dropped and why:
-[`DATA-CARD-training-corpus.md`](DATA-CARD-training-corpus.md).
+| disassembly lines | rows | one attempt | up to 32 tries |
+|---|---|---|---|
+| under 100 | 448 | 92.86% | 98.21% |
+| 100–199 | 112 | 65.18% | 85.71% |
+| 200–399 | 32 | 53.12% | 81.25% |
+| 400+ | 8 | 0.00% | 0.00% |
+
+Below ~100 lines it is on home ground. Accuracy starts dropping around 200, and above ~400 lines
+nothing certified at all, even with 32 tries. Sampling more buys roughly one bucket of headroom;
+it does not remove the limit. For big units, a symbolic decompiler is the better tool. The full
+seven-bucket curve is in [`EVAL.md`](EVAL.md).
+
+## Limits
+
+- **Long inputs.** The table above is the honest specification: trained on functions, not modules,
+  and it fails above ~400 disassembly lines.
+- **Python 3.12 only.** Trained and measured on 3.12; the checker refuses other minor versions by
+  design.
+- **If the `.pyc` was built with `-O`, compile at the same level or the check will not match.**
+  Wrong level collapses to ~24%, so try all three — it costs three compiles. At `-O` and above,
+  docstrings aren't in the `.pyc` at all, so docstring recovery can't be confirmed against one.
+- **A `.pyc` built by someone else can fail the check even when the answer is right** — about
+  0.33% of the time, because CPython patch releases compile the same source differently. It always
+  fails in the safe direction: "unknown" about a correct answer, never "confirmed" about a wrong
+  one.
+- **It has not been shown to work on real malware.** On the one packed sample we tried, the
+  entry-point module produced nothing certifiable. Extraction worked; decompiling the actual
+  malware logic did not.
+- Untested: Python 3.13, Nuitka, non-CPython builds, obfuscated bytecode.
+
+Details on all of these in [`EVAL.md`](EVAL.md) and [`ORACLE-LIMITS.md`](ORACLE-LIMITS.md).
+
+## Model details
+
+Fine-tuned from [`Qwen/Qwen2.5-Coder-1.5B-Instruct`](https://huggingface.co/Qwen/Qwen2.5-Coder-1.5B-Instruct)
+(Apache-2.0) with LoRA on 48,196 pairs of Python 3.12 disassembly → source, adapter merged. The
+corpus was filtered to permissive licences before training and is not redistributed — per-row
+attribution was not retained, so shipping it would strip required notices. Lineage:
+[`DATA-CARD-training-corpus.md`](DATA-CARD-training-corpus.md). Training settings: [`EVAL.md`](EVAL.md).
 
 ## Licence
 
@@ -372,18 +146,11 @@ reciprocally licensed. Full lineage, including what was dropped and why:
 
 Derived from `Qwen/Qwen2.5-Coder-1.5B-Instruct`, which is Apache-2.0. Under Apache-2.0 §4 we ship
 the licence, retain attribution, and state our changes (LoRA fine-tune, adapter merged; no
-architecture, vocabulary or tokenizer change). The upstream repository ships a `LICENSE` and no
-`NOTICE`, so there is no upstream notice text to carry forward. If you redistribute these weights
-or build derivatives, the same four obligations pass to you.
-
-The corpus constraint above binds the *corpus*, not these weights: trained parameters are not a
-copy of any source text, and the corpus was filtered to permissive licences before training, so no
-reciprocal terms exist upstream to propagate.
+architecture, vocabulary or tokenizer change). The same obligations pass to you if you
+redistribute these weights or build derivatives.
 
 Decompilation has obvious dual use. Apache-2.0 imposes no field-of-use restriction and we have not
-added one: a restrictive weights licence would not prevent misuse and would block the
-interoperability, incident-response and recovery work this is for. Complying with the law where
-you operate is your responsibility.
+added one. Complying with the law where you operate is your responsibility.
 
 ## Citation