Add files using upload-large-folder tool

README.md

@@ -0,0 +1,398 @@
+---
+license: apache-2.0
+base_model: Qwen/Qwen2.5-Coder-1.5B-Instruct
+library_name: transformers
+pipeline_tag: text-generation
+language:
+  - en
+tags:
+  - decompilation
+  - reverse-engineering
+  - python
+  - bytecode
+  - code
+  - verified-generation
+  - qwen2
+---
+
+# PyBytecode v3 — 1.5B
+
+**A 1.5B open model that decompiles Python 3.12 bytecode back to source, and ships with a sound
+verifier that proves when an output is correct.**
+
+Hand it a disassembled code object, get Python back. Then run the verifier: it recompiles the
+prediction and compares the resulting code object against the target, byte for byte. If they
+match, the decompilation is **proven** correct — not scored, not ranked, proven.
+
+That verifier is the distinguishing property of this release. Most generative models ask you to
+trust an aggregate accuracy. This one lets you check the single file in front of you.
+
+```
+verified   = recompile(prediction) yields a code object byte-identical to the target's
+           = PROVABLY correct. Sound: there is no such thing as a false "verified".
+unverified = UNKNOWN, not wrong. A correct decompilation that compiles differently
+             (a `while` where the original had a `for`) does not certify.
+```
+
+Three consequences worth being explicit about:
+
+- **Reported accuracy is a lower bound on correctness.** Everything counted here is proven; some
+  of what is not counted is right too.
+- **Best-of-N buys real accuracy, not a better guess.** Sample until something certifies, then
+  stop. The verifier decides when to stop, so extra budget cannot mislead you.
+- **The check is exact.** Comparison runs over the real code object, recursively, including
+  docstrings and `co_exceptiontable` — not over a textual disassembly. That is not decoration: an
+  earlier representation omitted the exception table's `end`, which made a `try:` body and a
+  `try/else:` body indistinguishable, and it once certified a wrong prediction.
+
+Weights are Apache-2.0. A GGUF build ships alongside for llama.cpp / LM Studio / Ollama.
+
+---
+
+## How well it works
+
+Strict L1 oracle throughout. Every figure carries a 95% interval and names the set it was measured
+on.
+
+### On the published benchmark — `csn-3.12-licensed`, n=600 across 117 repositories
+
+| | certified | 95% CI |
+|---|---|---|
+| **PyBytecode v3, greedy** | **506 / 600 = 84.33%** | [80.48, 87.94] |
+| **PyBytecode v3, verified best-of-32** | **562 / 600 = 93.67%** | [90.86, 96.08] |
+| Untuned `Qwen2.5-Coder-1.5B-Instruct`, greedy | **4 / 600 = 0.67%** | [0.16, 1.35] |
+
+Intervals are **repo-clustered** (bootstrap over repositories, 10,000 resamples), because rows
+from one repository share an author and a house style and are not independent. Design effects
+1.65 / 1.79 / 0.84.
+
+The benchmark ships with the model, licence-resolved per row at the exact commit, so these numbers
+are recomputable from files in this repository rather than taken on faith.
+
+### The untuned base scores ~0, which is what makes the benchmark trustworthy
+
+The same base model, given the same prompt, the same decoding budget and the same oracle, certifies
+**4 of 600** here and **0 of 400** on our earlier benchmark. Under looser oracles it is clearer
+still: the base produces **syntactically valid Python on 52% of rows** and reaches AST-identity on
+**0%**. It paraphrases the disassembly into plausible-looking code with invented helpers rather
+than inverting it.
+
+A strong general code model cannot guess its way to a single point on this task. The benchmark is
+not solvable by pattern-matching, and everything v3 scores was learned from the fine-tune.
+
+### The benchmark rebuild reproduced the result
+
+The published 600-row set was built from scratch: different rows (only 85 of 600 appear in the
+earlier 400-row set), 117 repositories instead of 24, and a 1% per-repo cap replacing a 15%
+concentration. It lands in the same place — certified@32 **93.67%** against **93.25%**, greedy
+**84.33%** against **83.75%**. The difference of **+0.42pp** carries a 95% interval of
+[−4.38, +5.22], so it is well inside noise, and the two size profiles are near-identical (median
+59 vs 62 rep lines; 93.3% vs 94.0% of rows under 200), which is the composition factor that would
+otherwise move the number. Standardised to the older set's size mix the new set reads 94.48%.
+A result that survives a rebuild of the set it was measured on is worth more than the same result
+quoted twice.
+
+### Which numbers came from where
+
+| Set | Published? | Why |
+|---|---|---|
+| `csn-3.12-licensed` (600) | **yes** | every row licence-resolved at its commit |
+| `pybytecode-mbpp-3.12` (383) | **yes** | MBPP, CC-BY-4.0 |
+| CSN-3.12 (400) | no | 45% of its rows are not redistributable |
+| Held-out (279) | no | per-row attribution was not retained |
+
+Our own results are measured on both the published set and the earlier pair. The figures in
+*Comparison with other systems* exist only on the earlier pair, because that is where those runs
+were done; they are labelled there because you cannot re-run them from what we shipped.
+
+---
+
+## How to use it well
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+tok = AutoTokenizer.from_pretrained("BlazingCustoms/pybytecode-v3-1.5b")
+model = AutoModelForCausalLM.from_pretrained(
+    "BlazingCustoms/pybytecode-v3-1.5b", torch_dtype="bfloat16", device_map="auto")
+
+INSTRUCTION = ("Decompile this Python 3.12 bytecode disassembly back into the original Python "
+               "source code. Output only the source code.")
+
+# `disasm` comes from harness/pybytecode_core/rep.py: disassemble_v2(code_object)
+msgs = [{"role": "user", "content": f"{INSTRUCTION}\n\n{disasm}"}]
+batch = tok.apply_chat_template(msgs, add_generation_prompt=True,
+                                return_tensors="pt", return_dict=True).to(model.device)
+prediction = tok.decode(model.generate(**batch, max_new_tokens=2048,
+                                       do_sample=False)[0][batch["input_ids"].shape[1]:],
+                        skip_special_tokens=True)
+```
+
+**Then certify it** — this is the step that matters:
+
+```python
+from harness.pybytecode_core.verify import code_fingerprint
+
+def certified(prediction: str, reference_code_object) -> bool:
+    got = compile(prediction, "<pred>", "exec", dont_inherit=True, optimize=0)
+    return code_fingerprint(got) == code_fingerprint(reference_code_object)
+```
+
+Greedy decoding (temperature 0) for single-shot; temperature ~0.8 when sampling for best-of-N.
+
+### Measure your input size first
+
+**This model works on function-sized units and degrades sharply past ~200 representation lines.**
+That is the single most useful thing to know before you start, and it costs one line to check:
+
+```python
+from harness.pybytecode_core.rep import disassemble_v2
+rep_lines = disassemble_v2(code_object).count("\n")   # the model's actual input length
+```
+
+| your input | what to expect |
+|---|---|
+| **under 100 rep lines** | the model's home ground — ~89–96% greedy, ~97–99% at best-of-32 |
+| **100–200** | still strong, but sampling starts to earn its cost |
+| **200–300** | roughly a coin flip greedy; best-of-32 recovers much of it |
+| **over ~300** | greedy rarely certifies; best-of-32 recovers some. Expect misses |
+| **over ~400** | we certified nothing here, greedy **or** at 32 samples |
+
+Above the knee a symbolic decompiler is the better tool, and the measurements are in *Comparison
+with other systems* below.
+
+The full curve, pooled over both earlier benchmarks (n=679) so the thin upper buckets carry as
+many rows as we can give them:
+
+| rep lines | rows | v3 greedy | v3 best-of-32 | untuned base | PyLingual k=32 |
+|---|---|---|---|---|---|
+| 0–49 | 311 | 94.21% | 98.71% | 0.96% | 99.04% |
+| 50–99 | 229 | 88.65% | 96.94% | 0.00% | 95.63% |
+| 100–199 | 105 | 78.10% | 88.57% | 0.00% | 84.76% |
+| 200–299 | 19 | 47.37% | 84.21% | 0.00% | 84.21% |
+| 300–399 | 11 | 18.18% | 54.55% | 0.00% | 81.82% |
+| 400–599 | 2 | 0.00% | 50.00% | 0.00% | 100.00% |
+| 600+ | 2 | 0.00% | 0.00% | 0.00% | 50.00% |
+
+- **The greedy knee is at ~200 rep lines**: 78.1% → 47.4% → 18.2% → 0% across four consecutive
+  buckets.
+- **Best-of-32 postpones the knee to ~300; it does not remove it.** Sampling 32 times buys roughly
+  one bucket of headroom.
+- Almost every point we score comes from small units. On the earlier CSN-3.12 set (n=400),
+  **97.61%** of greedy certifications and **95.98%** of best-of-32 certifications are units under
+  200 rep lines; on the published 600-row set it is **96.64%** of greedy certifications. The
+  headline accuracy is a statement about small units.
+- The top two buckets are n=2 each and carry nothing on their own. What carries the conclusion is
+  the monotone decline through n=311/229/105/19/11 below them.
+
+The same stratification on the **published** 600-row benchmark — recomputable from the files in
+this repo with `harness/size_curve.py` — reproduces the shape:
+
+| rep lines | rows | v3 greedy | 95% CI | v3 best-of-32 | 95% CI |
+|---|---|---|---|---|---|
+| 0–49 | 224 | 95.98% | [93.01, 98.51] | 98.21% | [96.31, 99.58] |
+| 50–99 | 224 | 89.73% | [85.17, 93.93] | 98.21% | [96.26, 99.57] |
+| 100–199 | 112 | 65.18% | [55.36, 74.14] | 85.71% | [77.57, 92.98] |
+| 200–299 | 27 | 51.85% | — | 77.78% | — |
+| 300–399 | 5 | 60.00% | — | 100.00% | — |
+| 400–599 | 5 | 0.00% | — | 0.00% | — |
+| 600+ | 3 | 0.00% | — | 0.00% | — |
+
+Intervals are repo-clustered, and omitted below 30 rows / 10 repositories rather than printed at a
+width that would imply precision we do not have. The 300–399 bucket reading above the one below it
+is n=5 noise, not a recovery. Median input in this benchmark is **59** rep lines; p90 is 167, max
+1,622.
+
+On this set the top two buckets stay at **0% even with 32 samples** — above ~400 rep lines,
+sampling did not rescue a single one of the 8 rows. **96.64%** of greedy certifications and
+**95.37%** of best-of-32 certifications come from units under 200 rep lines.
+
+### Decide how much budget to spend
+
+The best-of-32 budget is **1 greedy decode + 31 sampled candidates at temperature 0.8**, with
+verified early stop — sampling halts on the first candidate that certifies, so the mean cost is
+far below 32. Across the 94 greedy failures it took a mean of 15.16 samples and recovered 56:
+
+| budget | certified | |
+|---|---|---|
+| @1 (greedy alone) | 506 / 600 | 84.33% |
+| @2 | 525 / 600 | 87.50% |
+| @4 | 546 / 600 | 91.00% |
+| @8 | 555 / 600 | 92.50% |
+| @16 | 558 / 600 | 93.00% |
+| @32 | 562 / 600 | 93.67% |
+
+Most of the recovery arrives in the first few samples: @4 already captures two thirds of what @32
+gets. Because early stop depends only on the index of the first passing sample, this curve is
+exact rather than an estimate — the same cached generations give every point.
+
+---
+
+## What this model does not do
+
+- **Large units — this is the measured limit.** Certification falls from 94% under 50
+  representation lines to 18% at 300–399 and 0% above 400. See *Measure your input size*: the
+  curve is the honest specification of what this model does. It was trained on functions; a whole
+  module is a different regime in both size and shape, and we have not evaluated one.
+- **Python 3.12 only.** A scope statement, not the explanation for the size behaviour above:
+  trained and measured on 3.12, and the oracle refuses cross-minor input by design. Every number
+  on this card is in-version, so nothing here is contaminated by a version mismatch.
+- **It did not decompile real malware.** On the one wild PyInstaller-packed sample in scope, the
+  entry-point module produced **nothing certifiable** — the prediction did not even compile. 3 of
+  9 units in that sample certified, and those three were PyInstaller's own bootstrap plus a
+  bundled stdlib module. Extraction and representation work (1,510/1,510 code objects recovered);
+  end-to-end decompilation of real malware logic is **not demonstrated**.
+
+  That entry point was both 3.13 *and* 491 representation lines, so **size and version are
+  confounded in it and it is evidence for neither** on its own. It is reported because you should
+  know the result, not because it diagnoses anything. The benign in-version curve above is the
+  instrument, and it already shows the model failing at that size on clean 3.12 input.
+- **Annotated functions** are handicapped: the training corpus was compiled with PEP-563
+  stringised annotations inherited, real `.pyc` files are not, and the benchmarks deliberately do
+  not hand the model its training-time distribution back.
+
+## The oracle's limits, where you will meet them
+
+Full detail in [`ORACLE-LIMITS.md`](ORACLE-LIMITS.md).
+
+- **0.33% false-reject floor on foreign `.pyc`.** On 600 wild install-time `.pyc`, 598 certify;
+  2 do not, because CPython 3.12.3 and 3.12.13 generate different code for the same source. No
+  normalisation fixes that without abandoning the byte-identical guarantee. It fails to a **false
+  reject** — you are told "unknown" about a correct answer, never "verified" about a wrong one.
+- **Optimization level must match the producer's.** A `.pyc` built with `-O` is a different code
+  object. The wrong level does not degrade gracefully, it collapses to ~24%. Try all three; it
+  costs three compiles. But note that at `optimize>=1` docstrings are absent from the `.pyc`, so
+  **docstring recovery becomes unprovable** against such a file — a limit of the artifact, not of
+  the oracle.
+- **The 100% pre-flight the harness prints proves nothing about soundness.** It grades each
+  reference against itself, i.e. compares `compile(x)` with `compile(x)`, so any deterministic
+  function of the source scores 100% — including a stub that ignores the bytecode entirely. It
+  detects a broken harness and nothing more. The soundness evidence is the mutation test (0 true
+  survivors in 1,239 mutants) and 18 targeted blind-spot probes (18/18).
+- **Untested:** cross-minor 3.13, Nuitka, non-CPython builds, obfuscated bytecode.
+
+## Reproduce every number on this card
+
+No model, no GPU, no network, no third-party decompiler — CPython 3.12 standard library only:
+
+```bash
+cd harness
+python3 grade.py --bench ../benchmarks/csn-3.12-licensed/bench.jsonl --self-test-only --out /tmp/st.json
+python3 analyze_scores.py --bench ../benchmarks/csn-3.12-licensed/bench.jsonl \
+    --greedy ../generations/gen_v3_csn600.jsonl \
+    --base   ../generations/gen_base_csn600.jsonl \
+    --out /tmp/scores.json --rows-out /tmp/rows.jsonl
+python3 size_curve.py --bench ../benchmarks/csn-3.12-licensed/bench.jsonl \
+    --greedy ../generations/gen_v3_csn600.jsonl \
+    --base   ../generations/gen_base_csn600.jsonl \
+    --out /tmp/size_curve.json
+```
+
+Per-row verdicts for all 600 rows — repo, function, commit SHA, SPDX, tuned verdict, base verdict
+— land in `rows.jsonl`. Every aggregate above is recomputable from that file.
+
+## Comparison with other systems
+
+[PyLingual](https://github.com/syssec-utd/pylingual) is another system that performs this task, by
+symbolic reconstruction rather than generation. We ran it on the same sets, under the same oracle,
+at a matched budget.
+
+| CSN-3.12, n=400, 24 repos | certified | 95% CI (repo-clustered) |
+|---|---|---|
+| untuned base, greedy | 0 / 400 = 0.00% | — |
+| v3 greedy | 335 / 400 = 83.75% | [77.84, 88.89] |
+| v3 certified@32 | 373 / 400 = 93.25% | [88.82, 96.89] |
+| PyLingual k=32 | 377 / 400 = 94.25% | [91.06, 96.80] |
+| **union** | **393 / 400 = 98.25%** | [96.81, 99.51] |
+
+| Held-out, n=279 | certified | 95% CI (Wilson) |
+|---|---|---|
+| untuned base, greedy | 3 / 279 = 1.08% | [0.37, 3.11] |
+| v3 greedy | 254 / 279 = 91.04% | [87.11, 93.86] |
+| v3 certified@32 | 272 / 279 = 97.49% | [94.91, 98.78] |
+| PyLingual k=32 | 267 / 279 = 95.70% | [92.63, 97.52] |
+| **union** | **275 / 279 = 98.57%** | [96.37, 99.44] |
+
+**Neither difference is statistically significant** (exact paired McNemar, p = 0.6177 on CSN and
+p = 0.2266 on held-out). We claim no accuracy advantage, and none is claimed against us. On cost
+the comparison runs against us: ~1.5B parameters against their ~320M, and slower wall-clock.
+
+Docstring recovery on the 115 held-out rows carrying a real docstring: **115 / 115 = 100%**
+(PyLingual: 113/115 = 98.26%, under an oracle that does not score docstrings at all).
+
+The held-out interval is **Wilson, not clustered**: every row of that set carries the same
+placeholder repository value, so its clustering cannot be assessed. The CSN design effects of
+2.34–2.69 show why that matters — on a set where one repository supplied 15% of rows, a naive
+interval would have been far too narrow.
+
+These figures were measured on our earlier benchmark pair, which is **not** published. PyLingual
+has not been run on the 600-row licensed benchmark released alongside this model.
+
+### The two approaches fail on different inputs
+
+The more interesting measured result is not which number is larger, but that the two systems lose
+on different inputs, along an axis that turns out to be **unit size**:
+
+| rep lines | rows | PyBytecode greedy | PyBytecode @32 | PyLingual k=32 |
+|---|---|---|---|---|
+| 100–199 | 105 | 78.10% | 88.57% | 84.76% |
+| 200–299 | 19 | 47.37% | 84.21% | 84.21% |
+| 300–399 | 11 | **18.18%** | 54.55% | **81.82%** |
+| 400–599 | 2 | **0.00%** | 50.00% | **100.0%** |
+
+A generative model degrades as its input grows; a symbolic reconstruction does not. Two things
+follow, both useful:
+
+- **Large units are not intrinsically unsolvable.** A symbolic system handles them at 81.82% where
+  we score 18.18%. What our curve measures is a capability limit of a 1.5B model, not a property
+  of the task — which is a statement about where to spend the next round of scale.
+- **The union exceeds either system**, 98.25% and 98.57% above. Because verification is sound, the
+  union costs nothing to take: run either, keep the answer that certifies, fall through to the
+  other. Neither system can certify a wrong answer, so combining them cannot mislead you.
+
+## Training
+
+| | |
+|---|---|
+| Base | [`Qwen/Qwen2.5-Coder-1.5B-Instruct`](https://huggingface.co/Qwen/Qwen2.5-Coder-1.5B-Instruct) (Apache-2.0) |
+| Method | LoRA r=16, α=32, all attention + MLP projections, 1 epoch, lr 2e-4, merged into the base |
+| Rows | 48,196 pairs of Python 3.12 disassembly → source |
+| Corpus | `codeparrot/github-code-clean`, per-row filtered to 7 permissive licences; all GPL/LGPL/AGPL/MPL/EPL dropped before extraction |
+| Decontamination | shard-disjoint, repo-disjoint, and identifier-blind fingerprint-disjoint |
+
+**The training corpus is not distributed.** Per-row attribution was not retained during
+extraction, so the corpus cannot be redistributed without dropping required MIT/BSD/Apache
+notices. That is a property of the artifact, not of the licences — nothing in it is unlicensed or
+reciprocally licensed. Full lineage, including what was dropped and why:
+[`DATA-CARD-training-corpus.md`](DATA-CARD-training-corpus.md).
+
+## Licence
+
+**Apache-2.0.** See [`LICENSE`](LICENSE) and [`NOTICE`](NOTICE).
+
+Derived from `Qwen/Qwen2.5-Coder-1.5B-Instruct`, which is Apache-2.0. Under Apache-2.0 §4 we ship
+the licence, retain attribution, and state our changes (LoRA fine-tune, adapter merged; no
+architecture, vocabulary or tokenizer change). The upstream repository ships a `LICENSE` and no
+`NOTICE`, so there is no upstream notice text to carry forward. If you redistribute these weights
+or build derivatives, the same four obligations pass to you.
+
+The corpus constraint above binds the *corpus*, not these weights: trained parameters are not a
+copy of any source text, and the corpus was filtered to permissive licences before training, so no
+reciprocal terms exist upstream to propagate.
+
+Decompilation has obvious dual use. Apache-2.0 imposes no field-of-use restriction and we have not
+added one: a restrictive weights licence would not prevent misuse and would block the
+interoperability, incident-response and recovery work this is for. Complying with the law where
+you operate is your responsibility.
+
+## Citation
+
+```bibtex
+@software{pybytecode2026,
+  title  = {PyBytecode: verified neural decompilation for Python 3.12 bytecode},
+  author = {Blazing Customs},
+  year   = {2026},
+  note   = {Fine-tuned from Qwen2.5-Coder-1.5B-Instruct},
+  url    = {https://huggingface.co/BlazingCustoms/pybytecode-v3-1.5b}
+}
+```