Clarify model comparison and report English/Python perplexity

README.md

@@ -18,7 +18,7 @@ Model card written by GPT-6 Astra:
 
 # Swift 1.0 HyperQwen
 
-**Swift 1.0's efficient reasoning, combined with HyperQwen's optimized serving, delivered 39% lower average task completion time than HyperQwen serving the Qwen fast checkpoint in our RTX 3090 evaluation.**
+Swift 1.0's efficient reasoning, combined with HyperQwen's optimized serving, **delivered 39% lower average task completion time** than HyperQwen serving the Qwen fast checkpoint in our RTX 3090 evaluation.
 
 This checkpoint brings Swift 1.0 to HyperQwen's quantized serving and MTP speculative-decoding path. Across 630 tasks, it generated **42% fewer output tokens** than the Qwen fast reference. Swift reduces the amount of generation needed to finish a task; HyperQwen accelerates generation. The headline result is shorter task time, even where raw tokens per second is lower. Average task time here is per-request elapsed time at **two concurrent requests**.
 
@@ -36,35 +36,31 @@ Upstream checkpoint: [TheUnderscore/Swift-Qwen3.8-27b-W4A16-AWQ](https://hugging
 
 ## Performance
 
-| Measurement | Qwen fast | Swift 1.0 | Swift 1.5 INT8 heads | Swift 1.5 fast |
+| Measurement | Qwen — HyperQwen fast quant (W4A16 AutoRound) | Swift 1.0 | Swift 1.5 INT8 heads | Swift 1.5 fast |
 |---|---:|---:|---:|---:|
 | **Average request time ↓** | **108.1 s** | **66.2 s** | **72.2 s** | **68.2 s** |
 | **Average output tokens/task ↓** | **8,985** | **5,245** | **5,751** | **5,669** |
 | Median decode tokens/s ↑ | 112.1 | 105.9 | 104.0 | 107.2 |
 | Total output tokens, 630 tasks | 5.66M | 3.30M | 3.62M | 3.57M |
 
-All four checkpoints run through HyperQwen. Request times include failures and are not single-user latency measurements. Output counts include reasoning. TPS is a separate **single-request** test: four short prompts repeated twice, with 512 greedy output tokens each.
-
 ## Quality
 
-| Test | Qwen fast | Swift 1.0 | Swift 1.5 INT8 heads | Swift 1.5 fast |
+| Test | Qwen — HyperQwen fast quant (W4A16 AutoRound) | Swift 1.0 | Swift 1.5 INT8 heads | Swift 1.5 fast |
 |---|---:|---:|---:|---:|
 | **GSM8K — 200-question subset** | 97.5% | 98.0% | 98.0% | 97.5% |
 | **IFBench — 300 prompts, strict** | 74.0% | 73.3% | 73.7% | 72.3% |
 | **LiveCodeBench — 100-problem subset** | 90% | 89% | 89% | 91% |
 | **Custom tool-call/JSON checks — 30 tasks** | 29/30 | 28/30 | 30/30 | 30/30 |
-| Perplexity, English/Danish/Python ↓ | 8.143 | 8.215 | 8.252 | 8.318 |
+| Perplexity, English/Python ↓ | 6.551 | 6.605 | 6.643 | 6.679 |
 | Truncated answers, counted wrong | 2 | 1 | 1 | 0 |
 
 - **GSM8K:** the first 200 questions from the [test split](https://huggingface.co/datasets/openai/gsm8k), with thinking disabled.
 - **IFBench:** all 300 prompts in the pinned [IFBench test dataset](https://huggingface.co/datasets/allenai/IFBench_test), scored with the official strict prompt-level verifier.
 - **LiveCodeBench:** a frozen [v6-era dataset](https://huggingface.co/datasets/livecodebench/code_generation_lite) subset of Python stdin/stdout problems: 34 easy, 33 medium, 33 hard. Scored against supplied public/private tests with a custom judge; not a full official LiveCodeBench result.
 - **Tool-call/JSON checks:** 20 custom weather-tool tasks checking the function name, arguments, Celsius-to-Fahrenheit conversion and final JSON; plus 10 JSON inventory-filtering tasks. These are integration checks, not an external agent benchmark.
-- **Perplexity:** 32,646 scored tokens from English Wikipedia, Danish web text and Python source; lower is better.
-
-These are single-seed local results; small accuracy differences do not establish a universal ranking. Qwen fast is a quantized AutoRound reference, not BF16 Qwen. Swift 1.0 was the most token-efficient model on this task mix.
+- **Perplexity:** 18,729 scored tokens from English Wikipedia and Python source; lower is better. Recomputed from saved token log-probabilities, excluding Danish; see [subset results](evaluation/perplexity-english-python.json).
 
-**Evaluation setup:** RTX 3090 24 GB; FP8 KV cache; 150,000-token configured context; 128,000 output tokens per call. These are runtime settings, not fixed model properties, and the TPS test uses short prompts. All models used the same serving settings and task budgets. Thinking tests used xhigh effort, temperature 1.0, top_p 0.95, top_k 20 and seed 15027; GSM8K/tool checks were greedy.
+**Evaluation setup:** RTX 3090 24 GB; FP8 KV cache; 150,000-token configured context; 128,000 output tokens per call. These are runtime settings, not fixed model properties, and the TPS test uses short prompts. Task evaluation used two concurrent requests; TPS was measured with one request at a time. All models used the same serving settings and task budgets. Thinking tests used xhigh effort, temperature 1.0, top_p 0.95, top_k 20 and seed 15027; GSM8K/tool checks were greedy.
 
 ## Setup
 

RELEASE_MANIFEST.json

@@ -18,8 +18,8 @@
       "sha256": "9646e189bb9cfb1f35467eece46b59faf73c5eb0cc9800993343947cd875594d"
     },
     "README.md": {
-      "size": 5990,
-      "sha256": "3818b35f7c956576ad6110db8f10cfc70ef2f6fcd083e3e576d95df5422d30ac"
+      "size": 5767,
+      "sha256": "669f0e281c61bbb4edd7cb67a018af234364aceccf2119a3b0a9ac28f35d59c0"
     },
     "RUNTIME.md": {
       "size": 2588,
@@ -82,8 +82,8 @@
       "sha256": "c5129121b4ba333b0c4dd2694fb69076cdc75e4b9cda8a6572b3a449c32c3b5e"
     },
     "evaluation/code/swift15/report.py": {
-      "size": 4938,
-      "sha256": "873db1c8e86fb281b15f0b3c2e66adfb3cf5da652cf0139788acd00f8bd2ac02"
+      "size": 5287,
+      "sha256": "622af3dd5ab1eaa1843407c5bfb57e3111a154463c024a6685095a7aa293b482"
     },
     "evaluation/code/swift15/run.py": {
       "size": 9281,
@@ -428,6 +428,10 @@
     "vocab.json": {
       "size": 6722759,
       "sha256": "ce99b4cb2983d118806ce0a8b777a35b093e2000a503ebde25853284c9dfa003"
+    },
+    "evaluation/perplexity-english-python.json": {
+      "size": 38213,
+      "sha256": "5c0f9ab2df9a8d4e14ea846cd03e1c3cb577847a022a955f97ff57ef6392fe85"
     }
   }
 }

evaluation/code/swift15/report.py

@@ -3,6 +3,13 @@ import math
 from common import RUN, read_json, stamp
 
 
+def english_python_ppl(scores):
+    groups = [scores[k] for k in ["en", "code"] if k in scores]
+    if len(groups) != 2:
+        return None
+    return math.exp(sum(g["tokens"] * math.log(g["ppl"]) for g in groups) / sum(g["tokens"] for g in groups))
+
+
 def interval(correct, n):
     z = 1.96
     p = correct / n
@@ -20,8 +27,8 @@ def main():
              "The Swift 1.5 INT8-head baseline and INT4-head fast derivative share the same upstream AWQ body. "
              "A BF16 reference measurement is outside this four-checkpoint comparison.", "",
              "## Fixed-output speed and perplexity", "",
-             "512 output tokens/request, concurrency 1; natural task lengths are reported separately.", "",
-             "| Model | Decode tok/s (median) | TTFT (s) | EN PPL | DA PPL | Code PPL |",
+             "512 output tokens/request, concurrency 1; natural task lengths are reported separately. Perplexity uses English and Python only; Danish is excluded.", "",
+             "| Model | Decode tok/s (median) | TTFT (s) | EN PPL | Code PPL | EN/Python PPL |",
              "|---|---:|---:|---:|---:|---:|"]
     tags = ["qwen-fast", "swift10", "swift15-baseline", "swift15-fast"]
     for tag in tags:
@@ -30,7 +37,7 @@ def main():
         ppl = read_json(directory / "perplexity.json").get("scores", {}) if (directory / "perplexity.json").exists() else {}
         def f(v): return "pending" if v is None else f"{v:.3f}"
         lines.append("| " + " | ".join([tag, f(speed.get("median_decode_tps")), f(speed.get("median_ttft_seconds")),
-                                      *[f(ppl.get(k, {}).get("ppl")) for k in ["en", "da", "code"]]]) + " |")
+                                      *[f(ppl.get(k, {}).get("ppl")) for k in ["en", "code"]], f(english_python_ppl(ppl))]) + " |")
     lines += ["", "## Swift 1.5 multi-user serving", "",
               "Same fast checkpoint, no MTP, concurrency eight. INT8 applies to MLP activations only.", "",
               "| Runtime | Aggregate output tok/s | Combined PPL |", "|---|---:|---:|"]
@@ -38,7 +45,7 @@ def main():
         directory = RUN / "results" / tag
         if not (directory / "speed-c8.json").exists(): continue
         speed = read_json(directory / "speed-c8.json")
-        ppl = read_json(directory / "perplexity.json")["scores"]["all"]["ppl"] if (directory / "perplexity.json").exists() else None
+        ppl = english_python_ppl(read_json(directory / "perplexity.json")["scores"]) if (directory / "perplexity.json").exists() else None
         lines.append(f"| {tag} | {speed['aggregate_output_tps']:.2f} | {ppl if ppl is not None else 'pending'} |")
     for mode in ["pilot", "full"]:
         lines += ["", "## " + ("Sequential task latency pilot" if mode == "pilot" else "Quality suite (concurrency recorded per run)"), "",

evaluation/perplexity-english-python.json

@@ -0,0 +1,1632 @@
+{
+  "included_languages": [
+    "en",
+    "code"
+  ],
+  "excluded_languages": [
+    "da"
+  ],
+  "method": "exp(-sum(logprob_sum) / sum(tokens)) over retained windows; no model rerun",
+  "models": {
+    "qwen-fast": {
+      "source_file": "qwen-fast/perplexity.json",
+      "source_sha256": "9db1599421dc916b97602a8921dbd219b815a5a76fc7def41f508500eb10e7e7",
+      "input_sha256": "57c83ffe7c0dfba2b869f6a379d2df3f0bc0d055069b49569a2154336151a718",
+      "windows": [
+        {
+          "id": 0,
+          "language": "en",
+          "tokens": 283,
+          "logprob_sum": -592.9908143195717
+        },
+        {
+          "id": 1,
+          "language": "en",
+          "tokens": 313,
+          "logprob_sum": -694.9407490367958
+        },
+        {
+          "id": 2,
+          "language": "en",
+          "tokens": 284,
+          "logprob_sum": -731.2198149585729
+        },
+        {
+          "id": 3,
+          "language": "en",
+          "tokens": 286,
+          "logprob_sum": -760.9622869625673
+        },
+        {
+          "id": 4,
+          "language": "en",
+          "tokens": 279,
+          "logprob_sum": -585.9759548976872
+        },
+        {
+          "id": 5,
+          "language": "en",
+          "tokens": 251,
+          "logprob_sum": -624.1354073971652
+        },
+        {
+          "id": 6,
+          "language": "en",
+          "tokens": 258,
+          "logprob_sum": -606.1599571104307
+        },
+        {
+          "id": 7,
+          "language": "en",
+          "tokens": 258,
+          "logprob_sum": -601.0718723634782
+        },
+        {
+          "id": 8,
+          "language": "en",
+          "tokens": 274,
+          "logprob_sum": -629.9130605636892
+        },
+        {
+          "id": 9,
+          "language": "en",
+          "tokens": 278,
+          "logprob_sum": -605.0941830798547
+        },
+        {
+          "id": 10,
+          "language": "en",
+          "tokens": 274,
+          "logprob_sum": -713.5701698549092
+        },
+        {
+          "id": 11,
+          "language": "en",
+          "tokens": 288,
+          "logprob_sum": -759.5278018228346
+        },
+        {
+          "id": 12,
+          "language": "en",
+          "tokens": 277,
+          "logprob_sum": -623.4307517245397
+        },
+        {
+          "id": 13,
+          "language": "en",
+          "tokens": 291,
+          "logprob_sum": -699.0976419129438
+        },
+        {
+          "id": 14,
+          "language": "en",
+          "tokens": 260,
+          "logprob_sum": -672.0876630296007
+        },
+        {
+          "id": 15,
+          "language": "en",
+          "tokens": 250,
+          "logprob_sum": -761.0354307279922
+        },
+        {
+          "id": 16,
+          "language": "en",
+          "tokens": 252,
+          "logprob_sum": -754.8780198292225
+        },
+        {
+          "id": 17,
+          "language": "en",
+          "tokens": 272,
+          "logprob_sum": -788.3398015776256
+        },
+        {
+          "id": 18,
+          "language": "en",
+          "tokens": 240,
+          "logprob_sum": -668.2802309252293
+        },
+        {
+          "id": 19,
+          "language": "en",
+          "tokens": 244,
+          "logprob_sum": -561.2377929796639
+        },
+        {
+          "id": 20,
+          "language": "en",
+          "tokens": 274,
+          "logprob_sum": -682.5377916035941
+        },
+        {
+          "id": 21,
+          "language": "en",
+          "tokens": 299,
+          "logprob_sum": -738.1227494915511
+        },
+        {
+          "id": 22,
+          "language": "en",
+          "tokens": 318,
+          "logprob_sum": -823.5101455471158
+        },
+        {
+          "id": 23,
+          "language": "en",
+          "tokens": 263,
+          "logprob_sum": -677.3689384464233
+        },
+        {
+          "id": 24,
+          "language": "en",
+          "tokens": 239,
+          "logprob_sum": -711.9839977540105
+        },
+        {
+          "id": 25,
+          "language": "en",
+          "tokens": 293,
+          "logprob_sum": -493.05563317897304
+        },
+        {
+          "id": 26,
+          "language": "en",
+          "tokens": 303,
+          "logprob_sum": -678.5333020727912
+        },
+        {
+          "id": 27,
+          "language": "en",
+          "tokens": 276,
+          "logprob_sum": -623.330151065903
+        },
+        {
+          "id": 28,
+          "language": "en",
+          "tokens": 282,
+          "logprob_sum": -729.7613100720046
+        },
+        {
+          "id": 29,
+          "language": "en",
+          "tokens": 298,
+          "logprob_sum": -691.1857994162092
+        },
+        {
+          "id": 30,
+          "language": "en",
+          "tokens": 338,
+          "logprob_sum": -478.5514251983941
+        },
+        {
+          "id": 31,
+          "language": "en",
+          "tokens": 280,
+          "logprob_sum": -639.465224127729
+        },
+        {
+          "id": 32,
+          "language": "en",
+          "tokens": 310,
+          "logprob_sum": -813.8291402243576
+        },
+        {
+          "id": 33,
+          "language": "en",
+          "tokens": 280,
+          "logprob_sum": -798.3652476328898
+        },
+        {
+          "id": 34,
+          "language": "en",
+          "tokens": 302,
+          "logprob_sum": -693.1609880588003
+        },
+        {
+          "id": 35,
+          "language": "en",
+          "tokens": 286,
+          "logprob_sum": -523.3296906485411
+        },
+        {
+          "id": 36,
+          "language": "en",
+          "tokens": 244,
+          "logprob_sum": -581.809817118934
+        },
+        {
+          "id": 37,
+          "language": "en",
+          "tokens": 280,
+          "logprob_sum": -495.3885064668866
+        },
+        {
+          "id": 38,
+          "language": "en",
+          "tokens": 315,
+          "logprob_sum": -599.4133885340561
+        },
+        {
+          "id": 39,
+          "language": "en",
+          "tokens": 283,
+          "logprob_sum": -647.8761923574384
+        },
+        {
+          "id": 80,
+          "language": "code",
+          "tokens": 320,
+          "logprob_sum": -276.6699116966555
+        },
+        {
+          "id": 81,
+          "language": "code",
+          "tokens": 282,
+          "logprob_sum": -420.23796508818486
+        },
+        {
+          "id": 82,
+          "language": "code",
+          "tokens": 278,
+          "logprob_sum": -263.582931014716
+        },
+        {
+          "id": 83,
+          "language": "code",
+          "tokens": 273,
+          "logprob_sum": -418.2609401268255
+        },
+        {
+          "id": 84,
+          "language": "code",
+          "tokens": 253,
+          "logprob_sum": -199.9309399436836
+        },
+        {
+          "id": 85,
+          "language": "code",
+          "tokens": 239,
+          "logprob_sum": -465.6562252798758
+        },
+        {
+          "id": 86,
+          "language": "code",
+          "tokens": 272,
+          "logprob_sum": -374.48760858155583
+        },
+        {
+          "id": 87,
+          "language": "code",
+          "tokens": 253,
+          "logprob_sum": -291.8236611021075
+        },
+        {
+          "id": 88,
+          "language": "code",
+          "tokens": 317,
+          "logprob_sum": -284.14727575928714
+        },
+        {
+          "id": 89,
+          "language": "code",
+          "tokens": 298,
+          "logprob_sum": -303.1649754933601
+        },
+        {
+          "id": 90,
+          "language": "code",
+          "tokens": 280,
+          "logprob_sum": -329.05116950601234
+        },
+        {
+          "id": 91,
+          "language": "code",
+          "tokens": 265,
+          "logprob_sum": -394.77499111452255
+        },
+        {
+          "id": 92,
+          "language": "code",
+          "tokens": 305,
+          "logprob_sum": -219.57371839839698
+        },
+        {
+          "id": 93,
+          "language": "code",
+          "tokens": 310,
+          "logprob_sum": -331.6206139813031
+        },
+        {
+          "id": 94,
+          "language": "code",
+          "tokens": 278,
+          "logprob_sum": -251.4822827990156
+        },
+        {
+          "id": 95,
+          "language": "code",
+          "tokens": 301,
+          "logprob_sum": -308.4679439706525
+        },
+        {
+          "id": 96,
+          "language": "code",
+          "tokens": 311,
+          "logprob_sum": -184.7737082277912
+        },
+        {
+          "id": 97,
+          "language": "code",
+          "tokens": 312,
+          "logprob_sum": -226.99020833792702
+        },
+        {
+          "id": 98,
+          "language": "code",
+          "tokens": 325,
+          "logprob_sum": -197.7390554841084
+        },
+        {
+          "id": 99,
+          "language": "code",
+          "tokens": 291,
+          "logprob_sum": -325.6971021090387
+        },
+        {
+          "id": 100,
+          "language": "code",
+          "tokens": 321,
+          "logprob_sum": -453.9610157664529
+        },
+        {
+          "id": 101,
+          "language": "code",
+          "tokens": 318,
+          "logprob_sum": -451.3294726509339
+        },
+        {
+          "id": 102,
+          "language": "code",
+          "tokens": 316,
+          "logprob_sum": -502.03988517310944
+        },
+        {
+          "id": 103,
+          "language": "code",
+          "tokens": 307,
+          "logprob_sum": -297.40668610171537
+        },
+        {
+          "id": 104,
+          "language": "code",
+          "tokens": 254,
+          "logprob_sum": -418.13921085138236
+        },
+        {
+          "id": 105,
+          "language": "code",
+          "tokens": 275,
+          "logprob_sum": -457.8816399213333
+        }
+      ],
+      "tokens": 18729,
+      "ppl": 6.551020950595325
+    },
+    "swift10": {
+      "source_file": "swift10/perplexity.json",
+      "source_sha256": "fe393bd4962d43f9cef2a08ea40c0dd2f15e6695b87a09a20e184d448d941775",
+      "input_sha256": "57c83ffe7c0dfba2b869f6a379d2df3f0bc0d055069b49569a2154336151a718",
+      "windows": [
+        {
+          "id": 0,
+          "language": "en",
+          "tokens": 283,
+          "logprob_sum": -593.3995295942841
+        },
+        {
+          "id": 1,
+          "language": "en",
+          "tokens": 313,
+          "logprob_sum": -700.5331317378659
+        },
+        {
+          "id": 2,
+          "language": "en",
+          "tokens": 284,
+          "logprob_sum": -724.3817458400572
+        },
+        {
+          "id": 3,
+          "language": "en",
+          "tokens": 286,
+          "logprob_sum": -774.8060930467109
+        },
+        {
+          "id": 4,
+          "language": "en",
+          "tokens": 279,
+          "logprob_sum": -573.8085490163357
+        },
+        {
+          "id": 5,
+          "language": "en",
+          "tokens": 251,
+          "logprob_sum": -647.1657810044508
+        },
+        {
+          "id": 6,
+          "language": "en",
+          "tokens": 258,
+          "logprob_sum": -613.3406054319057
+        },
+        {
+          "id": 7,
+          "language": "en",
+          "tokens": 258,
+          "logprob_sum": -598.2336515752831
+        },
+        {
+          "id": 8,
+          "language": "en",
+          "tokens": 274,
+          "logprob_sum": -655.9989099322702
+        },
+        {
+          "id": 9,
+          "language": "en",
+          "tokens": 278,
+          "logprob_sum": -620.1450639520117
+        },
+        {
+          "id": 10,
+          "language": "en",
+          "tokens": 274,
+          "logprob_sum": -750.830778833566
+        },
+        {
+          "id": 11,
+          "language": "en",
+          "tokens": 288,
+          "logprob_sum": -780.6144631060088
+        },
+        {
+          "id": 12,
+          "language": "en",
+          "tokens": 277,
+          "logprob_sum": -614.716890099935
+        },
+        {
+          "id": 13,
+          "language": "en",
+          "tokens": 291,
+          "logprob_sum": -711.6267314220386
+        },
+        {
+          "id": 14,
+          "language": "en",
+          "tokens": 260,
+          "logprob_sum": -671.5577233368549
+        },
+        {
+          "id": 15,
+          "language": "en",
+          "tokens": 250,
+          "logprob_sum": -775.6550934422994
+        },
+        {
+          "id": 16,
+          "language": "en",
+          "tokens": 252,
+          "logprob_sum": -756.1478926279815
+        },
+        {
+          "id": 17,
+          "language": "en",
+          "tokens": 272,
+          "logprob_sum": -783.3444251654037
+        },
+        {
+          "id": 18,
+          "language": "en",
+          "tokens": 240,
+          "logprob_sum": -676.2792851193226
+        },
+        {
+          "id": 19,
+          "language": "en",
+          "tokens": 244,
+          "logprob_sum": -561.7464151983149
+        },
+        {
+          "id": 20,
+          "language": "en",
+          "tokens": 274,
+          "logprob_sum": -689.3211253857444
+        },
+        {
+          "id": 21,
+          "language": "en",
+          "tokens": 299,
+          "logprob_sum": -719.0637272568165
+        },
+        {
+          "id": 22,
+          "language": "en",
+          "tokens": 318,
+          "logprob_sum": -828.7790999828721
+        },
+        {
+          "id": 23,
+          "language": "en",
+          "tokens": 263,
+          "logprob_sum": -680.0565696356061
+        },
+        {
+          "id": 24,
+          "language": "en",
+          "tokens": 239,
+          "logprob_sum": -715.9142811552592
+        },
+        {
+          "id": 25,
+          "language": "en",
+          "tokens": 293,
+          "logprob_sum": -489.55537099707726
+        },
+        {
+          "id": 26,
+          "language": "en",
+          "tokens": 303,
+          "logprob_sum": -682.4112273987994
+        },
+        {
+          "id": 27,
+          "language": "en",
+          "tokens": 276,
+          "logprob_sum": -620.8538292451813
+        },
+        {
+          "id": 28,
+          "language": "en",
+          "tokens": 282,
+          "logprob_sum": -738.9185054079862
+        },
+        {
+          "id": 29,
+          "language": "en",
+          "tokens": 298,
+          "logprob_sum": -686.1773185838956
+        },
+        {
+          "id": 30,
+          "language": "en",
+          "tokens": 338,
+          "logprob_sum": -473.0945813098033
+        },
+        {
+          "id": 31,
+          "language": "en",
+          "tokens": 280,
+          "logprob_sum": -630.0955357118728
+        },
+        {
+          "id": 32,
+          "language": "en",
+          "tokens": 310,
+          "logprob_sum": -810.4693004913643
+        },
+        {
+          "id": 33,
+          "language": "en",
+          "tokens": 280,
+          "logprob_sum": -799.6141130025499
+        },
+        {
+          "id": 34,
+          "language": "en",
+          "tokens": 302,
+          "logprob_sum": -712.025332148276
+        },
+        {
+          "id": 35,
+          "language": "en",
+          "tokens": 286,
+          "logprob_sum": -538.7282043778414
+        },
+        {
+          "id": 36,
+          "language": "en",
+          "tokens": 244,
+          "logprob_sum": -593.2613173677673
+        },
+        {
+          "id": 37,
+          "language": "en",
+          "tokens": 280,
+          "logprob_sum": -498.5803779190901
+        },
+        {
+          "id": 38,
+          "language": "en",
+          "tokens": 315,
+          "logprob_sum": -604.9404760405196
+        },
+        {
+          "id": 39,
+          "language": "en",
+          "tokens": 283,
+          "logprob_sum": -652.8633822322918
+        },
+        {
+          "id": 80,
+          "language": "code",
+          "tokens": 320,
+          "logprob_sum": -274.7489258961019
+        },
+        {
+          "id": 81,
+          "language": "code",
+          "tokens": 282,
+          "logprob_sum": -430.6838706552262
+        },
+        {
+          "id": 82,
+          "language": "code",
+          "tokens": 278,
+          "logprob_sum": -260.0947107755287
+        },
+        {
+          "id": 83,
+          "language": "code",
+          "tokens": 273,
+          "logprob_sum": -411.2611253242603
+        },
+        {
+          "id": 84,
+          "language": "code",
+          "tokens": 253,
+          "logprob_sum": -197.21853261078218
+        },
+        {
+          "id": 85,
+          "language": "code",
+          "tokens": 239,
+          "logprob_sum": -461.1512319083995
+        },
+        {
+          "id": 86,
+          "language": "code",
+          "tokens": 272,
+          "logprob_sum": -377.2601323130457
+        },
+        {
+          "id": 87,
+          "language": "code",
+          "tokens": 253,
+          "logprob_sum": -282.7043757491738
+        },
+        {
+          "id": 88,
+          "language": "code",
+          "tokens": 317,
+          "logprob_sum": -288.68833832504083
+        },
+        {
+          "id": 89,
+          "language": "code",
+          "tokens": 298,
+          "logprob_sum": -290.93484990602883
+        },
+        {
+          "id": 90,
+          "language": "code",
+          "tokens": 280,
+          "logprob_sum": -305.8559028699442
+        },
+        {
+          "id": 91,
+          "language": "code",
+          "tokens": 265,
+          "logprob_sum": -387.24696855193054
+        },
+        {
+          "id": 92,
+          "language": "code",
+          "tokens": 305,
+          "logprob_sum": -197.82955161913992
+        },
+        {
+          "id": 93,
+          "language": "code",
+          "tokens": 310,
+          "logprob_sum": -351.8639510574991
+        },
+        {
+          "id": 94,
+          "language": "code",
+          "tokens": 278,
+          "logprob_sum": -236.71723654100515
+        },
+        {
+          "id": 95,
+          "language": "code",
+          "tokens": 301,
+          "logprob_sum": -317.9189716539063
+        },
+        {
+          "id": 96,
+          "language": "code",
+          "tokens": 311,
+          "logprob_sum": -186.151134865902
+        },
+        {
+          "id": 97,
+          "language": "code",
+          "tokens": 312,
+          "logprob_sum": -227.76212669502684
+        },
+        {
+          "id": 98,
+          "language": "code",
+          "tokens": 325,
+          "logprob_sum": -195.6774061962218
+        },
+        {
+          "id": 99,
+          "language": "code",
+          "tokens": 291,
+          "logprob_sum": -313.3808719475601
+        },
+        {
+          "id": 100,
+          "language": "code",
+          "tokens": 321,
+          "logprob_sum": -436.92633147341314
+        },
+        {
+          "id": 101,
+          "language": "code",
+          "tokens": 318,
+          "logprob_sum": -442.086307042553
+        },
+        {
+          "id": 102,
+          "language": "code",
+          "tokens": 316,
+          "logprob_sum": -522.1931145482638
+        },
+        {
+          "id": 103,
+          "language": "code",
+          "tokens": 307,
+          "logprob_sum": -305.42324257145077
+        },
+        {
+          "id": 104,
+          "language": "code",
+          "tokens": 254,
+          "logprob_sum": -407.6405011889765
+        },
+        {
+          "id": 105,
+          "language": "code",
+          "tokens": 275,
+          "logprob_sum": -499.1410040079517
+        }
+      ],
+      "tokens": 18729,
+      "ppl": 6.605178600890788
+    },
+    "swift15-baseline": {
+      "source_file": "swift15-baseline/perplexity.json",
+      "source_sha256": "6f63e857421957c76b3ca290ec8c95a62fd4374e3839adc2cdcdca54f14786fa",
+      "input_sha256": "57c83ffe7c0dfba2b869f6a379d2df3f0bc0d055069b49569a2154336151a718",
+      "windows": [
+        {
+          "id": 0,
+          "language": "en",
+          "tokens": 283,
+          "logprob_sum": -589.2026971864252
+        },
+        {
+          "id": 1,
+          "language": "en",
+          "tokens": 313,
+          "logprob_sum": -693.8275278248402
+        },
+        {
+          "id": 2,
+          "language": "en",
+          "tokens": 284,
+          "logprob_sum": -725.2991740150046
+        },
+        {
+          "id": 3,
+          "language": "en",
+          "tokens": 286,
+          "logprob_sum": -774.4817539830692
+        },
+        {
+          "id": 4,
+          "language": "en",
+          "tokens": 279,
+          "logprob_sum": -570.2485540384951
+        },
+        {
+          "id": 5,
+          "language": "en",
+          "tokens": 251,
+          "logprob_sum": -646.9325801496389
+        },
+        {
+          "id": 6,
+          "language": "en",
+          "tokens": 258,
+          "logprob_sum": -616.6254827602825
+        },
+        {
+          "id": 7,
+          "language": "en",
+          "tokens": 258,
+          "logprob_sum": -601.0592320801952
+        },
+        {
+          "id": 8,
+          "language": "en",
+          "tokens": 274,
+          "logprob_sum": -645.9870536324452
+        },
+        {
+          "id": 9,
+          "language": "en",
+          "tokens": 278,
+          "logprob_sum": -619.5700800593986
+        },
+        {
+          "id": 10,
+          "language": "en",
+          "tokens": 274,
+          "logprob_sum": -754.2683118742716
+        },
+        {
+          "id": 11,
+          "language": "en",
+          "tokens": 288,
+          "logprob_sum": -783.0617909351113
+        },
+        {
+          "id": 12,
+          "language": "en",
+          "tokens": 277,
+          "logprob_sum": -621.23286663424
+        },
+        {
+          "id": 13,
+          "language": "en",
+          "tokens": 291,
+          "logprob_sum": -711.5569263604993
+        },
+        {
+          "id": 14,
+          "language": "en",
+          "tokens": 260,
+          "logprob_sum": -672.5337460310639
+        },
+        {
+          "id": 15,
+          "language": "en",
+          "tokens": 250,
+          "logprob_sum": -772.983594376361
+        },
+        {
+          "id": 16,
+          "language": "en",
+          "tokens": 252,
+          "logprob_sum": -755.1737170430715
+        },
+        {
+          "id": 17,
+          "language": "en",
+          "tokens": 272,
+          "logprob_sum": -785.5630708901444
+        },
+        {
+          "id": 18,
+          "language": "en",
+          "tokens": 240,
+          "logprob_sum": -677.5972105439869
+        },
+        {
+          "id": 19,
+          "language": "en",
+          "tokens": 244,
+          "logprob_sum": -563.4146524730604
+        },
+        {
+          "id": 20,
+          "language": "en",
+          "tokens": 274,
+          "logprob_sum": -683.130450318291
+        },
+        {
+          "id": 21,
+          "language": "en",
+          "tokens": 299,
+          "logprob_sum": -729.0297191701102
+        },
+        {
+          "id": 22,
+          "language": "en",
+          "tokens": 318,
+          "logprob_sum": -832.9527902895788
+        },
+        {
+          "id": 23,
+          "language": "en",
+          "tokens": 263,
+          "logprob_sum": -678.3365666632744
+        },
+        {
+          "id": 24,
+          "language": "en",
+          "tokens": 239,
+          "logprob_sum": -710.4111919912539
+        },
+        {
+          "id": 25,
+          "language": "en",
+          "tokens": 293,
+          "logprob_sum": -489.4903313313025
+        },
+        {
+          "id": 26,
+          "language": "en",
+          "tokens": 303,
+          "logprob_sum": -675.9598828403468
+        },
+        {
+          "id": 27,
+          "language": "en",
+          "tokens": 276,
+          "logprob_sum": -618.1935117283683
+        },
+        {
+          "id": 28,
+          "language": "en",
+          "tokens": 282,
+          "logprob_sum": -745.1290100208425
+        },
+        {
+          "id": 29,
+          "language": "en",
+          "tokens": 298,
+          "logprob_sum": -689.4636160820846
+        },
+        {
+          "id": 30,
+          "language": "en",
+          "tokens": 338,
+          "logprob_sum": -467.5889795224348
+        },
+        {
+          "id": 31,
+          "language": "en",
+          "tokens": 280,
+          "logprob_sum": -636.7247954699196
+        },
+        {
+          "id": 32,
+          "language": "en",
+          "tokens": 310,
+          "logprob_sum": -804.4559548924517
+        },
+        {
+          "id": 33,
+          "language": "en",
+          "tokens": 280,
+          "logprob_sum": -804.8273709801942
+        },
+        {
+          "id": 34,
+          "language": "en",
+          "tokens": 302,
+          "logprob_sum": -701.7936646631133
+        },
+        {
+          "id": 35,
+          "language": "en",
+          "tokens": 286,
+          "logprob_sum": -537.7564877930354
+        },
+        {
+          "id": 36,
+          "language": "en",
+          "tokens": 244,
+          "logprob_sum": -588.5291649973951
+        },
+        {
+          "id": 37,
+          "language": "en",
+          "tokens": 280,
+          "logprob_sum": -497.2685717854147
+        },
+        {
+          "id": 38,
+          "language": "en",
+          "tokens": 315,
+          "logprob_sum": -604.644531319529
+        },
+        {
+          "id": 39,
+          "language": "en",
+          "tokens": 283,
+          "logprob_sum": -656.118171527778
+        },
+        {
+          "id": 80,
+          "language": "code",
+          "tokens": 320,
+          "logprob_sum": -272.59314406909556
+        },
+        {
+          "id": 81,
+          "language": "code",
+          "tokens": 282,
+          "logprob_sum": -430.0411255940244
+        },
+        {
+          "id": 82,
+          "language": "code",
+          "tokens": 278,
+          "logprob_sum": -259.4892357758499
+        },
+        {
+          "id": 83,
+          "language": "code",
+          "tokens": 273,
+          "logprob_sum": -414.21677607048605
+        },
+        {
+          "id": 84,
+          "language": "code",
+          "tokens": 253,
+          "logprob_sum": -206.03995015959322
+        },
+        {
+          "id": 85,
+          "language": "code",
+          "tokens": 239,
+          "logprob_sum": -463.0410933274834
+        },
+        {
+          "id": 86,
+          "language": "code",
+          "tokens": 272,
+          "logprob_sum": -385.2234219837974
+        },
+        {
+          "id": 87,
+          "language": "code",
+          "tokens": 253,
+          "logprob_sum": -287.78074731933157
+        },
+        {
+          "id": 88,
+          "language": "code",
+          "tokens": 317,
+          "logprob_sum": -285.0069392228546
+        },
+        {
+          "id": 89,
+          "language": "code",
+          "tokens": 298,
+          "logprob_sum": -295.7226561442835
+        },
+        {
+          "id": 90,
+          "language": "code",
+          "tokens": 280,
+          "logprob_sum": -316.34263046231933
+        },
+        {
+          "id": 91,
+          "language": "code",
+          "tokens": 265,
+          "logprob_sum": -386.3777894470843
+        },
+        {
+          "id": 92,
+          "language": "code",
+          "tokens": 305,
+          "logprob_sum": -220.36317087604692
+        },
+        {
+          "id": 93,
+          "language": "code",
+          "tokens": 310,
+          "logprob_sum": -361.5046516778
+        },
+        {
+          "id": 94,
+          "language": "code",
+          "tokens": 278,
+          "logprob_sum": -243.23737413886033
+        },
+        {
+          "id": 95,
+          "language": "code",
+          "tokens": 301,
+          "logprob_sum": -320.9270023946732
+        },
+        {
+          "id": 96,
+          "language": "code",
+          "tokens": 311,
+          "logprob_sum": -186.95426371483813
+        },
+        {
+          "id": 97,
+          "language": "code",
+          "tokens": 312,
+          "logprob_sum": -229.6340473171649
+        },
+        {
+          "id": 98,
+          "language": "code",
+          "tokens": 325,
+          "logprob_sum": -196.01683645374595
+        },
+        {
+          "id": 99,
+          "language": "code",
+          "tokens": 291,
+          "logprob_sum": -323.4737405007072
+        },
+        {
+          "id": 100,
+          "language": "code",
+          "tokens": 321,
+          "logprob_sum": -447.75894075650376
+        },
+        {
+          "id": 101,
+          "language": "code",
+          "tokens": 318,
+          "logprob_sum": -452.00106130935956
+        },
+        {
+          "id": 102,
+          "language": "code",
+          "tokens": 316,
+          "logprob_sum": -523.678944430956
+        },
+        {
+          "id": 103,
+          "language": "code",
+          "tokens": 307,
+          "logprob_sum": -308.9857649166101
+        },
+        {
+          "id": 104,
+          "language": "code",
+          "tokens": 254,
+          "logprob_sum": -407.14345852988436
+        },
+        {
+          "id": 105,
+          "language": "code",
+          "tokens": 275,
+          "logprob_sum": -509.09120340238167
+        }
+      ],
+      "tokens": 18729,
+      "ppl": 6.6431833085780445
+    },
+    "swift15-fast": {
+      "source_file": "swift15-fast/perplexity.json",
+      "source_sha256": "3fa2ac71c6c24072036837c3ae0afcf96ce03e85a09b276874c75432aeb091a1",
+      "input_sha256": "57c83ffe7c0dfba2b869f6a379d2df3f0bc0d055069b49569a2154336151a718",
+      "windows": [
+        {
+          "id": 0,
+          "language": "en",
+          "tokens": 283,
+          "logprob_sum": -588.6054017473216
+        },
+        {
+          "id": 1,
+          "language": "en",
+          "tokens": 313,
+          "logprob_sum": -694.9624558154828
+        },
+        {
+          "id": 2,
+          "language": "en",
+          "tokens": 284,
+          "logprob_sum": -725.915981323611
+        },
+        {
+          "id": 3,
+          "language": "en",
+          "tokens": 286,
+          "logprob_sum": -775.7597814282599
+        },
+        {
+          "id": 4,
+          "language": "en",
+          "tokens": 279,
+          "logprob_sum": -569.1412088309735
+        },
+        {
+          "id": 5,
+          "language": "en",
+          "tokens": 251,
+          "logprob_sum": -647.5557491235522
+        },
+        {
+          "id": 6,
+          "language": "en",
+          "tokens": 258,
+          "logprob_sum": -616.5903292949806
+        },
+        {
+          "id": 7,
+          "language": "en",
+          "tokens": 258,
+          "logprob_sum": -599.9040577597189
+        },
+        {
+          "id": 8,
+          "language": "en",
+          "tokens": 274,
+          "logprob_sum": -648.8657056111551
+        },
+        {
+          "id": 9,
+          "language": "en",
+          "tokens": 278,
+          "logprob_sum": -619.8002782169642
+        },
+        {
+          "id": 10,
+          "language": "en",
+          "tokens": 274,
+          "logprob_sum": -755.9231413470116
+        },
+        {
+          "id": 11,
+          "language": "en",
+          "tokens": 288,
+          "logprob_sum": -784.8980654559855
+        },
+        {
+          "id": 12,
+          "language": "en",
+          "tokens": 277,
+          "logprob_sum": -623.0504599666219
+        },
+        {
+          "id": 13,
+          "language": "en",
+          "tokens": 291,
+          "logprob_sum": -713.9815500143741
+        },
+        {
+          "id": 14,
+          "language": "en",
+          "tokens": 260,
+          "logprob_sum": -674.8367498576463
+        },
+        {
+          "id": 15,
+          "language": "en",
+          "tokens": 250,
+          "logprob_sum": -774.9568990021071
+        },
+        {
+          "id": 16,
+          "language": "en",
+          "tokens": 252,
+          "logprob_sum": -761.2714033735101
+        },
+        {
+          "id": 17,
+          "language": "en",
+          "tokens": 272,
+          "logprob_sum": -787.7554339407579
+        },
+        {
+          "id": 18,
+          "language": "en",
+          "tokens": 240,
+          "logprob_sum": -678.0826184380858
+        },
+        {
+          "id": 19,
+          "language": "en",
+          "tokens": 244,
+          "logprob_sum": -568.7623678010859
+        },
+        {
+          "id": 20,
+          "language": "en",
+          "tokens": 274,
+          "logprob_sum": -686.8405574182398
+        },
+        {
+          "id": 21,
+          "language": "en",
+          "tokens": 299,
+          "logprob_sum": -732.8032818522224
+        },
+        {
+          "id": 22,
+          "language": "en",
+          "tokens": 318,
+          "logprob_sum": -834.4363407381461
+        },
+        {
+          "id": 23,
+          "language": "en",
+          "tokens": 263,
+          "logprob_sum": -681.1934023356298
+        },
+        {
+          "id": 24,
+          "language": "en",
+          "tokens": 239,
+          "logprob_sum": -711.196296677168
+        },
+        {
+          "id": 25,
+          "language": "en",
+          "tokens": 293,
+          "logprob_sum": -489.5727469942158
+        },
+        {
+          "id": 26,
+          "language": "en",
+          "tokens": 303,
+          "logprob_sum": -677.6055709071516
+        },
+        {
+          "id": 27,
+          "language": "en",
+          "tokens": 276,
+          "logprob_sum": -617.7064340228862
+        },
+        {
+          "id": 28,
+          "language": "en",
+          "tokens": 282,
+          "logprob_sum": -747.541580047051
+        },
+        {
+          "id": 29,
+          "language": "en",
+          "tokens": 298,
+          "logprob_sum": -691.5382801084379
+        },
+        {
+          "id": 30,
+          "language": "en",
+          "tokens": 338,
+          "logprob_sum": -470.8770159313949
+        },
+        {
+          "id": 31,
+          "language": "en",
+          "tokens": 280,
+          "logprob_sum": -637.8394971183043
+        },
+        {
+          "id": 32,
+          "language": "en",
+          "tokens": 310,
+          "logprob_sum": -804.141806322179
+        },
+        {
+          "id": 33,
+          "language": "en",
+          "tokens": 280,
+          "logprob_sum": -807.7271480543714
+        },
+        {
+          "id": 34,
+          "language": "en",
+          "tokens": 302,
+          "logprob_sum": -702.7786499263748
+        },
+        {
+          "id": 35,
+          "language": "en",
+          "tokens": 286,
+          "logprob_sum": -540.5276784175985
+        },
+        {
+          "id": 36,
+          "language": "en",
+          "tokens": 244,
+          "logprob_sum": -590.4049851828822
+        },
+        {
+          "id": 37,
+          "language": "en",
+          "tokens": 280,
+          "logprob_sum": -503.33479981976416
+        },
+        {
+          "id": 38,
+          "language": "en",
+          "tokens": 315,
+          "logprob_sum": -606.4029330076664
+        },
+        {
+          "id": 39,
+          "language": "en",
+          "tokens": 283,
+          "logprob_sum": -657.8862001403468
+        },
+        {
+          "id": 80,
+          "language": "code",
+          "tokens": 320,
+          "logprob_sum": -271.8203841643988
+        },
+        {
+          "id": 81,
+          "language": "code",
+          "tokens": 282,
+          "logprob_sum": -430.6886993729022
+        },
+        {
+          "id": 82,
+          "language": "code",
+          "tokens": 278,
+          "logprob_sum": -257.2257569156982
+        },
+        {
+          "id": 83,
+          "language": "code",
+          "tokens": 273,
+          "logprob_sum": -413.2185932852717
+        },
+        {
+          "id": 84,
+          "language": "code",
+          "tokens": 253,
+          "logprob_sum": -207.92188511058242
+        },
+        {
+          "id": 85,
+          "language": "code",
+          "tokens": 239,
+          "logprob_sum": -466.56151770966244
+        },
+        {
+          "id": 86,
+          "language": "code",
+          "tokens": 272,
+          "logprob_sum": -386.5245793041249
+        },
+        {
+          "id": 87,
+          "language": "code",
+          "tokens": 253,
+          "logprob_sum": -288.5439935779359
+        },
+        {
+          "id": 88,
+          "language": "code",
+          "tokens": 317,
+          "logprob_sum": -284.2035732616263
+        },
+        {
+          "id": 89,
+          "language": "code",
+          "tokens": 298,
+          "logprob_sum": -295.2464475941133
+        },
+        {
+          "id": 90,
+          "language": "code",
+          "tokens": 280,
+          "logprob_sum": -319.10645955688994
+        },
+        {
+          "id": 91,
+          "language": "code",
+          "tokens": 265,
+          "logprob_sum": -388.01281921545444
+        },
+        {
+          "id": 92,
+          "language": "code",
+          "tokens": 305,
+          "logprob_sum": -222.33547965546757
+        },
+        {
+          "id": 93,
+          "language": "code",
+          "tokens": 310,
+          "logprob_sum": -362.4325922049678
+        },
+        {
+          "id": 94,
+          "language": "code",
+          "tokens": 278,
+          "logprob_sum": -244.08122701483012
+        },
+        {
+          "id": 95,
+          "language": "code",
+          "tokens": 301,
+          "logprob_sum": -322.9636465795079
+        },
+        {
+          "id": 96,
+          "language": "code",
+          "tokens": 311,
+          "logprob_sum": -185.14480259043967
+        },
+        {
+          "id": 97,
+          "language": "code",
+          "tokens": 312,
+          "logprob_sum": -230.45695572777822
+        },
+        {
+          "id": 98,
+          "language": "code",
+          "tokens": 325,
+          "logprob_sum": -198.365868733448
+        },
+        {
+          "id": 99,
+          "language": "code",
+          "tokens": 291,
+          "logprob_sum": -326.1211774999374
+        },
+        {
+          "id": 100,
+          "language": "code",
+          "tokens": 321,
+          "logprob_sum": -447.167507823165
+        },
+        {
+          "id": 101,
+          "language": "code",
+          "tokens": 318,
+          "logprob_sum": -456.6576964538326
+        },
+        {
+          "id": 102,
+          "language": "code",
+          "tokens": 316,
+          "logprob_sum": -525.123810860335
+        },
+        {
+          "id": 103,
+          "language": "code",
+          "tokens": 307,
+          "logprob_sum": -312.1443012760128
+        },
+        {
+          "id": 104,
+          "language": "code",
+          "tokens": 254,
+          "logprob_sum": -408.9510145225404
+        },
+        {
+          "id": 105,
+          "language": "code",
+          "tokens": 275,
+          "logprob_sum": -512.3511249634262
+        }
+      ],
+      "tokens": 18729,
+      "ppl": 6.679203244518718
+    }
+  }
+}