Add frozen selected-result prompt comparison and clarify model scope

.gitattributes

@@ -1,2 +1,4 @@
 *.pt filter=lfs diff=lfs merge=lfs -text
 *.whl filter=lfs diff=lfs merge=lfs -text
+comparison/records.jsonl.gz filter=lfs diff=lfs merge=lfs -text
+comparison/responses.jsonl.gz filter=lfs diff=lfs merge=lfs -text

MANIFEST.sha256

@@ -1,10 +1,15 @@
 cf525d607a58b825e4a124bd5511bcfa00d257a22525f0bec81ad4dbbd1d2cfb  .gitattributes
 9ab279df0ec2f39a82fe6cde3b4bf205a2fdd48b047e3cfdf77df44501b6d8a9  BASE_MODEL.json
 01c3d96207e01603bc83082edb665ddef6edde36d96a81caa70efa5fad444177  LICENSE
-200791cbfc25a84e0cef71e4bb7ed74b9dd9c766da5734e02e0d8077ef40b653  README.md
+89a2b36672b37365afbf6859b42d1265a5bc4fa3c7f3395973df98cd9b05c726  README.md
 6f0a5f6bbb660fbdbdd2695bdd93b9e5c53d4fcb79cb4d16a3f62a1b7bbbac9e  checkpoints/ACTION_HEAD_FINAL.pt
 75b595cafc25cb87bcfd0fdac33965ba5efaf80f691c802549693fe127c9cf7b  checkpoints/MEMORY_PATH_FINAL.pt
 60fa8569aec9d3344abb20b1bf8d3ade7ad11426b4498412f7567cc4bf4add49  checkpoints/P0_M2_CHECKPOINT_FINAL.pt
+6a4ec82defbc8196be35749a2d1902726020d5527d07c8259f05317d2bd2ab5e  comparison/benchmark.py
+adc0d5141f391fdbb7cd6f96bf3cfffd25be9d0a01baf3aba334dd0610cf5835  comparison/protocol.json
+987c3936971579b07fbee9a940960d88c9a0e95277bed51068284bca90aeda98  comparison/records.jsonl.gz
+0d78a9a2ba8b95f869efb0a89538d8a156786d60e879dcf2dc067aa71b5d06c0  comparison/responses.jsonl.gz
+e332d0bb19b1e9c61cce5075356f7214361d77f7a2baad692d654ee4d4a6bf5a  comparison/results.json
 f0ca13dc5c0bbfc7852df6f2cee36d4039f7c2197afb8840a3335211862d5e67  configs/strata_native_lm_integration_m1.json
 d0ed976b9409ba9123a7b3ca2e16616a4f3e19daa1bf18ca6b042f7918c7eb7c  configs/strata_native_lm_p0_m2_v1.json
 d97c93bb90ea2c2217820d7e1eafac61cdab16ff28a2189f97b7918b6ffebde2  configs/strata_native_lm_system_v1.json

README.md

@@ -15,7 +15,7 @@ tags:
 
 Licensed under [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/) for non-commercial use. Commercial use requires separate permission from the rights holders. The upstream base model and third-party dependencies retain their respective licenses.
 
-STRATA Native LM couples a frozen Qwen3-4B-Instruct-2507 backbone to a trained 20.44M-parameter memory interface. It generates answers from trusted structured observations supplied by tools, APIs, or applications. Structured writes and read addresses are supplied explicitly.
+STRATA Native LM couples a frozen Qwen3-4B-Instruct-2507 backbone to a trained 20.44M-parameter memory interface. It generates short factual responses from trusted structured observations supplied by tools, APIs, or applications. Structured writes and read addresses are supplied explicitly. The runtime executes joins; the language model verbalizes their selected results. Open-ended response generation was not evaluated.
 
 ```text
 structured observation
@@ -25,7 +25,9 @@ structured observation
   -> generated frame + exact value copying
 ```
 
-The model receives the current query and compact memory without previous source tokens, source-history KV-cache entries, or persisted source residuals. Stored value bytes are copied only after the response frame is complete. Integrity checks reject invalid, stale, deleted, or corrupted references.
+The reader decodes and latches an address-bound value code. During response-frame generation the actual value state is replaced by a fixed neutral state, and COPY validates the selected record. Its bytes are inserted after frame completion. No previous source tokens, source-history KV-cache entries, persisted source residuals, or actual value tokens enter frame generation. Integrity checks reject invalid, stale, deleted, or corrupted references.
+
+The 255 nonzero value codes are reused across records, not global value identifiers. Structured-service adapters assign a field index; persistent identity comes from the full instance/predicate/role address and version. The evaluated copy bank contains one runtime-selected record per request. Arbitrary application bytes remain in that record and are not encoded by the 8-bit code.
 
 ## Results
 
@@ -38,16 +40,42 @@ The model receives the current query and compact memory without previous source
 | Persistent-memory horizon | 128 independent 8,192-token windows |
 | Factual accuracy | 1.000 |
 | Copied-value byte correctness | 1.000 |
-| Held-out-schema accuracy | 1.000 |
-| 2/3/4-hop join accuracy | 1.000 |
+| Three schemas excluded from component training | 768 unique records; 11,520 evaluations; 1.000 |
+| Runtime-executed 2/3/4-hop joins | 1.000 |
 | Field-specific intervention following | 1.000 |
 | Integrity verification and execution replay | 1.000 |
-| Supporting proof-inclusive retained-state reductions | 86.46% / 86.72% |
+| Structured-service retained-state reduction, including values and integrity metadata | At least 86.72% |
 | Neural value-state reduction | 95.83% (128 vs. 3,072 bytes/value) |
 
 The neural-state figure excludes application value bytes and shared address/integrity metadata. The structured-service accounting includes those bytes. The horizon is persistence across independent windows, not a single Transformer context.
 
-STRATA and the realization-matched token-value arm both achieved 1.000 factual accuracy. On eight NVIDIA L40 GPUs, the evaluated implementation took 43.17 ms/record versus 564.82 ms/record for the source-history arm named `FULL_KV`. Both decoding loops disabled KV reuse; the latter used free generation. These timings describe that workload and implementation.
+STRATA and the realization-matched token-value arm both achieved 1.000 factual accuracy. The original uncached source-history timing remains in `evaluation.json` as an implementation diagnostic, not a speed claim against optimized cached serving. The three training-excluded schemas share the trained memory interface, operation inventory, and templated query format.
+
+## Selected-Result Prompt Comparison
+
+The same 4,096 unique selected current records were passed to the frozen base model as JSON in the current prompt, with within-request KV caching enabled. Both interfaces were evaluated twice on eight L40 GPUs without cross-request prefix caching.
+
+| Metric | STRATA | Prompt serialization |
+| --- | ---: | ---: |
+| Stored UTF-8 value reproduced exactly once | 4,096/4,096 | 4,051/4,096 |
+| Ordinary bindings and resolved joins | 3,328/3,328 | 3,328/3,328 |
+| Mean input tokens/request | 64.52 | 136.29 |
+| Batch-amortized ms/record | 37.81 | 117.51 |
+| Generation-cap cases | 0 | 8 |
+
+All 45 prompt failures were in the hard-value holdout: 36 explicit refusals, eight outputs capped at 256 generated tokens, and one other value mismatch. Prompt generation freely spells values; STRATA copies their bytes. These are measured interface tradeoffs, not an optimized-serving or retrieval-quality ranking. Both passes produced identical responses.
+
+The `comparison/` directory contains the selected records, sealed responses, protocol, results, and executable comparison. Recompute published scores without a GPU:
+
+```bash
+python3 comparison/benchmark.py --score-published
+```
+
+After installation and base-model download, reproduce both interfaces on eight GPUs:
+
+```bash
+python3 comparison/benchmark.py --base-model "$STRATA_BASE_MODEL" --output comparison-run
+```
 
 ## Install
 
@@ -90,6 +118,7 @@ The JSON response contains `answer`, `frame`, `status`, `payload_handle`, `recei
 | `checkpoints/` | Compact-memory reader, memory interface, and copy-action head |
 | `configs/` | Inference settings |
 | `evaluation.json` | Evaluation metrics, criteria, and source-artifact hashes |
+| `comparison/` | Selected-result baseline data, outputs, protocol, results, and runner |
 | `wheels/` | Runtime package |
 | `BASE_MODEL.json` | Required upstream model and immutable revision |
 | `load_and_answer.py` | Executable structured-read example |

comparison/benchmark.py

@@ -0,0 +1,167 @@
+#!/usr/bin/env python3
+"""Reproduce selected-result prompting versus the frozen STRATA response interface."""
+
+import argparse
+from collections import defaultdict
+import gzip
+import json
+import os
+from pathlib import Path
+import subprocess
+import sys
+import time
+
+ROOT = Path(__file__).resolve().parents[1]
+DATA = Path(__file__).resolve().parent
+sys.path.insert(0, str(ROOT))
+
+def selected_prompt(record, config):
+    selected = {key: record[key] for key in config['selected_fields']}
+    return config['prompt_prefix'] + json.dumps(selected, ensure_ascii=False, separators=(',', ':')) + config['prompt_suffix'] + record['query']
+
+def exact_value_once(text, value):
+    return text.encode('utf-8').count(value.encode('utf-8')) == 1
+
+def make_infer(torch, model, head, tokenizer, rows, records, config, system,
+               state_table, banks, versions, device):
+    from strata.eval.native_lm_benchmark import _prompt_ids
+    from strata.eval.native_lm_frame_separated_copy import frame_separated_generate
+    backbone = model.backbone
+
+    @torch.inference_mode()
+    def infer(arm, indices):
+        group = [rows[i] for i in indices]
+        torch.cuda.synchronize()
+        start = time.perf_counter()
+        if arm == 'PROMPT_SERIALIZATION':
+            sequences = [_prompt_ids(tokenizer, selected_prompt(records[i], config)) for i in indices]
+            width = max(map(len, sequences))
+            ids = torch.tensor([[tokenizer.pad_token_id] * (width - len(seq)) + seq for seq in sequences], device=device)
+            mask = torch.tensor([[0] * (width - len(seq)) + [1] * len(seq) for seq in sequences], device=device)
+            outputs = backbone.generate(input_ids=ids, attention_mask=mask, do_sample=False, use_cache=True, max_new_tokens=config['prompt_generation_max_new_tokens'], pad_token_id=tokenizer.pad_token_id, eos_token_id=tokenizer.eos_token_id)
+            texts = tokenizer.batch_decode(outputs[:, width:], skip_special_tokens=True)
+            generated_lengths = [len(tokens) - list(tokens).count(tokenizer.pad_token_id) for tokens in outputs[:, width:].cpu().tolist()]
+            truncated = [tokenizer.eos_token_id not in tokens for tokens in outputs[:, width:].cpu().tolist()]
+        else:
+            outputs, _ = frame_separated_generate(model, head, tokenizer, group, state_table, [banks[i] for i in indices], [0] * len(group), batch_size=len(group), max_actions=config['strata_max_actions'], frame_handle=system['frame']['canonical_frame_handle'], frame_surrogate=system['frame']['canonical_frame_surrogate'], terminator=system['frame']['structural_terminator'], current_versions=[versions[i] for i in indices])
+            texts = [output.text for output in outputs]
+            generated_lengths = [sum((action.startswith('GEN(') for action in output.actions)) for output in outputs]
+            truncated = [output.status == 'MAX_ACTIONS' for output in outputs]
+        torch.cuda.synchronize()
+        elapsed = time.perf_counter() - start
+        lengths = [len(_prompt_ids(tokenizer, selected_prompt(records[i], config) if arm == 'PROMPT_SERIALIZATION' else records[i]['query'])) for i in indices]
+        return (texts, lengths, generated_lengths, truncated, elapsed)
+    return infer
+
+def read_records():
+    with gzip.open(DATA/'records.jsonl.gz', 'rt', encoding='utf-8') as handle:
+        return [json.loads(line) for line in handle]
+
+
+def score(paths):
+    gold = {row['id']:row for row in read_records()}
+    scores = defaultdict(lambda: {'correct':0,'total':0,'input_tokens':0,'truncated':0})
+    timings = {}
+    for path in paths:
+        opener = gzip.open if str(path).endswith('.gz') else open
+        with opener(path, 'rt', encoding='utf-8') as handle:
+            for line in handle:
+                item = json.loads(line)
+                row = gold[item['id']]
+                value = scores[item['arm']]
+                value['correct'] += int(exact_value_once(item['text'],row['value']))
+                value['total'] += 1
+                value['input_tokens'] += item['input_tokens']
+                value['truncated'] += item['truncated']
+                timings[(item['arm'],item['rank'],item['repeat'],item['batch_offset'])] = item['batch_seconds']
+    for arm,value in scores.items():
+        value['accuracy'] = value['correct']/value['total']
+        value['mean_input_tokens'] = value['input_tokens']/value['total']
+        value['amortized_ms_per_record'] = sum(t for key,t in timings.items() if key[0]==arm)*1000/value['total']
+    return dict(scores)
+
+
+def worker(args, config):
+    import torch
+    from load_and_answer import load_model
+    from strata.data.native_lm_integration import NativeLMExample, address_codes, answer_text
+    from strata.modeling.exact_payload_realizer import PayloadAuthority
+    from strata.training.native_lm_integration import compact_state_table
+    torch.set_num_threads(2)
+    torch.cuda.set_device(args.rank)
+    torch.manual_seed(config['seed'])
+    device = torch.device('cuda',args.rank)
+    system,tokenizer,model,head,codec = load_model(ROOT,args.base_model,device)
+    state_table = compact_state_table(codec)
+    records = [row for row in read_records() if row['rank']==args.rank]
+    if args.limit is not None:
+        records = records[:args.limit]
+    rows = [NativeLMExample(example_id=r['id'],split='system-v1',schema=r['schema'],field=r['field'],
+                event=r['event'],predicate=r['predicate'],role=r['role'],value_type=r['value_type'],
+                payload_handle=r['payload_handle'],value=r['value'],
+                address_codes=address_codes(r['event'],r['predicate'],r['role']),query=r['query'],
+                full_history_query='',answer=answer_text(r['field'],r['value']),operation=r['operation'],age_windows=0)
+            for r in records]
+    versions = [r['event_version'] for r in records]
+    banks = [[PayloadAuthority.issue(event=r.event,predicate=r.predicate,role=r.role,
+                handle=r.payload_handle,payload=r.value,version=v)] for r,v in zip(rows,versions)]
+    infer = make_infer(torch,model,head,tokenizer,rows,records,config,system,state_table,banks,versions,device)
+    warm = list(range(min(config['batch_size'],len(rows))))
+    for arm in ['PROMPT_SERIALIZATION','STRATA_NATIVE_V1']:
+        infer(arm,warm)
+    path = args.output/f'rank-{args.rank}.jsonl'
+    with path.open('x',encoding='utf-8') as handle:
+        for repeat in range(config['repeats']):
+            for offset in range(0,len(rows),config['batch_size']):
+                indices = list(range(offset,min(offset+config['batch_size'],len(rows))))
+                arms = ['PROMPT_SERIALIZATION','STRATA_NATIVE_V1']
+                if (args.rank+repeat+offset//config['batch_size'])%2:
+                    arms.reverse()
+                for arm in arms:
+                    texts,lengths,generated,truncated,elapsed = infer(arm,indices)
+                    for index,text,length,n,cutoff in zip(indices,texts,lengths,generated,truncated):
+                        handle.write(json.dumps({'id':records[index]['id'],'rank':args.rank,'repeat':repeat,
+                            'arm':arm,'text':text,'input_tokens':length,'generated_tokens':n,'truncated':cutoff,
+                            'batch_offset':offset,'batch_seconds':elapsed,'batch_size':len(indices)},ensure_ascii=False)+'\n')
+
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--score-published',action='store_true')
+    parser.add_argument('--base-model',default=os.environ.get('STRATA_BASE_MODEL'))
+    parser.add_argument('--output',type=Path)
+    parser.add_argument('--rank',type=int,choices=range(8))
+    parser.add_argument('--limit',type=int)
+    args = parser.parse_args()
+    if args.score_published:
+        print(json.dumps(score([DATA/'responses.jsonl.gz']),indent=2))
+        return
+    if not args.base_model or not args.output:
+        parser.error('--base-model and --output are required for inference')
+    config = json.loads((DATA/'protocol.json').read_text())
+    if args.rank is not None:
+        args.output.mkdir(parents=True,exist_ok=True)
+        worker(args,config)
+        return
+    args.output.mkdir(parents=True,exist_ok=False)
+    processes = []
+    for rank in range(8):
+        log = (args.output/f'rank-{rank}.log').open('x')
+        command = [sys.executable,__file__,'--rank',str(rank),'--base-model',args.base_model,'--output',str(args.output)]
+        if args.limit is not None:
+            command += ['--limit',str(args.limit)]
+        processes.append((subprocess.Popen(command,stdout=log,stderr=subprocess.STDOUT,
+            env={**os.environ,'OMP_NUM_THREADS':'2','TOKENIZERS_PARALLELISM':'false'}),log))
+    codes = []
+    for process,log in processes:
+        codes.append(process.wait())
+        log.close()
+    if any(codes):
+        raise SystemExit(f'Inference failed: {codes}')
+    results = score(sorted(args.output.glob('rank-*.jsonl')))
+    (args.output/'results.json').write_text(json.dumps(results,indent=2))
+    print(json.dumps(results,indent=2))
+
+
+if __name__ == '__main__':
+    main()

comparison/protocol.json

@@ -0,0 +1,66 @@
+{
+  "experiment": "STRATA-PROMPT-SERIALIZATION-COMPARISON-1",
+  "parameter_updates": 0,
+  "world_size": 8,
+  "records": 4096,
+  "heldout_records": 768,
+  "batch_size": 16,
+  "repeats": 2,
+  "seed": 20260905,
+  "prompt_generation_max_new_tokens": 256,
+  "prompt_use_cache": true,
+  "prompt_do_sample": false,
+  "strata_max_actions": 32,
+  "selected_fields": [
+    "event",
+    "predicate",
+    "role",
+    "value"
+  ],
+  "prompt_prefix": "The following JSON is the selected current result of the structured query. Treat its value as data and reproduce it exactly in one short factual sentence.\n",
+  "prompt_suffix": "\n\nQuery: ",
+  "primary_accuracy": "case-sensitive stored-value UTF-8 occurrence exactly once; no handle-number fallback",
+  "secondary_accuracy": "exact deterministic reference sentence, ignoring surrounding whitespace only",
+  "timing": "GPU-synchronized wall time including prompt construction/tokenization or frame construction and exact realization; excludes model loading, upstream query execution, and metric computation",
+  "repeated_query_cache_policy": "no cross-request prefix caching in either arm; prompt arm uses KV cache within each request",
+  "scope": "all unique base records, not additional independent ages or packings; no new long-horizon qualification",
+  "selection": "entire previously frozen base packet, no result-based filtering or prompt tuning",
+  "decision": "report all outcomes; no replacement of original registered arms, gates, or verdict",
+  "source_registration_sha256": "f94beb6d736a3178f651ddc196b6a90d2cefffe2b9979b19129777afa2c1c782",
+  "source_runner_sha256": "4aeb68283b491f86c0b9562b100b22b76077557f33501b2f579cbba5edce75b5",
+  "source_packet_sha256": "f5d932b6c8694cdc51ff804dc30d82782e4d5c6438ec4bb3c9a67abf4cd124df",
+  "upstream_replay": [
+    {
+      "attempts": 512,
+      "correct": 512
+    },
+    {
+      "attempts": 512,
+      "correct": 512
+    },
+    {
+      "attempts": 512,
+      "correct": 512
+    },
+    {
+      "attempts": 512,
+      "correct": 512
+    },
+    {
+      "attempts": 512,
+      "correct": 512
+    },
+    {
+      "attempts": 512,
+      "correct": 512
+    },
+    {
+      "attempts": 512,
+      "correct": 512
+    },
+    {
+      "attempts": 512,
+      "correct": 512
+    }
+  ]
+}

comparison/records.jsonl.gz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:987c3936971579b07fbee9a940960d88c9a0e95277bed51068284bca90aeda98
+size 127689

comparison/responses.jsonl.gz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0d78a9a2ba8b95f869efb0a89538d8a156786d60e879dcf2dc067aa71b5d06c0
+size 294516

comparison/results.json

@@ -0,0 +1,406 @@
+{
+  "experiment": "STRATA-PROMPT-SERIALIZATION-COMPARISON-1",
+  "unique_records": 4096,
+  "repeats": 2,
+  "registration_sha256": "f94beb6d736a3178f651ddc196b6a90d2cefffe2b9979b19129777afa2c1c782",
+  "arms": {
+    "PROMPT_SERIALIZATION": {
+      "correct": 8102,
+      "total": 8192,
+      "reference_exact": 0,
+      "input_tokens": 1116502,
+      "truncated": 16,
+      "accuracy": 0.989013671875,
+      "mean_input_tokens": 136.291748046875,
+      "seconds": 962.6804165355861,
+      "amortized_ms_per_record": 117.51469928412916,
+      "median_batch_amortized_ms": 62.35561228822917,
+      "p95_batch_amortized_ms": 417.074111988768,
+      "repeated_response_identity": 1.0
+    },
+    "STRATA_NATIVE_V1": {
+      "correct": 8192,
+      "total": 8192,
+      "reference_exact": 8110,
+      "input_tokens": 528536,
+      "truncated": 0,
+      "accuracy": 1.0,
+      "mean_input_tokens": 64.5185546875,
+      "seconds": 309.70214548148215,
+      "amortized_ms_per_record": 37.80543768084499,
+      "median_batch_amortized_ms": 37.54055639728904,
+      "p95_batch_amortized_ms": 42.57394268643111,
+      "repeated_response_identity": 1.0
+    }
+  },
+  "slices": [
+    {
+      "arm": "PROMPT_SERIALIZATION",
+      "axis": "graph_depth",
+      "value": "1",
+      "correct": 6566,
+      "total": 6656,
+      "accuracy": 0.9864783653846154
+    },
+    {
+      "arm": "PROMPT_SERIALIZATION",
+      "axis": "graph_depth",
+      "value": "2",
+      "correct": 512,
+      "total": 512,
+      "accuracy": 1.0
+    },
+    {
+      "arm": "PROMPT_SERIALIZATION",
+      "axis": "graph_depth",
+      "value": "3",
+      "correct": 512,
+      "total": 512,
+      "accuracy": 1.0
+    },
+    {
+      "arm": "PROMPT_SERIALIZATION",
+      "axis": "graph_depth",
+      "value": "4",
+      "correct": 512,
+      "total": 512,
+      "accuracy": 1.0
+    },
+    {
+      "arm": "PROMPT_SERIALIZATION",
+      "axis": "kind",
+      "value": "binding",
+      "correct": 5120,
+      "total": 5120,
+      "accuracy": 1.0
+    },
+    {
+      "arm": "PROMPT_SERIALIZATION",
+      "axis": "kind",
+      "value": "heldout",
+      "correct": 1446,
+      "total": 1536,
+      "accuracy": 0.94140625
+    },
+    {
+      "arm": "PROMPT_SERIALIZATION",
+      "axis": "kind",
+      "value": "join",
+      "correct": 1536,
+      "total": 1536,
+      "accuracy": 1.0
+    },
+    {
+      "arm": "PROMPT_SERIALIZATION",
+      "axis": "schema",
+      "value": "approval",
+      "correct": 492,
+      "total": 494,
+      "accuracy": 0.9959514170040485
+    },
+    {
+      "arm": "PROMPT_SERIALIZATION",
+      "axis": "schema",
+      "value": "calendar",
+      "correct": 1024,
+      "total": 1024,
+      "accuracy": 1.0
+    },
+    {
+      "arm": "PROMPT_SERIALIZATION",
+      "axis": "schema",
+      "value": "crm",
+      "correct": 1536,
+      "total": 1536,
+      "accuracy": 1.0
+    },
+    {
+      "arm": "PROMPT_SERIALIZATION",
+      "axis": "schema",
+      "value": "deployment",
+      "correct": 516,
+      "total": 524,
+      "accuracy": 0.9847328244274809
+    },
+    {
+      "arm": "PROMPT_SERIALIZATION",
+      "axis": "schema",
+      "value": "incident",
+      "correct": 1536,
+      "total": 1536,
+      "accuracy": 1.0
+    },
+    {
+      "arm": "PROMPT_SERIALIZATION",
+      "axis": "schema",
+      "value": "inventory",
+      "correct": 1024,
+      "total": 1024,
+      "accuracy": 1.0
+    },
+    {
+      "arm": "PROMPT_SERIALIZATION",
+      "axis": "schema",
+      "value": "project",
+      "correct": 1536,
+      "total": 1536,
+      "accuracy": 1.0
+    },
+    {
+      "arm": "PROMPT_SERIALIZATION",
+      "axis": "schema",
+      "value": "shipment",
+      "correct": 438,
+      "total": 518,
+      "accuracy": 0.8455598455598455
+    },
+    {
+      "arm": "PROMPT_SERIALIZATION",
+      "axis": "schema_partition",
+      "value": "heldout",
+      "correct": 1446,
+      "total": 1536,
+      "accuracy": 0.94140625
+    },
+    {
+      "arm": "PROMPT_SERIALIZATION",
+      "axis": "schema_partition",
+      "value": "registered",
+      "correct": 6656,
+      "total": 6656,
+      "accuracy": 1.0
+    },
+    {
+      "arm": "PROMPT_SERIALIZATION",
+      "axis": "value_type",
+      "value": "datetime",
+      "correct": 654,
+      "total": 654,
+      "accuracy": 1.0
+    },
+    {
+      "arm": "PROMPT_SERIALIZATION",
+      "axis": "value_type",
+      "value": "entity",
+      "correct": 3434,
+      "total": 3436,
+      "accuracy": 0.9994179278230501
+    },
+    {
+      "arm": "PROMPT_SERIALIZATION",
+      "axis": "value_type",
+      "value": "enum",
+      "correct": 2008,
+      "total": 2008,
+      "accuracy": 1.0
+    },
+    {
+      "arm": "PROMPT_SERIALIZATION",
+      "axis": "value_type",
+      "value": "integer",
+      "correct": 612,
+      "total": 692,
+      "accuracy": 0.884393063583815
+    },
+    {
+      "arm": "PROMPT_SERIALIZATION",
+      "axis": "value_type",
+      "value": "location",
+      "correct": 956,
+      "total": 964,
+      "accuracy": 0.991701244813278
+    },
+    {
+      "arm": "PROMPT_SERIALIZATION",
+      "axis": "value_type",
+      "value": "string",
+      "correct": 438,
+      "total": 438,
+      "accuracy": 1.0
+    },
+    {
+      "arm": "STRATA_NATIVE_V1",
+      "axis": "graph_depth",
+      "value": "1",
+      "correct": 6656,
+      "total": 6656,
+      "accuracy": 1.0
+    },
+    {
+      "arm": "STRATA_NATIVE_V1",
+      "axis": "graph_depth",
+      "value": "2",
+      "correct": 512,
+      "total": 512,
+      "accuracy": 1.0
+    },
+    {
+      "arm": "STRATA_NATIVE_V1",
+      "axis": "graph_depth",
+      "value": "3",
+      "correct": 512,
+      "total": 512,
+      "accuracy": 1.0
+    },
+    {
+      "arm": "STRATA_NATIVE_V1",
+      "axis": "graph_depth",
+      "value": "4",
+      "correct": 512,
+      "total": 512,
+      "accuracy": 1.0
+    },
+    {
+      "arm": "STRATA_NATIVE_V1",
+      "axis": "kind",
+      "value": "binding",
+      "correct": 5120,
+      "total": 5120,
+      "accuracy": 1.0
+    },
+    {
+      "arm": "STRATA_NATIVE_V1",
+      "axis": "kind",
+      "value": "heldout",
+      "correct": 1536,
+      "total": 1536,
+      "accuracy": 1.0
+    },
+    {
+      "arm": "STRATA_NATIVE_V1",
+      "axis": "kind",
+      "value": "join",
+      "correct": 1536,
+      "total": 1536,
+      "accuracy": 1.0
+    },
+    {
+      "arm": "STRATA_NATIVE_V1",
+      "axis": "schema",
+      "value": "approval",
+      "correct": 494,
+      "total": 494,
+      "accuracy": 1.0
+    },
+    {
+      "arm": "STRATA_NATIVE_V1",
+      "axis": "schema",
+      "value": "calendar",
+      "correct": 1024,
+      "total": 1024,
+      "accuracy": 1.0
+    },
+    {
+      "arm": "STRATA_NATIVE_V1",
+      "axis": "schema",
+      "value": "crm",
+      "correct": 1536,
+      "total": 1536,
+      "accuracy": 1.0
+    },
+    {
+      "arm": "STRATA_NATIVE_V1",
+      "axis": "schema",
+      "value": "deployment",
+      "correct": 524,
+      "total": 524,
+      "accuracy": 1.0
+    },
+    {
+      "arm": "STRATA_NATIVE_V1",
+      "axis": "schema",
+      "value": "incident",
+      "correct": 1536,
+      "total": 1536,
+      "accuracy": 1.0
+    },
+    {
+      "arm": "STRATA_NATIVE_V1",
+      "axis": "schema",
+      "value": "inventory",
+      "correct": 1024,
+      "total": 1024,
+      "accuracy": 1.0
+    },
+    {
+      "arm": "STRATA_NATIVE_V1",
+      "axis": "schema",
+      "value": "project",
+      "correct": 1536,
+      "total": 1536,
+      "accuracy": 1.0
+    },
+    {
+      "arm": "STRATA_NATIVE_V1",
+      "axis": "schema",
+      "value": "shipment",
+      "correct": 518,
+      "total": 518,
+      "accuracy": 1.0
+    },
+    {
+      "arm": "STRATA_NATIVE_V1",
+      "axis": "schema_partition",
+      "value": "heldout",
+      "correct": 1536,
+      "total": 1536,
+      "accuracy": 1.0
+    },
+    {
+      "arm": "STRATA_NATIVE_V1",
+      "axis": "schema_partition",
+      "value": "registered",
+      "correct": 6656,
+      "total": 6656,
+      "accuracy": 1.0
+    },
+    {
+      "arm": "STRATA_NATIVE_V1",
+      "axis": "value_type",
+      "value": "datetime",
+      "correct": 654,
+      "total": 654,
+      "accuracy": 1.0
+    },
+    {
+      "arm": "STRATA_NATIVE_V1",
+      "axis": "value_type",
+      "value": "entity",
+      "correct": 3436,
+      "total": 3436,
+      "accuracy": 1.0
+    },
+    {
+      "arm": "STRATA_NATIVE_V1",
+      "axis": "value_type",
+      "value": "enum",
+      "correct": 2008,
+      "total": 2008,
+      "accuracy": 1.0
+    },
+    {
+      "arm": "STRATA_NATIVE_V1",
+      "axis": "value_type",
+      "value": "integer",
+      "correct": 692,
+      "total": 692,
+      "accuracy": 1.0
+    },
+    {
+      "arm": "STRATA_NATIVE_V1",
+      "axis": "value_type",
+      "value": "location",
+      "correct": 964,
+      "total": 964,
+      "accuracy": 1.0
+    },
+    {
+      "arm": "STRATA_NATIVE_V1",
+      "axis": "value_type",
+      "value": "string",
+      "correct": 438,
+      "total": 438,
+      "accuracy": 1.0
+    }
+  ]
+}