# Lex: AMD batch capacity

## Optional larger batches

```python
from decision_inference import predict_1k, predict_auto_1k

# native and records use the same objects as the Python usage examples.
outputs = predict_1k(native, records)       # unchanged default: up to 8
outputs = predict_auto_1k(native, records)  # opt in: up to 32 with the padding guard
```

The optional entry admits every complete input before the first forward. It uses consecutive batches up to 32 only when the request has one decision type and total padded tokens do not increase relative to B8. Mixed types or increased padding fall back to B8; requests of eight or fewer take the unchanged default path. Input order, candidates, full 1,024-token limit and FP32 weights remain unchanged. It does not share contextual activations across questions. Larger physical batches can change floating-point rounding and peak memory.

### Paired B8/B32 study

These are synchronized resident **Python API measurements**, not Studio or network latency. This earlier study compared default B8 with a homogeneous cap32 policy before the final padding guard was added. All eight measured points satisfy that guard, but **the final `predict_auto_1k` entry was validated separately and was not timed**. Existing default-B8 measurements above, if present, remain their original separate run.

| Workload | B8 p50 / p95 (ms) | Cap32 p50 / p95 (ms) | p50 ratio · paired 95% CI |
|---|---:|---:|---:|
| 242 tokens × 1 question | 13.92 / 14.01 | 13.92 / 14.05 | 1.0001 · [0.9975, 1.0020] |
| 242 tokens × 8 questions | 22.65 / 22.75 | 22.60 / 22.77 | 0.9981 · [0.9967, 0.9994] |
| 242 tokens × 16 questions | 38.86 / 39.42 | 33.39 / 33.55 | 0.8592 · [0.8578, 0.8603] |
| 242 tokens × 32 questions | 71.03 / 71.46 | 54.83 / 60.15 | 0.7718 · [0.7706, 0.7724] |
| 1024 tokens × 1 question | 18.09 / 18.24 | 18.08 / 18.21 | 0.9992 · [0.9978, 1.0003] |
| 1024 tokens × 8 questions | 65.74 / 65.89 | 65.72 / 66.52 | 0.9997 · [0.9990, 1.0004] |
| 1024 tokens × 32 questions | 242.37 / 243.72 | 217.63 / 218.37 | 0.8979 · [0.8961, 0.8986] |
| Multi-context · 32 decisions | 44.08 / 44.25 | 28.76 / 28.87 | 0.6523 · [0.6519, 0.6537] |

AMD ROCm gfx942, FP32 SDPA, one resident model and one request at a time. Each point/mode had 10 warmups and 50 timed requests; the same validation, transfer and auditing hooks were present in both modes. Ten paired blocks (five AB, five BA; five observations per mode/block) were retained. The reported p50 ratio is median(cap32)/median(B8); 2,000 whole-block bootstrap resamples within order strata used fixed seed 20260922. p95 has only 50 observations/mode and correspondingly limited precision. Model loading is excluded.

Choice count curves repeat the same synthetic four-candidate question with opaque ID changes. The multi-context fixture repeats eight existing contexts with two related questions twice (32 decisions). These are throughput shapes, not quality examples. The fixed 15% utility gate remained **FAIL** for both models: Q16 improved approximately 14%, despite a confidence interval below 1. Q32 and multi-context improvements do not retroactively change that outcome. The option is an explicit engineering choice, not a universal speed guarantee.

At 1024×32, peak allocated memory increased from approximately 2.652 to 3.824 GiB. Shared allocator reserved-memory observations are not independent per-mode estimates. Equal total padding does not imply equal peak memory.

### Public-entry validation

The actual imported package passed 46 forwards per model (92 total across Kai and Lex) over 225 technical row-occurrences/model, including all three types, dynamic candidate counts, enabled32, tail33, unequal-length fallback and mixed64. Late invalid and empty requests added zero forwards. The original strict bounds were 1e-4 logits, 2e-5 probability and normalized Score error, with zero hard-decision flips. Both models passed; fallback outputs were exact. All 489 parameters and 66 buffers retained identical contents and versions. No training, quality evaluation or timing was added in this integration check. Fixed repetition is not independent quality support.

[Exact paired measurements, numerical results and retained failure gates](BATCH_CAPACITY.json). No weights or default entry behavior changed, and no Studio speedup is claimed.