README-PCD.md

# LFM2.5-2.6B-RLCD

**Fast finite-choice structured inference using unchanged LiquidAI/LFM2.5-2.6B weights.**
This package implements Parallel Constrained Decoding (PCD): prefill shared context once,
branch the model's attention **and convolution** state, evaluate allowed answers in parallel,
and serialize a typed JSON object in Python.

**Inference-only, experimental, and uncalibrated.** No training, LoRA, reinforcement learning,
quantization, or saved weight modification was performed. The `RLCD` name follows the community
inference examples; it does **not** mean we reproduced TypeSafe/Jev's Reinforcement Learning for
Calibrated Decisions. We target the fast finite-choice interaction pattern, not proprietary
training, calibration, API parity, or an unmeasured speed claim.

## Results and release status

See [measured results, wrong answers, and limitations](docs/pcd-results.md), with raw JSON evidence
under `results/pcd/`. Only the measurements for the released source revision belong to this release.
This is a small development/diagnostic evaluation, **not held-out production validation**.

On the final single-L40S run, fast token PCD averaged **56.24 ms** on 12 development
cases versus **557.41 ms** for direct-answer AR JSON (~9.9x). On a single synthetic
28-boolean configuration, it took **106.98 ms versus 4,875.30 ms (~45.6x)**, with all
fields correct for both methods. These are warm GPU request times, not network latency.

**Accuracy is the important limitation:** token PCD reached 88.9% development field
accuracy but only **72.2% on six fresh audit cases**, versus **94.4%** for AR JSON.
It also failed both high-cardinality token-mode probes. The production accuracy gate
was **not met**. Sequence scoring correctly resolved those high-cardinality examples,
but the 255-choice case was slower than AR. See the report rather than extrapolating
the favorable 28-field speedup to every task.

Schema validity does not mean a correct decision. Do not use the returned probabilities as
validated automation thresholds. Test your own labeled workload; use human review or a stronger
reasoning model where mistakes matter. No comparison against Jev's service was performed.

## Two inference modes

### `token`: fast parallel choices

Each field receives an unambiguous atomic option code. Codes are verified against the actual
tokenizer. One cached branch per field produces a decision hidden state; the output head is
projected only onto the relevant option-token rows, not the entire 128,000-token vocabulary.
The selected code is mapped back to the original enum string or actual Python boolean.

This is a classifier-style next-token decision, not arbitrary free-text generation. It does not
collapse multi-token enum values onto an ambiguous first token: opaque codes distinguish them.
The output includes the distribution over allowed codes, explicitly marked **uncalibrated**.
Code/label wording and ordering can affect predictions.

### `sequence`: full-sequence scoring reference

Each candidate is a complete JSON value with a newline terminator. The whole JSON member is
canonically tokenized before factoring its shared token prefix, preserving space/quote merges.
All remaining candidate tokens are scored with teacher forcing and full-vocabulary normalized
log-likelihood; the score is not a first-token proxy. Shared-prefix choices, escaped strings,
and Unicode are supported. The scoring convention still has length and wording bias.

Both paths reuse the shared prompt once and fork isolated hybrid caches. Token mode normally
uses two backbone calls for up to 32 fields. Sequence mode uses
`1 + ceil(total_candidates / branch_batch_size)` calls. Parallelism reduces sequential steps;
it does not make memory, FLOPs, or latency constant in the input size.

## Install and use

Get the code without immediately downloading duplicate weight files:

```bash
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/monotykamary/LFM2.5-2.6B-RLCD
cd LFM2.5-2.6B-RLCD
uv venv --python 3.11
uv pip install --python .venv/bin/python -r requirements-pcd.txt
source .venv/bin/activate
```

The default engine loads the pinned original LiquidAI model. On a CUDA GPU:

```python
from pcd import Engine

schema = {
    "type": "object",
    "properties": {
        "topic": {
            "type": "string",
            "enum": ["billing", "technical", "shipping"],
            "description": "The main issue in the message",
        },
        "refund": {
            "type": "boolean",
            "description": "Whether the customer explicitly requests a refund",
        },
    },
    "required": ["topic", "refund"],
    "additionalProperties": False,
}
engine = Engine(device="cuda", dtype="float16", attention="sdpa")
result = engine.constrained(
    "I was charged twice. Please refund the duplicate charge.", schema, mode="token"
)
print(result["object"])          # Typed values; field decisions may still be wrong.
print(result["fields"])          # Scores, candidate distributions, and margins.
print(result["elapsed_ms"])
assert result["calibrated"] is False

reference = engine.constrained("Please refund my duplicate charge.", schema, mode="sequence")
```

To use the bundled weights instead, pass `model_id="monotykamary/LFM2.5-2.6B-RLCD"`
and `revision="main"` to `Engine`; pin the public commit SHA for reproducibility.
The original BF16 tensors are bundled unchanged, even though the measured engine uses an FP16
runtime cast. Plain `AutoModelForCausalLM.from_pretrained(...)` loads the original generative
model; it does **not** enable parallel constrained inference. No `trust_remote_code=True` is needed.

## Supported schema and limits

- Closed, flat object, with every property required and `additionalProperties: false`.
- Fields are booleans or nonempty unique string enums; descriptions are optional.
- No arbitrary numbers, free-text strings, arrays, nested/optional fields, or relational constraints.
- Defaults: 32 fields, 256 total candidates, 4,096 shared-prompt tokens (schema included),
  64 tokens per serialized value, and 32 branches per microbatch.
- Limits and unsupported schema keywords are rejected, not silently ignored.
- Successful calls guarantee syntax, types and enum membership through programmatic assembly.
  They do not guarantee truth, completeness of the answer space, or cross-field consistency.

Return values live in `result["object"]`; telemetry is separate and does not alter your schema.
`token` probabilities are a restricted-code softmax; `sequence` probabilities are normalized
candidate likelihoods. Neither is a calibration guarantee or evidence that all candidates include
the correct answer. Include an explicit unknown/other option where appropriate.

## LFM2.5-specific details

The pinned model has 8 full-attention layers, 22 short-convolution layers and 2,697,198,592
parameters. Branches copy **both** KV tensors and convolution state; mutable expanded views
are not shared. Request caches are discarded, not retained across users. GPU operations are
serialized within one engine instance to bound memory and avoid state races.

The native chat template always opens `<think>`. The PCD prompts explicitly supply an empty
closed reasoning span; this is not an officially supported `enable_thinking=False` mode.
Skipping reasoning can hurt accuracy. The original native reasoning behavior is preserved in
the unchanged model/tokenizer and is available through normal generation.

FP32 CUDA and local tiny-model checks establish cache/scoring equivalence; measured FP16
rounding tolerances and errors are in the results. The initial BF16 path failed the tighter
hidden-state comparison and is not the recommended validated serving precision.

## Run on Modal, frugally

The scripts use the existing `huggingface-cache` Volume, a separate results Volume, one L40S,
zero minimum containers, a maximum of one container per parameterization, and short idle
shutdown. No H100, multi-GPU job or training run is required.

```bash
modal run lfm25_pcd_modal.py --task prepare
modal run lfm25_pcd_modal.py --task validate --precision float32
modal run lfm25_pcd_modal.py --task benchmark --suite diagnostic --repeats 3
modal run lfm25_pcd_modal.py --task benchmark --suite stress --repeats 3
```

Preparation/publication use a Modal Secret named `huggingface` (`HF_TOKEN` preferred).
GPU inference only needs public cached weights; it is not given the write credential.
Benchmarks have a time budget and persist raw results. They do not perform automatic retries
or switch to a larger GPU on failure. Pricing and actual billed time depend on your workspace.

`PCDModel.infer.remote(context, schema, mode)` supports on-demand calls. An authenticated POST
Web Function is included (`PCDModel.extract`); it accepts `context`, `schema`, and optional `mode`.
It requires Modal proxy authentication (`Modal-Key` and `Modal-Secret` headers). Deploy with
`modal deploy lfm25_pcd_modal.py` only after validating your workload. This release does not
claim a production-ready always-on endpoint or measured network/cold-start latency.

## Tests and reproducibility

```bash
python -m pytest tests/pcd -q
```

Local tests use a small randomly initialized LFM2 model: no GPU and no full-model download.
They test cached/full scoring, branch isolation, both convolution cache paths, microbatch
invariance, schema rejection, token shifts and typed output. Modal validation repeats critical
checks on the real pinned model. Benchmarks retain outputs, errors, source hashes, dependency
versions, actual accelerator, precision, prompt sizes and memory measurements.

Upstream identity: `LiquidAI/LFM2.5-2.6B` at
`654f9463ce32b05d0429d76fe1f580b27d4c1ac0`.
The Hub release includes `BASE_MODEL_MANIFEST.json` and `ARTIFACTS.json` for model and code
provenance. The original model card is preserved as `UPSTREAM_README.md` and appended below
the constrained-inference documentation in the published README.

## Attribution and license

The 350M reference's hybrid-cache/full-sequence design informed this implementation:
[notnotsamuel/LFM2.5-350M-RLCD](https://huggingface.co/notnotsamuel/LFM2.5-350M-RLCD).
See [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md) and [LICENSE-CODE](LICENSE-CODE).

LiquidAI weights, tokenizer, configuration and original model documentation retain the
**LFM Open License v1.0**, including redistribution requirements and commercial revenue
conditions. The inference-code MIT license does not replace the model license.

TypeSafe describes Jev's actual RLCD training in its
[AI primer](https://docs.typesafe.ai/introduction/machine-learning-primer).
This is an independent PCD implementation, not an affiliated or equivalent Jev release.