ARCHITECTURE.md

Lex architecture

Lex fine-tunes Kai's 571,909,635-parameter encoder architecture. Three independently parameterized 22-layer paths serve Choice, Noul and Score. Each path has its own final normalization, two interaction layers and a candidate scorer; token embeddings and embedding normalization are shared.

A question and all of its candidate descriptions are encoded jointly with the context. Marker states supply scalar candidate scores, normalized over the valid candidates. A shared type embedding is added after each path's final normalization. Each needed encoder path processes the full padded batch; interaction heads gather rows of that type.

Choice returns a candidate and probabilities; Noul returns the probability of yes; Score returns the ordered distribution and its expected value. Candidate IDs preserve associations outside the token stream. Multiple questions use batches, with repeated context encoding.

The public API accepts complete packed inputs of at most 1,024 tokens per question. No field is silently truncated.

Main SVG · Residual layers · Attention and GEGLU · PDF atlas