overthelex/ua-legal-citation-grounded-14b

UA Legal Citation-Grounded Generator (14B)

A Ukrainian legal citation-grounded generator: given a legal question and a set of retrieved court-decision excerpts, it writes an answer where every factual claim is attributed to a source as [doc:ID], and it refuses when the provided sources are insufficient (instead of hallucinating). Built for the retrieval-augmented (RAG) leg of a legal chat pipeline over the Ukrainian ЄДРСР state court-decision registry.

• Base: Qwen/Qwen2.5-14B → domain continual-pretraining on ЄДРСР → SFT.
• Training data: overthelex/ua-legal-citation-grounded-sft (47,301 citation-grounded ChatML examples, 12.4% refusals, teacher-distilled + judge-filtered).
• Teacher: Qwen2.5-72B-Instruct (self-hosted; no proprietary API in the data).

What it does

Input: a question + a numbered context of court-decision excerpts (each tagged [doc:ID]). Output: a grounded Ukrainian legal answer citing only the [doc:ID] that appear in the context, or an explicit refusal if the sources do not support an answer.

Held-out evaluation (unseen queries, prod-representative fresh-corpus contexts)

Metric	Value
citation coverage (answers that cite)	95.7%
relevant citations	96.1%
distractor citations	2.35%
out-of-context (fabricated) citations	3.9%
refusal correctness (refuses when sources insufficient)	96.6%
avg citations / answer	3.1

How to use

⚠️ Two things matter for correct behavior:

1. Serve via HF transformers generate (this is the tested path). Do not serve via vLLM without validation — a vLLM decoding bug degenerates this model.
2. Use max_length ≥ 8192 when tokenizing. Legal contexts run ~4-5k tokens; truncating the prompt silently drops context and collapses citation coverage.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL = "overthelex/ua-legal-citation-grounded-14b"
tok = AutoTokenizer.from_pretrained(MODEL)
model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16, device_map="auto").eval()

SYSTEM = (
    "Ти — український юридичний асистент. Відповідай ВИКЛЮЧНО на основі наданих витягів "
    "із судових рішень ЄДРСР. Кожне фактичне твердження підкріплюй посиланням у форматі "
    "[doc:ID], де ID — це edrsr_doc_id відповідного джерела. Якщо у наданих джерелах немає "
    "достатньої підстави — прямо напиши, що наданих джерел недостатньо. Пиши українською."
)

context = [
    {"doc_id": "114714815", "text": "...витяг із рішення суду..."},
    {"doc_id": "77050963",  "text": "...ще один витяг..."},
]
question = "Як суд розподіляє спільне майно подружжя при розлученні?"

ctx = "\n\n".join(f"[doc:{c['doc_id']}] {c['text']}" for c in context)
user = (f"Питання: {question}\n\nДжерела (витяги з рішень ЄДРСР):\n{ctx}\n\n"
        "Дай обґрунтовану відповідь українською з посиланнями [doc:ID].")

msgs = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": user}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt",
                              truncation=True, max_length=8192).to(model.device)   # <-- 8192, not 3500
out = model.generate(ids, max_new_tokens=512, do_sample=False, pad_token_id=tok.pad_token_id)
answer = tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True)
print(answer)

Recommended guardrail (deterministic citation validator)

The model's citations are grounded, but strip any [doc:ID] not present in the context you provided (belt-and-suspenders against fabrication):

import re
def strip_ungrounded(answer, context_doc_ids):
    valid = {str(x) for x in context_doc_ids}
    def repl(m):
        kept = [i for i in re.findall(r"\d+", m.group(1)) if i in valid]
        return f"[doc:{', doc:'.join(kept)}]" if kept else ""
    return re.sub(r"\[doc:\s*([^\]]+?)\s*\]", repl, answer)

Intended use & limitations

• Intended: RAG-style legal drafting/QA assistant over ЄДРСР where answers must be source-attributed, and a research/reference aid for Ukrainian case law.
• Not intended: standalone legal advice without a qualified human in the loop; answering without a retrieved context; jurisdictions/languages other than Ukrainian law.
• Cites only what is in the provided context — quality depends on your retriever. Doc-ids are ЄДРСР edrsr_doc_id; the model learned the current (2026) corpus, so feed it current-corpus contexts.
• Covers civil / commercial / administrative-offence procedure best (training justice_kinds ЦПК/ГПК/КУпАП).

Training

CPT-14B (Qwen2.5-14B continually pretrained on ЄДРСР) → LoRA SFT (r16, α32, 2 epochs, lr 1e-4, max-len 4096, bf16, DeepSpeed ZeRO-2) on 47.3K citation-grounded ChatML examples. Dataset regenerated on the current ЄДРСР vector store with a self-hosted Qwen2.5-72B teacher and a faithfulness judge (programmatic citation check + LLM judge). Ukrainian legal-tech project (SecondLayer / legal.org.ua).