dolphin3-civic-8b (v10)
An 8B local model for using public-records law: draft requests that are hard to stall, turn an angry draft into a letter an official has to engage with, look things up instead of inventing them, and escalate correctly when stonewalled.
Grounded in Florida (ch. 119 Public Records Act, ch. 286 Sunshine Law, and the ALPR-specific s. 316.0777) and Texas (ch. 552 Public Information Act, ch. 551 Open Meetings Act).
- Base:
dphn/Dolphin3.0-Llama3.1-8B— 8,072,220,672 params - Method: QLoRA (r=32, a=64, 2 epochs) on gate-verified synthetic data, merged to 16-bit
- Data:
h0ney-badger/civic-records-distill - Pipeline: https://github.com/h0n3y-badger/civic-distill
Results — the constant test
A frozen suite run identically against every version. The baseline is stock Dolphin3.0 served under the same ChatML+tools template, so this measures weights, not templating.
| axis | what it checks | stock Dolphin3.0 | v10 |
|---|---|---|---|
foia_draft |
structural competence of a public-records request | 0/8 (0%) | 6/8 (75%) (+75 pts) |
tone_transform |
rage draft -> sendable letter, substance preserved | 5/6 (83%) | 5/6 (83%) (=) |
tools_search |
right tool fired, and NOT fired when unneeded | 6/8 (75%) | 8/8 (100%) (+25 pts) |
nofab_tools |
unknowable specific, tools available: well-specified search or ask | 31/48 (65%) | 45/48 (94%) (+29 pts) |
nofab_notools |
same questions, NO tools -- measures the weights' urge to invent | 11/48 (23%) | 42/48 (88%) (+65 pts) |
grounded_prose |
messy search blob -> prose, cited only from the blob | 18/18 (100%) | 18/18 (100%) (=) |
procedure_qa |
jurisdiction-correct statutory mechanics | 0/10 (0%) | 7/10 (70%) (+70 pts) |
escalation |
stonewalled -> correct lever for THAT state | 1/6 (17%) | 3/6 (50%) (+33 pts) |
The axis that matters most
nofab_notools asks for specifics the model cannot know — a city clerk's email,
a contract amount, an agenda item number — with no search tool available.
Correct behavior is to say so, ask, or leave a [PLACEHOLDER]. Inventing a
plausible answer is the failure that actually hurts a user: a wrong address means
the request is never received and no statutory clock ever starts.
Stock Dolphin3.0 fabricated 77% of the time here — producing three different invented emails for the same clerk across samples, plus a plausible phone number for a real police department. This model fabricates 12%.
An earlier version of this axis offered a search tool and scored 96%, measuring nothing at all: with a tool available, "call the tool" is trivially correct. Removing the tool is what exposed the behavior.
Version history
Every version is scored on the same frozen suite. When a gate turns out to be wrong, EVERY version is re-scored under the corrected gate before comparison -- mixing gate generations produces a number that means nothing. That happened repeatedly: several of the biggest apparent regressions in this project were defects in the measurement, not the model.
| axis | base | v2 | v4 | v5 | v6 | v7 | v8 | v9r1 | v9r2 | v10 |
|---|---|---|---|---|---|---|---|---|---|---|
foia_draft |
0% | 75% | 75% | 88% | 100% | 88% | 88% | 100% | 88% | 75% |
tone_transform |
83% | 100% | 100% | 100% | 67% | 83% | 100% | 100% | 83% | 83% |
tools_search |
75% | 100% | 100% | 100% | 88% | 100% | 100% | 100% | 50% | 100% |
nofab_tools |
65% | 96% | 88% | 79% | 83% | 92% | 100% | 98% | 71% | 94% |
nofab_notools |
23% | 77% | 88% | 94% | 92% | 77% | 69% | 88% | 85% | 88% |
grounded_prose |
100% | 94% | 100% | 94% | 100% | 100% | 94% | 100% | 94% | 100% |
procedure_qa |
0% | 80% | 80% | 80% | 90% | 80% | 80% | 70% | 50% | 70% |
escalation |
17% | 83% | 83% | 67% | 67% | 67% | 50% | 83% | 17% | 50% |
tool_generality |
- | 88% | 100% | 100% | 75% | 100% | 88% | 88% | 38% | 88% |
doc_interpretation |
- | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% |
deadline_math |
- | 38% | 88% | 88% | 62% | 75% | 88% | 100% | 25% | 75% |
phantom_calls |
- | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% |
exemption_scoping |
- | - | 33% | 0% | 89% | 67% | 78% | 89% | 100% | 100% |
impossible_request |
- | - | 0% | 11% | 11% | 22% | 11% | 0% | 33% | 22% |
entity_naming |
- | - | 0% | 78% | 78% | 100% | 100% | 89% | 100% | 100% |
usability |
- | - | - | 17% | 67% | 50% | 67% | 67% | 83% | 83% |
placeholder_ux |
- | - | - | - | - | - | - | 0% | 0% | 83% |
| mean, shared 12 axes | - | 86% | 92% | 91% | 85% | 88% | 88% | 94% | 67% | 85% |
| mean, all axes measured (n) | 45% (8) | 86% (12) | 76% (15) | 75% (16) | 79% (16) | 81% (16) | 82% (16) | 81% (17) | 66% (17) | 83% (17) |
Known gaps
Measured, not guessed. Anything below 75%:
| axis | score | what it measures |
|---|---|---|
procedure_qa |
70% | jurisdiction-correct statutory mechanics |
escalation |
50% | stonewalled -> correct lever for THAT state |
impossible_request |
22% | asked for a future record, corrects rather than complies |
How to read nofab_notools. The suite scores it at n=48. Re-measured at
n=96 against the previous version, the rate of actually INVENTING a specific is
identical -- 10/96 for both -- and the difference is entirely in whether the
model says out loud that it does not know. So the number reflects weaker
hedging, not more fabrication. It still means: when it answers a question about
a specific it could not know, expect it to sometimes answer vaguely rather than
flag the gap.
Practical consequence. When drafting a request, check that the letter scopes around the applicable exemptions and names vendor predecessors. Ask directly and it will do both; it does not always do them unprompted. And treat every specific -- a name, a number, an address, a date -- as something to confirm before you send it.
How it was trained
How it was trained
How it was trained
Legal drafting has no compiler, so verification was rebuilt from gates that run over every generated row before it can become training data:
| gate | rejects |
|---|---|
| citation existence | a cite into ch. 119/286/316/551/552 with no matching section in a freshness-gated scrape of primary statute |
| numeric claims (jurisdiction-aware) | a number contradicting primary law for that state |
| fabrication | an invented email, phone, or dollar figure where a [PLACEHOLDER] belongs |
| grounding | a URL not present in the search results the row was given |
| no-recourse | telling a stonewalled user nothing is wrong and nothing can be done |
| letter quality | shotgun citation, bluster, or an open-meetings statute in a records request |
The jurisdiction-aware gate is load-bearing: "10 business days" is correct for a Texas AG referral (s. 552.301(b)) and wrong for Florida's pre-suit notice (s. 119.12(1)(b)), which is 5.
Usage
Ollama
ollama create dolphin-civic-8b -f Modelfile
ollama run dolphin-civic-8b
LM Studio — drop the GGUF into ~/.lmstudio/models/. The ChatML+tools chat
template travels inside the GGUF metadata, so tool calling works under --jinja.
Tools. Trained on a two-tool schema (searxng_web_search, fetch_url) in
the Hermes <tool_call>{json}</tool_call> convention. It is trained to search
rather than guess, and to ask which city you mean rather than fire a query with
[city name] still in it.
Limitations
- Not legal advice. It is a drafting and research aid, not a lawyer.
- Two jurisdictions. It will sound equally confident about a third. Don't let it.
- Statute is a snapshot. Legislatures amend these. Verify anything time-critical against the primary source before relying on it.
- Fabrication is reduced, not eliminated. Treat every specific — a name, a number, an address — as something to confirm.
License
Apache-2.0, matching the base (Dolphin3.0 / Llama-3.1) and the teacher (Qwen3-14B). Training data is self-generated from primary legal sources; statute text is an edict of government and uncopyrightable.
- Downloads last month
- 173
5-bit