Luigi/qwen35-0.8b-arcsum

arcsum — Qwen3.5-0.8B agentic zh-TW meeting summariser

An on-device agent that reads a meeting transcript in ~2,500-token chunks, maintains a small external memory (ARC + POINTS) across steps, and emits one flowing Traditional Chinese summary. The model never sees conversation history: each step is (system, MEMORY + CHUNK) -> tool call, and a harness applies the edits deterministically.

Trained for the single-turn tool-call protocol: each reading step emits ONE batched update_memory call with arc / add / drop arguments.

Which file to use

file	what it is
Qwen3.5-0.8B.Q8_0.gguf	current default (qwen-tools-v5)
Qwen3.5-0.8B-mixed-e3.Q8_0.gguf	mixed-e3 best epoch — shipped and rolled back the same day, see below

Why mixed-e3 is not the default despite better gate numbers

mixed-e3 beat v5 on every offline gate measured (revision probe 3/27 -> 8/27, real-ASR "curated" 17/20 -> 19/20, all three G3 gates still passing). It was served in the reference demo on 2026-09-02 and rolled back the same day.

On a real zh-TW ASR meeting, in the demo's own serving configuration:

	points kept	churn events	summary
v5	4	0	304 chars
mixed-e3	1	4	553 chars

mixed-e3 dropped its only memory point and re-added a near-identical one on five consecutive steps, then synthesised 553 characters of confident competitive-strategy prose from that single point.

Two configuration-level evaluation gaps let this through, and neither is about choosing better metrics. First, the gates run with cache_prompt: false while the demo runs with the KV cache live across calls — and the prompt cache is known to change generation (measured: 167 vs 700 characters, same model, same seed, temperature 0). Under the gated configuration mixed-e3 handles the same transcript fine: 4 points, 0 churn. The regression exists only in the configuration the product actually uses. Second, the real-ASR check scored a meeting "curated" by summary LENGTH, which a confabulation passes easily — so the metric reporting 19/20 was rewarding the failure.

The mixed-e3 numbers below are retained because they are real; they are simply not sufficient to ship on.

mixed-e3 was measured head-to-head against v5 on the SAME probe corpus and the SAME 40 held-out meetings, on the same day:

	v5	mixed-e3
within-meeting revision probe (27 scenarios)	3/27	8/27
real zh-TW ASR meetings curated	17/20, 28% NOP	19/20, 15% NOP
G3 rouge1	+0.069, 28/40	+0.053, 30/40, p=0.002
G3 rouge2	+0.041, 29/40	+0.031, 27/40, p=0.038
G3 rougeL	+0.057, 35/40	+0.039, 31/40, p=0.000

It wins on the two deployment-facing axes and pays roughly 30% of the ROUGE margin.

This is the BEST-epoch export (checkpoint 626), not the last. That is not a general rule: v5 is worse at its best epoch, and a sibling build's lowest-loss checkpoint abstains on 95% of real meetings. Eval loss does not order these checkpoints on anything that is gated — the epoch was chosen by measurement.

Serving — read this first

Serve with --no-jinja. Every number here was measured that way (plain ChatML, prompt ending <|im_start|>assistant\n).

llama-server -m Qwen3.5-0.8B.Q8_0.gguf --no-jinja -c 4096 -C 0xFF

Serving under --jinja puts a <think> block in every prompt — a different configuration from the one these numbers came from.

For reproducibility also pin cache_prompt: false — llama.cpp's prompt cache changes generation (measured: 700 vs 167 characters, same seed and temperature).

Use repeat_penalty=1.1 on the synthesis call only. Reading steps emit a fixed op vocabulary and a repetition penalty there punishes the format's own tokens.

Measured results (mixed-e3)

40 held-out MeetingBank-derived zh-TW meetings, agent vs a fair map-reduce baseline (same model, same chunk size), plus 20 real zh-TW legislative meetings for the ASR check.

gate	result
G2 faithfulness	PASS — 24 vs 53 inversions, 40/40 paired, 0 judge failures
G3 rouge1	PASS — 30/10, +0.053, p=0.002
G3 rouge2	PASS — 27/13, +0.031, p=0.038
G3 rougeL	PASS — 31/8, +0.039, p=0.000
real zh-TW ASR	19/20 meetings curated, 15% NOP rate
G1 within-meeting revision	FAIL — 8/27 on an independent 27-scenario probe
G4 latency	not measured for this build — see below

Known limitations

• Within-meeting revision fails (8/27). If a decision is taken and then reversed later in the same meeting, the summary may report the superseded decision as if it still stood. Do not rely on this capability. The failure has been traced: the identifying detail now survives revision far more often than before, but the model frequently omits the NEW outcome. MeetingBank contains essentially no within-meeting reversals, so this was learned only from synthetic supervision.
• G4 latency was measured for the PREVIOUS checkpoint, not this one: 19.0 min/meeting on an Oppo Reno 7 (CPU-only, Q8, all 8 cores) against a 20-minute ceiling — a 3% margin, and a contended phone measured 21.6 min, over budget. mixed-e3 writes longer summaries and has not been re-timed on the device. Treat the budget as unverified for this file.
• G2 was measured on this build's last-epoch sibling (same pool, one epoch apart), not byte-for-byte on this file. The G3 and ASR numbers above ARE this file.
• G2's per-claim rate favours the baseline. The agent wins on absolute inversions partly because it asserts far fewer claims.
• zh-TW only. English is source material, never a product language.

Under the project's all-or-nothing ship rule, G1's failure means the recorded decision is still ship the baseline. This model is published as the best measured checkpoint, not as a cleared-for-production artifact.

Provenance

Base: Qwen/Qwen3.5-0.8B (text tower only; the vision encoder is not used and is dropped at export). The MTP head is restored from the base checkpoint before GGUF conversion — llama.cpp requires those 15 tensors at load time even though training never modifies them.

Q4_K_M is deliberately not published: measured against Q8_0 on the same 40 meetings it keeps under half the quality margin and writes ~30% shorter summaries.