mtp-test-plan.md

# MTP Viability Test Plan: Quant x Temperature x Task Type

## Goal

Map the complete MTP speculative decoding viability landscape for Qwen3.6-27B on M2 Max 96GB.
Produce a recommendation table for each quant/temperature/task combination showing whether
MTP helps, hurts, or is neutral, with measured speedups.

## Test Parameters

- **Token budget**: `-n 2500` per test (expect 1500-2500 actual, model may stop early)
- **Context**: `-c 4096` (sufficient for prompt + 2500 generation)
- **MTP config**: `--spec-type mtp --spec-draft-n-max 3` (N=3)
- **Chat template**: `--jinja --chat-template-file <path> -sys '<system prompt>'`
- **Exit**: `printf '/exit\n'` piped to stdin
- **Verbosity**: `-v` to capture acceptance stats
- **GPU layers**: `-ngl 99`

## Variables

### Quants (4 levels)
Q4_K_M (16G), Q5_K_M (18G), Q6_K (21G), Q8_0 (27G)

Rationale: Skip iQ quants (IQ2_M, IQ3_M, IQ4_XS) — they showed inconsistent results
and imatrix has no measurable effect on acceptance. Focus on Q quants for clean data.

### Temperatures (3 levels)
0.0 (greedy/deterministic), 0.3 (low randomness), 0.7 (standard creative)

### Task Types (5 categories, determinism-ordered)

Tasks are ordered from most deterministic (highest expected acceptance) to least.

#### T1: Code Generation (very high determinism)
Syntax is rigid, most tokens are forced by grammar. Greedy draft ≈ verification.
```
Write a complete Python implementation of an LRU cache class with get(key), put(key, value),
delete(key), size(), and clear(). Use a doubly-linked list combined with a hash map for O(1)
operations. Include type hints, docstrings, and 10 unit tests using pytest with good coverage
of edge cases.
```
Expected: ~1500-2000 tokens

#### T2: Factual Explanation (high determinism)
Technical content with specific terminology and logical structure. One dominant token per position.
```
Explain in detail how the Transformer architecture works. Cover: token embedding, positional
encoding (sinusoidal and RoPE), multi-head self-attention including Q/K/V projections and
scaled dot-product attention, feed-forward networks, layer normalization (pre-norm vs post-norm),
residual connections, and the difference between encoder-decoder and decoder-only variants.
Include the key mathematical equations for attention and positional encoding.
```
Expected: ~1500-2500 tokens

#### T3: Technical Analysis (medium determinism)
Constrained by technical facts but allows structural freedom in organization and phrasing.
```
Analyze the tradeoffs of quantization in large language models. Cover: memory reduction,
inference speed gains on bandwidth-limited hardware, quality degradation patterns, different
quantization schemes (uniform quantization, k-quant mixtures, importance matrix guided),
perplexity impact at different bit widths, and the practical recommendations for selecting
a quantization level for deployment. Be thorough and technical.
```
Expected: ~1500-2500 tokens

#### T4: Expository Essay (low determinism)
Structured but high freedom in content organization, word choice, and emphasis.
```
Write a detailed 1500-word essay on the history and cultivation of lychee. Cover its origins
in southern China, spread to Southeast Asia and beyond, ideal growing conditions, major
commercial varieties, harvesting and post-harvest handling challenges, and economic importance
in global trade.
```
Expected: ~1500-2000 tokens

#### T5: Creative Writing (very low determinism)
Maximum freedom — many equally valid continuations at every position. Worst case for MTP.
```
Write a short story (about 1500 words) about a lone astronaut aboard a generation ship who
discovers that the ship's AI has been secretly altering the crew's memories over centuries.
The story should build tension gradually and have a surprising but logically consistent ending.
```
Expected: ~1500-2000 tokens

---

## Already Completed

| Quant | Temp | Task | Type | Speed | Acceptance | Source |
|-------|------|------|------|-------|------------|--------|
| Q4_K_M | 0.0 | Essay | MTP | 15.56 | 59.4% | Previous |
| Q4_K_M | 0.0 | Essay | Base | 14.96 | — | Previous |
| Q4_K_M | 0.3 | Essay | MTP | 15.18 | 57.2% | Previous |
| Q4_K_M | 0.3 | Essay | Base | 15.09 | — | Previous |
| Q4_K_M | 0.7 | Essay | MTP | 14.95 | 55.8% | Previous |
| Q4_K_M | 0.7 | Essay | Base | 15.07 | — | Previous |
| Q4_K_M+imat | 0.7 | Essay | MTP | 14.86 | 55.0% | Previous |
| Q5_K_M | 0.7 | Essay | MTP | 13.83 | 55.1% | Previous |
| Q6_K | 0.7 | Essay | MTP | 14.97 | 54.8% | Previous |
| Q8_0 | 0.7 | Essay | MTP | 18.33 | 55.7% | Previous |
| Q8_0 | 0.7 | Essay | Base | ? | — | Running |
| IQ3_M | 0.7 | Essay | MTP | 12.48 | 54.1% | Previous |

---

## Test Plan

### Phase 1: Task Type Effect (16 tests, ~45 min)

Test all 4 quants × 4 new task types (T1-T3, T5) at temp 0.7 with MTP.
Essay (T4) already done for all quants.

| # | Quant | Task | Temp | Type | Notes |
|---|-------|------|------|------|-------|
| 1 | Q4_K_M | Code | 0.7 | MTP | |
| 2 | Q4_K_M | Factual | 0.7 | MTP | |
| 3 | Q4_K_M | Analysis | 0.7 | MTP | |
| 4 | Q4_K_M | Creative | 0.7 | MTP | |
| 5 | Q5_K_M | Code | 0.7 | MTP | |
| 6 | Q5_K_M | Factual | 0.7 | MTP | |
| 7 | Q5_K_M | Analysis | 0.7 | MTP | |
| 8 | Q5_K_M | Creative | 0.7 | MTP | |
| 9 | Q6_K | Code | 0.7 | MTP | |
| 10 | Q6_K | Factual | 0.7 | MTP | |
| 11 | Q6_K | Analysis | 0.7 | MTP | |
| 12 | Q6_K | Creative | 0.7 | MTP | |
| 13 | Q8_0 | Code | 0.7 | MTP | |
| 14 | Q8_0 | Factual | 0.7 | MTP | |
| 15 | Q8_0 | Analysis | 0.7 | MTP | |
| 16 | Q8_0 | Creative | 0.7 | MTP | |

### Phase 2: Temperature x Task Interaction (8 tests, ~25 min)

Test code (T1, most deterministic) and creative (T5, least deterministic) at temp 0.0
and 0.3 for Q4_K_M and Q8_0 (representing low and high quality bounds).

| # | Quant | Task | Temp | Type | Notes |
|---|-------|------|------|------|-------|
| 17 | Q4_K_M | Code | 0.0 | MTP | Best case: low quant + deterministic task |
| 18 | Q4_K_M | Code | 0.3 | MTP | |
| 19 | Q4_K_M | Creative | 0.0 | MTP | Low quant + creative at greedy |
| 20 | Q4_K_M | Creative | 0.3 | MTP | |
| 21 | Q8_0 | Code | 0.0 | MTP | Best case: high quant + deterministic task |
| 22 | Q8_0 | Code | 0.3 | MTP | |
| 23 | Q8_0 | Creative | 0.0 | MTP | High quant + creative at greedy |
| 24 | Q8_0 | Creative | 0.3 | MTP | |

### Phase 3: Missing Baselines (3 tests, ~10 min)

One baseline per quant at temp 0.7. Task type doesn't significantly affect baseline speed.

| # | Quant | Temp | Type | Notes |
|---|-------|------|------|-------|
| 25 | Q5_K_M | 0.7 | Base | |
| 26 | Q6_K | 0.7 | Base | |
| 27 | IQ3_M | 0.7 | Base | Optional, lower priority |

Q4_K_M baselines already done. Q8_0 baseline running.

### Phase 4: Fill-in Tests (conditional, ~15 min)

Based on Phase 1-3 results, test additional combinations to clarify transitions:

- If code at Q4_K_M/temp 0.7 is above breakeven: test Q4_K_M/factual at temp 0.3
- If creative at Q8_0/temp 0.7 is above breakeven: verify with analysis task
- Test any quant/task/temperature combinations near the breakeven threshold

Up to 6 additional tests, TBD based on results.

---

## Total: 27 core tests + ~6 conditional = ~33 tests, ~90 min

## Test Execution

### Command Template

```bash
CLI="/Volumes/ssd/ai/llm-dev/llama.cpp-mtp/build/bin/llama-cli"
MODEL="<quant-file>"
TEMPLATE="/Volumes/ssd/ai/llm-models/froggeric/Qwen-Fixed-Chat-Templates/qwen3.6/chat_template-v9.jinja"
SYS='You are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|think_off|>'
PROMPT="<task-prompt>"

# MTP test
printf '/exit\n' | $CLI -m "$MODEL" \
  --spec-type mtp --spec-draft-n-max 3 \
  -c 4096 -n 2500 --temp <temp> -ngl 99 \
  --jinja --chat-template-file "$TEMPLATE" -sys "$SYS" -p "$PROMPT" \
  -v 2>&1 | tee <output-file> | grep -E "accept|draft|statistics|tokens per second|eval time|prompt eval" | tail -10

# Baseline test
printf '/exit\n' | $CLI -m "$MODEL" \
  -c 4096 -n 2500 --temp <temp> -ngl 99 \
  --jinja --chat-template-file "$TEMPLATE" -sys "$SYS" -p "$PROMPT" \
  -v 2>&1 | tee <output-file> | grep -E "tokens per second|eval time|prompt eval" | tail -5
```

### Execution Strategy

Run tests sequentially (single GPU). Chain commands in groups:
- Phase 1: 4 quants × 4 tasks = 16 sequential tests, ~45 min
- Phase 2: 8 sequential tests, ~25 min
- Phase 3: 3 sequential tests, ~10 min
- Phase 4: conditional

Use `tee` to capture full output for post-hoc analysis if grep misses something.

### Metrics to Extract

From each test output:
- **eval time** and **tokens generated** → decode speed (tok/s)
- **draft acceptance rate** (MTP tests only)
- **#gen tokens** and **#acc tokens** from statistics line

---

## Analysis Plan

### 1. Acceptance Rate Heatmap

```
            Code  Factual  Analysis  Essay  Creative
Q4_K_M 0.0   ?      ?        ?       59.4%    ?
Q4_K_M 0.3   ?      ?        ?       57.2%    ?
Q4_K_M 0.7   ?      ?        ?       55.8%    ?
Q5_K_M 0.7   ?      ?        ?       55.1%    ?
Q6_K  0.7   ?      ?        ?       54.8%    ?
Q8_0  0.0   ?      ?        ?        ?       ?
Q8_0  0.3   ?      ?        ?        ?       ?
Q8_0  0.7   ?      ?        ?       55.7%    ?
```

### 2. Speedup Table

```
            Code     Factual  Analysis  Essay    Creative
Q4_K_M     ?%       ?%       ?%       -0.8%    ?%
Q5_K_M     ?%       ?%       ?%       ?%       ?%
Q6_K      ?%       ?%       ?%       ?%       ?%
Q8_0      ?%       ?%       ?%       ?%       ?%
```

Speedup = (MTP_speed / Base_speed - 1) × 100%
Breakeven acceptance for N=3 ≈ 59% (but depends on MTP overhead ratio)

### 3. Final Recommendation Matrix

For each quant, recommend:
- **Green**: MTP recommended (measurable speedup)
- **Yellow**: MTP neutral (marginal, task-dependent)
- **Red**: MTP not recommended (net slowdown)

Organized by use case:
- Coding assistant (temp 0.0-0.3)
- General chatbot (temp 0.7)
- Technical writing (temp 0.3-0.5)
- Creative writing (temp 0.7-1.0)