---
language:
- en
license: apache-2.0
base_model: microsoft/deberta-v3-small
pipeline_tag: text-classification
tags:
- llm-routing
- routing
- model-selection
- onnx
---

# Gen Router Tier Classifier

## The Problem

Every LLM-powered application has to decide, for each incoming prompt, which
model should answer it. Get that decision wrong in either direction and it
costs you:

- **Always use the most capable model** and you pay frontier-tier pricing
  and latency for questions a much cheaper model could have answered just as
  well — "what's the capital of France?" doesn't need a reasoning-heavy,
  expensive model.
- **Always use a cheap, fast model** and you save money on easy traffic but
  silently degrade quality on the hard prompts that actually need deep
  reasoning, domain expertise, or careful multi-step work — a failure mode
  that's easy to miss until a user notices a bad answer.
- **Route by hand-written rules** (keyword lists, prompt length, source of
  the request) and you get a system that's brittle and doesn't generalize —
  short prompts can be genuinely hard, and long prompts can be trivial.

This model exists to make that per-request decision automatically and
cheaply: given a prompt, predict whether it belongs on a **fast**, tier,
**balanced** tier, or **frontier** tier model — *before* any generation
happens — so a routing layer can send each request to the right-sized model.
Because the classifier itself is small (see benchmark below), the routing
decision adds negligible cost and latency compared to running the prompt
through an oversized model unnecessarily.

## What It Predicts

| Tier | Intended for |
|---|---|
| `fast` | Simple, single-step tasks — factual lookups, short translations, basic formatting/classification |
| `balanced` | Multi-step tasks needing real reasoning — code generation, summarization, structured writing |
| `frontier` | Expert-level or deep multi-step reasoning — complex system design, proofs, research synthesis, adversarial edge cases |

## Test Set Results

| Metric | Value |
|---|---|
| Accuracy | 0.890 |
| F1 Macro | 0.797 |
| F1 Fast | 0.928 |
| F1 Balanced | 0.848 |
| F1 Frontier | 0.615 |

## Label Mapping

```json
{
  "0": "fast",
  "1": "balanced",
  "2": "frontier"
}
```

## Training

- 4 epochs, lr=2e-5, batch size 32, max_length=320
- Class-weighted cross-entropy loss (inverse frequency) to counter frontier
  being a minority class
- Best checkpoint selected by validation F1 macro (monotonically improved
  every epoch: 0.708 → 0.788 → 0.792 → 0.798)

## Comprehensive Benchmark

### Per-Source Accuracy (test set)

| Source | n | Accuracy |
|---|---|---|
| flan | 4,108 | 97.4% |
| openhermes | 5,929 | 87.5% |
| evol | 2,479 | 85.0% |
| sharegpt | 1,431 | 78.3% |

### Calibration

Expected Calibration Error (ECE): **0.053**.

### Inference Latency — PyTorch vs ONNX (single-sample, batch=1)

| Config | p50 | p95 | p99 | Mean |
|---|---|---|---|---|
| PyTorch GPU (T4) | 15.83ms | 23.62ms | 43.95ms | 16.99ms |
| **ONNX GPU (T4)** | **4.93ms** | 10.86ms | 12.48ms | **5.67ms** |
| PyTorch CPU | 158.17ms | 412.04ms | 483.39ms | 204.57ms |
| **ONNX CPU** | **77.77ms** | 360.32ms | 488.04ms | **111.39ms** |

The ONNX export runs **~3x faster on GPU** and **~2x faster on CPU** than the
native PyTorch model, with accuracy verified identical on the full test set
(89.02% for both). CPU tail latency (p95/p99) reflects a constrained
benchmarking environment and is not representative of a dedicated server.

### Error Analysis

- Overall accuracy: 89.0% (test set, n=13,947)
- Critical failure rate (frontier prompt routed to `fast`): **3.6%**
- Over-provisioning rate (fast prompt routed to `frontier`): **0.17%**

The error asymmetry favors safety: the model is far more likely to
over-provision an easy prompt (wasting some cost) than under-provision a
hard one (risking a bad answer) — the direction you want a router to err in.

## Usage (PyTorch)

```python
from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

repo_id = "appriai/gen-router-t1"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForSequenceClassification.from_pretrained(repo_id)

inputs = tokenizer("Explain quantum entanglement in detail.", return_tensors="pt")
with torch.no_grad():
    logits = model(**inputs).logits
tier = model.config.id2label[logits.argmax(-1).item()]
print(tier)  # "frontier"
```

## Usage (ONNX — faster, recommended for production)

```python
import onnxruntime as ort
from transformers import AutoTokenizer
from huggingface_hub import hf_hub_download

repo_id = "appriai/gen-router-t1"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
onnx_path = hf_hub_download(repo_id, "onnx/model.onnx")
session = ort.InferenceSession(onnx_path, providers=["CUDAExecutionProvider", "CPUExecutionProvider"])

enc = tokenizer("Explain quantum entanglement in detail.", return_tensors="np")
inputs = {i.name: enc[i.name] for i in session.get_inputs()}
logits = session.run(["logits"], inputs)[0]
```