Instructions to use Andrew0425/AgenticASR-Refiner with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Andrew0425/AgenticASR-Refiner with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Andrew0425/AgenticASR-Refiner")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Andrew0425/AgenticASR-Refiner")
model = AutoModelForCausalLM.from_pretrained("Andrew0425/AgenticASR-Refiner", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Andrew0425/AgenticASR-Refiner with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Andrew0425/AgenticASR-Refiner"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Andrew0425/AgenticASR-Refiner",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Andrew0425/AgenticASR-Refiner

SGLang

How to use Andrew0425/AgenticASR-Refiner with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Andrew0425/AgenticASR-Refiner" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Andrew0425/AgenticASR-Refiner",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Andrew0425/AgenticASR-Refiner" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Andrew0425/AgenticASR-Refiner",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Andrew0425/AgenticASR-Refiner with Docker Model Runner:
```
docker model run hf.co/Andrew0425/AgenticASR-Refiner
```

AgenticASR-Refiner / onnx-int4 /README.md

Andrew0425

onnx-int4 (#2)

a786ce7 about 1 month ago

preview code

Raw

History Blame Contribute Delete

2.13 kB

AgenticASR-Refiner ONNX (INT4)

INT4 weight-only quantization of the Optimum ONNX export of Andrew0425/AgenticASR-Refiner (Llama-based ASR transcript refiner, 24 layers, hidden 1536, GQA 16/2, head_dim 128, vocab 130560).

model.onnx + model.onnx.data — INT4 (MatMulNBits, block size 32, symmetric, accuracy level 4), ~1.3 GB. Graph I/O is identical to the fp32 export (input_ids / attention_mask / position_ids / past_key_values.*).
Quantized with onnxruntime 1.24.4 MatMulNBitsQuantizer (onnxruntime.quantization.matmul_nbits_quantizer).
Requires ONNX Runtime >= 1.20 (CPU EP supports MatMulNBits).

Tokenization uses the original repo tokenizer (tokenizer.json at the repo root).

Verified generation (ONNX Runtime 1.28, CPU)

Input	Output
我今天去了公司然后然后开了个会，明天再去见张总	我今天去了公司然后开了个会，明天再去见张总
你好你好你好我是那个小李啊电话是13800138000	你好我是小李，电话是13800138000

Usage

import numpy as np
import onnxruntime as ort
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("Andrew0425/AgenticASR-Refiner")
session = ort.InferenceSession("model.onnx", providers=["CPUExecutionProvider"])
input_names = [i.name for i in session.get_inputs()]
kv_names = [n for n in input_names if n.startswith("past_key_values")]

prompt = tokenizer.apply_chat_template(
    [{"role": "system", "content": "你是 ASR 文本纠错助手。保留原意，最小修改。"},
     {"role": "user", "content": "我今天去了公司然后然后开了个会"}],
    tokenize=False, add_generation_prompt=True,
)
ids = tokenizer(prompt).input_ids
pasts = [np.zeros((1, 2, 0, 128), dtype=np.float32) for _ in kv_names]

# prefill, then loop decode: feed input_ids/attention_mask/position_ids + pasts

Note: optimum-onnx 0.1.0 cannot yet run this checkpoint (dummy KV-cache shape uses hidden_size // num_heads = 96 instead of head_dim = 128); use the raw ONNX Runtime loop above, or a future fixed version of optimum-onnx.