Abdullahu5mani/granite-speech-4.1-2b-nar-cuda

Granite Speech 4.1 2B NAR CUDA INT4 for Taurscribe

This repository contains a Taurscribe-optimized ONNX Runtime CUDA artifact derived from IBM's original ibm-granite/granite-speech-4.1-2b-nar model.

This is not a newly trained model. No additional training or fine-tuning was performed. The artifact keeps the same underlying IBM Granite Speech model, but optimizes the ONNX representation and Taurscribe runtime path for fast local transcription on NVIDIA CUDA GPUs.

What Changed

Starting from the original IBM Granite Speech 4.1 2B NAR ONNX bundle, Taurscribe applied:

1. INT4 weight-only quantization to selected heavy ONNX MatMul weights.
2. MatMul to MatMulNBits conversion for compressed matrix multiplication weights.
3. An editor.onnx graph patch that adds ArgMax and outputs token_ids instead of full vocabulary logits.
4. Stale external-data pruning after quantization.
5. Runtime support in Taurscribe for ONNX Runtime CUDA Execution Provider.

Intended Runtime

Expected Taurscribe runtime placement:

encoder.onnx      -> CUDA
projector.onnx    -> CUDA
embed_tokens.onnx -> CUDA
editor.onnx       -> CUDA

CUDA is selected by Taurscribe/Rust through ONNX Runtime's CUDA Execution Provider. The ONNX files describe the graph; the app chooses where to run it.

Why This Exists

The original editor returns full logits shaped roughly like:

batch x sequence x vocabulary

The vocabulary is large, so copying full logits from GPU to CPU is expensive. Taurscribe only needs the winning token IDs, so this artifact makes editor.onnx output token_ids directly.

Source / Attribution

Base model: ibm-granite/granite-speech-4.1-2b-nar
Original provider: IBM Granite Team
Original license: Apache License 2.0
Optimization/package: Taurscribe project

Files

Core files:

encoder.onnx
encoder.onnx.data
projector.onnx
projector.onnx.data
embed_tokens.onnx
editor.onnx
editor.onnx.data
tokenizer.json
preprocessor_config.json
processor_config.json
tokenizer_config.json
generation_config.json
manifest.json
NOTICE

See manifest.json for checksums and artifact metadata.

Hardware Recommendation

This artifact is intended for NVIDIA CUDA GPUs. For CPU-only, AMD/DirectML, or Apple Silicon systems, use a separate portable or platform-specific artifact.