naranor/DeBERTa-v3-small-NLI-Attentions

DeBERTa-v3-small NLI ONNX with Attention Weights (INT8)

This repository provides a quantized INT8 ONNX export of cross-encoder/nli-deberta-v3-small. This version is specifically modified to expose raw attention weights from DeBERTa's unique Disentangled Attention mechanism, enabling advanced semantic analysis, intent routing, and context pruning.

Key Features

• Architecture: DeBERTa v3 fine-tuned for Natural Language Inference (NLI).
• Modification: Custom ONNX graph export that includes attentions.{0..5} as additional outputs.
• Versatility: Returns both logits (for NLI/classification routing) AND full attention matrices (batch, heads, seq_len, seq_len).
• Optimization: INT8 quantization for efficient CPU-based inference.

Intended Use

This model is intended to serve as a lightweight, lightning-fast "Intent Router" and "Context Pruner" for downstream applications that need to analyze context without full LLM generation. By analyzing the attention entropy and cross-attention resonance, applications can determine what the user is asking (Intent Routing) and which parts of the context are relevant (Attention Pruning) purely through the encoder's perspective.

Quick Inference (ONNX Runtime)

import onnxruntime as ort
from transformers import AutoTokenizer
import numpy as np

# Load model and tokenizer
session = ort.InferenceSession("model_quantized.onnx")
tokenizer = AutoTokenizer.from_pretrained("cross-encoder/nli-deberta-v3-small")

# Prepare input
inputs = tokenizer("Your task description here", return_tensors="np")
onnx_inputs = {
    "input_ids": inputs["input_ids"],
    "attention_mask": inputs["attention_mask"]
}

# Run with attentions
outputs = session.run(None, onnx_inputs)
# Last hidden state is outputs[0], logits/pooler output might be next, followed by attentions

Attribution & Original Work

• Original Model: nli-deberta-v3-small by cross-encoder.
• Base Model: DeBERTa-v3-small by Microsoft.

License

This model is licensed under the MIT License. You are free to use, modify, and distribute this model for any purpose, including commercial applications, provided you include the original license notice.

---

License details: MIT License