README.md

---
language:
- en
license: apache-2.0
tags:
- sparse-encoder
- onnx
- quantized
- int8
- splade
- financial
- inference-optimized
base_model: oneryalcin/fin-sparse-encoder-doc-v1
pipeline_tag: feature-extraction
library_name: onnxruntime
---

# fin-sparse-encoder-doc-v1-onnx

ONNX + INT8 quantized version of [oneryalcin/fin-sparse-encoder-doc-v1](https://huggingface.co/oneryalcin/fin-sparse-encoder-doc-v1) for CPU-efficient document encoding.

This is the **document encoder path only** — it produces sparse SPLADE vectors for indexing financial documents (SEC filings, earnings call transcripts). Query encoding uses a separate IDF lookup table (sub-ms, no neural model needed).

## Model Variants

| File | Format | Size | Use Case |
|---|---|---|---|
| `model.onnx` | FP32 | 647.9 MB | Maximum accuracy, GPU or high-memory CPU |
| `model_quantized.onnx` | INT8 | 166.7 MB | **Recommended for CPU deployment** |

## Performance

### Domain Evaluation (Financial Documents)

The parent model ([fin-sparse-encoder-doc-v1](https://huggingface.co/oneryalcin/fin-sparse-encoder-doc-v1)) was evaluated on 2,028 held-out financial test examples:

| Metric | Base Model | Fine-tuned | Delta |
|:---|:---|:---|:---|
| **acc@1** | 39.9% | **55.2%** | +15.2% |
| **acc@3** | 69.2% | **84.0%** | +14.8% |
| **ndcg@10** | 0.681 | **0.781** | +10.0% |
| median_rank | 2.0 | **1.0** | -1.0 |

### Inference Latency (seq_len=512, 1 thread)

Benchmarked on Apple M-series CPU. Server CPUs with AVX512-VNNI will see larger INT8 speedups (~2-3x).

| Backend | p50 (ms) | p95 (ms) | Model Size |
|---|---|---|---|
| PyTorch FP32 | 186.3 | 192.8 | ~620 MB |
| ONNX FP32 | 211.7 | 218.9 | 647.9 MB |
| **ONNX INT8** | **164.4** | **166.9** | **166.7 MB** |

## Usage

```python
import numpy as np
import onnxruntime as ort
from transformers import AutoTokenizer

# Load
tokenizer = AutoTokenizer.from_pretrained("oneryalcin/fin-sparse-encoder-doc-v1-onnx")
sess = ort.InferenceSession("model_quantized.onnx", providers=["CPUExecutionProvider"])

# Encode a document
text = "Revenue increased 12% year over year to $4.2 billion in Q4 2023."
inputs = tokenizer(text, return_tensors="np", padding="max_length", max_length=512, truncation=True)
logits = sess.run(None, {"input_ids": inputs["input_ids"], "attention_mask": inputs["attention_mask"]})[0]

# SpladePooling: log1p_relu activation (matches OpenSearch v3 models)
masked = logits * inputs["attention_mask"][..., None]
pooled = masked.max(axis=1)
sparse_vector = np.log1p(np.log1p(np.maximum(pooled, 0.0)))  # [1, 30522]

# Convert to token->weight dict (for inverted index)
nonzero = np.nonzero(sparse_vector[0])[0]
token_weights = {tokenizer.decode([tid]): float(sparse_vector[0, tid]) for tid in nonzero}
print(f"Active dimensions: {len(token_weights)}")
print(f"Top tokens: {sorted(token_weights.items(), key=lambda x: -x[1])[:10]}")
```

## Architecture

```
Input text
  → Tokenizer (max_length=512)
  → ONNX model (MLM logits) [batch, seq, 30522]
  → SpladePooling: log(1 + log(1 + ReLU(max_over_seq(logits * mask))))
  → Sparse vector [batch, 30522]
```

Base model: `opensearch-project/opensearch-neural-sparse-encoding-doc-v3-gte` (Alibaba-NLP/new-impl architecture).

Fine-tuned on [financial-filings-sparse-retrieval-training](https://huggingface.co/datasets/oneryalcin/financial-filings-sparse-retrieval-training) (18K examples, 2 epochs).

## Export Details

- Exported via `torch.onnx.export` (legacy tracer, opset 17)
- INT8: dynamic quantization via `onnxruntime.quantization.quantize_dynamic` (per-channel, QInt8)
- Numerical verification: FP32 ONNX max diff vs PyTorch = 0.000057