File size: 3,648 Bytes
fb65c5c | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 | ---
language:
- en
license: apache-2.0
tags:
- sparse-encoder
- onnx
- quantized
- int8
- splade
- financial
- inference-optimized
base_model: oneryalcin/fin-sparse-encoder-doc-v1
pipeline_tag: feature-extraction
library_name: onnxruntime
---
# fin-sparse-encoder-doc-v1-onnx
ONNX + INT8 quantized version of [oneryalcin/fin-sparse-encoder-doc-v1](https://huggingface.co/oneryalcin/fin-sparse-encoder-doc-v1) for CPU-efficient document encoding.
This is the **document encoder path only** — it produces sparse SPLADE vectors for indexing financial documents (SEC filings, earnings call transcripts). Query encoding uses a separate IDF lookup table (sub-ms, no neural model needed).
## Model Variants
| File | Format | Size | Use Case |
|---|---|---|---|
| `model.onnx` | FP32 | 647.9 MB | Maximum accuracy, GPU or high-memory CPU |
| `model_quantized.onnx` | INT8 | 166.7 MB | **Recommended for CPU deployment** |
## Performance
### Domain Evaluation (Financial Documents)
The parent model ([fin-sparse-encoder-doc-v1](https://huggingface.co/oneryalcin/fin-sparse-encoder-doc-v1)) was evaluated on 2,028 held-out financial test examples:
| Metric | Base Model | Fine-tuned | Delta |
|:---|:---|:---|:---|
| **acc@1** | 39.9% | **55.2%** | +15.2% |
| **acc@3** | 69.2% | **84.0%** | +14.8% |
| **ndcg@10** | 0.681 | **0.781** | +10.0% |
| median_rank | 2.0 | **1.0** | -1.0 |
### Inference Latency (seq_len=512, 1 thread)
Benchmarked on Apple M-series CPU. Server CPUs with AVX512-VNNI will see larger INT8 speedups (~2-3x).
| Backend | p50 (ms) | p95 (ms) | Model Size |
|---|---|---|---|
| PyTorch FP32 | 186.3 | 192.8 | ~620 MB |
| ONNX FP32 | 211.7 | 218.9 | 647.9 MB |
| **ONNX INT8** | **164.4** | **166.9** | **166.7 MB** |
## Usage
```python
import numpy as np
import onnxruntime as ort
from transformers import AutoTokenizer
# Load
tokenizer = AutoTokenizer.from_pretrained("oneryalcin/fin-sparse-encoder-doc-v1-onnx")
sess = ort.InferenceSession("model_quantized.onnx", providers=["CPUExecutionProvider"])
# Encode a document
text = "Revenue increased 12% year over year to $4.2 billion in Q4 2023."
inputs = tokenizer(text, return_tensors="np", padding="max_length", max_length=512, truncation=True)
logits = sess.run(None, {"input_ids": inputs["input_ids"], "attention_mask": inputs["attention_mask"]})[0]
# SpladePooling: log1p_relu activation (matches OpenSearch v3 models)
masked = logits * inputs["attention_mask"][..., None]
pooled = masked.max(axis=1)
sparse_vector = np.log1p(np.log1p(np.maximum(pooled, 0.0))) # [1, 30522]
# Convert to token->weight dict (for inverted index)
nonzero = np.nonzero(sparse_vector[0])[0]
token_weights = {tokenizer.decode([tid]): float(sparse_vector[0, tid]) for tid in nonzero}
print(f"Active dimensions: {len(token_weights)}")
print(f"Top tokens: {sorted(token_weights.items(), key=lambda x: -x[1])[:10]}")
```
## Architecture
```
Input text
→ Tokenizer (max_length=512)
→ ONNX model (MLM logits) [batch, seq, 30522]
→ SpladePooling: log(1 + log(1 + ReLU(max_over_seq(logits * mask))))
→ Sparse vector [batch, 30522]
```
Base model: `opensearch-project/opensearch-neural-sparse-encoding-doc-v3-gte` (Alibaba-NLP/new-impl architecture).
Fine-tuned on [financial-filings-sparse-retrieval-training](https://huggingface.co/datasets/oneryalcin/financial-filings-sparse-retrieval-training) (18K examples, 2 epochs).
## Export Details
- Exported via `torch.onnx.export` (legacy tracer, opset 17)
- INT8: dynamic quantization via `onnxruntime.quantization.quantize_dynamic` (per-channel, QInt8)
- Numerical verification: FP32 ONNX max diff vs PyTorch = 0.000057
|