---
title: thinletter
emoji: 🪶
colorFrom: pink
colorTo: yellow
sdk: static
pinned: false
---

# thinletter — a query encoder as a 120 MiB file, in the browser, open source

We take an embedding model, leave its document index exactly as it is, and compile the **query side** into a file a tenth of the size that
answers in the browser in about 100 ms. Everything behind the numbers is public under Apache-2.0: the quantiser, the container compiler,
the WebGPU runtime, the verification recipe and every released file. Web: [thinletter.io](https://thinletter.io) · code and report:
[github.com/rosecky/embedding-quantization-public](https://github.com/rosecky/embedding-quantization-public) · demo:
[thinletter.io/demo](https://thinletter.io/demo) or the Space [thinletter/demo](https://huggingface.co/spaces/thinletter/demo) · contact: info@thinletter.io

## Quickstart: a released client in your page

```js
import { loadVqwClient } from 'https://thinletter.io/vqweb/vqweb.js';   // or: npm install @thinletterio/vqweb

const client = await loadVqwClient('https://huggingface.co/thinletter/qwen3-embedding-0.6b-vq-clients/resolve/main/qwen3-0.6b-vq3.5d2-webfaq_q-vocfull.vqw');
const q = await client.embed('Kolik stojí parkování v centru Prahy?');   // Float32Array(1024), L2-normalised: dot it with your Qwen3-Embedding-0.6B index
```

`@thinletterio/vqweb` ([npm](https://www.npmjs.com/package/@thinletterio/vqweb), Apache-2.0) is the open WebGPU runtime as a package: download with
progress, a cache in the browser's private file system, tokenizer and prompt from the container header, pipelines, warm-up. Needs WebGPU
with `shader-f16` (Chrome / Edge), no CPU fallback; about 100 ms per query on an integrated GPU. The scalar GGUF files run with
[wllama](https://github.com/ngxson/wllama) or llama.cpp.

## What holds

| number | what it is | where to check |
|---|---|---|
| **98.2 %** | of full-precision nDCG@10 from a 120 MiB, 2.1-bit vector-quantised harrier-0.6b client, no retraining: above the trained BitNet-270m (140 MiB, 97.0 %), below any file llama.cpp can produce (178 MiB, 93.9 %). The browser reproduces the simulation (0.7375 vs 0.7380). | [harrier-0.6b-vq-clients](https://huggingface.co/thinletter/harrier-0.6b-vq-clients), [BeIR/scifact](https://huggingface.co/datasets/BeIR/scifact) |
| **4.3×** | faster than llama.cpp's WebGPU path on the same integrated GPU: 103 ms per query for the 119 MiB vector-quantised client against 444 ms for the 235 MiB Q3_K file, in our open WebGPU runtime (`client/vqweb`). | report §3.4b in the repository |
| **96.8 %** | of fp32 nDCG@10 on the public Czech benchmark WebFAQ-cs from a 237 MiB vector-quantised Qwen3-Embedding-0.6B client (2-d codebooks, 3.6 bits per weight), browser = simulation, 112 ms per query. Czech-calibrated scalar clients: bge-m3 99.2 % (355 MiB), Qwen3 98.7 % (340 MiB) on a Czech legal index. | [qwen3-embedding-0.6b-vq-clients](https://huggingface.co/thinletter/qwen3-embedding-0.6b-vq-clients), [PaDaS-Lab/webfaq-retrieval](https://huggingface.co/datasets/PaDaS-Lab/webfaq-retrieval) |
| **+0.010** | nDCG@10 at 1.8 bits from a method of ours: ship the query prompt's keys and values in full precision (2 MB) and calibrate the compressed model with them in place; three seeds on SciFact and on a Czech legal index, every paired interval above zero. | report §3.7; the `*p` files in [harrier-0.6b-vq-clients](https://huggingface.co/thinletter/harrier-0.6b-vq-clients) |
| **96–101 %** | kept by the 3.4-bit harrier-0.6b scalar client on four English corpora, 235 MiB instead of 1 143 MiB. Not ours to claim: llama.cpp's own quantiser gets there too, and we say so. | [harrier-0.6b-query-clients](https://huggingface.co/thinletter/harrier-0.6b-query-clients) |

## Repositories

Query-side clients: each file is a query encoder for exactly one document encoder and its settings (the model card says which).
The index does not change.

| repository | base model · licence | files | what holds (test split, the model's own fp32 index) |
|---|---|---|---|
| [harrier-0.6b-query-clients](https://huggingface.co/thinletter/harrier-0.6b-query-clients) | [microsoft/harrier-oss-v1-0.6b](https://huggingface.co/microsoft/harrier-oss-v1-0.6b) · MIT | 3 GGUF, 192–235 MiB | Q3_K generic-English 98.9 / 99.3 / 101 / 99.2 % on [SciFact](https://huggingface.co/datasets/BeIR/scifact) / [NFCorpus](https://huggingface.co/datasets/BeIR/nfcorpus) / [ArguAna](https://huggingface.co/datasets/BeIR/arguana) / [SciDocs](https://huggingface.co/datasets/BeIR/scidocs); Q2_K SciDocs-calibrated 94.5 % |
| [harrier-0.6b-vq-clients](https://huggingface.co/thinletter/harrier-0.6b-vq-clients) | microsoft/harrier-oss-v1-0.6b · MIT | 8 `.vqw`, 105–129 MiB, 1.8–2.1 bits per weight | SciDocs 94.7 / 91.3 %, SciFact 98.2 / 96.5 %; prompt-K/V variants SciDocs 96.6 / 93.8 %, SciFact 98.1 / 95.8 %; browser-verified; run in the open WebGPU runtime |
| [qwen3-embedding-0.6b-query-clients](https://huggingface.co/thinletter/qwen3-embedding-0.6b-query-clients) | [Qwen/Qwen3-Embedding-0.6B](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B) · Apache-2.0 | 4 GGUF, 340–385 MiB, English and Czech calibration | Q4_K_M + 4-bit table: 99.3–100 % on the four English corpora, 98.7 % on a Czech legal index; Q5_K_M 99.5 % Czech |
| [qwen3-embedding-0.6b-vq-clients](https://huggingface.co/thinletter/qwen3-embedding-0.6b-vq-clients) | Qwen/Qwen3-Embedding-0.6B · Apache-2.0 | 1 `.vqw`, 237 MiB, 2-d codebooks 3.57 bpw + full token table | Czech: 96.8 % of fp32 on [WebFAQ-cs](https://huggingface.co/datasets/PaDaS-Lab/webfaq-retrieval) (cosine 0.946, overlap 0.755), browser = simulation, 112 ms on an integrated GPU |
| [bge-m3-query-clients](https://huggingface.co/thinletter/bge-m3-query-clients) | [BAAI/bge-m3](https://huggingface.co/BAAI/bge-m3) · MIT | 2 GGUF, 321–355 MiB, Czech calibration | 99.2 % / 98.3 % on the Czech legal index, 99.6–100 % on SciFact |
| [demo](https://huggingface.co/spaces/thinletter/demo) (Space) | — | the browser demo, static | search SciDocs (25 656 abstracts) or WebFAQ-cs (71 529 Czech passages) locally with a scalar or a vector-quantised client; models and indexes download from thinletter.io |
| [llm-weight-compression-evidence](https://huggingface.co/datasets/thinletter/llm-weight-compression-evidence) | dataset · MIT | per-window evidence | which objective a post-training quantiser should optimise (companion project) |

Each model repository has a summary post in its Community tab with the numbers, what was found along the way, and the limits. The `.vqw` files load with `@thinletterio/vqweb` (above); the GGUF files with llama.cpp / wllama.

## How we measure

Test split, the model's own fp32 index, native runtime (llama.cpp) or the browser. Every run is pre-registered with a prediction and a
kill rule; comparisons change one thing at a time at the same calibration budget; differences are paired bootstrap intervals over queries
(10 000 draws) read against the variance of the calibration draw (~0.01 nDCG@10) and between machines (±0.006): under 0.01 is a tie.
Negative results stay published (a codebook-budget idea that did not work, a 2.6-bit headline that held on one corpus only, the 2-bit cliff
of Qwen3-Embedding: 65–81 % for the same run on two machines).

## Verify on your own index

The [public repository](https://github.com/rosecky/embedding-quantization-public) has the recipe: import your corpus and existing
document vectors, generate synthetic queries, quantise (llama-quantize, the GPTQ exporter, or the vector quantiser + `.vqw` compiler),
evaluate the client against the unchanged index with a paired interval. Everything in the report is reproducible from it; only the raw
per-run results, the research log and a customer's data stay out.

## Licences

Code Apache-2.0 (NOTICE lists wllama MIT, @huggingface/tokenizers Apache-2.0, a Q2_K decoder ported from llama.cpp MIT). Weights under
their base model's licence (MIT / Apache-2.0). Demo corpora: SciDocs (BEIR, CC BY 4.0), WebFAQ-cs (CC BY 4.0). Files derived from
non-commercial bases (jina-embeddings-v5, CC BY-NC) are reported as numbers only.