mimelens-001 cell: small/byte/s2

README.md

@@ -1,16 +1,20 @@
 ---
 license: mit
 library_name: transformers
+language:
+  - en
 tags:
   - file-type-detection
   - mime-classification
   - binary-content
+  - binary-analysis
   - position-agnostic
   - libmagic
+  - forensics
+  - packet-inspection
   - byte-level
   - mimelens
-language: en
-pipeline_tag: feature-extraction
+pipeline_tag: text-classification
 model-index:
   - name: mimelens-001-small-byte-s2
     results:
@@ -18,8 +22,8 @@ model-index:
           type: feature-extraction
           name: MIME-125 classification (libmagic 125-class taxonomy)
         dataset:
-          name: magic-bpe magic-frags (4 KB head of 64 KB random chunks, n=4,096)
-          type: mjbommar/magic-bpe-stratified
+          name: magic-frags (4 KB head of 64 KB random chunks, n=4,096)
+          type: custom
         metrics:
           - name: top-1 accuracy
             type: accuracy
@@ -32,92 +36,132 @@ model-index:
             value: 0.7187
         source:
           name: "MimeLens paper (Bommarito 2026), Appendix A"
-          url: https://github.com/mjbommar/binary-embedding-paper
+          url: https://github.com/mjbommar/mimelens-training
 ---
 
-# MimeLens-001 / small / byte / s2
+# mimelens-001-small-byte-s2
 
-**One cell from the [MimeLens-001](https://huggingface.co/mjbommar/mimelens-001) family** — `14.16` M backbone params, `byte` input pipeline, seed `2`. Pretrained MLM-only on 33 GB of position-arbitrary binary content for fine-grained file-content-type classification under [libmagic](https://github.com/file/file)'s 125-class MIME taxonomy.
+A 14.16M-backbone-parameter BERT-style encoder for position-agnostic file-content-type detection from binary data. It reads a byte window taken from *any* offset in a file (the first ~1{,}022 tokens of whatever you pass) and produces a 384-dimensional embedding that classifiers map to one of [libmagic](https://github.com/file/file)'s 125 MIME labels. Designed for inputs where you only have a chunk: a forensic-carved fragment, a random disk-block read, a streaming HTTP upload, a single network packet payload.
+
+- **🔗 Model**: [`mjbommar/mimelens-001-small-byte-s2`](https://huggingface.co/mjbommar/mimelens-001-small-byte-s2)
+- **👥 Family**: [`mjbommar/mimelens-001`](https://huggingface.co/mjbommar/mimelens-001) (36 released cells: 28 parent + 8 short-sequence)
+- **📄 Paper**: *MimeLens: Position-Agnostic Content-Type Detection for Binary Fragments* (Bommarito 2026)
+- **💻 Training code**: [`mjbommar/mimelens-training`](https://github.com/mjbommar/mimelens-training)
+- **📊 Pretraining corpus**: [`mjbommar/binary-30k-tokenized`](https://huggingface.co/datasets/mjbommar/binary-30k-tokenized) plus magic-corpus extracts, packed binaries, a [`glaurung`](https://github.com/mjbommar/glaurung)-sourced binary corpus, and Windows drivers (33 GB stratified; the full corpus is not redistributable)
+
+---
+
+## What MimeLens does
+
+MimeLens classifies file content type from a byte window taken at any offset, not just the header of a complete file.
+
+Existing tools assume whole-file access at a known offset:
+
+- [`libmagic`](https://github.com/file/file) and [Apache Tika](https://tika.apache.org/) match handcrafted magic-byte signatures, almost always anchored at the file head.
+- [Magika](https://github.com/google/magika) (Google) is a small (~1 M-parameter) feedforward network over three 512-byte windows (head, middle, tail) of a known-bounded file.
+- TrID, PRONOM/Siegfried/DROID similarly require a complete file.
+
+These break down on a fragment. MimeLens is pretrained MLM-only on 1024-token windows sampled *uniformly at random* across files and 64 KB fragments, with no privileged head-of-file position. One checkpoint handles streaming, partial-arrival, mid-file, packet-payload, and forensic-carved inputs uniformly. The trade-off is CPU latency (roughly two orders of magnitude slower than Magika at the medium size; hardware-dependent) in exchange for libmagic's 125-class taxonomy plus position arbitrariness.
+
+The family ships 28 parent cells (3 sizes × 4 vocabs × 2-3 seeds at seq\_len=1024) plus an 8-cell short-sequence extension (medium tier × 4 vocabs × 2 seeds at seq\_len=256). This README documents one of them.
+
+> **Short-sequence sibling available.** If your inputs are sub-KB (DNS payloads, sub-MTU packets, small forensic fragments), use `mjbommar/mimelens-001-small-byte-s2-seq256` instead. Same architecture, 4× shorter context, ~5× lower CPU latency, BPE-cell accuracy ties or beats this cell on the magic-files probe-fit. See paper Appendix B.5.
 
-A single 4 KB byte buffer in (of which the first 1,022 body tokens are consumed), one of libmagic's 125 MIME labels out, regardless of where in a source file the buffer came from.
 
-For the family overview, decision tree (which cell to load?), and full cube results, see [`mjbommar/mimelens-001`](https://huggingface.co/mjbommar/mimelens-001).
 
-## How to use
+---
+
+## Overview
+
+- **This cell**: `small` tier, `byte` input pipeline, seed `2`
+- **Backbone**: 14.16M parameters (8 layers, hidden 384, 6 attention heads, head dim 64, RoPE, RMSNorm, no biases, no dropout)
+- **Input vocabulary**: `byte`. Raw 256-byte vocabulary plus 5 special tokens (CLS, SEP, PAD, UNK, MASK); id = byte_value + 5. The model reads exactly the first 1,022 bytes that arrive.
+- **Output**: 384-dim mean-pooled body-token embedding
+- **Label space**: [libmagic](https://github.com/file/file) 125-class MIME taxonomy (full list in paper Appendix)
+- **Pretraining**: MLM-only, 30% mask ratio, 33 GB stratified multi-source binary corpus, 22,888 gradient updates, single RTX 4060 Ti, ~10.7 h wall-clock
+- **License**: MIT
+
+## Headline benchmarks (this cell)
+
+| Benchmark | Value |
+|---|---|
+| MIME-125 top-1 (magic-frags, 4 KB head, n=4,096)            | **0.766** |
+| MIME-125 macro-F1 (magic-frags, 4 KB head)                  | 0.617 |
+| kNN R@1 (magic-frags, 3,147-file gallery / 949 queries)     | 0.719 |
+
+Full evaluation (within-cube bootstrap CIs, adversarial sweep, calibration, real-network curves, disk-block matrix, baselines against libmagic 5.46 and TrID 2.24) is in the [paper](https://github.com/mjbommar/mimelens-training).
+
+---
+
+## Quick start
+
+This cell publishes the encoder only (no classifier head baked in). Use it to extract embeddings, then fit a probe, run kNN over a labelled gallery, or fine-tune a head:
 
 ```python
 import torch
-from transformers import AutoModel
+from transformers import AutoModel, AutoTokenizer
 
-repo = "mjbommar/mimelens-001-small-byte-s2"
+repo  = "mjbommar/mimelens-001-small-byte-s2"
 model = AutoModel.from_pretrained(repo, trust_remote_code=True).eval()
-cfg = model.config
+tok   = AutoTokenizer.from_pretrained(repo)
 
-# Byte cell: tokenization is trivial — id == byte_value + byte_offset.
 window = open("path/to/file", "rb").read(4096)
-ids = [cfg.cls_token_id] + [b + cfg.byte_offset for b in window][:1022] + [cfg.sep_token_id]
-input_ids = torch.tensor([ids])
-
+inputs = tok(window.decode("latin-1"), max_length=1024, truncation=True,
+             padding="max_length", return_tensors="pt")
 with torch.no_grad():
-    out = model(input_ids=input_ids, attention_mask=torch.ones_like(input_ids))
-
-embedding = out.pooler_output     # (1, 384)  mean-pooled body-token embedding
-# Downstream: a frozen LR probe, a kNN over a labeled gallery, or fine-tune a classification head.
-# See the paper for the standard evaluation protocol.
+    embedding = model(**inputs).pooler_output         # (1, 384)
 ```
 
-## What this cell is
+The pre-fit LR probe weights for this cell are not bundled here. The deployed cells and per-size winners (e.g. `mimelens-001-medium-bpe-16k-s1`) ship a baked classifier head for a one-line `pipeline()` path.
 
-- **Family**: [MimeLens-001](https://huggingface.co/mjbommar/mimelens-001) — 28 pretrained checkpoints across 3 sizes × 4 vocabularies × 2 seeds, plus one matched-tokens-seen ablation.
-- **Size**: `small` — 14.16 M backbone params, 8 layers, hidden 384, 6 attention heads, head dim 64.
-- **Input pipeline**: `byte` (raw 256-byte vocabulary plus 5 special tokens (cls, sep, pad, unk, mask); the model reads exactly the first 1022 bytes that arrive in a 4 KB window.).
-- **Seed**: `2` (1 of 2 for this (size, vocab) combination).
-- **Pretraining**: 22,888 gradient updates, MLM-only, 30% mask ratio, 1024-token windows sampled uniformly at random across files and 64 KB fragments. AdamW + cosine LR (peak 5e-4, 2,000-step warmup, 10% floor), bf16 mixed precision, single RTX 4060 Ti.
-- **License**: MIT.
 
-## Evaluation
+---
 
-Numbers below are for **this specific cell** on the `magic-frags` held-out test set (4 KB head of 64 KB random chunks, n=4,096). The within-cube comparison (3 sizes × 4 vocabs × 2-3 seeds, bootstrap CIs, adversarial sweep, calibration, real-network and disk-block validations) is in the [paper](https://github.com/mjbommar/binary-embedding-paper).
+## Choosing a window
 
-| Benchmark | This cell |
-|---|---|
-| MIME-125 top-1 (magic-frags 4 KB head, n=4,096)        | **0.766** |
-| MIME-125 macro-F1 (magic-frags 4 KB head)              | 0.617 |
-| kNN R@1 (magic-frags, 3,147-file gallery / 949 queries) | 0.719 |
+The model reads the first ~1{,}022 tokens of whatever you pass — a prefix of the buffer (the first 1{,}022 bytes for this byte cell), not the whole window.
+
+- **Magic-byte / compressed types** (PNG, ZIP, GZIP, JPEG): a **short head window (256 B--1 KB) classifies better than 4 KB**. A long high-entropy body dilutes the header signal within the fixed token budget, and the model returns `application/octet-stream` on a mostly-opaque window — correct behaviour for genuinely high-entropy input, not a bug.
+- **Fragments / packets**: you cannot choose the offset, so pass what you have. This is the regime MimeLens is built for.
+
+---
 
 ## Recommended deployment regimes
 
 See the family hub README ([`mjbommar/mimelens-001`](https://huggingface.co/mjbommar/mimelens-001)) for the regime decision tree.
 
+---
+
 ## Training
 
-This cell is one point of the pre-registered 3 × 4 × 2 factorial cube described in the [MimeLens paper](https://github.com/mjbommar/binary-embedding-paper). Salient details:
+This cell is one point of the 3 × 4 × 2 factorial cube described in the paper.
 
-- **33 GB stratified multi-source binary corpus** (binary-30k + magic-frags + glaurung + Windows drivers).
-- **Position-arbitrary windowing**: 1024-token windows sampled uniformly at random across files and 64 KB fragments — no privileged "head of file" position. This is what makes MimeLens work on streaming / partial / random-offset inputs that whole-file detectors were not designed for.
-- **MLM-only** objective, 30% mask ratio (BERT replacement schedule: 80% `[MASK]`, 10% random, 10% original); tied input/output embeddings.
-- **Mean-pool over body tokens** for downstream tasks; the BERT-style `cls_pool` linear projection is *not* used because under MLM-only training it receives no gradient and remains at random init across all 28 cube cells (paper §3.4 verifies this).
-- **Wall-clock**: ~10.7 h on a single RTX 4060 Ti.
+- **Corpus** (33 GB, stratified multi-source): [`binary-30k`](https://huggingface.co/datasets/mjbommar/binary-30k-tokenized) (assorted ELF/PE/Mach-O), magic-frags (random 64 KB chunks across libmagic's full corpus), assorted packed/raw binaries, a [`glaurung`](https://github.com/mjbommar/glaurung)-sourced binary corpus, Windows drivers.
+- **Position-arbitrary windowing**: 1024-token windows sampled uniformly at random across files and 64 KB fragments. **No privileged "head of file" position.** This is the design choice that makes MimeLens work on streaming / partial / random-offset inputs.
+- **Objective**: MLM with 30% mask ratio (BERT replacement schedule: 80% `[MASK]`, 10% random, 10% original); tied input/output embeddings.
+- **Pooling**: mean-pool over body tokens for downstream tasks. The BERT-style `cls_pool` linear projection is *not* used: under MLM-only training it receives no gradient and remains byte-identical to its random initialisation across all 28 cube cells (paper §3.4 verifies this; left in the saved weights for architectural completeness only).
+- **Optimisation**: AdamW + cosine LR (peak 5e-4, 2,000-step warmup, 10% floor), bf16 mixed precision, gradient clipping at $\|g\|_2 \leq 1$, effective batch 128 at sequence length 1024, 22,888 gradient updates.
+- **Hardware**: single RTX 4060 Ti (16 GB), ~10.7 h wall-clock for this cell.
 
-## Honest caveats
+---
 
-- This is one cell of a 28-cell cube. Within-cube comparisons in the paper come with bootstrap CIs at n=2 seeds; some marginal orderings (byte vs bpe-16k at the top of medium) are within seed noise and should be read as ties.
+## Caveats
+
+- This is one cell of a 28-cell parent cube (36 released cells including the 8-cell short-sequence extension). Within-cube comparisons in the paper carry bootstrap CIs at n=2 seeds; some marginal orderings (byte vs bpe-16k at the largest size) are within seed noise and should be read as ties.
 - The training corpus is one 33 GB stratified multi-source binary sample. Results may not transfer to substantially different corpora.
-- All numbers are computed on data derived from a single labelling pipeline (libmagic-pinned via the [magic-bpe](https://github.com/mjbommar/magic-bpe) project). Cross-validation against PRONOM, Siegfried, DROID, or IANA reference files is a documented limitation.
-- CPU latency at the `medium` size is ~348× slower than Magika; for sub-millisecond whole-file triage on broad categories, Magika is purpose-built and is the right default. MimeLens occupies a different point on the deployment surface (position-arbitrary inputs + libmagic's 125-class taxonomy) rather than a drop-in replacement.
-- End-to-end fine-tuning on the production label distribution may shift these numbers and should be evaluated before deployment. The frozen-probe numbers reported above are not claimed as a lower bound on fine-tuned performance.
+- All numbers are computed on data labelled by a single pipeline (libmagic-pinned). Cross-validation against PRONOM, Siegfried, DROID, or IANA reference files is a documented limitation.
+- CPU latency at the `medium` size is ~155× slower than Magika v1.1 on a desktop CPU (hardware-dependent). For sub-millisecond whole-file triage on broad categories, Magika is purpose-built and is the right tool. MimeLens occupies a different point on the deployment surface (position-arbitrary inputs + libmagic's 125-class taxonomy), not a drop-in replacement.
+- End-to-end fine-tuning on the production label distribution may shift these numbers and should be evaluated before deployment. The frozen-probe numbers above are not claimed as a lower bound on fine-tuned performance.
+
+---
 
 ## Citation
 
 ```bibtex
 @misc{bommarito2026mimelens,
-  title  = {MimeLens: Pretrained Encoders for Fine-Grained Content-Type Detection},
+  title  = {MimeLens: Position-Agnostic Content-Type Detection for Binary Fragments},
   author = {Bommarito II, Michael J.},
   year   = {2026},
-  note   = {https://github.com/mjbommar/binary-embedding-paper},
+  note   = {https://github.com/mjbommar/mimelens-training},
 }
-```
-
-## Acknowledgments
-
-Thanks to the [magic-bpe](https://github.com/mjbommar/magic-bpe) project and the [binary-tokenizer-001](https://huggingface.co/mjbommar/binary-tokenizer-001-16k) family for the labelled corpus and BPE tokenizers this work builds on, and to the [Magika](https://github.com/google/magika) team for releasing a public package that made the §3 calibration possible.
+```

config.json

@@ -8,7 +8,7 @@
   },
   "model_type": "mimelens",
   "torch_dtype": "float32",
-  "vocab_size": 261,
+  "vocab_size": 263,
   "hidden_size": 384,
   "num_hidden_layers": 8,
   "num_attention_heads": 6,
@@ -22,7 +22,7 @@
   "cls_token_id": 4,
   "sep_token_id": 5,
   "mask_token_id": 6,
-  "byte_offset": 5,
+  "byte_offset": 7,
   "cls_pool_dim": 256,
   "mimelens_cell_id": "small/byte/s2",
   "mimelens_vocab_pipeline": "byte",

configuration_mimelens.py

@@ -28,8 +28,9 @@ class MimeLensConfig(PretrainedConfig):
     paper repository (https://github.com/mjbommar/binary-embedding-paper).
 
     Args:
-        vocab_size: int — full vocabulary including 5 special tokens. byte
-            cells: 261 (256 bytes + 5 specials). BPE cells: 4101 / 16391 / 65543.
+        vocab_size: int — full vocabulary including 7 special tokens (start, end,
+            pad, unk, cls, sep, mask). byte cells: 263 (256 bytes + 7 specials).
+            BPE cells: 4103 / 16391 / 65543.
         hidden_size: int — transformer model dimension (256 / 384 / 512 for
             tiny / small / medium).
         num_hidden_layers: int — layer count (4 / 8 / 12 for tiny / small /
@@ -46,7 +47,7 @@ class MimeLensConfig(PretrainedConfig):
         pad_token_id / cls_token_id / sep_token_id / mask_token_id: int —
             special-token indices, matching binary_embedding.constants.
         byte_offset: int — for byte cells, ord(b)+byte_offset gives the token
-            id. Fixed at 5 (after the 5 special tokens). Unused for BPE cells.
+            id. Fixed at 7 (after the 7 special tokens). Unused for BPE cells.
         cls_pool_dim: int — output dim of the cls_pool layer. Note: this layer
             receives no gradient under MLM-only training (see paper §3.4); the
             mean-pool over body tokens is the trained pooling, not cls_pool.
@@ -82,7 +83,7 @@ class MimeLensConfig(PretrainedConfig):
         cls_token_id: int = 4,
         sep_token_id: int = 5,
         mask_token_id: int = 6,
-        byte_offset: int = 5,
+        byte_offset: int = 7,
         cls_pool_dim: int = 256,
         initializer_range: float = 0.02,
         mimelens_cell_id: str = "medium/bpe-16k/s1",

modeling_mimelens.py

@@ -32,7 +32,7 @@ import torch
 import torch.nn as nn
 import torch.nn.functional as F
 from transformers import PreTrainedModel
-from transformers.modeling_outputs import BaseModelOutputWithPooling
+from transformers.modeling_outputs import BaseModelOutputWithPooling, SequenceClassifierOutput
 
 from .configuration_mimelens import MimeLensConfig
 
@@ -272,3 +272,103 @@ class MimeLensModel(PreTrainedModel):
         attention_mask = torch.tensor([attn], dtype=torch.long, device=device)
         with torch.inference_mode():
             return self(input_ids, attention_mask=attention_mask).pooler_output
+
+
+class MimeLensForSequenceClassification(PreTrainedModel):
+    """MimeLens encoder + a 125-class libmagic-MIME classifier head.
+
+    Lets users do, in one line:
+
+        from transformers import pipeline
+        clf = pipeline("text-classification",
+                       model="mjbommar/mimelens-001-medium-bpe-16k-s1",
+                       trust_remote_code=True)
+        clf(open("some.bin", "rb").read(4096).decode("latin-1"))
+        # → [{"label": "text/x-python", "score": 0.91}, ...]
+
+    The classifier head is the same logistic-regression probe the paper
+    reports on the magic-files corpus, re-fit on the full 4,096-file
+    labelled set and baked into `model.safetensors` as `classifier.weight`
+    and `classifier.bias`. Labels live in `config.id2label` / `config.label2id`.
+
+    For embedding-only use, load via `AutoModel.from_pretrained(...)` instead,
+    which returns mean-pooled embeddings and ignores the classifier head.
+    """
+
+    config_class = MimeLensConfig
+    base_model_prefix = "mimelens"
+
+    def __init__(self, config: MimeLensConfig):
+        super().__init__(config)
+        self.config = config
+        self.num_labels = getattr(config, "num_labels", 125)
+        # The encoder body, identical to MimeLensModel — same parameter names so
+        # the encoder weights load from the same safetensors keys.
+        self.embed = nn.Embedding(config.vocab_size, config.hidden_size)
+        self.layers = nn.ModuleList([Layer(config) for _ in range(config.num_hidden_layers)])
+        self.final_norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
+        self.cls_pool = nn.Linear(config.hidden_size, config.cls_pool_dim, bias=False)
+        # The 125-way classifier head.
+        self.classifier = nn.Linear(config.hidden_size, self.num_labels)
+
+        self._rope_cache: Optional[tuple[torch.Tensor, torch.Tensor]] = None
+        self._rope_cache_meta: Optional[tuple[torch.device, torch.dtype, int]] = None
+        self.post_init()
+
+    def _init_weights(self, module):
+        if isinstance(module, nn.Linear):
+            module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
+            if module.bias is not None:
+                module.bias.data.zero_()
+        elif isinstance(module, nn.Embedding):
+            module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
+            if module.padding_idx is not None:
+                module.weight.data[module.padding_idx].zero_()
+
+    def _get_rope(self, seq_len: int, device: torch.device, dtype: torch.dtype):
+        meta = (device, dtype, seq_len)
+        if self._rope_cache_meta != meta:
+            self._rope_cache = _build_rope_cache(seq_len, self.config.head_dim,
+                                                 self.config.rope_theta,
+                                                 device=device, dtype=dtype)
+            self._rope_cache_meta = meta
+        return self._rope_cache
+
+    def forward(
+        self,
+        input_ids: torch.LongTensor,
+        attention_mask: Optional[torch.Tensor] = None,
+        labels: Optional[torch.LongTensor] = None,
+        return_dict: bool = True,
+    ):
+        B, S = input_ids.shape
+        x = self.embed(input_ids)
+
+        if attention_mask is None:
+            attention_mask = torch.ones(B, S, device=input_ids.device, dtype=torch.long)
+        attn_mask = attention_mask.to(x.dtype)
+        attn_mask = (1.0 - attn_mask).masked_fill((1.0 - attn_mask).bool(),
+                                                    torch.finfo(x.dtype).min)
+        attn_mask = attn_mask.view(B, 1, 1, S)
+
+        cos, sin = self._get_rope(S, device=x.device, dtype=x.dtype)
+        for layer in self.layers:
+            x = layer(x, cos, sin, attn_mask)
+        x = self.final_norm(x)
+
+        lens = attention_mask.sum(dim=1, keepdim=True)
+        positions = torch.arange(S, device=x.device).unsqueeze(0)
+        body_mask = (positions >= 1) & (positions < (lens - 1))
+        body_mask_f = body_mask.to(x.dtype).unsqueeze(-1)
+        pooled = (x * body_mask_f).sum(dim=1) / body_mask_f.sum(dim=1).clamp(min=1)
+
+        # Cast pooled to classifier dtype (bf16 encoder + fp32 classifier is common).
+        logits = self.classifier(pooled.to(self.classifier.weight.dtype))
+
+        loss = None
+        if labels is not None:
+            loss = F.cross_entropy(logits, labels)
+
+        if not return_dict:
+            return (loss, logits) if loss is not None else (logits,)
+        return SequenceClassifierOutput(loss=loss, logits=logits)

tokenizer.json

@@ -0,0 +1,372 @@
+{
+  "version": "1.0",
+  "truncation": {
+    "direction": "Right",
+    "max_length": 1024,
+    "strategy": "LongestFirst",
+    "stride": 0
+  },
+  "padding": {
+    "strategy": "BatchLongest",
+    "direction": "Right",
+    "pad_to_multiple_of": null,
+    "pad_id": 2,
+    "pad_type_id": 0,
+    "pad_token": "[PAD]"
+  },
+  "added_tokens": [],
+  "normalizer": null,
+  "pre_tokenizer": {
+    "type": "Split",
+    "pattern": {
+      "Regex": "[\\s\\S]"
+    },
+    "behavior": "Isolated",
+    "invert": false
+  },
+  "post_processor": {
+    "type": "TemplateProcessing",
+    "single": [
+      {
+        "SpecialToken": {
+          "id": "[CLS]",
+          "type_id": 0
+        }
+      },
+      {
+        "Sequence": {
+          "id": "A",
+          "type_id": 0
+        }
+      },
+      {
+        "SpecialToken": {
+          "id": "[SEP]",
+          "type_id": 0
+        }
+      }
+    ],
+    "pair": [
+      {
+        "SpecialToken": {
+          "id": "[CLS]",
+          "type_id": 0
+        }
+      },
+      {
+        "Sequence": {
+          "id": "A",
+          "type_id": 0
+        }
+      },
+      {
+        "SpecialToken": {
+          "id": "[SEP]",
+          "type_id": 0
+        }
+      },
+      {
+        "Sequence": {
+          "id": "B",
+          "type_id": 1
+        }
+      },
+      {
+        "SpecialToken": {
+          "id": "[SEP]",
+          "type_id": 1
+        }
+      }
+    ],
+    "special_tokens": {
+      "[CLS]": {
+        "id": "[CLS]",
+        "ids": [
+          4
+        ],
+        "tokens": [
+          "[CLS]"
+        ]
+      },
+      "[SEP]": {
+        "id": "[SEP]",
+        "ids": [
+          5
+        ],
+        "tokens": [
+          "[SEP]"
+        ]
+      }
+    }
+  },
+  "decoder": null,
+  "model": {
+    "type": "WordLevel",
+    "vocab": {
+      "<|start|>": 0,
+      "<|end|>": 1,
+      "[PAD]": 2,
+      "[UNK]": 3,
+      "[CLS]": 4,
+      "[SEP]": 5,
+      "[MASK]": 6,
+      "\u0000": 7,
+      "\u0001": 8,
+      "\u0002": 9,
+      "\u0003": 10,
+      "\u0004": 11,
+      "\u0005": 12,
+      "\u0006": 13,
+      "\u0007": 14,
+      "\b": 15,
+      "\t": 16,
+      "\n": 17,
+      "\u000b": 18,
+      "\f": 19,
+      "\r": 20,
+      "\u000e": 21,
+      "\u000f": 22,
+      "\u0010": 23,
+      "\u0011": 24,
+      "\u0012": 25,
+      "\u0013": 26,
+      "\u0014": 27,
+      "\u0015": 28,
+      "\u0016": 29,
+      "\u0017": 30,
+      "\u0018": 31,
+      "\u0019": 32,
+      "\u001a": 33,
+      "\u001b": 34,
+      "\u001c": 35,
+      "\u001d": 36,
+      "\u001e": 37,
+      "\u001f": 38,
+      " ": 39,
+      "!": 40,
+      "\"": 41,
+      "#": 42,
+      "$": 43,
+      "%": 44,
+      "&": 45,
+      "'": 46,
+      "(": 47,
+      ")": 48,
+      "*": 49,
+      "+": 50,
+      ",": 51,
+      "-": 52,
+      ".": 53,
+      "/": 54,
+      "0": 55,
+      "1": 56,
+      "2": 57,
+      "3": 58,
+      "4": 59,
+      "5": 60,
+      "6": 61,
+      "7": 62,
+      "8": 63,
+      "9": 64,
+      ":": 65,
+      ";": 66,
+      "<": 67,
+      "=": 68,
+      ">": 69,
+      "?": 70,
+      "@": 71,
+      "A": 72,
+      "B": 73,
+      "C": 74,
+      "D": 75,
+      "E": 76,
+      "F": 77,
+      "G": 78,
+      "H": 79,
+      "I": 80,
+      "J": 81,
+      "K": 82,
+      "L": 83,
+      "M": 84,
+      "N": 85,
+      "O": 86,
+      "P": 87,
+      "Q": 88,
+      "R": 89,
+      "S": 90,
+      "T": 91,
+      "U": 92,
+      "V": 93,
+      "W": 94,
+      "X": 95,
+      "Y": 96,
+      "Z": 97,
+      "[": 98,
+      "\\": 99,
+      "]": 100,
+      "^": 101,
+      "_": 102,
+      "`": 103,
+      "a": 104,
+      "b": 105,
+      "c": 106,
+      "d": 107,
+      "e": 108,
+      "f": 109,
+      "g": 110,
+      "h": 111,
+      "i": 112,
+      "j": 113,
+      "k": 114,
+      "l": 115,
+      "m": 116,
+      "n": 117,
+      "o": 118,
+      "p": 119,
+      "q": 120,
+      "r": 121,
+      "s": 122,
+      "t": 123,
+      "u": 124,
+      "v": 125,
+      "w": 126,
+      "x": 127,
+      "y": 128,
+      "z": 129,
+      "{": 130,
+      "|": 131,
+      "}": 132,
+      "~": 133,
+      "": 134,
+      "€": 135,
+      "": 136,
+      "‚": 137,
+      "ƒ": 138,
+      "„": 139,
+      "…": 140,
+      "†": 141,
+      "‡": 142,
+      "ˆ": 143,
+      "‰": 144,
+      "Š": 145,
+      "‹": 146,
+      "Œ": 147,
+      "": 148,
+      "Ž": 149,
+      "": 150,
+      "": 151,
+      "‘": 152,
+      "’": 153,
+      "“": 154,
+      "”": 155,
+      "•": 156,
+      "–": 157,
+      "—": 158,
+      "˜": 159,
+      "™": 160,
+      "š": 161,
+      "›": 162,
+      "œ": 163,
+      "": 164,
+      "ž": 165,
+      "Ÿ": 166,
+      " ": 167,
+      "¡": 168,
+      "¢": 169,
+      "£": 170,
+      "¤": 171,
+      "¥": 172,
+      "¦": 173,
+      "§": 174,
+      "¨": 175,
+      "©": 176,
+      "ª": 177,
+      "«": 178,
+      "¬": 179,
+      "­": 180,
+      "®": 181,
+      "¯": 182,
+      "°": 183,
+      "±": 184,
+      "²": 185,
+      "³": 186,
+      "´": 187,
+      "µ": 188,
+      "¶": 189,
+      "·": 190,
+      "¸": 191,
+      "¹": 192,
+      "º": 193,
+      "»": 194,
+      "¼": 195,
+      "½": 196,
+      "¾": 197,
+      "¿": 198,
+      "À": 199,
+      "Á": 200,
+      "Â": 201,
+      "Ã": 202,
+      "Ä": 203,
+      "Å": 204,
+      "Æ": 205,
+      "Ç": 206,
+      "È": 207,
+      "É": 208,
+      "Ê": 209,
+      "Ë": 210,
+      "Ì": 211,
+      "Í": 212,
+      "Î": 213,
+      "Ï": 214,
+      "Ð": 215,
+      "Ñ": 216,
+      "Ò": 217,
+      "Ó": 218,
+      "Ô": 219,
+      "Õ": 220,
+      "Ö": 221,
+      "×": 222,
+      "Ø": 223,
+      "Ù": 224,
+      "Ú": 225,
+      "Û": 226,
+      "Ü": 227,
+      "Ý": 228,
+      "Þ": 229,
+      "ß": 230,
+      "à": 231,
+      "á": 232,
+      "â": 233,
+      "ã": 234,
+      "ä": 235,
+      "å": 236,
+      "æ": 237,
+      "ç": 238,
+      "è": 239,
+      "é": 240,
+      "ê": 241,
+      "ë": 242,
+      "ì": 243,
+      "í": 244,
+      "î": 245,
+      "ï": 246,
+      "ð": 247,
+      "ñ": 248,
+      "ò": 249,
+      "ó": 250,
+      "ô": 251,
+      "õ": 252,
+      "ö": 253,
+      "÷": 254,
+      "ø": 255,
+      "ù": 256,
+      "ú": 257,
+      "û": 258,
+      "ü": 259,
+      "ý": 260,
+      "þ": 261,
+      "ÿ": 262
+    },
+    "unk_token": "[UNK]"
+  }
+}

tokenizer_config.json

@@ -0,0 +1,54 @@
+{
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "model_max_length": 1024,
+  "padding_side": "right",
+  "truncation_side": "right",
+  "pad_token": "[PAD]",
+  "unk_token": "[UNK]",
+  "cls_token": "[CLS]",
+  "sep_token": "[SEP]",
+  "mask_token": "[MASK]",
+  "clean_up_tokenization_spaces": false,
+  "added_tokens_decoder": {
+    "2": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "rstrip": false,
+      "single_word": false,
+      "normalized": false,
+      "special": true
+    },
+    "3": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "rstrip": false,
+      "single_word": false,
+      "normalized": false,
+      "special": true
+    },
+    "4": {
+      "content": "[CLS]",
+      "lstrip": false,
+      "rstrip": false,
+      "single_word": false,
+      "normalized": false,
+      "special": true
+    },
+    "5": {
+      "content": "[SEP]",
+      "lstrip": false,
+      "rstrip": false,
+      "single_word": false,
+      "normalized": false,
+      "special": true
+    },
+    "6": {
+      "content": "[MASK]",
+      "lstrip": false,
+      "rstrip": false,
+      "single_word": false,
+      "normalized": false,
+      "special": true
+    }
+  }
+}