Initial DNABERT-4mer Hugging Face port

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,188 @@
+---
+library_name: transformers
+tags:
+- biology
+- DNA
+- language-model
+- genomics
+license: apache-2.0
+---
+
+# DNABERT-4mer
+
+Minimal HuggingFace port of the **4-mer** variant of
+[DNABERT](https://huggingface.co/zhihan1996/DNA_bert_4) -- a BERT-base
+masked language model pre-trained on the human reference genome using
+overlapping 4-mer tokenization.
+
+**This repo contains only weights and tokenizer files.** The model code is loaded
+automatically from `Taykhoom/BERT-updated` via `trust_remote_code=True`.
+
+## Architecture
+
+Standard BERT-base with a 4-mer DNA vocabulary.
+
+| Parameter | Value |
+|---|---|
+| Layers | 12 |
+| Attention heads | 12 |
+| Embedding dimension | 768 |
+| FFN hidden dimension | 3072 (GELU) |
+| Vocabulary size | 261 (5 special + 256 DNA 4-mers) |
+| Positional encoding | Learned absolute |
+| Normalization | Post-LayerNorm (epsilon 1e-12) |
+| Architecture | Bidirectional BERT encoder |
+| Max sequence length | 512 tokens (510 k-mers; 513 nucleotides) |
+| Runtime parameters | 87,034,629 |
+
+### Tokenization
+
+Input sequences must be pre-split into overlapping 4-mers (stride 1) with spaces
+between tokens before calling the tokenizer. For example:
+
+```
+ATCGATG  ->  ATCG TCGA CGAT GATG
+```
+
+```python
+def seq_to_kmers(seq, k=4):
+    return " ".join(seq[i:i+k] for i in range(len(seq) - k + 1))
+```
+
+## Pretraining
+
+- **Objective:** Masked Language Modeling
+- **Data:** Human reference genome (GRCh38)
+- **Source checkpoint:** `pytorch_model.bin` from [zhihan1996/DNA_bert_4](https://huggingface.co/zhihan1996/DNA_bert_4)
+
+## Parity Verification
+
+All 13 representation levels (embedding + 12 transformer layers) verified
+against the source implementation (max abs diff = 9.79e-5); MLM logits match
+with max abs diff = 2.52e-4. The source `dnabert_layer.BertModel` is a direct
+subclass of `transformers.BertModel` with no modifications.
+Verified on GPU with PyTorch 2.7.1 / CUDA 12.9.
+
+## Related Models
+
+See the full [DNABERT collection](https://huggingface.co/collections/Taykhoom/dnabert-6a20958f8ce004ea4e985e7b).
+
+| Model | Architecture | Notes |
+|---|---|---|
+| [DNABERT-3mer](https://huggingface.co/Taykhoom/DNABERT-3mer) | BERT + k-mer | k=3 |
+| **[DNABERT-4mer](https://huggingface.co/Taykhoom/DNABERT-4mer)** | **BERT + k-mer** | **k=4** |
+| [DNABERT-5mer](https://huggingface.co/Taykhoom/DNABERT-5mer) | BERT + k-mer | k=5 |
+| [DNABERT-6mer](https://huggingface.co/Taykhoom/DNABERT-6mer) | BERT + k-mer | k=6 |
+| [DNABERT-2](https://huggingface.co/Taykhoom/DNABERT2) | MosaicBERT + BPE + ALiBi | Multi-species pre-trained |
+| [DNABERT-S](https://huggingface.co/Taykhoom/DNABERT-S) | MosaicBERT + BPE + ALiBi | Species-aware |
+
+
+## Usage
+
+### Embedding generation
+
+```python
+import torch
+from transformers import AutoTokenizer, AutoModel
+
+def seq_to_kmers(seq, k=4):
+    return " ".join(seq[i:i+k] for i in range(len(seq) - k + 1))
+
+tokenizer = AutoTokenizer.from_pretrained("Taykhoom/DNABERT-4mer", trust_remote_code=True)
+model = AutoModel.from_pretrained("Taykhoom/DNABERT-4mer", trust_remote_code=True)
+model.eval()
+
+sequences = ["ATCGATCGATCG", "GCTAGCTAGCTA"]
+kmer_seqs = [seq_to_kmers(s) for s in sequences]
+enc = tokenizer(kmer_seqs, return_tensors="pt", padding=True)
+
+with torch.no_grad():
+    out = model(**enc)
+
+cls_emb   = out.last_hidden_state[:, 0, :]   # (batch, 768)
+token_emb = out.last_hidden_state             # (batch, seq_len, 768)
+
+# Mean-pool DNA k-mers only (exclude CLS, SEP, and padding)
+content_mask = enc["attention_mask"].bool()
+content_mask[:, 0] = False
+sep_positions = enc["attention_mask"].sum(dim=1) - 1
+batch_indices = torch.arange(len(sequences), device=content_mask.device)
+content_mask[batch_indices, sep_positions] = False
+mean_emb = (
+    token_emb * content_mask.unsqueeze(-1)
+).sum(dim=1) / content_mask.sum(dim=1, keepdim=True)
+
+# Intermediate layers
+out_all = model(**enc, output_hidden_states=True)
+layer6_emb = out_all.hidden_states[6]
+```
+
+Sequences shorter than 4 nucleotides contain no k-mer tokens and therefore
+cannot be mean-pooled or assigned a k-mer pseudo-likelihood.
+
+### MLM logits
+
+```python
+from transformers import AutoModelForMaskedLM
+
+model = AutoModelForMaskedLM.from_pretrained(
+    "Taykhoom/DNABERT-4mer", trust_remote_code=True
+)
+tokens = seq_to_kmers("ATCGATCG", k=4).split()
+tokens[2] = tokenizer.mask_token
+enc = tokenizer(" ".join(tokens), return_tensors="pt")
+
+with torch.no_grad():
+    logits = model(**enc).logits   # (1, seq_len, 261)
+```
+
+### Faster attention backends
+
+```python
+# SDPA (PyTorch 2.0+)
+model = AutoModel.from_pretrained("Taykhoom/DNABERT-4mer", trust_remote_code=True,
+                                   attn_implementation="sdpa")
+
+# Flash Attention 2 (requires flash-attn)
+model = AutoModel.from_pretrained("Taykhoom/DNABERT-4mer", trust_remote_code=True,
+                                   attn_implementation="flash_attention_2",
+                                   dtype=torch.float16)
+```
+
+### Fine-tuning
+
+For sequence-level tasks, mean-pool only k-mer positions as above or use the
+CLS token embedding as input to a prediction head.
+
+## Implementation Notes
+
+The original DNABERT codebase has `BertModel` as a thin subclass of
+`transformers.BertModel` with no modifications. This HF port uses
+[Taykhoom/BERT-updated](https://huggingface.co/Taykhoom/BERT-updated) which adds
+`attn_implementation="sdpa"` and `attn_implementation="flash_attention_2"`
+support — these were not part of the original codebase.
+
+## Citation
+
+```bibtex
+@article{ji2021_dnabert,
+  title   = {{DNABERT}: pre-trained Bidirectional Encoder Representations from Transformers model for {DNA}-language in genome},
+  author  = {Ji, Yanrong and Zhou, Zhihan and Liu, Han and Davuluri, Ramana V},
+  journal = {Bioinformatics},
+  volume  = {37},
+  number  = {15},
+  pages   = {2112--2120},
+  year    = {2021},
+  doi     = {10.1093/bioinformatics/btab083}
+}
+```
+
+## Credits
+
+Original DNABERT model and code by Ji et al. Source: [GitHub](https://github.com/jerryji1993/DNABERT).
+The HF conversion code was authored primarily by [Claude Code](https://claude.ai/code)
+and reviewed manually by Taykhoom Dalal.
+
+## License
+
+Apache License 2.0, following the original repository.

config.json

@@ -0,0 +1,26 @@
+{
+  "architectures": [
+    "BertForMaskedLM"
+  ],
+  "model_type": "bert_updated",
+  "auto_map": {
+    "AutoConfig": "Taykhoom/BERT-updated--configuration_bert_updated.BertUpdatedConfig",
+    "AutoModel": "Taykhoom/BERT-updated--modeling_bert.BertModel",
+    "AutoModelForMaskedLM": "Taykhoom/BERT-updated--modeling_bert.BertForMaskedLM"
+  },
+  "vocab_size": 261,
+  "hidden_size": 768,
+  "num_hidden_layers": 12,
+  "num_attention_heads": 12,
+  "intermediate_size": 3072,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "attention_probs_dropout_prob": 0.1,
+  "max_position_embeddings": 512,
+  "type_vocab_size": 2,
+  "initializer_range": 0.02,
+  "layer_norm_eps": 1e-12,
+  "pad_token_id": 0,
+  "kmer": 4,
+  "model_max_length": 512
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f148b8f5ee0b4e92a3c3a17d27a0f0be78ef9d0f3231fb43fb211f475bbbf768
+size 348162252

special_tokens_map.json

@@ -0,0 +1 @@
+{"unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]"}

tokenizer_config.json

@@ -0,0 +1,10 @@
+{
+  "do_lower_case": false,
+  "model_max_length": 512,
+  "tokenizer_class": "BertTokenizer",
+  "unk_token": "[UNK]",
+  "sep_token": "[SEP]",
+  "pad_token": "[PAD]",
+  "cls_token": "[CLS]",
+  "mask_token": "[MASK]"
+}

vocab.txt

@@ -0,0 +1,261 @@
+[PAD]
+[UNK]
+[CLS]
+[SEP]
+[MASK]
+AAAA
+AAAT
+AAAC
+AAAG
+AATA
+AATT
+AATC
+AATG
+AACA
+AACT
+AACC
+AACG
+AAGA
+AAGT
+AAGC
+AAGG
+ATAA
+ATAT
+ATAC
+ATAG
+ATTA
+ATTT
+ATTC
+ATTG
+ATCA
+ATCT
+ATCC
+ATCG
+ATGA
+ATGT
+ATGC
+ATGG
+ACAA
+ACAT
+ACAC
+ACAG
+ACTA
+ACTT
+ACTC
+ACTG
+ACCA
+ACCT
+ACCC
+ACCG
+ACGA
+ACGT
+ACGC
+ACGG
+AGAA
+AGAT
+AGAC
+AGAG
+AGTA
+AGTT
+AGTC
+AGTG
+AGCA
+AGCT
+AGCC
+AGCG
+AGGA
+AGGT
+AGGC
+AGGG
+TAAA
+TAAT
+TAAC
+TAAG
+TATA
+TATT
+TATC
+TATG
+TACA
+TACT
+TACC
+TACG
+TAGA
+TAGT
+TAGC
+TAGG
+TTAA
+TTAT
+TTAC
+TTAG
+TTTA
+TTTT
+TTTC
+TTTG
+TTCA
+TTCT
+TTCC
+TTCG
+TTGA
+TTGT
+TTGC
+TTGG
+TCAA
+TCAT
+TCAC
+TCAG
+TCTA
+TCTT
+TCTC
+TCTG
+TCCA
+TCCT
+TCCC
+TCCG
+TCGA
+TCGT
+TCGC
+TCGG
+TGAA
+TGAT
+TGAC
+TGAG
+TGTA
+TGTT
+TGTC
+TGTG
+TGCA
+TGCT
+TGCC
+TGCG
+TGGA
+TGGT
+TGGC
+TGGG
+CAAA
+CAAT
+CAAC
+CAAG
+CATA
+CATT
+CATC
+CATG
+CACA
+CACT
+CACC
+CACG
+CAGA
+CAGT
+CAGC
+CAGG
+CTAA
+CTAT
+CTAC
+CTAG
+CTTA
+CTTT
+CTTC
+CTTG
+CTCA
+CTCT
+CTCC
+CTCG
+CTGA
+CTGT
+CTGC
+CTGG
+CCAA
+CCAT
+CCAC
+CCAG
+CCTA
+CCTT
+CCTC
+CCTG
+CCCA
+CCCT
+CCCC
+CCCG
+CCGA
+CCGT
+CCGC
+CCGG
+CGAA
+CGAT
+CGAC
+CGAG
+CGTA
+CGTT
+CGTC
+CGTG
+CGCA
+CGCT
+CGCC
+CGCG
+CGGA
+CGGT
+CGGC
+CGGG
+GAAA
+GAAT
+GAAC
+GAAG
+GATA
+GATT
+GATC
+GATG
+GACA
+GACT
+GACC
+GACG
+GAGA
+GAGT
+GAGC
+GAGG
+GTAA
+GTAT
+GTAC
+GTAG
+GTTA
+GTTT
+GTTC
+GTTG
+GTCA
+GTCT
+GTCC
+GTCG
+GTGA
+GTGT
+GTGC
+GTGG
+GCAA
+GCAT
+GCAC
+GCAG
+GCTA
+GCTT
+GCTC
+GCTG
+GCCA
+GCCT
+GCCC
+GCCG
+GCGA
+GCGT
+GCGC
+GCGG
+GGAA
+GGAT
+GGAC
+GGAG
+GGTA
+GGTT
+GGTC
+GGTG
+GGCA
+GGCT
+GGCC
+GGCG
+GGGA
+GGGT
+GGGC
+GGGG