aaa961/modernbert-embed-base-legal-matryoshka-corrected_train_set_anchor_positive_2026_03_09

SentenceTransformer based on aaa961/modernbert-embed-base-legal-matryoshka-2_2026-03-06

This is a sentence-transformers model finetuned from aaa961/modernbert-embed-base-legal-matryoshka-2_2026-03-06 on the json dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: aaa961/modernbert-embed-base-legal-matryoshka-2_2026-03-06
• Maximum Sequence Length: 8192 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • json

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False, 'architecture': 'ModernBertModel'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("aaa961/modernbert-embed-base-legal-matryoshka-corrected_train_set_anchor_positive_2026_03_09")
# Run inference
sentences = [
    "To whom did the CIA's government counsel refer the plaintiff's counsel?",
    'CIA’s in-house journal Studies in Intelligence, see supra Part I.B.4, plaintiff’s counsel contacted \ngovernment counsel for the CIA, who referred plaintiff’s counsel to the FBI.  See Pl.’s First Mot. \nto Compel at 1.  In January 2012, an FBI field agent met with plaintiff’s counsel, at which time \nplaintiff’s counsel signed a non-disclosure agreement as to any classified material contained in',
    'favorably to his profile as a baseball player, and the fourth and fifth entries refer to his \ndefamation suit. While the fourth entry is headlined “Northwestern baseball player sued for \nintentional infliction of,” the text for that entry states: “For over two years, Chad Readey has \nbeen the victim of a vicious, coordinated effort to assassinate his character based on',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.7737, 0.4363],
#         [0.7737, 1.0000, 0.4764],
#         [0.4363, 0.4764, 1.0000]])

Evaluation

Metrics

Information Retrieval

• Dataset: dim_768
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 768
  }
  

Metric	Value
cosine_accuracy@1	0.391
cosine_accuracy@3	0.4297
cosine_accuracy@5	0.5116
cosine_accuracy@10	0.6059
cosine_precision@1	0.391
cosine_precision@3	0.3725
cosine_precision@5	0.2893
cosine_precision@10	0.1859
cosine_recall@1	0.1365
cosine_recall@3	0.3672
cosine_recall@5	0.4619
cosine_recall@10	0.5898
cosine_ndcg@10	0.4931
cosine_mrr@10	0.4355
cosine_map@100	0.4815

Information Retrieval

• Dataset: dim_512
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 512
  }
  

Metric	Value
cosine_accuracy@1	0.3849
cosine_accuracy@3	0.4204
cosine_accuracy@5	0.5147
cosine_accuracy@10	0.5981
cosine_precision@1	0.3849
cosine_precision@3	0.3622
cosine_precision@5	0.2847
cosine_precision@10	0.1815
cosine_recall@1	0.137
cosine_recall@3	0.3624
cosine_recall@5	0.4581
cosine_recall@10	0.5766
cosine_ndcg@10	0.4852
cosine_mrr@10	0.4297
cosine_map@100	0.4769

Information Retrieval

• Dataset: dim_256
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 256
  }
  

Metric	Value
cosine_accuracy@1	0.3787
cosine_accuracy@3	0.4189
cosine_accuracy@5	0.4961
cosine_accuracy@10	0.5904
cosine_precision@1	0.3787
cosine_precision@3	0.3622
cosine_precision@5	0.2816
cosine_precision@10	0.1796
cosine_recall@1	0.1328
cosine_recall@3	0.3592
cosine_recall@5	0.4498
cosine_recall@10	0.5681
cosine_ndcg@10	0.4777
cosine_mrr@10	0.4235
cosine_map@100	0.4673

Information Retrieval

• Dataset: dim_128
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 128
  }
  

Metric	Value
cosine_accuracy@1	0.3385
cosine_accuracy@3	0.3725
cosine_accuracy@5	0.456
cosine_accuracy@10	0.524
cosine_precision@1	0.3385
cosine_precision@3	0.3199
cosine_precision@5	0.2572
cosine_precision@10	0.16
cosine_recall@1	0.1209
cosine_recall@3	0.3162
cosine_recall@5	0.4109
cosine_recall@10	0.5066
cosine_ndcg@10	0.4272
cosine_mrr@10	0.3785
cosine_map@100	0.4224

Information Retrieval

• Dataset: dim_64
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 64
  }
  

Metric	Value
cosine_accuracy@1	0.2457
cosine_accuracy@3	0.2782
cosine_accuracy@5	0.3462
cosine_accuracy@10	0.4142
cosine_precision@1	0.2457
cosine_precision@3	0.238
cosine_precision@5	0.1932
cosine_precision@10	0.1249
cosine_recall@1	0.0873
cosine_recall@3	0.2336
cosine_recall@5	0.3108
cosine_recall@10	0.3981
cosine_ndcg@10	0.3255
cosine_mrr@10	0.2823
cosine_map@100	0.3215

Training Details

Training Dataset

json

• Dataset: json
• Size: 5,822 training samples
• Columns: anchor and positive
• Approximate statistics based on the first 1000 samples:

  	anchor	positive
  type	string	string
  details	min: 7 tokens mean: 16.67 tokens max: 39 tokens	min: 29 tokens mean: 97.89 tokens max: 170 tokens

• Samples:

  anchor	positive
  What is the case citation for Colón v. Romero Barceló?	13 Colón v. Romero Barceló, 112 DPR 573 (1982). KLAN202300916 11 Posteriormente, en López Tristani v. Maldonado Carrero, supra, la Alta Curia puntualizó que la doctrina civilista reconocía el derecho a la propia imagen como un derecho de la personalidad, protegido por principios constitutivos del ordenamiento jurídico.
  Who became a major investor/shareholder?	exchange for 397,219 shares of DR’s Series Seed-1 Preferred Stock, becoming a major investor/shareholder.7 Senetas invested in DR because DR “had a leading medical application for artificial intelligence and machine learning, and had assembled a team of the most expert people around the world that were assisting in
  Where is information about fee category not included, according to the CIA's declarant?	sort its incoming FOIA requests based on fee categories.” First Lutz Decl. ¶ 11. The CIA’s declarant also states that “this information [i.e., fee category] is not included in the electronic system,” though the CIA’s declarant also avers that “[f]ee category is not a mandatory field,” and thus “this information is often not included in a FOIA request record.” Id. The plaintiff focuses

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          768,
          512,
          256,
          128,
          64
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 32
• num_train_epochs: 4
• learning_rate: 2e-05
• lr_scheduler_type: cosine
• warmup_steps: 0.1
• optim: adamw_torch_fused
• gradient_accumulation_steps: 16
• bf16: True
• tf32: True
• eval_strategy: epoch
• per_device_eval_batch_size: 16
• load_best_model_at_end: True
• warmup_ratio: 0.1
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• per_device_train_batch_size: 32
• num_train_epochs: 4
• max_steps: -1
• learning_rate: 2e-05
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: None
• warmup_steps: 0.1
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 16
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: True
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: True
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: trackio
• eval_strategy: epoch
• per_device_eval_batch_size: 16
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: True
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 42
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: []
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• warmup_ratio: 0.1
• local_rank: -1
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	dim_768_cosine_ndcg@10	dim_512_cosine_ndcg@10	dim_256_cosine_ndcg@10	dim_128_cosine_ndcg@10	dim_64_cosine_ndcg@10
0.8791	10	4.4842	-	-	-	-	-
1.0	12	-	0.4931	0.4852	0.4777	0.4272	0.3255
1.7033	20	4.5952	-	-	-	-	-
2.0	24	-	0.4931	0.4852	0.4777	0.4272	0.3255
2.5275	30	4.4715	-	-	-	-	-
3.0	36	-	0.4931	0.4852	0.4777	0.4272	0.3255
3.3516	40	4.6128	-	-	-	-	-
4.0	48	-	0.4931	0.4852	0.4777	0.4272	0.3255

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.12.11
• Sentence Transformers: 5.2.3
• Transformers: 5.3.0
• PyTorch: 2.5.1+cu121
• Accelerate: 1.13.0
• Datasets: 4.6.1
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}