jd-rodriguezp1234 commited on
Commit
8dc7cc9
·
verified ·
1 Parent(s): b26f396

Load tokenizer from the repo now that it ships with one

Browse files
Files changed (1) hide show
  1. README.md +4 -3
README.md CHANGED
@@ -91,7 +91,7 @@ from transformers import AutoTokenizer, AutoModelForSequenceClassification
91
  import torch
92
 
93
  name = "Flaglab/ESNLIR-XLM-RoBERTa-premise-only"
94
- tok = AutoTokenizer.from_pretrained("FacebookAI/xlm-roberta-base")
95
  model = AutoModelForSequenceClassification.from_pretrained(name)
96
 
97
  # premise only, by design
@@ -104,8 +104,9 @@ with torch.no_grad():
104
  print(model.config.id2label[int(logits.argmax(-1))])
105
  ```
106
 
107
- > The tokenizer is not stored in this repository load it from the base model
108
- > `FacebookAI/xlm-roberta-base`, as shown above.
 
109
 
110
  ## Citation
111
 
 
91
  import torch
92
 
93
  name = "Flaglab/ESNLIR-XLM-RoBERTa-premise-only"
94
+ tok = AutoTokenizer.from_pretrained(name)
95
  model = AutoModelForSequenceClassification.from_pretrained(name)
96
 
97
  # premise only, by design
 
104
  print(model.config.id2label[int(logits.argmax(-1))])
105
  ```
106
 
107
+ > The tokenizer bundled here is an unmodified copy of the one from
108
+ > [`FacebookAI/xlm-roberta-base`](https://huggingface.co/FacebookAI/xlm-roberta-base); vocabulary size matches this model's
109
+ > embedding table exactly.
110
 
111
  ## Citation
112