Update README.md

README.md

@@ -6,4 +6,89 @@ pipeline_tag: text-classification
 tags:
 - transformer
 - tokenizer
----
+---
+---
+
+language:
+- he
+- en
+pipeline_tag: text-classification
+tags:
+- transformer
+- tokenizer
+
+---
+
+# Model Overview
+
+**Model Name:** T5 Hebrew-to-English Translation Tokenizer  
+**Model Type:** Tokenizer for Transformer-based models  
+**Base Model:** T5 (Text-to-Text Transfer Transformer)  
+**Preprocessing:** Custom Tokenizer using SentencePieceBPETokenizer  
+**Training Data:** Custom Hebrew-English dataset curated for translation tasks  
+**Intended Use:** This tokenizer is intended for machine translation tasks, specifically Hebrew-to-English translations.  
+
+## Model Description
+
+This tokenizer was trained on a Hebrew-to-English dataset using `SentencePieceBPETokenizer`. It is optimized for handling Hebrew text tokenization and can be paired with a Transformer model, such as T5, for sequence-to-sequence translation tasks. It handles preprocessing tasks like tokenization, padding, and truncation effectively.
+
+## Performance
+
+- **Task:** Hebrew-to-English Translation (Tokenizer only)
+- **Dataset:** A custom dataset containing parallel Hebrew-English sentences
+- **Metrics:**
+    - Vocabulary size: 30,000 tokens
+    - Tokenization accuracy: Not applicable (Tokenizer-specific metric)
+
+## Usage
+
+### How to Use the Tokenizer
+
+To use this tokenizer, you can load it using the Hugging Face Transformers library:
+
+```python
+from transformers import AutoTokenizer
+
+# Load the tokenizer
+tokenizer = AutoTokenizer.from_pretrained("tejagowda/t5-hebrew-translation", use_fast=False)
+
+# Example: Tokenizing a Hebrew sentence
+hebrew_text = "\u05D0\u05EA\u05D4\u05D3 \u05E2\u05DC \u05D4\u05D7\u05D5\u05DE\u05E8\u05D4."
+inputs = tokenizer(hebrew_text, return_tensors="pt")
+
+print("Tokens:", inputs["input_ids"])
+```
+
+### Example Usage with a Pretrained Model
+
+To perform translation, you can pair this tokenizer with a pretrained T5 model:
+
+```python
+from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
+
+# Load the tokenizer and model
+tokenizer = AutoTokenizer.from_pretrained("tejagowda/t5-hebrew-translation", use_fast=False)
+model = AutoModelForSeq2SeqLM.from_pretrained("t5-small")  # Replace with fine-tuned model if available
+
+# Hebrew text to translate
+hebrew_text = "\u05EA\u05D0\u05E8 \u05D0\u05EA \u05DE\u05D1\u05E0\u05D4 \u05E9\u05DC \u05D0\u05D8\u05D5\u05DD."
+
+# Tokenize and translate
+inputs = tokenizer(hebrew_text, return_tensors="pt")
+outputs = model.generate(inputs["input_ids"], max_length=100)
+
+# Decode the output
+english_translation = tokenizer.decode(outputs[0], skip_special_tokens=True)
+
+print("Translation:", english_translation)
+```
+
+## Limitations
+
+- The tokenizer itself does not perform translation; it must be paired with a translation model.
+- Performance depends on the quality of the paired model and training data.
+
+## License
+
+This tokenizer is licensed under the Apache 2.0 License. See the LICENSE file for more details.
+