Download decode_examples.py from Xerv-AI/CRAYON-tokenizer: direct link, hf CLI and curl.
- Browser
- Download file 203 Bytes
-
https://huggingface.co/Xerv-AI/CRAYON-tokenizer/resolve/main/decode_examples.py
- Command line
-
hf download hf://Xerv-AI/CRAYON-tokenizer/decode_examples.py
-
curl -L -o decode_examples.py https://huggingface.co/Xerv-AI/CRAYON-tokenizer/resolve/main/decode_examples.py
203 Bytes
| from crayon import CrayonVocab | |
| vocab = CrayonVocab(device="auto") | |
| vocab.load_profile("lite") | |
| text = "Hello, world!" | |
| tokens = vocab.tokenize(text) | |
| print(tokens) | |
| decode=vocab.decode(tokens) | |
| print(decode) |