Download decode_examples.py from Xerv-AI/CRAYON-tokenizer: direct link, hf CLI and curl.
- Browser
- Download file 203 Bytes
-
https://huggingface.co/Xerv-AI/CRAYON-tokenizer/resolve/708f4a32d1dad5ca970aa34e3bd3e309eacaf4cd/decode_examples.py
- Command line
-
hf download hf://Xerv-AI/CRAYON-tokenizer@708f4a32d1dad5ca970aa34e3bd3e309eacaf4cd/decode_examples.py
-
curl -L -o decode_examples.py https://huggingface.co/Xerv-AI/CRAYON-tokenizer/resolve/708f4a32d1dad5ca970aa34e3bd3e309eacaf4cd/decode_examples.py
203 Bytes
| from crayon import CrayonVocab | |
| vocab = CrayonVocab(device="auto") | |
| vocab.load_profile("lite") | |
| text = "Hello, world!" | |
| tokens = vocab.tokenize(text) | |
| print(tokens) | |
| decode=vocab.decode(tokens) | |
| print(decode) |