Download src/crayon/unicode/__init__.py from Xerv-AI/CRAYON-tokenizer: direct link, hf CLI and curl.
- Browser
- Download file 368 Bytes
-
https://huggingface.co/Xerv-AI/CRAYON-tokenizer/resolve/708f4a32d1dad5ca970aa34e3bd3e309eacaf4cd/src/crayon/unicode/__init__.py
- Command line
-
hf download hf://Xerv-AI/CRAYON-tokenizer@708f4a32d1dad5ca970aa34e3bd3e309eacaf4cd/src/crayon/unicode/__init__.py
-
curl -L -o __init__.py https://huggingface.co/Xerv-AI/CRAYON-tokenizer/resolve/708f4a32d1dad5ca970aa34e3bd3e309eacaf4cd/src/crayon/unicode/__init__.py
368 Bytes
| """ | |
| Crayon Unicode Processing Module. | |
| Implements the high-performance text normalization and multilingual support | |
| strategies defined in Section 5 of the XERV Crayon Engineering Treatise. | |
| """ | |
| from .normalizer import unicode_normalize_nfc_optimized | |
| from .multilingual import MultilingualProcessor | |
| __all__ = ["unicode_normalize_nfc_optimized", "MultilingualProcessor"] |