Import Error appears with `transformers>=5.0.0`
#8
by 6gsn - opened
Traceback (most recent call last):
...
File "/<workspace>/.cache/huggingface/modules/transformers_modules/line_hyphen_corporation/line_hyphen_distilbert_hyphen_base_hyphen_japanese/73d6f79438b9bfc325b27ddc6cfc637395e1408b/distilbert_japanese_tokenizer.py", line 29, in <module>
from transformers.tokenization_utils import PreTrainedTokenizer, _is_control, _is_punctuation, _is_whitespace
ImportError: cannot import name '_is_control' from 'transformers.tokenization_utils' (unknown location)
The refactoring of Tokenizer on transformers v5.0.0 might affect this error 🤔