load model failed: Exception: expected value at line 1 column 1
File "/assistant_vePFS/algorithm/model/Search_model/jina-embeddings-v4/modeling_jina_embeddings_v4.py", line 145, in init
self.processor = JinaEmbeddingsV4Processor.from_pretrained(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/Qwen/lib/python3.12/site-packages/transformers/processing_utils.py", line 1185, in from_pretrained
args = cls._get_arguments_from_pretrained(pretrained_model_name_or_path, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/Qwen/lib/python3.12/site-packages/transformers/processing_utils.py", line 1248, in _get_arguments_from_pretrained
args.append(attribute_class.from_pretrained(pretrained_model_name_or_path, **kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/Qwen/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 2025, in from_pretrained
return cls._from_pretrained(
^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/Qwen/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 2278, in _from_pretrained
tokenizer = cls(*init_inputs, **init_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/Qwen/lib/python3.12/site-packages/transformers/models/qwen2/tokenization_qwen2_fast.py", line 120, in init
super().init(
File "/root/miniconda3/envs/Qwen/lib/python3.12/site-packages/transformers/tokenization_utils_fast.py", line 117, in init
fast_tokenizer = TokenizerFast.from_file(fast_tokenizer_file)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Exception: expected value at line 1 column 1
Hi @vivian345 , you should pull the tokenizer file with git lfs pull tokenizer.json. The issue was that tokenizer.json wasn’t automatically pulled when cloning the model. I just fixed this, so next time you clone, you won’t have to pull it yourself.