daeren007/tuxiangmoxing / tokenizer_2 /special_tokens_map.json
daeren007's picture
download
raw
462 Bytes
{
"bos_token": {
"content": "<|startoftext|>",
"lstrip": false,
"normalized": true,
"rstrip": false,
"single_word": false
},
"eos_token": {
"content": "<|endoftext|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"pad_token": "!",
"unk_token": {
"content": "<|endoftext|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
}
}

Xet Storage Details

Size:
462 Bytes
·
Xet hash:
8da9c42b12581d66f33d1d66c094b54af17505d727f8781603820311baa2973a

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.