subword-gpt-7m / tokenizer_config.json

Add config, tokenizer config, and README

093384d verified 9 days ago

192 Bytes

Invalid JSON:Expected ',' or '}' after property value in JSONat line 5, column 18

	{
	"tokenizer_class": "GPT2TokenizerFast",
	"model_max_length": 512,
	"pad_token": null,
	"eos_token": "":[[ctrl:endoftext]]",
	"bos_token": "":[[ctrl:endoftext]]",
	"unk_token": null
	}