pipenetwork commited on
Commit
d957535
·
verified ·
1 Parent(s): ed0954b

Refresh model card

Browse files
Files changed (1) hide show
  1. README.md +5 -5
README.md CHANGED
@@ -100,15 +100,15 @@ from deepseek_v4_mlx.load import load
100
  from deepseek_v4_mlx.generate import greedy_generate, load_tokenizer
101
 
102
  model, args = load("/path/to/this/repo")
103
- tok = load_tokenizer() # DeepSeek-V3's tokenizer see below
104
  ids = tok("The capital of France is")["input_ids"]
105
  print(tok.decode(greedy_generate(model, args, ids, max_new_tokens=32)))
106
  ```
107
 
108
- **The base repo ships no tokenizer** only `encoding/encoding_dsv4.py`, which renders
109
- prompt *strings*. DeepSeek-V3's tokenizer is the correct one: same padded `vocab_size`
110
- 129280 over 128815 real tokens, `bos=0`/`eos=1` matching V4's `generation_config.json`,
111
- and every V4 special token present.
112
 
113
  ## What this port implements
114
 
 
100
  from deepseek_v4_mlx.generate import greedy_generate, load_tokenizer
101
 
102
  model, args = load("/path/to/this/repo")
103
+ tok = load_tokenizer("/path/to/this/repo") # bundledno download
104
  ids = tok("The capital of France is")["input_ids"]
105
  print(tok.decode(greedy_generate(model, args, ids, max_new_tokens=32)))
106
  ```
107
 
108
+ **The tokenizer is bundled in this repo**, so nothing is fetched at runtime. (The base repo
109
+ shipped no tokenizer at first only `encoding/encoding_dsv4.py`, a prompt-*string* renderer;
110
+ DeepSeek has since published one. DeepSeek-V3's tokenizer, used here in the interim, produces
111
+ **identical ids** for text and identical special-token ids, so builds made either way agree.)
112
 
113
  ## What this port implements
114