In discussion 44, 46 and 23, people say to use either a different repo like AtomicChat's, a custom llama.cpp fork, or a custom Unsloth model modification to split the engram into its own file in order to use engram SSD offload. But in this video, he offloads engram to SSD using the original UD-Q3_K_XL. Which is correct? Has it been implemented in mainline llama.cpp by now?
I had search answer for this question too. Google AI tell me that I need two flags: --load-mode mmap --lazy-mode on. Now I reask Google, and It say, that now in llama.cpp was added more efficiently flag: --lazy-mode on-direct. When you start model, check how mach ram it use, and it did not write to ssd. I use ud-q4-k-xl( 111gb) on 128gb ram and 32gb vram. When model start answer, it start to fill ram, and fill it up to 80gb approximately. I did not see any writes to ssd where model is. So, ngram stay on ssd, I`m guess.
Yes, mainline llama.cpp supports it now, no fork or split file needed. The engram table is the per_layer_token_embd tensor (about 26.8 GiB, same size in every UD quant). llama.cpp#27794 (merged Aug 27) added on-demand row reads for it, and #27969 renamed the flag to --lazy-mode / -lzm.
On current master the only values are on, auto and off (there is no on-direct). The default auto already lazy-reads marked tensors larger than 4 GiB, so the original UD-Q3_K_XL works as-is. Since #28326, auto falls back to off if a device has no mmap support (some iGPUs), so on those pass --lazy-mode on explicitly. You don't need --load-mode mmap for it either.
To confirm, look for tensor per_layer_token_embd (size = ... MiB) lazy read enabled in the load log. The RAM growth @drondt sees while generating should be the OS page cache for touched rows, which can be evicted, rather than a resident copy.
