LuffyTheFox commited on
Commit
d983c80
·
1 Parent(s): 193d99a

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +1 -1
README.md CHANGED
@@ -30,7 +30,7 @@ tags:
30
 
31
  > **What is Genesis?** Genesis is post training data regeneration and calibrarion algorythm for neural networks (LLM) in GGUF format that I made with AI help during almost half a year of development. It's optimized, architecture independent, works with any model in GGUF format and based on mathematical statistics. I don't train or finetune models, I repair **purity of signal** in them instead on Google Collab Free on Tesla T4 GPU via Python based on how models learns information. On first stage I scan ssm_conv1d tensors in model, they handle long context memory. I repair balance between heads in them. On second stage, I scan blocks in model via chunks via 3 parameters and pick best one that fits to weight distribution in tensor. Best picked chunk replaces zero chunks in broken tensor without touching learned structure in model. On third stage I scan model and detect noise in tensors via custom SVD. During scanning I exclude token_embd.weight, output.weight, 1D tensors, bias and norms. Then I reduce training noise in tensors via custom SVD based on [Marchenko–Pastur law](https://en.wikipedia.org/wiki/Marchenko%E2%80%93Pastur_distribution) with preserved training data, 99% of siginal and learned gradient.
32
 
33
- `Many tensors in this model were singular with huge condition number for matrices. They distorted the signal instead of transmitting it correctly. I fixed it as much as I can and reduced condition number for matrices for stable inference during image generation.`
34
 
35
  ## Any questions?
36
 
 
30
 
31
  > **What is Genesis?** Genesis is post training data regeneration and calibrarion algorythm for neural networks (LLM) in GGUF format that I made with AI help during almost half a year of development. It's optimized, architecture independent, works with any model in GGUF format and based on mathematical statistics. I don't train or finetune models, I repair **purity of signal** in them instead on Google Collab Free on Tesla T4 GPU via Python based on how models learns information. On first stage I scan ssm_conv1d tensors in model, they handle long context memory. I repair balance between heads in them. On second stage, I scan blocks in model via chunks via 3 parameters and pick best one that fits to weight distribution in tensor. Best picked chunk replaces zero chunks in broken tensor without touching learned structure in model. On third stage I scan model and detect noise in tensors via custom SVD. During scanning I exclude token_embd.weight, output.weight, 1D tensors, bias and norms. Then I reduce training noise in tensors via custom SVD based on [Marchenko–Pastur law](https://en.wikipedia.org/wiki/Marchenko%E2%80%93Pastur_distribution) with preserved training data, 99% of siginal and learned gradient.
32
 
33
+ `Many tensors in this model were singular with huge condition number for matrices. They distorted the signal distribution between tensors instead of transmitting it correctly. I fixed it as much as I can for base model and text encoder and reduced condition number for matrices for stable inference during image generation.`
34
 
35
  ## Any questions?
36