Commit ·
d983c80
1
Parent(s): 193d99a
Update README.md
Browse files
README.md
CHANGED
|
@@ -30,7 +30,7 @@ tags:
|
|
| 30 |
|
| 31 |
> **What is Genesis?** Genesis is post training data regeneration and calibrarion algorythm for neural networks (LLM) in GGUF format that I made with AI help during almost half a year of development. It's optimized, architecture independent, works with any model in GGUF format and based on mathematical statistics. I don't train or finetune models, I repair **purity of signal** in them instead on Google Collab Free on Tesla T4 GPU via Python based on how models learns information. On first stage I scan ssm_conv1d tensors in model, they handle long context memory. I repair balance between heads in them. On second stage, I scan blocks in model via chunks via 3 parameters and pick best one that fits to weight distribution in tensor. Best picked chunk replaces zero chunks in broken tensor without touching learned structure in model. On third stage I scan model and detect noise in tensors via custom SVD. During scanning I exclude token_embd.weight, output.weight, 1D tensors, bias and norms. Then I reduce training noise in tensors via custom SVD based on [Marchenko–Pastur law](https://en.wikipedia.org/wiki/Marchenko%E2%80%93Pastur_distribution) with preserved training data, 99% of siginal and learned gradient.
|
| 32 |
|
| 33 |
-
`Many tensors in this model were singular with huge condition number for matrices. They distorted the signal instead of transmitting it correctly. I fixed it as much as I can and reduced condition number for matrices for stable inference during image generation.`
|
| 34 |
|
| 35 |
## Any questions?
|
| 36 |
|
|
|
|
| 30 |
|
| 31 |
> **What is Genesis?** Genesis is post training data regeneration and calibrarion algorythm for neural networks (LLM) in GGUF format that I made with AI help during almost half a year of development. It's optimized, architecture independent, works with any model in GGUF format and based on mathematical statistics. I don't train or finetune models, I repair **purity of signal** in them instead on Google Collab Free on Tesla T4 GPU via Python based on how models learns information. On first stage I scan ssm_conv1d tensors in model, they handle long context memory. I repair balance between heads in them. On second stage, I scan blocks in model via chunks via 3 parameters and pick best one that fits to weight distribution in tensor. Best picked chunk replaces zero chunks in broken tensor without touching learned structure in model. On third stage I scan model and detect noise in tensors via custom SVD. During scanning I exclude token_embd.weight, output.weight, 1D tensors, bias and norms. Then I reduce training noise in tensors via custom SVD based on [Marchenko–Pastur law](https://en.wikipedia.org/wiki/Marchenko%E2%80%93Pastur_distribution) with preserved training data, 99% of siginal and learned gradient.
|
| 32 |
|
| 33 |
+
`Many tensors in this model were singular with huge condition number for matrices. They distorted the signal distribution between tensors instead of transmitting it correctly. I fixed it as much as I can for base model and text encoder and reduced condition number for matrices for stable inference during image generation.`
|
| 34 |
|
| 35 |
## Any questions?
|
| 36 |
|