---
license: apache-2.0
datasets:
- wikimedia/wikipedia
- HPLT/HPLT2.0_cleaned
language:
- hu
---

A wikipédia kenlm kvantálás nélküli, a HPLT -a 22 -q 8 -b 8 kapcsolókkal lett kvantálva.

A HPLT adatbázis 50%-a lett felhasználva:
```
=== 1/5 Counting and sorting n-grams ===
Reading /home/sarpba/2TB/hplt_magyar_korpusz_felezett.txt
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
****************************************************************************************************
Unigram tokens 13847057702 types 96225600
=== 2/5 Calculating and sorting adjusted counts ===
Chain sizes: 1:1154707200 2:29655070720 3:55603257344 4:88965210112 5:129740939264 6:177930420224
Statistics:
1 96225600 D1=0.764073 D2=1.03791 D3+=1.27263
2 1265844926 D1=0.806391 D2=1.0989 D3+=1.30792
3 1575497442/3590763606 D1=0.874748 D2=1.16937 D3+=1.29281
4 877443852/5312014485 D1=0.925102 D2=1.26541 D3+=1.29521
5 838676466/5932092458 D1=0.953571 D2=1.35662 D3+=1.27285
6 439341856/6030728712 D1=0.585162 D2=1.55164 D3+=1.71841
Memory estimate for binary LM:
type     GB
probing 111 assuming -p 1.5
probing 137 assuming -r models -p 1.5
trie     68 without quantization
trie     42 assuming -q 8 -b 8 quantization 
trie     57 assuming -a 22 array pointer compression
trie     31 assuming -a 22 -q 8 -b 8 array pointer compression and quantization
=== 3/5 Calculating and sorting initial probabilities ===
Chain sizes: 1:1154707200 2:20253518816 3:31509948840 4:21058652448 5:23482941048 6:14058939392
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
*******#############################################################################################
=== 4/5 Calculating and writing order-interpolated probabilities ===
Chain sizes: 1:1154707200 2:20253518816 3:31509948840 4:21058652448 5:23482941048 6:14058939392
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
####################################################################################################
=== 5/5 Writing ARPA model ===
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
****************************************************************************************************
Name:lmplz	VmPeak:473506760 kB	VmRSS:0 kB	RSSMax:462636584 kB	user:18683.3	sys:7247.18	CPU:25930.5	real:21271.2
```