README.md

metadata

language:
  - ko
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
tags:
  - korean
  - causal-lm
  - decoder-only
  - from-scratch
  - instruction-tuned
  - 1.2b
model-index:
  - name: Jumini-Ko-1.2B
    results:
      - task:
          type: text-generation
          name: Korean Knowledge (HAE-RAE Bench)
        dataset:
          type: HAERAE-HUB/HAE_RAE_BENCH_1.0
          name: HAE-RAE Bench
        metrics:
          - type: acc
            name: accuracy (5-shot)
            value: 21.9
      - task:
          type: text-generation
          name: Korean Reading (Belebele-Ko)
        dataset:
          type: facebook/belebele
          name: Belebele (kor_Hang)
        metrics:
          - type: acc
            name: accuracy (5-shot)
            value: 27.9
      - task:
          type: text-generation
          name: KMMLU
        dataset:
          type: HAERAE-HUB/KMMLU
          name: KMMLU
        metrics:
          - type: acc
            name: accuracy (5-shot)
            value: 24.3
      - task:
          type: text-generation
          name: KoBEST
        dataset:
          type: skt/kobest_v1
          name: KoBEST
        metrics:
          - type: acc
            name: accuracy (5-shot)
            value: 49.5

Jumini-Ko-1.2B

Jumini-Ko-1.2B is a 1.26B-parameter Korean decoder-only language model trained from scratch — its architecture, tokenizer, data pipeline, and training loop were all built in-house, and it is not a fine-tune of any existing model. It is a compact, Korean-specialized model designed to run on commodity hardware.

Among the evaluated open non-flagship Korean baselines (polyglot-ko-1.3b, Tri-1.9B), Jumini-Ko-1.2B is the strongest on Korean knowledge (HAE-RAE) and reading comprehension (Belebele-Ko) — despite being the smallest model compared. The flagship EXAONE-4.0-1.2B, trained on far more data/compute, is stronger on all four benchmarks.

Highlights

• 🇰🇷 Korean-specialized, from scratch — Llama-3-style architecture (RoPE, GQA, SwiGLU, RMSNorm), 128K byte-level BPE tokenizer, trained from random initialization.
• 🥇 Beats the size-matched polyglot-ko-1.3b and the larger Tri-1.9B on HAE-RAE and Belebele-Ko (5-shot), the two Korean-language benchmarks emphasized here. (It trails polyglot-ko-1.3b on KoBEST commonsense and KMMLU, and the flagship EXAONE-4.0-1.2B overall.)
• 🔬 A data-centric recipe — we show that which corpus you continue-pretrain on decides which capability improves (web → commonsense, Wikipedia → knowledge).
• 📦 Edge-friendly — 1.26B parameters; runs comfortably on a single consumer GPU.

Benchmark Results

Korean benchmarks via the EleutherAI lm-evaluation-harness, 5-shot, accuracy (%). All models evaluated under identical settings. Bold = best, underline = second best.

Benchmark	Jumini-Ko-1.2B (1.26B)	polyglot-ko-1.3b (1.43B)	Tri-1.9B (1.9B)	EXAONE-4.0-1.2B† (1.28B)
HAE-RAE (Korean knowledge)	21.9	18.7	18.9	30.0
Belebele-Ko (reading)	27.9	22.4	22.9	44.7
KMMLU (knowledge)	24.3	27.8	16.6	32.6
KoBEST (commonsense)	49.5	55.9	50.1	50.6

_{† EXAONE-4.0-1.2B is a strong flagship model trained on vastly more data/compute, shown as an aspirational reference. Against the open same-tier baselines (polyglot-ko-1.3b, Tri-1.9B), Jumini leads on the Korean-specific HAE-RAE and Belebele-Ko while being the smallest model.}

Jumini also beats polyglot-ko-1.3b on 4 of 5 HAE-RAE subtasks (history, loan-word, rare-word, standard-nomenclature). It trails polyglot-ko-1.3b on commonsense (KoBEST) and broad knowledge (KMMLU). Full per-subtask numbers are in the technical report.

Quickstart

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "properly59/Jumini-Ko-1.2B"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype=torch.float16, device_map="auto")

prompt = "### 질문:\n대한민국의 수도는 어디인가요?\n\n### 답변:\n"
ids = tok(tok.bos_token + prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
out = model.generate(**ids, max_new_tokens=128, do_sample=True, temperature=0.8,
                     min_p=0.05, repetition_penalty=1.2, no_repeat_ngram_size=3,
                     pad_token_id=tok.pad_token_id)
print(tok.decode(out[0][ids.input_ids.shape[1]:], skip_special_tokens=True))

Model Details

Architecture	Decoder-only Transformer (Llama-3 family)
Parameters	1.26B (hidden 2048, 28 layers, 32 Q / 8 KV heads, SwiGLU 4096)
Position encoding	RoPE (θ = 500,000)
Tokenizer	Byte-level BPE, 128,000 vocab
Context length	4,096
Precision	bf16 / fp16
License	Apache-2.0

Training

A three-stage, fully-documented pipeline on top of the from-scratch base:

1. Continued pre-training on a high-quality Korean mixture (FineWeb-2 kor_Hang, KOREAN-WEBTEXT, Korean Wikipedia), document-boundary packed.
2. Encyclopedic annealing on Korean Wikipedia (LR → 0) — the most token-efficient route to Korean knowledge.
3. Supervised fine-tuning on a 132K permissively-licensed Korean instruction mixture (KoAlpaca, OpenOrca-KO, KOpen-Platypus, KULLM-v2), with completion-only loss and explicit EOS supervision.

All continued-pretraining and instruction data are public corpora used only for post-training; no external pretrained weights are used. A benchmark decontamination check found 0.00% of benchmark items substantially covered (≥50% of 25-character shingles) by the instruction data.

Intended Use & Limitations

Intended for Korean text generation, QA, summarization, and research on small-model training. As a compact model trained from scratch under a constrained budget, its factual accuracy is limited and it can produce incorrect content; greedy decoding is best paired with a repetition penalty. It trails much larger / higher-budget Korean models (e.g., EXAONE) on knowledge tasks and has not undergone safety alignment. Use for research and non-critical applications only.

Citation

@techreport{jumini2026,
  title  = {Jumini-Ko-1.2B Technical Report},
  author = {Cho, Ju-min},
  year   = {2026},
  note   = {https://huggingface.co/properly59/Jumini-Ko-1.2B}
}