cadazar/han2han-it

Han2Han IT

Instruction-tuned checkpoint of Han2Han (han2han-ul2-base-1-it, step 43153). Han2Han is a 169M-parameter encoder-decoder model that learns script-invariant representations of Korean text: a document written in Hanja and its Hangul transcription land at the same point in embedding space. The recipe (jamo and character-level embedding fusion, morpheme-aware denoising, bidirectional Hanja-Hangul transcription) is described in the paper, accepted to Findings of EMNLP 2026.

This repo holds the PyTorch weights, the tokenizer with its chat template, and the modeling code needed to load them through the transformers Auto classes with trust_remote_code=True. Training code, the Flax model, the Flax-to-PyTorch converter, and the evaluation pipeline live in the GitHub repo.

Intended use

Han2Han is meant for representations: classification and fine-tuning on downstream tasks, and sentence embeddings that treat Hanja and Hangul spellings of the same text alike. The model generates text, but generation is not what it was built or evaluated for, and improving it is follow-up work:

• Prompt layout. The task prompt goes in the system message and the text alone in the user message, which is how the instruction tuning presented every task. With the instruction written into the user message instead, Hangul to Hanja restoration returns the Hangul input unchanged.
• Hangul to Hanja restoration (한글을 한자로 전사하시오:) works on article-length input. Given the first 478 characters of a 1922 newspaper article in Hangul, greedy decoding restores 195 of its 251 Hanja (78%). The article comes from a corpus used in pre-training, so this is an illustration and not a held-out measurement. On single sentences the model mostly returns Hangul: across 256 newspaper sentences it restores 24% of the Hanja, and 145 of the outputs contain no Hanja at all.
• Hanja to Hangul transcription (한자를 한글로 전사하시오:) works on sentences and articles, in either layout, but can leave some Hanja untranscribed.
• Stopping. Give generate() a token limit: twice the input's tokens plus 8 leaves room for a correct answer. In fp32 the outputs measured here end on their own; in bf16 the article restoration ran on to the limit, repeating itself.

hanja_transcription_demo.ipynb in the GitHub repo was written with the other layout (the instruction in the user message), so the Hangul to Hanja outputs in its sections 4 to 9 understate what this checkpoint does. Its section 10 runs the layout described here.

Usage

Runtime requirements: torch and transformers (tested with torch 2.14.1 and transformers 5.18.0 on CPU). The model class ships in this repo, so loading needs trust_remote_code=True; the tokenizer itself runs no custom code, but without the flag AutoTokenizer stops to ask.

Sentence embeddings

output_sentence_embeddings=True returns the mean-pooled encoder states, one 640-dimensional vector per input.

import torch
import torch.nn.functional as F
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

repo = "cadazar/han2han-it"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForSeq2SeqLM.from_pretrained(repo, trust_remote_code=True).eval()

texts = [
    "會場을 一巡하고 돌아올 때까지도 日本人側 畵家까지도 一人도 發見할 수가 없었다.",
    "회장을 일순하고 돌아올 때까지도 일본인측 화가까지도 일인도 발견할 수가 없었다.",
    "南畵나 四君子에서는 破墨의 妙法으로 白雪을 象徵할 수 있다.",
]
inputs = tokenizer(texts, return_tensors="pt", padding=True)
with torch.no_grad():
    embeddings = model(**inputs, output_sentence_embeddings=True)[0]
embeddings = F.normalize(embeddings, dim=-1)
print(embeddings @ embeddings.T)
# tensor([[1.0000, 0.9444, 0.8789],
#         [0.9444, 1.0000, 0.8459],
#         [0.8789, 0.8459, 1.0000]])

The first two rows are the same 1920s newspaper sentence in mixed script and in Hangul; the third is a different sentence.

Chat

AutoModelForCausalLM loads Han2HanForCausalLM, a wrapper that gives the encoder-decoder the interface chat tooling expects: generate() takes the rendered chat prompt as one sequence and returns the prompt followed by the reply. pipeline("text-generation") works with it, and so does the transformers server:

transformers serve --trust-remote-code
transformers chat cadazar/han2han-it --system-prompt "한자를 한글로 전사하시오:"

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "cadazar/han2han-it"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True).eval()


def run(task_prompt, text):
    messages = [
        {"role": "system", "content": task_prompt},
        {"role": "user", "content": text},
    ]
    inputs = tokenizer.apply_chat_template(
        messages, add_generation_prompt=True, return_tensors="pt", return_dict=True
    )
    budget = 2 * len(tokenizer(text).input_ids) + 8
    with torch.no_grad():
        output = model.generate(**inputs, max_new_tokens=budget, do_sample=False)
    return tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True)


print(run(
    "한자를 한글로 전사하시오:",
    "會場을 一巡하고 돌아올 때까지도 日本人側 畵家까지도 一人도 發見할 수가 없었다.",
))
# 회장을 一巡하고 돌아올 때까지도 일본인측 화가까지도 일인도 발견할 수가 없었다.

hangul = (
    "미술전람 계획 총독부 사안으로 명년 초차 개설총독부에서는 조선에 재 한 미술의 발달을 비보할 "
    "목적으로 동경의 제국미술원전람회를 방하 여 매년 일차 미술전람회를 개할 방침을 내정하고 시 "
    "이십육일 오전 십시 총독부 제이회의실에 박영효 후, 민병석 자, 서화협회의 정대유, 김돈희, 이도영 "
    "외 제씨, 서화연구회의 김규진 씨, 일본인측 서화가로  고목배수 외 수씨 기타 서화가에 관계있는 "
    "인사를 초청하고 수야 정무총감 이하 학무당국자가 회합하여 차에 관한 상의회를 개한바"
)
print(run("한글을 한자로 전사하시오:", hangul))
# 美術覽 計劃 總督府 事項으로 明年 初次 開設總督府에서는 朝鮮에 在 한 美術의 發達을 備報할
# 爲하야 東京의 帝國美術園展覽會를 訪하 여 每年 一次 美術를 開할 方針을 內定하고 時
# 二十六日 午前 十時 總督府 第二回 議室에 朴永孝 後, 閔丙錫 者, 書畵協會의 政大裕, 金敦熙,
# 李道英 外 諸氏, 書畵硏究會의 金奎鎭 氏, 日本人側 書畵家로 高木培 外 수씨 其他 書畵家에
# 關係있는 人事를 招請하고 壽野 政務總監 以下 學務當局者가 會合하여 차에 關한 商會를 開한바

The outputs are shown as generated (each is one line). In the first, 一巡 should have become 일순. The second is the opening of the 1922 article; the original reads 美術展覽 計劃 總督府 事案으로 明年 初次 開設總督府에서는 朝鮮에 在 한 美術의 發達을 裨補할 목적으로 東京의 帝國美術院展覽會를 倣하 여 ..., so the output drops or swaps some characters (美術覽, 事項, 帝國美術園) and writes same-sounding ones for several names.

Prompt format, as used by the SFT collator in the GitHub repo and by the chat template:

• <|system|>{system}<|user|>{user}<|end_of_turn|>, with <|assistant|>{reply}<|end_of_turn|><|user|>{user}<|end_of_turn|> for each further turn. The system part is optional in the format, but it is where this checkpoint expects the task prompt. Each direction was trained with three wordings: 한자를 한글로 전사하시오:, 다음 한문을 한글로 옮기시오:, 한자 표기를 한글 독음으로 변환하시오: for Hanja to Hangul, and 한글을 한자로 전사하시오:, 다음 한글 텍스트에 적절한 한자를 추가하시오:, 한자 표기가 필요한 부분에 한자를 병기하시오: for Hangul to Hanja.
• The generation prompt is <|assistant|>, or <|think|> with enable_thinking=True, in which case the reply is reasoning, then <|assistant|>, then the answer. A turn ends with <|end_of_turn|> (eos_token_id 10).

The wrapper splits the prompt at its last <|end_of_turn|>: everything up to and including it is the encoder input, and the generation prompt after it starts the decoder. A prompt without a generation prompt raises. forward() is not wrapped and stays encoder-decoder.

Encoder-decoder generation

AutoModelForSeq2SeqLM loads the model itself. generate() is the standard transformers one, with a KV cache; greedy decoding, beam search, and sampling all work. The encoder takes the prompt up to <|end_of_turn|>, and the decoder starts from <|assistant|> (decoder_start_token_id 9).

from transformers import AutoModelForSeq2SeqLM

model = AutoModelForSeq2SeqLM.from_pretrained(repo, trust_remote_code=True).eval()
prompt = (
    "<|system|>한자를 한글로 전사하시오:"
    "<|user|>會場을 一巡하고 돌아올 때까지도 日本人側 畵家까지도 一人도 發見할 수가 없었다.<|end_of_turn|>"
)
inputs = tokenizer(prompt, return_tensors="pt")
with torch.no_grad():
    output = model.generate(**inputs, max_new_tokens=64)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Tokenizer

tokenizer.json is a tokenizers-library build of the SentencePiece model (spiece.model, kept here as the source), made by scripts/build_hf_tokenizer.py in the GitHub repo. Calling the tokenizer does not add BOS or EOS tokens, and special tokens written into the text are mapped to their ids. It encodes like the SentencePiece wrapper the training code uses, with one known difference: where two segmentations of a span have exactly the same score (runs of digits, mostly), the same pieces can come out in a different order.

Files

File	Contents
model.safetensors	fp32 weights, 169.2M parameters, plus the jbu / cbu subword bucket tables
config.json, generation_config.json	model and generation config, with auto_map entries for the Auto classes
tokenizer.json, tokenizer_config.json, chat_template.jinja	fast tokenizer (38400 pieces) and chat template
spiece.model	the SentencePiece model tokenizer.json was built from
modeling_han2han.py, han2han_config.py	modeling code from the GitHub repo at commit af1330e

modeling_han2han.py (modeling_han2han_pytorch.py there) differs in two places: the config import is relative, and the module-level register_han2han import is replaced by the auto_map entries. han2han_config.py is the GitHub copy at commit 7304c38; it has not changed since. config.json names AutoModelForCausalLM under architectures so that transformers serve, which looks that name up in the transformers namespace, can load the model.

Instruction tuning

Fine-tuned from the Han2Han pre-trained checkpoint on instruction following, chain-of-thought reasoning, Hanja-Hangul article transcription, and summarization data (configs/it-muon-stage_1.yaml in the GitHub repo).

Citation

@inproceedings{han2han2026,
  title     = {Han2Han: Efficient Language-Specific Character Representation
               through Script-Aware Pre-Training for Historical Text Analysis},
  author    = {Adams, Cellik and Jo, EunKyoung and Kim, Juae},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2026},
  year      = {2026}
}

License

Apache License 2.0, the same as the GitHub repo.