Qwen3.8-27B-Satoori-KO-Synth

A Korean regional dialect transformation model trained without a single character of the source corpus. The training sentences are colloquial Korean utterances generated by our own model, converted to dialect by a rule-based transformer.

Read it together with the real-data sibling, Qwen3.8-27B-Satoori-KO. The pair exists to answer one question: how far can you get without the restricted data?

⛔ This is not a conversational model. All training data is single-turn transformation and classification, so conversational ability in dialect was neither trained nor measured.

Attribution (required notice)

⛔ Although no source text appears in the training set, the transformation rules were mined from word-level labels in the source corpora. Measured, 77.2% of transformed words in the training set pass through AI-Hub-derived rules; only 4.6% of transformations are explained by public resources alone. We therefore give the same attribution as the real-data model.

• Korean dialect speech data (Gangwon 118, Gyeongsang 119, Jeolla 120, Jeju 121, Chungcheong 122)
• Korean dialect speech data, middle-aged and elderly speakers (71517, 71558)
• Source: AI Hub (https://aihub.or.kr)

⛔ This repository contains no source data and no derived data (including any dialect-to-standard correspondence lexicon). Weights only.

Performance

KoDialectBench v0, median of three repeats, measured on the same engine as the real-data model.

Task	Metric	base	this model	real-data model
Comprehension	chrF	50.7	69.4	80.1
Comprehension	exact match %	2.2	11.6	24.7
Identification	accuracy %	24.7	74.6	79.4
Generation	dialectness	0.034	0.133	0.141
Generation	region match	0.103	0.688	0.593
Generation	ADI2	0.012	0.111	0.107
Generation	meaning-preservation chrF	68.7	79.5	75.8

Recovery, taking the real-data training gain as 100:

Axis	Recovery
Comprehension	52.7% (chrF 63.6, exact match 41.8, averaged)
Identification	91.2%
Generation, marker lexicon	105.6% ⛔ see below
Generation, reference chrF	72.4%
Axis-balanced mean (reference generation)	75.8%

⛔ The marker-lexicon generation figure is not independent of this model's training pipeline. Every marker the metric counts is a form the transformation rules can emit: removing the emittable forms from the scoring lexicon leaves zero held-out markers in all five regions. Rescored with chrF against held-out human reference sentences, without using the transformation-derived scoring inventory, generation recovery is 72.4%, and no synthetic arm exceeds the real-data reference. Treat 105.6% (and region match 119.3%, ADI2 105.2%) as metric alignment, not as better dialect.

⛔ Relatedly, synthetic between-region separation measures 0.870 against 0.753 for real data. The regions are more distinct from each other than real dialects are, which distorts the neighbour-confusion structure rather than improving it.

⛔ Comprehension sits at half. Neither improving source quality nor tripling the training set moved comprehension recovery out of the 52-54% band.

Training

A LoRA (r=8, alpha=16, all-linear, lr 2e-5, one epoch) on Qwen3.8-27B, then merged. The training set is 26,600 synthetic examples with the three tasks mixed evenly. Standard sentences were drawn from 19,510 colloquial utterances generated by our in-house Qwen3.8-27B-Human-KO.

Limitations

Shares every limitation of the real-data model (lower-bound metric, transcribed text only, a single training run).

Beyond those, regional variance is worse than in the real-data model. Gangwon identification is 62.0%, below both the real-data model (82.5%) and our own earlier synthetic version (72.0%). We traced this: synthetic Chungcheong's marker ambiguity runs 25pp above the real rate and absorbs predictions from neighbouring regions. It is not yet fixed.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

REPO = "ThakiCloud/Qwen3.8-27B-Satoori-KO-Synth"
tok = AutoTokenizer.from_pretrained(REPO)
m = AutoModelForCausalLM.from_pretrained(REPO, dtype=torch.bfloat16, device_map="cuda")

SYS = "너는 한국어 전문가다. 요청한 것만 정확히 답하고 설명·머리말·따옴표를 붙이지 않는다."

def ask(user, max_new=96):
    msgs = [{"role": "system", "content": SYS}, {"role": "user", "content": user}]
    enc = tok.apply_chat_template(msgs, add_generation_prompt=True,
                                  return_tensors="pt", enable_thinking=False)
    ids = (enc["input_ids"] if hasattr(enc, "keys") else enc)
    ids = (ids if ids.dim() > 1 else ids.unsqueeze(0)).to(m.device)
    out = m.generate(input_ids=ids, max_new_tokens=max_new, do_sample=False,
                     pad_token_id=tok.eos_token_id)
    return tok.decode(out[0][ids.shape[-1]:], skip_special_tokens=True).strip()

# Generation: standard -> dialect
ask("다음 표준어 문장을 경상도 방언으로 바꿔라. 뜻은 그대로 두고 말투만 바꾼다. "
    "바뀐 문장만 출력한다.\n\n어제 비가 많이 와서 길이 다 젖었더라")

# Comprehension: dialect -> standard
ask("다음 방언 문장을 표준어로 바꿔라. 바뀐 문장만 출력한다.\n\n어제 비가 마이 와가꼬 길이 다 젖었더라")

# Identification
ask("다음 문장이 어느 지역 방언인지 하나만 고른다. 지역 이름만 출력한다.\n"
    "보기: 강원도 / 경상도 / 전라도 / 제주도 / 충청도\n\n어제 비가 하영 와가주 길이 다 젖었더라", 16)

Prompt wording matters: these are the exact instruction templates the model was trained on. Rephrasing them degrades output.

Measured on out-of-domain register

The benchmark draws from the same conversational corpora the real-data arm trained on. To see what happens outside that, we wrote six sentences ourselves (customer service, public notice, tourism, drama dialogue, game NPC, everyday speech) and ran both models over all five regions.

	real-data model	synthetic model
Region differentiation (5 regions give 5 distinct outputs)	77%	97%
Round-trip exact recovery	93%	83%
Correctly identifies its own generated dialect	40%	63%

⛔ The real-data model collapsed on formal customer-service register: for "주문하신 상품은 내일 오후에 도착할 예정입니다" it returned the identical sentence for all five regions, ignoring the requested region entirely. The synthetic model, whose source sentences were model-generated across a wider register range, kept the regions apart.

⛔ Differentiating is not the same as differentiating correctly: the synthetic model produced a Jeju ending for a Jeolla request. And both models identify their own output poorly.

⛔ Six scenes, one greedy run. This is a demonstration, not a benchmark — do not rank the two models on these numbers.

Benchmark

https://huggingface.co/datasets/ThakiCloud/kodialectbench

---

한국어

한국어 사투리 변환 모델입니다. 학습 문장에 원문을 한 글자도 쓰지 않았습니다 — 우리 모델이 생성한 구어체 표준어 발화를 규칙 변환기로 사투리로 바꾼 합성 데이터로만 학습했습니다. 실데이터 학습 모델 (Satoori-KO)과 짝으로 읽으십시오. "실데이터 없이 어디까지 가능한가" 를 재기 위한 모델입니다.

⛔ 대화 모델이 아닙니다. 학습 데이터가 전부 단일턴 변환·분류라, 사투리로 자연스럽게 대화하는 능력은 학습된 적도 측정된 적도 없습니다.

출처(필수 고지): ⛔ 학습 문장에는 원문을 쓰지 않았지만 문장을 사투리로 바꾼 변환 규칙이 AI 허브 어절 라벨에서 나온 것입니다. 실측하면 변환 어절의 77.2%가 AI 허브 파생 규칙을 경유합니다(공개 자원만으로 설명되는 변환은 4.6%). 한국어 방언 발화 데이터(강원 118 · 경상 119 · 전라 120 · 제주 121 · 충청 122), 중·노년층 한국어 방언 발화 데이터(71517 · 71558). 출처: AI 허브 (https://aihub.or.kr). ⛔ 원본·재가공 데이터는 들어 있지 않습니다. 가중치만 배포합니다.

회수율(실데이터 이득을 100으로): 이해 52.7% · 식별 91.2% · 생성은 지표에 따라 갈립니다 — 어휘집 기준 105.6%, 참조 기준 72.4%. 축별 균등가중(참조 기준) 75.8%.

⛔ 어휘집 기반 생성 점수는 이 모델의 학습 파이프라인과 독립이 아닙니다. 채점 어휘집에서 변환 규칙이 산출 가능한 형태를 빼면 5개 권역 전부 남는 마커가 0개입니다. 어휘집을 쓰지 않는 참조 chrF 로 다시 재면 생성 회수율은 72.4% 이고 어떤 합성 팔도 실데이터를 넘지 않습니다. 105.6%(및 권역일치 119.3%)는 지표 정렬로 읽으십시오.

⛔ 강원 T2가 62.0%로 이전 합성 버전(72.0%)보다도 낮습니다. 합성 충청의 표지 모호도가 실제보다 25pp 높아 다른 권역을 흡수하는 것을 확인했고 아직 안 고쳤습니다.

Paper adapters (adapters/)

The two LoRA adapters behind the primary comparison of Metric--Construction Coupling Inflates Measured Synthetic Dialect Recovery are in adapters/, 234 MB each:

folder	arm	rules used
coupled-8600	SK5	full marker inventory
disjoint-8600	SK12-D9b	80% of it; held-out 20% used only for scoring

Both train on 8,600 synthetic examples with matched task-by-region composition, source register and recipe. Merging each onto Qwen/Qwen3.8-27B reproduces the paper's Figure 3.

⛔ Merge before serving. Attaching a LoRA adapter directly is a silent no-op on this architecture — no error, output identical to the base model. Use the bundled adapters/merge_lora.py and check the final line reads MERGE_OK {"n_applied": 496, ..., "verify": "PASS"}.

python adapters/merge_lora.py --base Qwen/Qwen3.8-27B \
  --adapter adapters/disjoint-8600 --out ./merged

⛔ The adapters are trained weights. No source data, dialect-to-standard correspondence list, or marker lexicon is included — those may not be redistributed.