jp-sns-jev7-estimator

Japanese SNS multi-axis scoring model that estimates seven continuous dimensions from text.

This repository contains an INT8 ONNX student model based on line-corporation/line-distilbert-base-japanese.

Outputs

The model returns seven sigmoid scores in [0, 1], in this order:

• insult
• threat
• obscene
• identity_attack
• sexual_explicit
• targetedness
• indirect_hostility

These values are continuous distilled teacher scores, not calibrated probabilities. They should not automatically be interpreted as binary toxic / non-toxic labels.

Important semantic notes:

• targetedness measures how clearly a specific target is addressed; it is not itself toxicity.
• obscene and sexual_explicit describe content characteristics and are not necessarily hostility.
• Binary thresholds should be validated separately for the intended domain and use case.

Model details

• Base model: line-corporation/line-distilbert-base-japanese
• Language: Japanese
• Architecture: DistilBERT-based sequence classifier
• Output dimensions: 7
• Maximum sequence length: 192
• Export: ONNX
• Quantization: dynamic INT8
• License: Apache-2.0
• Training objective: BCEWithLogitsLoss with continuous soft targets

Evaluation

The following metrics are from the bundled held-out test evaluation.

Axis	MAE	RMSE	Pearson	Spearman	F1@0.5
insult	0.0480	0.0907	0.7894	0.7971	0.5818
threat	0.0063	0.0219	0.5298	0.5217	0.0000
obscene	0.0274	0.0551	0.7886	0.7845	0.4658
identity_attack	0.0094	0.0374	0.7148	0.6098	0.5000
sexual_explicit	0.0358	0.0727	0.7306	0.6982	0.5323
targetedness	0.1045	0.1566	0.8941	0.9040	0.8372
indirect_hostility	0.0421	0.0805	0.8117	0.7941	0.5606

Macro metrics:

• Mean MAE: 0.039071
• Mean RMSE: 0.073569
• Mean Pearson: 0.751287
• Mean Spearman: 0.729915
• Mean F1@0.5: 0.496796
• Mean Accuracy@0.5: 0.972623

F1@0.5 and accuracy are auxiliary metrics only. Rare axes such as threat can be highly imbalanced, so thresholded classification metrics may be misleading.

ONNX validation

Validation subset size: 1000 / 5980 test samples.

Drift versus the PyTorch model:

• FP32 ONNX mean absolute drift: 0.00003698
• FP32 ONNX max absolute drift: 0.00076278
• INT8 ONNX mean absolute drift: 0.00654990
• INT8 ONNX max absolute drift: 0.12767627

CPU benchmark

Bundled local benchmark:

• Batch size: 32
• ms / batch: 1531.72
• Throughput: 20.89 texts/sec

Actual speed depends on CPU, ONNX Runtime version, thread settings, sequence length, and batch size.

Usage

Install dependencies:

pip install numpy onnxruntime transformers fugashi unidic-lite ipadic

Example:

from pathlib import Path
import json

import numpy as np
import onnxruntime as ort
from transformers import AutoTokenizer

ROOT = Path(".")

meta = json.loads(
    (ROOT / "model_meta.json").read_text(encoding="utf-8")
)

axes = meta["axes"]

tokenizer = AutoTokenizer.from_pretrained(
    ROOT / "tokenizer",
    local_files_only=True,
    trust_remote_code=True,
)

session = ort.InferenceSession(
    str(ROOT / "model_int8.onnx"),
    providers=["CPUExecutionProvider"],
)

text = "入力する日本語テキスト"

encoded = tokenizer(
    [text],
    return_tensors="np",
    padding=True,
    truncation=True,
    max_length=meta["max_length"],
)

scores = session.run(
    ["scores"],
    {
        "input_ids": encoded["input_ids"].astype(np.int64),
        "attention_mask": encoded["attention_mask"].astype(np.int64),
    },
)[0][0]

print(dict(zip(axes, map(float, scores))))

A standalone example is also included as inference_example.py.

Suggested interpretation

For exploratory hostility scoring, a simple baseline is to treat these four axes as the core hostile-language dimensions:

insult
threat
identity_attack
indirect_hostility

For example, an exploratory composite can be defined as the maximum of those four values. This is not a calibrated toxicity probability and should not be presented as a validated binary classifier without a separately labeled validation set.

Intended uses

Possible uses include:

• exploratory analysis of Japanese SNS text
• ranking or filtering by individual dimensions
• large-scale corpus scoring
• research on offensive / hostile language
• preprocessing for downstream NLP analysis

Limitations

• The model is a teacher-score estimator, not a model trained directly on human gold toxicity labels.
• Scores are not calibrated probabilities.
• Performance can shift across platforms, communities, dialects, slang, and time periods.
• Irony, quotation, coded language, reclaimed terms, and community-specific expressions can be misread.
• Rare dimensions can have unstable thresholded metrics.
• Do not use this model as the sole basis for moderation, punitive decisions, clinical judgments, or decisions affecting individual rights.

Files

• model_int8.onnx — quantized ONNX model
• model_meta.json — axis order and model metadata
• tokenizer/ — tokenizer files
• inference_example.py — standalone CPU inference example
• test_metrics.json — test-set metrics
• onnx_validation.json — ONNX / quantization validation
• benchmark_cpu.json — CPU benchmark

License

Apache License 2.0.