README.md

---
language:
  - ja
license: apache-2.0
base_model: openai/whisper-base
tags:
  - whisper
  - japanese
  - anime
  - speech-recognition
  - fine-tuned
  - automatic-speech-recognition
datasets:
  - joujiboi/japanese-anime-speech-v2
metrics:
  - cer
pipeline_tag: automatic-speech-recognition
widget:
  - src: https://huggingface.co/sky9262/whisper-base-japanese-anime/resolve/main/samples/sample_1305.wav
    example_title: Cheerful Greeting
  - src: https://huggingface.co/sky9262/whisper-base-japanese-anime/resolve/main/samples/sample_1370.wav
    example_title: Over-the-top Shock
  - src: https://huggingface.co/sky9262/whisper-base-japanese-anime/resolve/main/samples/sample_1740.wav
    example_title: Desperate Urgency
model-index:
  - name: whisper-base-japanese-anime
    results:
      - task:
          type: automatic-speech-recognition
          name: Automatic Speech Recognition
        dataset:
          name: joujiboi/japanese-anime-speech-v2
          type: joujiboi/japanese-anime-speech-v2
          split: sfw
        metrics:
          - type: cer
            value: 31.2
            name: CER (final)
          - type: cer
            value: 44.3
            name: CER (baseline)
---

# whisper-base-japanese-anime

**[Try the live demo on Spaces](https://huggingface.co/spaces/sky9262/whisper-base-japanese-anime)** — record audio or use pre-loaded anime samples to compare base vs fine-tuned model.

A fine-tuned version of [openai/whisper-base](https://huggingface.co/openai/whisper-base) for **Japanese anime speech recognition**.

## Model Description

This model was fine-tuned on the **SFW split** of [joujiboi/japanese-anime-speech-v2](https://huggingface.co/datasets/joujiboi/japanese-anime-speech-v2) — a dataset of Japanese anime voice clips paired with transcriptions. The goal is to improve Whisper's accuracy on anime-style Japanese speech, which includes expressive intonation, character voices, and domain-specific vocabulary.

| Property | Value |
|---|---|
| **Base model** | `openai/whisper-base` (74M params) |
| **Language** | Japanese (`ja`) |
| **Task** | `transcribe` |
| **Domain** | Anime speech |
| **Training samples** | ~269k SFW samples |

## Training Details

Training was done incrementally across **26 sequential ranges** of ~10k samples each (10k → 269k), with 800 gradient steps per range. This progressive approach allowed monitoring CER at each stage.

### Training Hyperparameters

| Parameter | Value |
|---|---|
| Learning rate | `1e-6` |
| LR scheduler | Linear |
| Warmup steps | 200 |
| Batch size (train) | 20 |
| Batch size (eval) | 16 |
| Gradient accumulation | 1 |
| Max steps per range | 800 |
| Precision | FP16 |
| Optimizer | AdamW (fused) |
| Weight decay | 0.0 |
| Total training ranges | 26 |

### Training Progression

| Stage | Samples | Val Loss | CER |
|---|---|---|---|
| Baseline (10k-20k) | 10k | 0.892 | 44.3% |
| Mid (120k-130k) | 120k | 0.740 | 33.5% |
| Late (210k-220k) | 210k | 0.621 | 31.2% |
| **Final (260k-269k)** | **269k** | **0.563** | **31.2%** |

### Key Results

- **CER reduction**: 44.3% → 31.2% (**~30% relative improvement**)
- **Validation loss reduction**: 0.892 → 0.563 (**37% drop**)
- Best CER achieved at 250k-260k range: **30.6%**
- Validation loss steadily decreased throughout training

## Usage

```python
import torch
from transformers import WhisperProcessor, WhisperForConditionalGeneration

# Load model and processor
processor = WhisperProcessor.from_pretrained(
    "sky9262/whisper-base-japanese-anime",
    language="Japanese",
    task="transcribe",
)
model = WhisperForConditionalGeneration.from_pretrained(
    "sky9262/whisper-base-japanese-anime"
)

device = "cuda" if torch.cuda.is_available() else "cpu"
model.to(device)

# Transcribe audio
import soundfile as sf

audio_array, sampling_rate = sf.read("your_audio.wav")

inputs = processor.feature_extractor(
    audio_array,
    sampling_rate=sampling_rate,
    return_tensors="pt",
).input_features.to(device)

with torch.no_grad():
    predicted_ids = model.generate(
        inputs,
        language="ja",
        task="transcribe",
    )

transcription = processor.tokenizer.batch_decode(
    predicted_ids,
    skip_special_tokens=True,
)[0]

print(transcription)
```

## Audio Samples & Transcriptions

15 expressive anime voice samples (unseen test set) comparing **base `openai/whisper-base`** vs. **fine-tuned model**.
These clips feature shouting, panic, teasing, anger, and more -- areas where the base model often hallucinates or loops.

### 1. Cheerful Greeting (sample 1305, 3.4s)
<audio controls src="https://huggingface.co/sky9262/whisper-base-japanese-anime/resolve/main/samples/sample_1305.wav"></audio>

| | Text |
|---|---|
| **Base Model** | 先生 おはろございまーす |
| **Fine-tuned** | 先生、おはろーございまーす |

### 2. Panicked Scream (sample 1340, 5.5s)
<audio controls src="https://huggingface.co/sky9262/whisper-base-japanese-anime/resolve/main/samples/sample_1340.wav"></audio>

| | Text |
|---|---|
| **Base Model** | 今日はスカートをめくらないでくださいましー! |
| **Fine-tuned** | 今日はわわー！スカートを目くらないでくださいましー！ |

### 3. Crying for Help — JP/EN Mix (sample 1343, 5.8s)
<audio controls src="https://huggingface.co/sky9262/whisper-base-japanese-anime/resolve/main/samples/sample_1343.wav"></audio>

| | Text |
|---|---|
| **Base Model** | いやー、助けてください、ニーさん! Help you! |
| **Fine-tuned** | やー、助けてください、兄さん！ヘルプユー |

### 4. Playful Teasing (sample 1369, 5.9s)
<audio controls src="https://huggingface.co/sky9262/whisper-base-japanese-anime/resolve/main/samples/sample_1369.wav"></audio>

| | Text |
|---|---|
| **Base Model** | うーん? もうしかして、ネギラってくれるのかねー? |
| **Fine-tuned** | うーん、もしかして、ネギラってくれるのかねー |

### 5. Over-the-top Shock (sample 1370, 8.0s)
<audio controls src="https://huggingface.co/sky9262/whisper-base-japanese-anime/resolve/main/samples/sample_1370.wav"></audio>

| | Text |
|---|---|
| **Base Model** | な、なに?だいまおがだいにケータイに返ししたです!ですー! |
| **Fine-tuned** | な、なにー！だいまおーが、だいに携帯に変身したです、ですー！ |

### 6. Disappointed Sigh (sample 1416, 3.8s)
<audio controls src="https://huggingface.co/sky9262/whisper-base-japanese-anime/resolve/main/samples/sample_1416.wav"></audio>

| | Text |
|---|---|
| **Base Model** | えぇ…もう終わりなのか? |
| **Fine-tuned** | えぇ～、もう終わりなのか？ |

### 7. Stuttering Shock (sample 1469, 4.1s)
<audio controls src="https://huggingface.co/sky9262/whisper-base-japanese-anime/resolve/main/samples/sample_1469.wav"></audio>

| | Text |
|---|---|
| **Base Model** | ななな! 伸びゆきとのは痛いなによ! |
| **Fine-tuned** | ななななな、信行きとのは一体なにを！ |

### 8. Childish Insistence (sample 1502, 5.9s)
<audio controls src="https://huggingface.co/sky9262/whisper-base-japanese-anime/resolve/main/samples/sample_1502.wav"></audio>

| | Text |
|---|---|
| **Base Model** | いやぁ、行くったら行くの? あるいも一緒に遊ぶですよ! |
| **Fine-tuned** | やぁ、行くったら行くの！アンリーも一緒に遊ぶですよ |

### 9. Furious Struggle (sample 1515, 6.0s) ⚠️ Base model hallucinates
<audio controls src="https://huggingface.co/sky9262/whisper-base-japanese-anime/resolve/main/samples/sample_1515.wav"></audio>

| | Text |
|---|---|
| **Base Model** | え、話せブレフモノ!おぉ、も、アドデカイスのか!へっへっへっへっへっへっへっ... **(loops 200+ times)** |
| **Fine-tuned** | えへっ、話せぶれ物、おうも後で返すのかっ、ふわああっ！ |

### 10. Whining / Pouting (sample 1542, 5.0s)
<audio controls src="https://huggingface.co/sky9262/whisper-base-japanese-anime/resolve/main/samples/sample_1542.wav"></audio>

| | Text |
|---|---|
| **Base Model** | だってだって!成功がかわいくないんだもん! |
| **Fine-tuned** | だってだって、せっくくが可愛くないんだもん！ |

### 11. Alarmed Demand (sample 1627, 2.9s)
<audio controls src="https://huggingface.co/sky9262/whisper-base-japanese-anime/resolve/main/samples/sample_1627.wav"></audio>

| | Text |
|---|---|
| **Base Model** | まんじゃ、痛い何があった? |
| **Fine-tuned** | なんじゃ、一体何があった！ |

### 12. Cute Excitement (sample 1661, 3.7s) ⚠️ Base model hallucinates
<audio controls src="https://huggingface.co/sky9262/whisper-base-japanese-anime/resolve/main/samples/sample_1661.wav"></audio>

| | Text |
|---|---|
| **Base Model** | もーもーもーもーもーもー... **(loops 200+ times)** |
| **Fine-tuned** | やふぅ！もうふもふもふですー |

### 13. Angry Scolding (sample 1728, 4.6s)
<audio controls src="https://huggingface.co/sky9262/whisper-base-japanese-anime/resolve/main/samples/sample_1728.wav"></audio>

| | Text |
|---|---|
| **Base Model** | たかものは、そんなことができたら、笑顔が特にやっておるわ! |
| **Fine-tuned** | 高者の、そんなことができたら、妾が特にやっておるわ |

### 14. Desperate Urgency (sample 1740, 4.5s)
<audio controls src="https://huggingface.co/sky9262/whisper-base-japanese-anime/resolve/main/samples/sample_1740.wav"></audio>

| | Text |
|---|---|
| **Base Model** | ハイヤト!しっかりしろ!ハイヤト! |
| **Fine-tuned** | 隼斗、しっかりしろ、隼斗っ！ |

### 15. Elongated Refusal (sample 1768, 3.0s)
<audio controls src="https://huggingface.co/sky9262/whisper-base-japanese-anime/resolve/main/samples/sample_1768.wav"></audio>

| | Text |
|---|---|
| **Base Model** | んだーめーですー |
| **Fine-tuned** | ダメです！ |

---

<details>
<summary>⚠️ <b>NSFW Samples (Click to expand)</b> — Contains explicit anime voice content. Viewer discretion advised.</summary>

<br>

> **Warning**: The following 15 samples are from the NSFW split and contain sexually explicit voice acting.
> They are included to demonstrate the model's ability to transcribe emotionally extreme and non-standard speech
> where the base model completely fails (repetition loops, hallucinations).

### NSFW 1. Apologetic Crying (nsfw #6, 6.2s)
<audio controls src="https://huggingface.co/sky9262/whisper-base-japanese-anime/resolve/main/samples/nsfw_sample_6.wav"></audio>

| | Text |
|---|---|
| **Base Model** | ううううううくれない!ごめんない! |
| **Fine-tuned** | やぁ、やぁああっ、くっくりゃない、ごめんなないっ！ |

### NSFW 2. Teasing / Seductive (nsfw #16, 5.7s)
<audio controls src="https://huggingface.co/sky9262/whisper-base-japanese-anime/resolve/main/samples/nsfw_sample_16.wav"></audio>

| | Text |
|---|---|
| **Base Model** | なぁ、ニーモクーン スマした顔して、エッチに興味あるの? |
| **Fine-tuned** | なー、ニーもくん、済ました顔して、エッチに興味あるの |

### NSFW 3. Flustered Embarrassment (nsfw #29, 5.9s)
<audio controls src="https://huggingface.co/sky9262/whisper-base-japanese-anime/resolve/main/samples/nsfw_sample_29.wav"></audio>

| | Text |
|---|---|
| **Base Model** | いやいや、ウイキを見出しながら近づかないでください。なんだか一致です。 |
| **Fine-tuned** | やややん、息を見出しながら近づかないでください。なんだかエッジーです |

### NSFW 4. Confused Protest (nsfw #33, 6.0s)
<audio controls src="https://huggingface.co/sky9262/whisper-base-japanese-anime/resolve/main/samples/nsfw_sample_33.wav"></audio>

| | Text |
|---|---|
| **Base Model** | 圧倒状況って何ですか?私エッチな動物さじゃないですよ! |
| **Fine-tuned** | 熱情聞ってなんですか、私、エッチな動物さんじゃないですよー |

### NSFW 5. Exhibitionist Excitement (nsfw #46, 6.8s)
<audio controls src="https://huggingface.co/sky9262/whisper-base-japanese-anime/resolve/main/samples/nsfw_sample_46.wav"></audio>

| | Text |
|---|---|
| **Base Model** | あさあ、とんどんご覧になって下さい!もっと私を見たからさ!あ、あ、あ、あ! |
| **Fine-tuned** | あさー、どんどんごらになってください。もっと私を見たからさ、あっ |

### NSFW 6. Victory Panting (nsfw #60, 4.8s) ⚠️ Base model hallucinates
<audio controls src="https://huggingface.co/sky9262/whisper-base-japanese-anime/resolve/main/samples/nsfw_sample_60.wav"></audio>

| | Text |
|---|---|
| **Base Model** | うううう... **(loops 400+ times)** |
| **Fine-tuned** | はぁああああ…勝ちましたっ！ |

### NSFW 7. Embarrassed Protest (nsfw #75, 5.6s)
<audio controls src="https://huggingface.co/sky9262/whisper-base-japanese-anime/resolve/main/samples/nsfw_sample_75.wav"></audio>

| | Text |
|---|---|
| **Base Model** | えぇ、おぉ、いい誘いでください!私、エッチさんじゃないですよ! |
| **Fine-tuned** | やふぅぅぅぅぅぅ、やめてください。私、エッチさんじゃないですよー |

### NSFW 8. Short Shock (nsfw #84, 2.1s) ⚠️ Base model hallucinates
<audio controls src="https://huggingface.co/sky9262/whisper-base-japanese-anime/resolve/main/samples/nsfw_sample_84.wav"></audio>

| | Text |
|---|---|
| **Base Model** | えぇえ!あ、嘘!嘘!あ、あ、あ... **(loops 200+ times)** |
| **Fine-tuned** | やや ややっ、そ、そ、そやだっ |

### NSFW 9. Angry Embarrassment (nsfw #89, 6.7s) ⚠️ Base model hallucinates
<audio controls src="https://huggingface.co/sky9262/whisper-base-japanese-anime/resolve/main/samples/nsfw_sample_89.wav"></audio>

| | Text |
|---|---|
| **Base Model** | うっうう... **(loops 400+ times)** |
| **Fine-tuned** | ぱ、バクバクバクバクバクバクッ！エイチのことを言うでない！もーもーもーもーもーもー |

### NSFW 10. Overwhelmed Scream (nsfw #356, 2.7s) ⚠️ Base model hallucinates
<audio controls src="https://huggingface.co/sky9262/whisper-base-japanese-anime/resolve/main/samples/nsfw_sample_356.wav"></audio>

| | Text |
|---|---|
| **Base Model** | うう... **(loops 400+ times)** |
| **Fine-tuned** | にゅうぅ～～～～～～～～っ |

### NSFW 11. Surprised Protest (nsfw #359, 6.9s) ⚠️ Fine-tuned model loops
<audio controls src="https://huggingface.co/sky9262/whisper-base-japanese-anime/resolve/main/samples/nsfw_sample_359.wav"></audio>

| | Text |
|---|---|
| **Base Model** | おい!いやぁ!あぁ!あぁ!いや、今そこないじゃん! |
| **Fine-tuned** | んにゃっ、やっ、やっ、やっ... **(loops 140+ times, 444 chars)** |

### NSFW 12. Intense Overwhelm (nsfw #404, 9.2s) ⚠️ Both models hallucinate
<audio controls src="https://huggingface.co/sky9262/whisper-base-japanese-anime/resolve/main/samples/nsfw_sample_404.wav"></audio>

| | Text |
|---|---|
| **Base Model** | あーーー... **(loops 400+ times)** |
| **Fine-tuned** | ああああああああああっ、それ、それ、ダメ、ダメッ |

### NSFW 13. Sudden Panic (nsfw #439, 3.3s) ⚠️ Base model hallucinates
<audio controls src="https://huggingface.co/sky9262/whisper-base-japanese-anime/resolve/main/samples/nsfw_sample_439.wav"></audio>

| | Text |
|---|---|
| **Base Model** | お、お、え、え、え... **(loops 200+ times)** |
| **Fine-tuned** | んっ、あっ、えんっ、えんっ、えんっ、えんっ！ |

### NSFW 14. Desperate Plea (nsfw #456, 7.0s)
<audio controls src="https://huggingface.co/sky9262/whisper-base-japanese-anime/resolve/main/samples/nsfw_sample_456.wav"></audio>

| | Text |
|---|---|
| **Base Model** | うえ、あわたって本当にでもやるし、お願いしましょ!このまま、このまま! |
| **Fine-tuned** | あ、もうだめ、本当にダメです。お願いします。お願いします。こんなもん、こんなもんもんっ！ |

### NSFW 15. Falling Panic (nsfw #484, 4.6s) ⚠️ Base model hallucinates
<audio controls src="https://huggingface.co/sky9262/whisper-base-japanese-anime/resolve/main/samples/nsfw_sample_484.wav"></audio>

| | Text |
|---|---|
| **Base Model** | あ、あ、あ... **(loops 200+ times)** |
| **Fine-tuned** | あっ、あっ、あっ、あっ、あっ、ひゃあんっ、落ちる、落ちちゃいませんっ |

---

#### Pure Moaning / Non-verbal Vocalization

> These 5 samples contain **no intelligible words** — only moaning, panting, and gasping.
> The base model completely hallucinates on all of them. The fine-tuned model also struggles but produces shorter, more contained output.

### NSFW 16. Soft Panting (nsfw #136, 8.8s) ⚠️ Both loop
<audio controls src="https://huggingface.co/sky9262/whisper-base-japanese-anime/resolve/main/samples/nsfw_sample_136.wav"></audio>

| | Text |
|---|---|
| **Base Model** | ふっふっふっふっふっ... **(loops 200+ times)** |
| **Fine-tuned** | んっ、はっ、はっ、はっ、はっ、はっ... **(15 repeats, 44 chars vs base 444)** |

### NSFW 17. Intense Climax Scream (nsfw #176, 5.3s) ⚠️ Both loop
<audio controls src="https://huggingface.co/sky9262/whisper-base-japanese-anime/resolve/main/samples/nsfw_sample_176.wav"></audio>

| | Text |
|---|---|
| **Base Model** | うっうっうっうっ... **(loops 200+ times)** |
| **Fine-tuned** | うぅ、うぅ、うぅ、うぅ、うぅ、うぅ、うぅ、うぅ **(8 repeats, 23 chars vs base 444)** |

### NSFW 18. Subdued Gasping (nsfw #272, 8.5s) ⚠️ Both loop
<audio controls src="https://huggingface.co/sky9262/whisper-base-japanese-anime/resolve/main/samples/nsfw_sample_272.wav"></audio>

| | Text |
|---|---|
| **Base Model** | あ!あ!あ!あ!... **(loops 200+ times)** |
| **Fine-tuned** | あ、あんっ、あんっ、あんっ、あんっ... **(11 repeats, 41 chars vs base 444)** |

### NSFW 19. Exhausted Breathing (nsfw #384, 9.9s) ⚠️ Both loop
<audio controls src="https://huggingface.co/sky9262/whisper-base-japanese-anime/resolve/main/samples/nsfw_sample_384.wav"></audio>

| | Text |
|---|---|
| **Base Model** | ううう... **(loops 400+ times)** |
| **Fine-tuned** | んっ、はぁ…はぁ…はぁ…はぁ…はぁ…はぁ… **(11 repeats, 35 chars vs base 444)** |

### NSFW 20. High-pitched Climax (nsfw #468, 5.5s) ⚠️ Both loop
<audio controls src="https://huggingface.co/sky9262/whisper-base-japanese-anime/resolve/main/samples/nsfw_sample_468.wav"></audio>

| | Text |
|---|---|
| **Base Model** | あ!あ!あ!あ!... **(loops 200+ times)** |
| **Fine-tuned** | あ、あ、ああ、あああ... **(loops 400+ times, 883 chars — worst case, similar to base)** |

</details>

## Evaluation

Evaluated on unseen test samples from the SFW split using **Character Error Rate (CER)**, the standard metric for Japanese ASR (since Japanese lacks clear word boundaries).

CER is computed as:

```
CER = (substitutions + insertions + deletions) / len(reference)
```

## Limitations

- **Domain-specific**: Optimized for anime-style Japanese speech. Performance on other Japanese speech domains (news, conversation, etc.) may differ from the base model.
- **Base model size**: This is whisper-base (74M params). Larger variants (small, medium, large) would likely achieve better CER.
- **SFW only**: Trained only on the SFW portion of the dataset.
- **Single speaker variability**: Anime speech has high variability in tone, pitch, and speaking style which can still cause errors.

## Citation

If you use this model, please cite the original Whisper paper and the training dataset:

```bibtex
@article{radford2022whisper,
  title={Robust Speech Recognition via Large-Scale Weak Supervision},
  author={Radford, Alec and Kim, Jong Wook and Xu, Tao and Brockman, Greg and McLeavey, Christine and Sutskever, Ilya},
  journal={arXiv preprint arXiv:2212.04356},
  year={2022}
}
```

Dataset: [joujiboi/japanese-anime-speech-v2](https://huggingface.co/datasets/joujiboi/japanese-anime-speech-v2)