marcosremar2/minimind-o-ptbr-talker-curriculum-100k-poc

MiniMind-O / Tucano2 PT-BR Mimi Talker POC

Experimental repository for the PT-BR speech fine-tuning work around MiniMind-O, Tucano2 1.5B and Mimi codec tokens.

This is not production-ready. The current result proves that the pipeline can connect text, Tucano2 hidden states, a Mimi-token Talker and Mimi decoding, but end-to-end speech for new Tucano2 responses is still unstable.

Current Goal

Build a small Portuguese voice assistant stack:

Whisper tiny/base
→ Tucano2 1.5B Q4 as Portuguese Thinker
→ small Mimi Talker
→ Mimi decoder

The near-term target is a Talker that can convert short PT-BR tutor/dialogue responses into intelligible Mimi audio tokens.

What Is Included

MiniMind-O PT-BR curriculum artifacts

Original MiniMind-O native checkpoints from the earlier PT-BR text/T2A curriculum:

checkpoints/sft_ptbr_curriculum_100k_textfix_768.pth
checkpoints/sft_ptbr_curriculum_100k_t2a_repair_768.pth
checkpoints/sft_ptbr_curriculum_100k_mixed_768.pth

Dataset artifacts:

data/ptbr_curriculum_100k.jsonl
data/sft_text_ptbr_curriculum_100k.parquet

Tucano2 hidden-state to Mimi Talker artifacts

Latest experimental Talker checkpoints:

checkpoints_tucano2_mimi/tucano_hidden_diagonal_bridge13_8k_h768_l8_duration_stop_heads_only.pt
checkpoints_tucano2_mimi/tucano_hidden_diagonal_bridge13_8k_h768_l8_prev_noise_from_duration_stop.pt

The best current checkpoint for duration control is:

checkpoints_tucano2_mimi/tucano_hidden_diagonal_bridge13_8k_h768_l8_duration_stop_heads_only.pt

The prev_noise_from_duration_stop checkpoint is an experimental hardening run. It did not improve the end-to-end result yet, but it is included for reproducibility.

Tools:

tools_tucano2_mimi/talker_text_mimi/

Evaluation WAVs and manifests:

eval_tucano2_mimi/

Implemented Architecture

The current Tucano2 → Mimi Talker uses:

• Tucano2 bridge layer 13 hidden states.
• Diagonal 8-stream Mimi codebook schedule, inspired by MiniMind-O.
• Causal Transformer decoder.
• 8 separate Mimi codebook heads.
• Optional text auxiliary head.
• duration_head to predict how many Mimi frames to generate.
• stop_head to predict where free-run generation should stop.
• prev-code noise training option to harden free-run generation against previous-token errors.

Latest Results

Duration / stop heads

Training only the new duration/stop heads while freezing the audio body worked as intended:

trainable parameters: 126,114 / 92,219,042
duration_mae: about 56 frames → about 3-5 frames
stop_mae: about 49 positions → about 2-4 positions

On offset-5000 diagnostic samples, predicted free-run lengths became close to target:

37 target frames → 40 generated frames
35 target frames → 34 generated frames
45 target frames → 46 generated frames

But the free-run audio still drifts in content.

Prev-code noise hardening

Implemented and tested:

--prev-code-noise-prob 0.03
--prev-code-noise-mode random

The training run stayed numerically healthy, but the end-to-end Whisper transcripts were still not acceptable. This means the next bottleneck is not duration; it is autoregressive code stability/content preservation.

End-to-End Status

The full chain runs:

prompt
→ Tucano2 generated text
→ Tucano2 response hidden states
→ Mimi Talker
→ Mimi decoder
→ WAV
→ Whisper validation

Quality status:

technical pipeline: passes
duration control: improved
teacher-forced audio: often good
free-run audio: still unstable
new Tucano2 responses: not production usable

Reproduction Commands

These commands assume the original workspace layout:

/workspace/minimind-o

Train the duration/stop heads only:

python tools_ptbr/talker_text_mimi/train_tucano_hidden_mimi_talker_diagonal_ss.py \
  --from-checkpoint out_talker_text_mimi/tucano_hidden_diagonal_bridge13_8k_h768_l8_tf6000_ss3000_p005/talker_tucano_hidden.pt \
  --cache-index data_ptbr/talker_text_mimi/tucano_hidden_cache_bridge13_16k/index.jsonl \
  --train-jsonl data_ptbr/talker_text_mimi/tucano2_contextual_text_mimi_full.jsonl \
  --out-dir out_talker_text_mimi/tucano_hidden_diagonal_bridge13_8k_h768_l8_duration_stop_heads_only \
  --limit-examples 16384 \
  --max-steps 2000 \
  --batch-size 16 \
  --hidden-size 768 \
  --decoder-layers 8 \
  --heads 12 \
  --learning-rate 1e-3 \
  --audio-loss-weight 0 \
  --duration-loss-weight 0.20 \
  --stop-loss-weight 0.05 \
  --train-duration-stop-only \
  --fp16

Train the prev-code-noise hardening run:

python tools_ptbr/talker_text_mimi/train_tucano_hidden_mimi_talker_diagonal_ss.py \
  --from-checkpoint out_talker_text_mimi/tucano_hidden_diagonal_bridge13_8k_h768_l8_duration_stop_heads_only/talker_tucano_hidden.pt \
  --cache-index data_ptbr/talker_text_mimi/tucano_hidden_cache_bridge13_16k/index.jsonl \
  --train-jsonl data_ptbr/talker_text_mimi/tucano2_contextual_text_mimi_full.jsonl \
  --out-dir out_talker_text_mimi/tucano_hidden_diagonal_bridge13_8k_h768_l8_prev_noise_from_duration_stop \
  --limit-examples 8192 \
  --max-steps 1200 \
  --batch-size 4 \
  --hidden-size 768 \
  --decoder-layers 8 \
  --heads 12 \
  --learning-rate 1e-5 \
  --ss-prob 0.02 \
  --prev-code-noise-prob 0.03 \
  --prev-code-noise-mode random \
  --duration-loss-weight 0.02 \
  --stop-loss-weight 0.01 \
  --fp16

Run end-to-end evaluation:

python tools_ptbr/talker_text_mimi/e2e_tucano_to_mimi_audio.py \
  --talker-checkpoint out_talker_text_mimi/tucano_hidden_diagonal_bridge13_8k_h768_l8_duration_stop_heads_only/talker_tucano_hidden.pt \
  --out-dir eval_ptbr/e2e_tucano2_h768_mimi_duration_stop_heads_only \
  --prompt "Como eu peço um café de forma educada?" \
  --prompt "Me corrija se eu disser: eu vai ao mercado." \
  --prompt "Me dê uma frase curta para praticar o som do R." \
  --use-duration-head \
  --use-stop-head \
  --whisper-model base

Next Technical Step

The next real step is not more duration training. It is content-stability training for free-run Mimi code generation:

1. Rollout/free-run loss or DAgger-style training.
2. Stronger corruption curriculum on previous Mimi codes.
3. Stage codebooks: stabilize codebooks 0-2 before fine acoustic codebooks.
4. Add validation that rejects checkpoints where E2E Whisper transcripts mix languages or become unsafe/irrelevant.

Caveats

• Checkpoints are native MiniMind/Talker .pth files, not standalone Transformers model directories.
• The Tucano2 1.5B model and Mimi decoder are not bundled here.
• Dataset caches such as Tucano hidden-state shards are not fully uploaded in this bundle.
• Some generated audio samples are unintelligible or contain mixed-language artifacts.
• This repository is intended for research/debugging, not deployment.

See SHA256SUMS.txt and file_manifest.json for file sizes and checksums.