marcosremar2/ptbr-omni-qwen3-venus-iasini-v1

PT-BR Omni Qwen3/Tucano Prototype - venus_iasini v1

Private research snapshot for the PT-BR omni voice pipeline:

audio question -> Whisper tiny -> audio bridge -> Tucano 2 -> answer text -> speech output

This repository preserves the useful training and benchmark artifacts from the GPU run. It does not include raw YouTube/reference audio. It includes generated validation WAVs, training manifests, checkpoints, and the exact Python tools used for the experiments.

What Works

• The audio bridge understands the Portuguese audio question and feeds Tucano coherently.
• The Kokoro fast answer baseline is intelligible and reaches sub-1s first playable audio after warmup.
• Qwen3 decoder/token layout is validated with ground-truth Qwen3 codes: target codes decode to intelligible PT-BR audio.

Current Limitation

The small Tucano -> Qwen3 talker is not quality-approved yet. It can produce a first chunk in about 0.8s, but the full generated audio is not intelligible. The next technical step is improving the talker/code predictor, not replacing the Qwen3 decoder.

Key Results

Path	Warm protocol	Result
Kokoro fallback answer	3 warmups + 3 measured	median first playable audio 0.822s
Audio bridge + Tucano text	3 warmups + 3 measured	median to answer text 0.760s
Small Qwen3 talker first chunk	3 warmups + 3 measured	median first chunk 0.796s
Qwen3 teacher full answer	3 warmups + 3 measured	median first playable audio 7.141s
Qwen3 ground-truth code decode	3 probe rows	ASR similarity about 0.864-0.943

Important Checkpoints

• checkpoints/tucano_qwen3_tts_venus_iasini_audio_bridge_aqa_kokoro_v1/audio_bridge.pt
  • Whisper/audio-feature bridge trained on the PT-BR audio Q/A bootstrap curriculum.
• checkpoints/tucano_qwen3_tts_venus_iasini_talker_aqa_answers_v1/talker.pt
  • AR C0 Qwen3 talker fine-tune. Kept for reproducibility; not production-quality.
• checkpoints/tucano_qwen3_tts_venus_iasini_talker_direct_aqa_answers_v1/talker.pt
  • Direct Qwen3 talker fine-tune used in first-chunk experiments. Fast, but not intelligible enough.

Training Manifests

• data/audio_qa/venus_iasini/manifests/venus_iasini_audio_qa_curriculum.jsonl
• data/audio_qa/venus_iasini/manifests/venus_iasini_audio_qa_kokoro_bootstrap_train.jsonl
• data/audio_qa/venus_iasini/manifests/venus_iasini_audio_qa_answers_qwen3_codes.jsonl

Each manifest has 96 examples. The Qwen3-code manifest stores answer text, speaker id, and flattened Qwen3 audio codes.

Benchmarks And Audio Probes

• artifacts/benchmark_page/index.html
  • Local benchmark dashboard copied from the run.
• artifacts/qwen3_omni_venus_kokoro_fast_answer_bench/
  • Fast intelligible fallback, voice pf_dora, not venus_iasini.
• artifacts/qwen3_omni_venus_gt_code_decode_probe/
  • Ground-truth Qwen3 token decode probes validating the decoder/token layout.
• artifacts/qwen3_omni_venus_first_chunk_small_talker_bench/
  • True first-chunk benchmark for the small Qwen3 talker.
• artifacts/qwen3_omni_venus_small_talker_full_audio_probe/
  • Full-audio probes showing the current small talker intelligibility failure.
• artifacts/qwen3_omni_venus_audio_qa_gpu_functional_qwen_teacher/
  • Functional Qwen3 teacher voice-clone baseline; quality OK, too slow for real-time UX.
• artifacts/qwen3_omni_venus_audio_qa_warm_first_chunk_bench/
  • Warm Qwen3 teacher benchmark.

Reproduction Notes

Expected base models:

• Polygl0t/Tucano2-qwen-1.5B-Instruct
• Qwen/Qwen3-TTS-Tokenizer-12Hz
• faster-whisper tiny
• Kokoro for the fast fallback baseline

Main scripts are under tools/talker_text_qwen3_tts/.

Status

This is a private experimental checkpoint. Treat the bridge and the benchmark harness as the working pieces. Treat the Qwen3 talker checkpoints as reproducibility artifacts that need another training iteration before mobile export.