docs/DESIGN.md

qwen-voice-studio — Design

Authored by the project brain (qwen-voice-brain) and committed by the lead. Decisions logged in DECISIONS.md; implementation tasks in PLAN.md.

1. Purpose & principles

Open-source voice studio on the Qwen3-TTS-12Hz-1.7B family. Priorities, in order: (1) fidelity — never trade audio quality for latency or convenience; (2) feature-completeness — every capability the three checkpoints expose; (3) one codebase, two platforms — Apple-silicon MPS locally, HF ZeroGPU (CUDA, RTX Pro 6000 Blackwell) deployed, with platform variance isolated in one policy object.

2. Models & capabilities

Checkpoint	Mode	API
Qwen3-TTS-12Hz-1.7B-Base	Voice cloning (ref audio+text, or x-vector-only); LoRA target	generate_voice_clone, create_voice_clone_prompt
…-CustomVoice	9 preset voices + natural-language instruct; batch	generate_custom_voice
…-VoiceDesign	Voice from text description	generate_voice_design

Shared: 10 languages + Auto (via get_supported_languages()), sampling knobs incl. subtalker_*, max_new_tokens (default 2048), 24 kHz output (always use the API-returned sr; never hardcode). Speakers via get_supported_speakers() at runtime. Codec Qwen3-TTS-Tokenizer-12Hz auto-downloaded.

3. Platform matrix

	Local (M5 Max, 128 GB)	HF ZeroGPU Space (private)
Device	mps	cuda (half RTX Pro 6000, 48 GB)
Load	from_pretrained(device_map="mps", dtype=bf16, attn_implementation="sdpa")	from_pretrained(dtype=bf16, attn="sdpa") → module-level .to("cuda") (CUDA emulation; per ZeroGPU docs, never lazy-move inside @spaces.GPU)
Env	PYTORCH_ENABLE_MPS_FALLBACK=1	@spaces.GPU(duration=callable), size default large
Python / torch	3.12 / 2.13 (dev-validated)	3.12.12 / torch pin ladder: 2.13.* → fallback 2.11.0 + local re-smoke
attn / compile	sdpa / no torch.compile	sdpa / no torch.compile (AOT-inductor = post-v1 only)
Residency	all-3-resident (adaptive degrade, §6)	all-3 on cuda at startup; QVS_RESIDENCY=fork_move fallback flag

spaces is an optional import: try: import spaces except ImportError: <no-op decorator shim>. The decorator is effect-free off-ZeroGPU — one app.py, no entrypoint branching.

4. Module architecture

qvs/
  config.py    # Settings; env flags QVS_RESIDENCY, QVS_FORCE_SINGLE_RESIDENT,
               # QVS_DTYPE_OVERRIDE; pinned version constants; paths (data/voices, data/adapters)
  device.py    # DevicePolicy {platform, device, dtype, attn_impl, load_strategy};
               # detect ZeroGPU (spaces import / env) vs mps vs cpu; seed_all(seed);
               # memory gauge: macOS = vm_stat committed (wired+active+compressed) + pageout delta;
               # cuda = torch.cuda.memory_allocated; Watchdog(warn=72GB, abort=76GB, sample during gen)
  registry.py  # ModelRegistry: sequential load of 3 checkpoints per DevicePolicy;
               # per-load committed-delta logging; residency modes all_resident | single_on_demand;
               # adaptive degrade (§6); get(mode) accessor; codec-copy count reported at startup
  engine.py    # THE seam: synthesize(SynthesisRequest) -> AudioResult{wav, sr, meta}.
               # Request covers all 3 modes + voice source (ref_pair | prompt | xvector) + adapter flag
               # + all gen params incl. subtalker_*; per-request seeding via device.seed_all;
               # duration_estimate(request) used by UI and @spaces.GPU dynamic duration
  lora.py      # AdapterManager: resolve HF repo-id or upload → snapshot; locate adapter_config.json
               # by subfolder scan (Darija: talker_lora/); validate declared base model (warn+override);
               # attach at base.model.talker.model; enable/disable/unload; one active adapter;
               # surface bundled speaker_embedding.pt to VoiceLibrary; introspection (r, alpha, targets)
  voices.py    # VoiceLibrary: VoicePrompt {kind: ref_pair | full_prompt(.pt) | xvector(2048-d [+speaker_id]),
               # name, meta.json sidecar} stored under data/voices (gitignored);
               # sources: user save, create_voice_clone_prompt cache, adapter-bundled, Design→Clone bridge
  audio.py     # save/load wav (returned sr), duration/RMS/silence utils — shared by app AND tests
ui/            # clone.py, presets.py, design.py, lora_lab.py, library.py, shared.py (Advanced accordion
               # builder: sampling + subtalker + max_new_tokens + seed + language; status strip)
app.py         # wires tabs; gr.queue(default_concurrency_limit=1); handlers decorated @spaces.GPU

5. UI decomposition (5 tabs)

1. Clone — ref audio + ref text | x_vector_only_mode toggle | active-voice picker (from Library) | adapter dropdown (state owned by LoRA Lab) | Advanced | Generate.
2. Preset Voices — speaker dropdown (runtime introspection), instruct textbox, Advanced; batch: multiline one-per-line → zip (built last).
3. Voice Design — instruct (required), text, Advanced; "Send to Library / use as Clone ref" bridge (canonical pattern for making a designed voice reusable).
4. LoRA Lab — repo-id fetch / file upload, subfolder autodetect, config display (r/α/targets/base), attach + on/off toggle, unload, "save bundled speaker to Library", one canned-sentence quick-test routed through synthesize(). No generation UI duplication.
5. Voice Library — list/rename/delete prompts of all three kinds; "set active for Clone".

Global status strip: device, dtype, attn, per-model residency, committed memory, active adapter.

6. Memory policy (local hard gate)

Gauge = vm_stat committed ("Memory Used"), NOT process RSS (undercounts on MPS). Budget: baseline ≈55 GB (operator workload) + all-3 ≈14.6 GB → peak ≈69.6 GB measured. Watchdog: warn ≥72 GB, hard-abort ≥76 GB, sampled every few seconds during loads AND generation; pageout/swap delta growth = early warning, log loudly. Adaptive startup: baseline >65 GB before loading → degrade to single_on_demand + loud log. fp32 contingency: if MPS-bf16 ever fails quality checks, fp32 all-3 (~+28 GB) breaches the gate → fp32 forces single_on_demand. Hard rule: exactly one model-holding process, ever. Playwright drives the running app; no parallel pytest workers loading models.

7. ZeroGPU deployment design

Module-level: build models, .to("cuda") at import (emulation mode). Handlers: @spaces.GPU(duration=estimate) where estimate = clamp(15 + k·expected_new_tokens, 30, 120); calibrate k from first Space runs; log GPU-seconds per request. First acceptance on Space: two consecutive generations with next(model.parameters()).device logged inside the fork both times → proves no orphan; if orphaned, flip QVS_RESIDENCY=fork_move (build CPU, idempotent move inside fork) — flag flip, not redesign. Quota: PRO 40 min/day; overage = pre-paid credits = operator approval required.

8. LoRA design + acceptance

Mechanism: unmerged PEFT attach at base.model.talker.model (q/k/v/o_proj, per Darija r8/α16), toggle = enable/disable_adapter_layers; removal = unload(); never merge_and_unload in product flow. Attach implementation decision tree: (A) keep PeftModel wrapper (talker.model = PeftModel(...)), toggle via enable_adapters()/disable_adapters() — test with generate smoke; if it works, ship A. (B) if A breaks the generate path → peft.inject_adapter_in_model (model identity & module tree stay native), toggle by iterating LoraLayers. (C) last resort → merge_and_unload with documented one-way semantics ("toggle off" = reload Base talker; no repeated merge/reload cycles). Acceptance (fixed seed, adapter-on vs -off): (i) generate_voice_clone works post-attach; (ii) toggle OFF→ON changes output audibly, <1 s; (iii) after unload, same-seed generation matches pre-attach baseline; (iv) Darija adapter + bundled speaker_embedding.pt {(2048,), speaker_id 3000} through the x-vector path yields the Darija speaker's voice, audibly distinct from adapter-off. Uploads: accept zip/safetensors+config; store under data/adapters.

9. Testing strategy

Layered: (0) env/version print + memory gauge sanity; (1) headless engine smokes per mode (wav exists, duration>0.5 s, RMS above silence floor, sr==API sr, no NaN); (2) MPS silent-corruption check ONCE: mel/STFT path (modeling_qwen3_tts.py:447/459) MPS-vs-CPU cosine >0.999 IF stft executes natively (FALLBACK=1 only rescues unimplemented ops — implemented-but-buggy fails silently) + clone-resemblance ear check; (3) Playwright full UI walkthrough, every tab + param, programmatic audio asserts (screenshots prove UI state, never audio); (4) same suite against the private Space. Determinism policy: same-seed repeatability asserted same-device only; MPS vs CUDA never bit-identical — assert properties, not waveforms.

10. Pinned versions

py 3.12; torch 2.13 (dev) with Space ladder →2.11.0; transformers==4.57.3 (5.x breaks qwen_tts); qwen-tts==0.1.1; peft==0.19.1; accelerate==1.12.0; huggingface_hub<1.0; gradio 6.17.3; spaces (Space only). requirements.txt = Space runtime; requirements-dev.txt = + playwright, pytest, psutil.

11. Risks register (accepted)

bf16-on-MPS numerics (mitigated: A/B smoke + fp32 contingency); torch 2.13 Space rejection (mitigated: pin ladder + re-smoke trigger); module-level cuda orphan regression (mitigated: fork_move flag + device-log acceptance); adapter key-prefix mismatch (mitigated: inspect-first + remap); Space cold-boot re-downloads ~14 GB (accepted: private personal Space; persistent storage = money = operator call); streaming absent (accepted: wrapper limitation, documented).