| --- |
| library_name: mlx |
| license: other |
| base_model: Lightricks/LTX-2.3 |
| tags: |
| - mlx |
| - apple-silicon |
| - safetensors |
| - video-generation |
| - quantized |
| - ram-quantization |
| - mixed-precision |
| pipeline_tag: text-to-video |
| --- |
| |
| # baa-ai/LTX-2.3-MLX-RAM-12GB |
|
|
| Mixed-precision MLX quantization of [Lightricks/LTX-2.3](https://huggingface.co/Lightricks/LTX-2.3), targeting ~12 GB RAM using the [RAM Pipeline](https://github.com/tomkay/RAM) — a sensitivity-driven knapsack allocator that assigns each tensor its optimal bit width independently. |
|
|
| Runs on Apple Silicon (M1 and later). Uses [ltx-2-mlx](https://github.com/dgrauet/ltx-2-mlx) for inference. |
|
|
| ## Model Details |
|
|
| | Property | Value | |
| |---|---| |
| | Base model | Lightricks/LTX-2.3 (22B joint audio-video DiT) | |
| | Architecture | LTX-2.3-22B-AudioVideoTransformer | |
| | Transformer blocks | 48 | |
| | Quantization method | RAM mixed-precision MCKP knapsack | |
| | Sensitivity metric | Kernel CKA (kernel-csk) | |
| | Group size | 64 | |
| | Quantized layers | 1,415 of 1,760 transformer tensors | |
| | Transformer file size | 9.3 GB | |
|
|
| ## Quantization |
|
|
| The transformer uses per-layer mixed precision. Bit widths are assigned per-tensor by solving a Multiple-Choice Knapsack Problem (MCKP) against a budget constraint, using Kernel Centered Kernel Alignment (CKA) as the sensitivity signal. |
|
|
| **Bit distribution (transformer layers):** |
|
|
| | Bits | Layers | % | |
| |---|---|---| |
| | 2 | 29 | 2.1% | |
| | 3 | 225 | 15.9% | |
| | 4 | 348 | 24.6% | |
| | 5 | 631 | 44.6% | |
| | 6 | 125 | 8.8% | |
| | 8 | 57 | 4.0% | |
|
|
| Average: **4.6 bpw** across quantized layers. Highest-sensitivity tensors (first/last layers, K/V projections, vision bridge) are protected at higher bit widths; lower-sensitivity layers are pushed to 2–3 bit. |
|
|
| The connector, VAE, audio VAE, and vocoder are kept at full BF16 precision. |
|
|
| ## Requirements |
|
|
| - macOS 13+ on Apple Silicon (M1 / M2 / M3 / M4) |
| - Python 3.10+ |
| - ~14 GB unified memory recommended (12 GB transformer + overhead) |
|
|
| ```bash |
| pip install mlx mlx-lm flask ltx-pipelines-mlx ltx-core-mlx |
| ``` |
|
|
| ## Quick Start — Web App |
|
|
| The included `webapp.py` provides a browser UI with live generation log, video playback, and job history. To compare against the 24 GB version, download it alongside this model. |
|
|
| ```bash |
| # Download this model |
| git clone https://huggingface.co/baa-ai/LTX-2.3-MLX-RAM-12GB |
| |
| # Optional: also download the 24 GB version for side-by-side comparison |
| git clone https://huggingface.co/baa-ai/LTX-2.3-MLX-RAM-24GB |
| |
| # Run the web app (single model) |
| cd LTX-2.3-MLX-RAM-12GB |
| python webapp.py |
| |
| # Or with comparison enabled |
| python webapp.py --compare-dir ../LTX-2.3-MLX-RAM-24GB |
| |
| # Open http://localhost:7860 |
| ``` |
|
|
| The web app lets you: |
| - Enter a prompt and set resolution, duration, FPS, and seed |
| - Watch the generation log stream live |
| - Play the generated video directly in the browser |
| - Switch between models for comparison (if `--compare-dir` is set) |
| - Browse job history and reload past settings |
|
|
| ## Quick Start — CLI |
|
|
| ```bash |
| python generate.py \ |
| --model-dir /path/to/LTX-2.3-MLX-RAM-12GB \ |
| --prompt "A serene mountain lake at sunrise, mist over calm water, pine trees reflected" \ |
| --height 480 --width 704 --num-frames 65 \ |
| --output output.mp4 |
| ``` |
|
|
| **Common resolution + frame count combinations:** |
|
|
| | Label | Height | Width | Frames | Duration @ 24fps | |
| |---|---|---|---|---| |
| | Tiny (test) | 256 | 256 | 33 | 1.4s | |
| | 480p short | 480 | 704 | 65 | 2.7s | |
| | 480p medium | 480 | 704 | 97 | 4.0s | |
| | 720p short | 720 | 1280 | 65 | 2.7s | |
|
|
| Frame count must follow the formula `32k + 1` (33, 65, 97, 129, …). |
|
|
| ## Files in This Repo |
|
|
| | File | Size | Description | |
| |---|---|---| |
| | `transformer-distilled.safetensors` | 9.3 GB | RAM mixed-precision quantized DiT transformer | |
| | `connector.safetensors` | 5.9 GB | Audio/video embedding connectors (BF16) | |
| | `vae_decoder.safetensors` | 777 MB | Video VAE decoder (BF16) | |
| | `vae_encoder.safetensors` | 608 MB | Video VAE encoder (BF16) | |
| | `audio_vae.safetensors` | 102 MB | Audio VAE (BF16) | |
| | `vocoder.safetensors` | 246 MB | Vocoder + bandwidth extension (BF16) | |
| | `spatial_upscaler_x2_v1_1.safetensors` | 950 MB | Optional 2× spatial upscaler | |
| | `split_model.json` | — | Pipeline model structure descriptor | |
| | `generate.py` | — | Single-file generation script (CLI) | |
| | `webapp.py` | — | Flask web UI for interactive generation | |
|
|
| ## Technical Notes |
|
|
| The [RAM Pipeline](https://github.com/tomkay/RAM) probes each tensor's sensitivity to quantization at 2/3/4/5/6/8 bits using Kernel CKA on calibration activations, builds a per-tensor sensitivity manifest, then solves a MCKP knapsack to find the Pareto-optimal bit assignment for the target memory budget. |
|
|
| Key invariants applied during allocation: |
| - **2-bit veto**: tensors with cosine divergence > 0.0001 at 2-bit are never assigned 2 bits |
| - **Saturation gate**: tensors whose quality plateaus early receive no further bit upgrades |
| - **Architectural priors**: first/last layer (3×), global-attention K/V (3×), vision bridge (2×) — these tensors resist low-bit allocation |
|
|
| Inference uses a custom `apply_mixed_precision_quantization` that detects each layer's bit width from its packed weight shape and calls `mlx.nn.quantize` once per unique bit width, enabling variable-precision loading into the `ltx-2-mlx` pipeline without any format modifications. |
|
|
| ## Attribution |
|
|
| - Base model: [Lightricks/LTX-2.3](https://huggingface.co/Lightricks/LTX-2.3) — see their license |
| - MLX inference: [dgrauet/ltx-2-mlx](https://github.com/dgrauet/ltx-2-mlx) |
| - Reference MLX weights: [dgrauet/ltx-2.3-mlx-q8](https://huggingface.co/dgrauet/ltx-2.3-mlx-q8) |
|
|
