docs/SCALING.md

# Scaling MOSAIC correctly

MOSAIC is an **ordered streaming recursion**. The state at time `t+1` depends on the state at `t`. Therefore this benchmark does **not** claim that a single temporal stream can be data-parallelized by splitting adjacent samples across GPUs and averaging states afterward.

The supported scale-out unit is an **independent ordered stream**: independent seeds, hyperparameter trials, sensors, assets, users, trajectories, or dataset shards whose ordering is meaningful within each shard. `benchmarks/run_scaling.py` vectorizes several independent streams per GPU and `torchrun` assigns independent streams to each GPU. Only final scalar metrics are gathered.

For one genuinely huge chronological stream, use `benchmarks/run_memmap.py`; it memory-maps a `.npy` matrix and transfers one sample at a time to a GPU. Future distributed work should first derive and validate a mathematically sound state-merge operator before claiming within-stream data parallelism.

## Scale ladder

1. CPU reference: run the original NumPy tests and smoke benchmarks.
2. DGX Spark: run `./scripts/run_dgx_spark_smoke.sh`.
3. Single GPU sweep: increase `d`, `rank`, `streams-per-gpu`, and `steps`.
4. Multi-GPU node: `NPROC=8 ./scripts/run_multi_gpu.sh --d 8192 --rank 64 --streams-per-gpu 16 --steps 100000`.
5. Multi-node: use the cluster's normal `torchrun` rendezvous parameters and invoke `benchmarks/run_scaling.py` directly.

Report GPU model, PyTorch/CUDA versions, world size, dtype, `d`, `r`, streams/GPU, steps, samples/s, distortion, orthogonality error, minimum SPD eigenvalue, and inverse-consistency error.