Upload folder using huggingface_hub

README.md

@@ -13,40 +13,42 @@ license: apache-2.0
 
 # jmportilla/ais-major-scratch-10k
 
-`jmportilla/ais-major-scratch-10k` is an experimental AIS-style motion in-betweening checkpoint for the Animaj/MIB rig-controller dataset format. It is trained from scratch with a modified AIS-BiLSTM architecture intended to improve timing awareness and reduce noisy per-controller gating behavior.
+`jmportilla/ais-major-scratch-10k` is an experimental AIS-style motion in-betweening checkpoint for the Animaj/MIB rig-controller dataset format. It was trained from scratch with a modified AIS-BiLSTM architecture designed to improve timing awareness and reduce noisy per-controller gating behavior.
 
 This is **not** a generic Hugging Face Transformers model. It is a PyTorch checkpoint intended to be loaded through the `motionib` code in this project.
 
-## What This Model Does
+## Model Purpose
 
 The model predicts dense animation controller curves from sparse input keyposes. Given a sequence where only selected block/keypose frames are visible, the model fills the missing in-between frames for a 596-dimensional rig-controller vector.
 
-The original AIS-BiLSTM baseline blends two paths:
+The released AIS-BiLSTM baseline blends:
 
 - an interpolation path between surrounding keyposes
 - a synthesis path predicted from a Bi-LSTM hidden sequence
 - a learned beta gate that blends interpolation and synthesis per controller dimension
 
-This model keeps that core idea but changes how the model reasons about time and gate structure.
+This model keeps that core AIS idea but adds stronger timing and spatial gate structure.
 
-## How It Improves On The Base Architecture
+## Architectural Improvements Over The Base AIS-BiLSTM
 
-Compared with `AnimajSAS/AIS_BI_LSTM_v0`, this checkpoint uses `model.type = improved_ais_bilstm`.
+Compared with `AnimajSAS/AIS_BI_LSTM_v0`, this checkpoint uses:
 
-Main architectural changes:
+```text
+model.type = improved_ais_bilstm
+```
+
+Main changes:
 
-- **Temporal progress conditioning**
-  The model receives explicit timing features for every frame:
+- **Explicit temporal progress conditioning**
+  Each frame receives four deterministic timing features:
   `phase`, `segment_len`, `dist_prev`, and `dist_next`.
   These tell the prediction heads where the frame sits between the previous and next observed keypose.
 
 - **Grouped beta gating**
-  The original AIS head predicts a separate beta gate for each of the 596 controller dimensions. This variant predicts `64` beta groups and expands them over the controller vector. The goal is to reduce noisy per-dimension gate switching and encourage related controller dimensions to make more coherent interpolation-vs-synthesis decisions.
+  The released AIS head predicts a separate beta gate for each of the 596 controller dimensions. This variant predicts `64` beta groups and expands them across the controller vector. The goal is to reduce noisy per-dimension gate switching and encourage related controller dimensions to choose interpolation-vs-synthesis behavior more coherently.
 
-- **Same AIS interpolation/synthesis structure**
-  The model still predicts:
-  `alpha`, `p_interp`, `p_synth`, `beta`, and final `pred`.
-  This keeps it close to the AIS design while adding stronger inductive bias.
+- **AIS structure retained**
+  The model still predicts `alpha`, `p_interp`, `p_synth`, `beta`, and final `pred`, so it remains close to the AIS design while adding improved inductive bias.
 
 Training objective changes:
 
@@ -54,7 +56,7 @@ Training objective changes:
 - velocity loss: `0.05`
 - acceleration loss: `0.01`
 - spectral loss: `0.001`
-- beta/alpha gate total-variation regularization: `0.001`
+- gate total-variation regularization: `0.001`
 - keypose reconstruction loss: `10.0`
 
 ## Architecture Configuration
@@ -72,7 +74,7 @@ model:
   beta_groups: 64
 ```
 
-Training run:
+Training configuration:
 
 ```yaml
 max_steps: 10000
@@ -89,37 +91,46 @@ The source checkpoint packaged here came from:
 runs/ais_major_scratch_10k/checkpoints/best.pt
 ```
 
-## Benchmark Summary
+## Full Benchmark Results
 
-The following results come from the local `motionib.official_benchmark` adapter on 20 clips per test set. Lower is better for all metrics.
+The table below comes from the local `motionib.official_benchmark` adapter using the full available public benchmark splits:
 
-Important caveat: this adapter is designed to closely match the public paper protocol, but the upstream `animaj-lab/mib-ais` Lightning evaluator remains the final oracle for paper-level claims.
+- `held_out_algorithmic`: 201 clips
+- `held_out_random`: 201 clips, 90% random masking
+- `production`: 56 clips
 
-### Main Comparison
+Lower is better for all listed metrics.
+
+Important caveat: this adapter is designed to closely match the public protocol, but the upstream `animaj-lab/mib-ais` Lightning evaluator remains the final oracle for paper-level claims.
 
 | Test set | Model | Clips | Shifted distance | NPSS | Missing L1 | Full L1 |
 | --- | --- | ---: | ---: | ---: | ---: | ---: |
-| held_out_algorithmic | base | 20 | 0.563570 | 1.616445 | 0.025400 | 0.022524 |
-| held_out_algorithmic | improved fine-tune | 20 | 0.312933 | 0.497523 | 0.012193 | 0.010505 |
-| held_out_algorithmic | major scratch 10k | 20 | **0.303092** | **0.494308** | 0.013189 | 0.011197 |
-| held_out_random | base | 20 | 0.332714 | 0.962153 | 0.019904 | 0.017668 |
-| held_out_random | improved fine-tune | 20 | 0.299334 | **0.906343** | 0.016428 | 0.014603 |
-| held_out_random | major scratch 10k | 20 | **0.296655** | 0.906478 | **0.016400** | **0.014578** |
-| production | base | 20 | 0.320340 | 1.064096 | 0.027204 | 0.020606 |
-| production | improved fine-tune | 20 | 0.258483 | **0.999074** | 0.019686 | 0.014293 |
-| production | major scratch 10k | 20 | **0.246954** | 1.021747 | **0.018992** | **0.013732** |
+| held_out_algorithmic | base | 201 | 0.465652 | 2.182326 | 0.024197 | 0.021374 |
+| held_out_algorithmic | improved fine-tune | 201 | 0.221540 | **0.897697** | **0.011201** | **0.009611** |
+| held_out_algorithmic | major scratch 10k | 201 | **0.211539** | 0.901307 | 0.012029 | 0.010211 |
+| held_out_random | base | 201 | 0.367792 | 1.359091 | 0.021008 | 0.018642 |
+| held_out_random | improved fine-tune | 201 | 0.310235 | 1.326799 | 0.017078 | 0.015172 |
+| held_out_random | major scratch 10k | 201 | **0.293350** | **1.266688** | **0.016796** | **0.014917** |
+| production | base | 56 | 0.257322 | 0.671743 | 0.016206 | 0.012205 |
+| production | improved fine-tune | 56 | 0.165956 | **0.485759** | 0.011167 | 0.008027 |
+| production | major scratch 10k | 56 | **0.156249** | 0.493669 | **0.010951** | **0.007838** |
 
-### Improvement Over Base
+## Improvement Over The Released Base Model
 
-For this sampled benchmark, `major_scratch_10k` improves over the released base checkpoint by:
+For the full available benchmark splits, `major_scratch_10k` improves over the released base checkpoint by:
 
-| Test set | Shifted distance change | NPSS change | Missing L1 change | Full L1 change |
+| Test set | Shifted distance | NPSS | Missing L1 | Full L1 |
 | --- | ---: | ---: | ---: | ---: |
-| held_out_algorithmic | -46.2% | -69.4% | -48.1% | -50.3% |
-| held_out_random | -10.8% | -5.8% | -17.6% | -17.5% |
-| production | -22.9% | -4.0% | -30.2% | -33.4% |
+| held_out_algorithmic | -54.6% | -58.7% | -50.3% | -52.2% |
+| held_out_random | -20.2% | -6.8% | -20.1% | -20.0% |
+| production | -39.3% | -26.5% | -32.4% | -35.8% |
+
+The scratch model is strongest on **shifted distance**, the timing-tolerant pose error metric used as the main paper-style comparison signal. On the full benchmark:
 
-The strongest gains are on **shifted distance**, which is the most relevant paper-style metric for timing-tolerant in-betweening quality. On production clips, the scratch model improves shifted distance and L1 substantially, while the fine-tuned model remains slightly better on NPSS in this 20-clip sample.
+- `major_scratch_10k` has the best shifted distance on all three test sets.
+- `major_scratch_10k` has the best NPSS on `held_out_random`.
+- The fine-tuned improved model has slightly better NPSS on `held_out_algorithmic` and `production`.
+- `major_scratch_10k` has the best missing-frame L1 on `held_out_random` and `production`.
 
 ## Metric Definitions
 
@@ -144,7 +155,7 @@ The strongest gains are on **shifted distance**, which is the most relevant pape
   Full training configuration for this checkpoint.
 
 - `official_protocol_summary_metrics.csv`
-  Aggregate benchmark metrics.
+  Aggregate benchmark metrics for the full available benchmark splits.
 
 - `official_protocol_clip_metrics.csv`
   Per-clip benchmark metrics.
@@ -166,7 +177,7 @@ model.eval()
 
 ## Expected Input Format
 
-This model expects the same tensor layout used by the `motionib` training code:
+This model expects the tensor layout used by the `motionib` training code:
 
 - `input_seq`: shape `[batch, time, 597]`
   596 masked controller dimensions plus one missing-frame mask channel.
@@ -210,27 +221,22 @@ From the project repository:
 python -m motionib.official_benchmark \
   --config configs/train/ais_repro.yaml \
   --checkpoint base=artifacts/hf_model/model.safetensors \
+  --checkpoint improved=runs/ais_improved_from_base_10k/checkpoints/best.pt \
   --checkpoint major_scratch=model.pt \
   --test-set all \
-  --max-clips 20 \
+  --max-clips 0 \
   --device auto \
-  --output-dir reports/official_protocol_compare
-```
-
-For a full run, use:
-
-```bash
---max-clips 0
+  --output-dir reports/official_protocol_compare_full
 ```
 
 ## Limitations
 
 - This is an experimental research checkpoint, not a production-ready animation system.
 - It was trained for only `10,000` steps from scratch. Longer scratch training may improve results.
-- The benchmark table uses a 20-clip sample per test set.
-- The official upstream Lightning evaluator should be used before making publication-level claims.
-- The model is tied to the released Animaj/MIB controller-vector format and does not generalize to arbitrary rigs without retraining/adaptation.
+- The official upstream Lightning evaluator should still be used before making publication-level claims.
+- The model is tied to the released Animaj/MIB controller-vector format and does not generalize to arbitrary rigs without retraining or adaptation.
+- The benchmark compares controller-vector prediction quality, not rendered character quality or animator retake time.
 
-## Citation / Related Work
+## Related Work
 
 This model is based on the AIS motion in-betweening setup and the public `AnimajSAS/AIS_BI_LSTM_v0` release. The architecture here is a local experimental variant that keeps the AIS interpolation/synthesis idea while adding temporal conditioning and grouped gate structure.

official_protocol_summary_metrics.csv

@@ -1,10 +1,10 @@
 test_set,model,clips,frames,mean_mask_ratio,shifted_distance,shifted_distance_median,shifted_distance_std,npss,npss_median,npss_std,full_l1,full_l1_median,full_l1_std,missing_l1,missing_l1_median,missing_l1_std
-held_out_algorithmic,base,20,2761,0.9005592126546578,0.5635700441298277,0.3973293047348959,0.5789553313958574,1.6164449568404926,0.7376146166286656,2.2313882746102753,0.022524041222641244,0.013523494824767113,0.024741861034447702,0.02539981409208849,0.01724134013056755,0.02791094618888122
-held_out_algorithmic,improved,20,2761,0.9005592126546578,0.31293257113134654,0.10416547441939256,0.49493006900668024,0.4975234434345155,0.19858078204558233,0.5986477767547608,0.010504539695102722,0.004716081544756889,0.01642234100953195,0.012192885919648688,0.005016518756747246,0.018367657231855836
-held_out_algorithmic,major_scratch,20,2761,0.9005592126546578,0.3030916049633498,0.10337592970656015,0.49330662595866864,0.4943081554823948,0.2142453324106498,0.5705970864084018,0.011196957573702093,0.004716202616691589,0.016538662774409327,0.013189485580369364,0.005016664741560817,0.018767806271011808
-held_out_random,base,20,2761,0.8824006015763473,0.332714187904665,0.16745417321036601,0.32247388332645976,0.9621530503120319,0.5530451889773913,1.0722316053179077,0.017667744151549414,0.008342467714101076,0.01874697560355533,0.019904192874673755,0.009538945741951466,0.021340581803422742
-held_out_random,improved,20,2761,0.8824006015763473,0.2993338749895051,0.14788039218607119,0.2864001494693676,0.9063433045806659,0.5543289849114983,0.9381369842839581,0.014602789636410308,0.005932004889473319,0.01630367203772323,0.016428220353554936,0.006658609490841627,0.018502989742684876
-held_out_random,major_scratch,20,2761,0.8824006015763473,0.2966546864184923,0.14784876577822054,0.2824031458742327,0.9064781272101436,0.5700359829499448,0.9387977765841757,0.014577854810340796,0.005925189005210996,0.016287218175314574,0.016399642699980177,0.006650809198617935,0.0184832842062924
-production,base,20,2012,0.7261683858264012,0.3203401765177655,0.22620787763375994,0.27399511657853765,1.0640955803737175,0.29989785372904165,1.5086611587002994,0.020606417080853135,0.013092817272990942,0.02319715492552421,0.02720442715799436,0.021532909013330936,0.028064481908314405
-production,improved,20,2012,0.7261683858264012,0.2584831672478787,0.17314157383655832,0.28787366397231107,0.9990735727197935,0.16207499589778165,1.4526559209453267,0.014292532837134785,0.009731731843203306,0.017846792527678713,0.01968566266295966,0.014810709282755852,0.021957319492503215
-production,major_scratch,20,2012,0.7261683858264012,0.2469537931085076,0.17087776462414056,0.28174928594805565,1.0217469455666672,0.1621899613085547,1.5094304201873916,0.013732498313765973,0.009767297189682722,0.01698585724705308,0.0189922235236736,0.013348973356187344,0.020972357275608075
+held_out_algorithmic,base,201,36540,0.892028525146237,0.4656521011399281,0.3791014449349765,0.3931737711154798,2.182326368485508,0.7365788845032173,3.6982533269284126,0.02137352209573334,0.014951972290873528,0.021946424559886976,0.02419732287988389,0.01711910031735897,0.025027325669826395
+held_out_algorithmic,improved,201,36540,0.892028525146237,0.22154000033319526,0.1473078812118125,0.2394588125189726,0.8976974981173184,0.34226874801054263,2.897946425232194,0.009610943387250794,0.006420005578547716,0.012836809854484668,0.011201015984054843,0.007405770476907492,0.01526556534868378
+held_out_algorithmic,major_scratch,201,36540,0.892028525146237,0.211539394814392,0.14533001110695376,0.2244646735629202,0.9013070676510134,0.35591952813450073,2.8483004516204717,0.010211404527409236,0.006433078087866306,0.01288174601664427,0.012029349021813416,0.007274171803146601,0.015535323926010824
+held_out_random,base,201,36540,0.8838676223733377,0.3677920032213693,0.22544339724949428,0.5257536387901985,1.3590905637842359,0.6595144758917957,1.9684007231330012,0.018641641307465817,0.012350904755294323,0.025518993097248317,0.021007639195958838,0.0138154411688447,0.02896018894485555
+held_out_random,improved,201,36540,0.8838676223733377,0.31023521993764497,0.18431830354260675,0.3688995320175478,1.326798513182525,0.7495140768163046,1.9821226153940659,0.015172160252748753,0.010188964195549488,0.01753981186045945,0.01707846266567934,0.011620927602052689,0.019821468952575746
+held_out_random,major_scratch,201,36540,0.8838676223733377,0.29334952945146275,0.18911407124251128,0.315516622310533,1.2666883682965036,0.746628022353424,1.682939913115968,0.014917402200206478,0.010201090015470982,0.01586107673236794,0.016796488320814935,0.011706330813467503,0.01790831811317601
+production,base,56,6654,0.7455420896844516,0.25732181862747094,0.21369302312533062,0.2380894102099856,0.6717432309989391,0.21827519930314782,1.1608108462091937,0.012205303612712928,0.0069318481255322695,0.016240797811617465,0.01620627434736629,0.00917418161407113,0.019941806284414724
+production,improved,56,6654,0.7455420896844516,0.16595623373969537,0.11044380088092802,0.19826644216103612,0.485758588189008,0.14760217784959556,0.965039190147313,0.008026533659371385,0.005082797259092331,0.011989678573781458,0.011166616229469841,0.006560873007401824,0.01518657816613101
+production,major_scratch,56,6654,0.7455420896844516,0.1562489603380366,0.094447316165842,0.1935570872599423,0.493668955278865,0.1476071641119691,0.9995527694713989,0.007838312734163213,0.004962481092661619,0.011422469790005494,0.010951255138414646,0.006517714820802212,0.014543937000781687