Upload folder using huggingface_hub

README.md

@@ -0,0 +1,236 @@
+---
+library_name: pytorch
+tags:
+  - motion-inbetweening
+  - animation
+  - rig-controllers
+  - bilstm
+  - ais
+  - pytorch
+base_model: AnimajSAS/AIS_BI_LSTM_v0
+license: apache-2.0
+---
+
+# jmportilla/ais-major-scratch-10k
+
+`jmportilla/ais-major-scratch-10k` is an experimental AIS-style motion in-betweening checkpoint for the Animaj/MIB rig-controller dataset format. It is trained from scratch with a modified AIS-BiLSTM architecture intended to improve timing awareness and reduce noisy per-controller gating behavior.
+
+This is **not** a generic Hugging Face Transformers model. It is a PyTorch checkpoint intended to be loaded through the `motionib` code in this project.
+
+## What This Model Does
+
+The model predicts dense animation controller curves from sparse input keyposes. Given a sequence where only selected block/keypose frames are visible, the model fills the missing in-between frames for a 596-dimensional rig-controller vector.
+
+The original AIS-BiLSTM baseline blends two paths:
+
+- an interpolation path between surrounding keyposes
+- a synthesis path predicted from a Bi-LSTM hidden sequence
+- a learned beta gate that blends interpolation and synthesis per controller dimension
+
+This model keeps that core idea but changes how the model reasons about time and gate structure.
+
+## How It Improves On The Base Architecture
+
+Compared with `AnimajSAS/AIS_BI_LSTM_v0`, this checkpoint uses `model.type = improved_ais_bilstm`.
+
+Main architectural changes:
+
+- **Temporal progress conditioning**
+  The model receives explicit timing features for every frame:
+  `phase`, `segment_len`, `dist_prev`, and `dist_next`.
+  These tell the prediction heads where the frame sits between the previous and next observed keypose.
+
+- **Grouped beta gating**
+  The original AIS head predicts a separate beta gate for each of the 596 controller dimensions. This variant predicts `64` beta groups and expands them over the controller vector. The goal is to reduce noisy per-dimension gate switching and encourage related controller dimensions to make more coherent interpolation-vs-synthesis decisions.
+
+- **Same AIS interpolation/synthesis structure**
+  The model still predicts:
+  `alpha`, `p_interp`, `p_synth`, `beta`, and final `pred`.
+  This keeps it close to the AIS design while adding stronger inductive bias.
+
+Training objective changes:
+
+- weighted L1 reconstruction
+- velocity loss: `0.05`
+- acceleration loss: `0.01`
+- spectral loss: `0.001`
+- beta/alpha gate total-variation regularization: `0.001`
+- keypose reconstruction loss: `10.0`
+
+## Architecture Configuration
+
+```yaml
+model:
+  type: improved_ais_bilstm
+  pose_dim: 596
+  input_dim: 597
+  temporal_dim: 4
+  hidden_size: 512
+  num_layers: 2
+  dropout: 0.3003
+  synthesis_hidden: 1024
+  beta_groups: 64
+```
+
+Training run:
+
+```yaml
+max_steps: 10000
+batch_size: 8
+gradient_accumulation_steps: 8
+optimizer: AdamW
+lr: 0.0001
+weight_decay: 0.001
+```
+
+The source checkpoint packaged here came from:
+
+```text
+runs/ais_major_scratch_10k/checkpoints/best.pt
+```
+
+## Benchmark Summary
+
+The following results come from the local `motionib.official_benchmark` adapter on 20 clips per test set. Lower is better for all metrics.
+
+Important caveat: this adapter is designed to closely match the public paper protocol, but the upstream `animaj-lab/mib-ais` Lightning evaluator remains the final oracle for paper-level claims.
+
+### Main Comparison
+
+| Test set | Model | Clips | Shifted distance | NPSS | Missing L1 | Full L1 |
+| --- | --- | ---: | ---: | ---: | ---: | ---: |
+| held_out_algorithmic | base | 20 | 0.563570 | 1.616445 | 0.025400 | 0.022524 |
+| held_out_algorithmic | improved fine-tune | 20 | 0.312933 | 0.497523 | 0.012193 | 0.010505 |
+| held_out_algorithmic | major scratch 10k | 20 | **0.303092** | **0.494308** | 0.013189 | 0.011197 |
+| held_out_random | base | 20 | 0.332714 | 0.962153 | 0.019904 | 0.017668 |
+| held_out_random | improved fine-tune | 20 | 0.299334 | **0.906343** | 0.016428 | 0.014603 |
+| held_out_random | major scratch 10k | 20 | **0.296655** | 0.906478 | **0.016400** | **0.014578** |
+| production | base | 20 | 0.320340 | 1.064096 | 0.027204 | 0.020606 |
+| production | improved fine-tune | 20 | 0.258483 | **0.999074** | 0.019686 | 0.014293 |
+| production | major scratch 10k | 20 | **0.246954** | 1.021747 | **0.018992** | **0.013732** |
+
+### Improvement Over Base
+
+For this sampled benchmark, `major_scratch_10k` improves over the released base checkpoint by:
+
+| Test set | Shifted distance change | NPSS change | Missing L1 change | Full L1 change |
+| --- | ---: | ---: | ---: | ---: |
+| held_out_algorithmic | -46.2% | -69.4% | -48.1% | -50.3% |
+| held_out_random | -10.8% | -5.8% | -17.6% | -17.5% |
+| production | -22.9% | -4.0% | -30.2% | -33.4% |
+
+The strongest gains are on **shifted distance**, which is the most relevant paper-style metric for timing-tolerant in-betweening quality. On production clips, the scratch model improves shifted distance and L1 substantially, while the fine-tuned model remains slightly better on NPSS in this 20-clip sample.
+
+## Metric Definitions
+
+- **Shifted distance**
+  Paper-style shifted temporal L1. It compares each ground-truth segment to nearby predicted segments, reducing over-penalization from small timing shifts. Lower is better.
+
+- **NPSS**
+  Normalized Power Spectrum Similarity. It compares temporal frequency/rhythm distribution between prediction and ground truth. Lower is better.
+
+- **Missing L1**
+  Mean absolute controller error on masked/in-between frames only. This isolates the frames the model actually has to predict.
+
+- **Full L1**
+  Mean absolute controller error over the full sequence, including observed keypose frames.
+
+## Files
+
+- `model.pt`
+  PyTorch training checkpoint.
+
+- `training_config.yaml`
+  Full training configuration for this checkpoint.
+
+- `official_protocol_summary_metrics.csv`
+  Aggregate benchmark metrics.
+
+- `official_protocol_clip_metrics.csv`
+  Per-clip benchmark metrics.
+
+- `README.md`
+  This model card.
+
+## Loading
+
+```python
+import torch
+from motionib.models.improved_ais import build_improved_ais_bilstm
+
+checkpoint = torch.load("model.pt", map_location="cpu", weights_only=False)
+model = build_improved_ais_bilstm(checkpoint["config"]["model"], device="cpu")
+model.load_state_dict(checkpoint["model"])
+model.eval()
+```
+
+## Expected Input Format
+
+This model expects the same tensor layout used by the `motionib` training code:
+
+- `input_seq`: shape `[batch, time, 597]`
+  596 masked controller dimensions plus one missing-frame mask channel.
+
+- `prev_pose`: shape `[batch, time, 596]`
+  Previous observed keypose pose for each frame.
+
+- `next_pose`: shape `[batch, time, 596]`
+  Next observed keypose pose for each frame.
+
+- `temporal_features`: shape `[batch, time, 4]`
+  `phase`, `segment_len`, `dist_prev`, `dist_next`.
+
+- `observed_mask`: optional shape `[batch, time]`
+  Used to hard-copy observed keyposes at inference.
+
+- `keypose_values`: optional shape `[batch, time, 596]`
+  Ground-truth values at observed keypose frames.
+
+Example forward pass:
+
+```python
+with torch.no_grad():
+    output = model(
+        input_seq,
+        prev_pose,
+        next_pose,
+        temporal_features=temporal_features,
+        observed_mask=observed_mask,
+        keypose_values=keypose_values,
+    )
+
+predicted_sequence = output["pred"]
+```
+
+## Reproducing The Benchmark
+
+From the project repository:
+
+```bash
+python -m motionib.official_benchmark \
+  --config configs/train/ais_repro.yaml \
+  --checkpoint base=artifacts/hf_model/model.safetensors \
+  --checkpoint major_scratch=model.pt \
+  --test-set all \
+  --max-clips 20 \
+  --device auto \
+  --output-dir reports/official_protocol_compare
+```
+
+For a full run, use:
+
+```bash
+--max-clips 0
+```
+
+## Limitations
+
+- This is an experimental research checkpoint, not a production-ready animation system.
+- It was trained for only `10,000` steps from scratch. Longer scratch training may improve results.
+- The benchmark table uses a 20-clip sample per test set.
+- The official upstream Lightning evaluator should be used before making publication-level claims.
+- The model is tied to the released Animaj/MIB controller-vector format and does not generalize to arbitrary rigs without retraining/adaptation.
+
+## Citation / Related Work
+
+This model is based on the AIS motion in-betweening setup and the public `AnimajSAS/AIS_BI_LSTM_v0` release. The architecture here is a local experimental variant that keeps the AIS interpolation/synthesis idea while adding temporal conditioning and grouped gate structure.

model.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:562c5340e42e52f5b16af0345323b3121d8050483a08eea2ad513b20855b07fa
+size 158375923

official_protocol_clip_metrics.csv

@@ -0,0 +1,181 @@
+test_set,model,clip_id,episode_id,scene_id,frames,unmasked_frames,missing_frames,mask_ratio,shifted_distance,npss,full_l1,missing_l1
+held_out_algorithmic,base,0,ep0001,scn0029,132,4,128,0.9696969696969697,0.03286344965280437,0.6189754786257916,0.004636986181139946,0.00478189205750823
+held_out_algorithmic,improved,0,ep0001,scn0029,132,4,128,0.9696969696969697,0.03316539235468676,1.054197425104389,0.004271413665264845,0.004404895938932896
+held_out_algorithmic,major_scratch,0,ep0001,scn0029,132,4,128,0.9696969696969697,0.033088874905856686,1.0456638525449964,0.004273704253137112,0.0044072577729821205
+held_out_algorithmic,base,1,ep0001,scn0034,102,6,96,0.9411764705882353,0.12578869034659187,0.20736011101742116,0.0071944547817111015,0.007644108030945063
+held_out_algorithmic,improved,1,ep0001,scn0034,102,6,96,0.9411764705882353,0.08925839274559381,0.047372724699636416,0.002271684119477868,0.002413664013147354
+held_out_algorithmic,major_scratch,1,ep0001,scn0034,102,6,96,0.9411764705882353,0.08927991536428344,0.0473899158526196,0.002272108569741249,0.0024141157045960426
+held_out_algorithmic,base,2,ep0002,scn0010b,105,9,96,0.9142857142857143,0.12353035411797464,0.09557619797304123,0.004367070738226175,0.004776483867317438
+held_out_algorithmic,improved,2,ep0002,scn0010b,105,9,96,0.9142857142857143,0.09005952905863523,0.041905568286094985,0.0018843567231670022,0.0020610152278095484
+held_out_algorithmic,major_scratch,2,ep0002,scn0010b,105,9,96,0.9142857142857143,0.08989156175500414,0.04250921714224456,0.0018754878547042608,0.002051314804702997
+held_out_algorithmic,base,3,ep0002,scn0011,242,10,232,0.9586776859504132,0.18437728214263915,0.7179870073625015,0.009074000641703606,0.009465120732784271
+held_out_algorithmic,improved,3,ep0002,scn0011,242,10,232,0.9586776859504132,0.03087509088218212,0.1584002987476757,0.001272849622182548,0.001327713718637824
+held_out_algorithmic,major_scratch,3,ep0002,scn0011,242,10,232,0.9586776859504132,0.030901210165582598,0.15829018961847036,0.001272978144697845,0.0013278478290885687
+held_out_algorithmic,base,4,ep0003,scn0049,42,14,28,0.6666666666666666,0.5908028273670761,0.17048635313186036,0.009024176746606827,0.013536266051232815
+held_out_algorithmic,improved,4,ep0003,scn0049,42,14,28,0.6666666666666666,0.56173927254147,0.19705801534524786,0.008375207893550396,0.012562813237309456
+held_out_algorithmic,major_scratch,4,ep0003,scn0049,42,14,28,0.6666666666666666,0.7724806975435328,0.7248868493423034,0.012235715053975582,0.018353572115302086
+held_out_algorithmic,base,5,ep0005,scn0044,72,10,62,0.8611111111111112,0.24370090770535172,0.4731558358008925,0.014951972290873528,0.01736357994377613
+held_out_algorithmic,improved,5,ep0005,scn0044,72,10,62,0.8611111111111112,0.0852576931938529,0.2252957812725467,0.008805965073406696,0.01022628229111433
+held_out_algorithmic,major_scratch,5,ep0005,scn0044,72,10,62,0.8611111111111112,0.0853188858833164,0.2280588313684925,0.008813237771391869,0.010234727524220943
+held_out_algorithmic,base,6,ep0007,scn0006,93,5,88,0.946236559139785,0.6629841613272826,2.6143315352241694,0.02537209913134575,0.026813695207238197
+held_out_algorithmic,improved,6,ep0007,scn0006,93,5,88,0.946236559139785,0.6130185946822166,1.6549973073177955,0.00986396986991167,0.010424423962831497
+held_out_algorithmic,major_scratch,6,ep0007,scn0006,93,5,88,0.946236559139785,0.6134727721412977,1.6508393742772325,0.009863358922302723,0.01042377669364214
+held_out_algorithmic,base,7,ep0008,scn0008,56,4,52,0.9285714285714286,2.558446021943257,2.346930371585338,0.06996681541204453,0.0753488764166832
+held_out_algorithmic,improved,7,ep0008,scn0008,56,4,52,0.9285714285714286,2.2693369152217078,2.2664621649969585,0.07226374745368958,0.0778224915266037
+held_out_algorithmic,major_scratch,7,ep0008,scn0008,56,4,52,0.9285714285714286,2.2691832667794722,2.2503575192484595,0.0720587894320488,0.07760177552700043
+held_out_algorithmic,base,8,ep0009,scn0019,249,13,236,0.9477911646586346,0.4105429514669455,4.800654393933147,0.028508519753813744,0.030078906565904617
+held_out_algorithmic,improved,8,ep0009,scn0019,249,13,236,0.9477911646586346,0.15880705542289295,0.6267548306122971,0.005241209175437689,0.0055299196392297745
+held_out_algorithmic,major_scratch,8,ep0009,scn0019,249,13,236,0.9477911646586346,0.15877368427239932,0.6263470906720456,0.005241935607045889,0.005530687049031258
+held_out_algorithmic,base,9,ep0010,scn0005,197,8,189,0.9593908629441624,0.43287690945446783,0.7572422258948296,0.021879514679312706,0.022805631160736084
+held_out_algorithmic,improved,9,ep0010,scn0005,197,8,189,0.9593908629441624,0.05788674656980731,0.03822354942577832,0.0015803503338247538,0.0016472435090690851
+held_out_algorithmic,major_scratch,9,ep0010,scn0005,197,8,189,0.9593908629441624,0.057908166459133034,0.038127873510103764,0.0015800819965079427,0.001646963763050735
+held_out_algorithmic,base,10,ep0010,scn0029,139,6,133,0.9568345323741008,0.32226105628313717,0.37359936340634936,0.009100641123950481,0.009511196985840797
+held_out_algorithmic,improved,10,ep0010,scn0029,139,6,133,0.9568345323741008,0.18470735212306041,0.1654620500644707,0.004767487291246653,0.004982561804354191
+held_out_algorithmic,major_scratch,10,ep0010,scn0029,139,6,133,0.9568345323741008,0.18472125771996026,0.16529708607719862,0.004766704980283976,0.004981744568794966
+held_out_algorithmic,base,11,ep0011,scn0005,138,5,133,0.9637681159420289,0.08176727244194518,0.023180581032958125,0.0010145026026293635,0.0010526417754590511
+held_out_algorithmic,improved,11,ep0011,scn0005,138,5,133,0.9637681159420289,0.10429027158740564,0.01649796850905911,0.00039853615453466773,0.00041351871914230287
+held_out_algorithmic,major_scratch,11,ep0011,scn0005,138,5,133,0.9637681159420289,0.10280184511174546,0.01661230649340926,0.0003915932320524007,0.0004063147644046694
+held_out_algorithmic,base,12,ep0011,scn0027,70,21,49,0.7,1.1154276172766524,1.7317846049768026,0.028200045228004456,0.04028577730059624
+held_out_algorithmic,improved,12,ep0011,scn0027,70,21,49,0.7,0.7696772733431184,1.2498765583906284,0.02458919771015644,0.0351274237036705
+held_out_algorithmic,major_scratch,12,ep0011,scn0027,70,21,49,0.7,0.270425960588991,0.5701260651677527,0.027994710952043533,0.03999244421720505
+held_out_algorithmic,base,13,ep0011,scn0051,281,11,270,0.9608540925266904,0.3791014449349765,2.773493999854869,0.019862845540046692,0.020672069862484932
+held_out_algorithmic,improved,13,ep0011,scn0051,281,11,270,0.9608540925266904,0.0608275464896498,0.1996733151121248,0.0026411516591906548,0.002748754108324647
+held_out_algorithmic,major_scratch,13,ep0011,scn0051,281,11,270,0.9608540925266904,0.06087087927193477,0.20043183345280713,0.0026429560966789722,0.002750632120296359
+held_out_algorithmic,base,14,ep0012,scn0012,56,4,52,0.9285714285714286,0.048860431745134554,0.01884139332409065,0.0010954395402222872,0.0011797042097896338
+held_out_algorithmic,improved,14,ep0012,scn0012,56,4,52,0.9285714285714286,0.046530184303892066,0.04294611216338562,0.0008097777026705444,0.000872068339958787
+held_out_algorithmic,major_scratch,14,ep0012,scn0012,56,4,52,0.9285714285714286,0.046549079747035586,0.04218840741284535,0.0008137770346365869,0.0008763752994127572
+held_out_algorithmic,base,15,ep0013,scn0006,42,6,36,0.8571428571428571,0.9670798623043558,1.325173970752675,0.09597998112440109,0.11197663098573685
+held_out_algorithmic,improved,15,ep0013,scn0006,42,6,36,0.8571428571428571,0.22987017424210257,0.35461180108198775,0.034774430096149445,0.04057016596198082
+held_out_algorithmic,major_scratch,15,ep0013,scn0006,42,6,36,0.8571428571428571,0.22928012194840805,0.35233488607050795,0.03467738255858421,0.040456946939229965
+held_out_algorithmic,base,16,ep0013,scn0030,246,14,232,0.943089430894309,1.3737301171519036,9.86499669879169,0.0641232579946518,0.06799276918172836
+held_out_algorithmic,improved,16,ep0013,scn0030,246,14,232,0.943089430894309,0.2858906880367634,0.7522139155365076,0.011963882483541965,0.012685840018093586
+held_out_algorithmic,major_scratch,16,ep0013,scn0030,246,14,232,0.943089430894309,0.286084248922592,0.7766097401418152,0.012343176640570164,0.013088024221360683
+held_out_algorithmic,base,17,ep0014,scn0013,144,11,133,0.9236111111111112,0.5486563233768239,0.4592648336865353,0.016799960285425186,0.018189430236816406
+held_out_algorithmic,improved,17,ep0014,scn0013,144,11,133,0.9236111111111112,0.10404067725137947,0.19748824897903983,0.004664675798267126,0.005050475709140301
+held_out_algorithmic,major_scratch,17,ep0014,scn0013,144,11,133,0.9236111111111112,0.10395001430137485,0.19657466602080448,0.004665700253099203,0.005051584914326668
+held_out_algorithmic,base,18,ep0014,scn0042,92,27,65,0.7065217391304348,0.6844875435543876,1.1713499040598983,0.012095017358660698,0.01711910031735897
+held_out_algorithmic,improved,18,ep0014,scn0042,92,27,65,0.7065217391304348,0.41044359055594504,0.4672616328475102,0.007933491840958595,0.011228942312300205
+held_out_algorithmic,major_scratch,18,ep0014,scn0042,92,27,65,0.7065217391304348,0.5038531303214722,0.5650409892508473,0.014439095743000507,0.02043687179684639
+held_out_algorithmic,base,19,ep0017,scn0016,263,6,257,0.9771863117870723,0.38411565800284625,1.7845142763749937,0.0072335232980549335,0.007402400951832533
+held_out_algorithmic,improved,19,ep0017,scn0016,263,6,257,0.9771863117870723,0.07296898202056742,0.19376960019717565,0.0017174092354252934,0.0017575046513229609
+held_out_algorithmic,major_scratch,19,ep0017,scn0016,263,6,257,0.9771863117870723,0.07299652606360475,0.1884764159829388,0.0017166563775390387,0.0017567339818924665
+held_out_random,base,0,ep0001,scn0029,132,22,110,0.8333333333333334,0.11837995973856826,0.31456006552125615,0.002530273050069809,0.0030363276600837708
+held_out_random,improved,0,ep0001,scn0029,132,22,110,0.8333333333333334,0.03007269591877335,0.16773771457925443,0.0009666995028965175,0.0011600394500419497
+held_out_random,major_scratch,0,ep0001,scn0029,132,22,110,0.8333333333333334,0.03021920923339693,0.22719903346807147,0.0009225935791619122,0.0011071121552959085
+held_out_random,base,1,ep0001,scn0034,102,15,87,0.8529411764705882,0.024688683515009673,0.045043490606928395,0.002710463711991906,0.003177785314619541
+held_out_random,improved,1,ep0001,scn0034,102,15,87,0.8529411764705882,0.02043995922026427,0.029278163731641493,0.001232622191309929,0.0014451434835791588
+held_out_random,major_scratch,1,ep0001,scn0034,102,15,87,0.8529411764705882,0.020377311499222464,0.02928147537531515,0.0012325998395681381,0.0014451169408857822
+held_out_random,base,2,ep0002,scn0010b,105,13,92,0.8761904761904762,0.1522713850798278,0.09806366001755813,0.0032247186172753572,0.0036803854163736105
+held_out_random,improved,2,ep0002,scn0010b,105,13,92,0.8761904761904762,0.14362870310914927,0.11276498269449498,0.0021422875579446554,0.002445002319291234
+held_out_random,major_scratch,2,ep0002,scn0010b,105,13,92,0.8761904761904762,0.1435992348947209,0.12008417656419945,0.0021380612161010504,0.0024401783011853695
+held_out_random,base,3,ep0002,scn0011,242,22,220,0.9090909090909091,0.03184852902670853,1.2857389392224616,0.008836601860821247,0.00972026214003563
+held_out_random,improved,3,ep0002,scn0011,242,22,220,0.9090909090909091,0.7401043433297909,2.1750528636158544,0.005638482049107552,0.006202330347150564
+held_out_random,major_scratch,3,ep0002,scn0011,242,22,220,0.9090909090909091,0.7402772531513203,2.1735983087104525,0.00563895795494318,0.006202853284776211
+held_out_random,base,4,ep0003,scn0049,42,3,39,0.9285714285714286,0.609465709952421,1.2753628148229483,0.023237397894263268,0.025024889037013054
+held_out_random,improved,4,ep0003,scn0049,42,3,39,0.9285714285714286,0.6082745041958121,1.3059569356936034,0.02280866540968418,0.024563178420066833
+held_out_random,major_scratch,4,ep0003,scn0049,42,3,39,0.9285714285714286,0.6082767220430596,1.0920033261713158,0.022779034450650215,0.02453126758337021
+held_out_random,base,5,ep0005,scn0044,72,7,65,0.9027777777777778,0.2734459905655353,0.5574795767258411,0.011315353214740753,0.012533930130302906
+held_out_random,improved,5,ep0005,scn0044,72,7,65,0.9027777777777778,0.26419794235330124,0.5387492927201225,0.011386370286345482,0.012612595222890377
+held_out_random,major_scratch,5,ep0005,scn0044,72,7,65,0.9027777777777778,0.2641934911151985,0.5688400543827279,0.011385265737771988,0.012611369602382183
+held_out_random,base,6,ep0007,scn0006,93,15,78,0.8387096774193549,0.6211634153183901,0.9633376175720667,0.007848333567380905,0.009357629343867302
+held_out_random,improved,6,ep0007,scn0006,93,15,78,0.8387096774193549,0.6905632383137379,0.9794576385603216,0.008459774777293205,0.01008665468543768
+held_out_random,major_scratch,6,ep0007,scn0006,93,15,78,0.8387096774193549,0.660109143704176,0.9740276193849131,0.008413145318627357,0.010031058453023434
+held_out_random,base,7,ep0008,scn0008,56,6,50,0.8928571428571429,0.49425243561466536,0.5486108012289413,0.01819061115384102,0.02037348598241806
+held_out_random,improved,7,ep0008,scn0008,56,6,50,0.8928571428571429,0.3948406480252743,0.8211810981069969,0.027841884642839432,0.031182911247015
+held_out_random,major_scratch,7,ep0008,scn0008,56,6,50,0.8928571428571429,0.3948851173122724,0.8212131018575842,0.027836909517645836,0.0311773382127285
+held_out_random,base,8,ep0009,scn0019,249,17,232,0.9317269076305221,0.18263696134090424,3.9526577223732122,0.03242313861846924,0.03479897230863571
+held_out_random,improved,8,ep0009,scn0019,249,17,232,0.9317269076305221,0.1521320812629931,3.261115145246889,0.022791706025600433,0.024461787194013596
+held_out_random,major_scratch,8,ep0009,scn0019,249,17,232,0.9317269076305221,0.15209829666172014,3.2356453921462522,0.022787850350141525,0.024457650259137154
+held_out_random,base,9,ep0010,scn0005,197,19,178,0.9035532994923858,0.038602783335982084,0.11303196295118186,0.004128917120397091,0.00456964410841465
+held_out_random,improved,9,ep0010,scn0005,197,19,178,0.9035532994923858,0.04103642952382003,0.3361841743644,0.0034962105564773083,0.003869401291012764
+held_out_random,major_scratch,9,ep0010,scn0005,197,19,178,0.9035532994923858,0.04106436593017555,0.3165920568819174,0.003497623372823,0.003870965214446187
+held_out_random,base,10,ep0010,scn0029,139,21,118,0.8489208633093526,0.12729283344424977,0.14074081110870987,0.004550701007246971,0.005360572133213282
+held_out_random,improved,10,ep0010,scn0029,139,21,118,0.8489208633093526,0.11968488296894711,0.13878818767696033,0.004008193966001272,0.004721516277641058
+held_out_random,major_scratch,10,ep0010,scn0029,139,21,118,0.8489208633093526,0.11968017415485427,0.13865909594204304,0.004009322728961706,0.0047228457406163216
+held_out_random,base,11,ep0011,scn0005,138,20,118,0.855072463768116,0.03658159331490214,0.011178852314861654,0.00027315004263073206,0.00031944666989147663
+held_out_random,improved,11,ep0011,scn0005,138,20,118,0.855072463768116,0.03220094434725933,0.02190016565483427,0.00035725426278077066,0.00041780585888773203
+held_out_random,major_scratch,11,ep0011,scn0005,138,20,118,0.855072463768116,0.032128553407696575,0.021913856590827115,0.0003587798273656517,0.0004195899819023907
+held_out_random,base,12,ep0011,scn0027,70,8,62,0.8857142857142857,1.0476753868554767,1.2920496478923302,0.03423325717449188,0.038650453090667725
+held_out_random,improved,12,ep0011,scn0027,70,8,62,0.8857142857142857,0.8872213661670685,1.159652342509766,0.031764086335897446,0.03586268052458763
+held_out_random,major_scratch,12,ep0011,scn0027,70,8,62,0.8857142857142857,0.8721251017168948,1.161282807516924,0.031677696853876114,0.035765141248703
+held_out_random,base,13,ep0011,scn0051,281,30,251,0.8932384341637011,0.047013151148955025,0.4382760703842266,0.005455251317471266,0.0061072735115885735
+held_out_random,improved,13,ep0011,scn0051,281,30,251,0.8932384341637011,0.04473059296511524,0.5699086771028742,0.0042670066468417645,0.004777007736265659
+held_out_random,major_scratch,13,ep0011,scn0051,281,30,251,0.8932384341637011,0.044711084644578424,0.5712319115171619,0.004266340285539627,0.004776261281222105
+held_out_random,base,14,ep0012,scn0012,56,8,48,0.8571428571428571,0.1355741052858291,0.107112321281465,0.004221740178763866,0.004925363231450319
+held_out_random,improved,14,ep0012,scn0012,56,8,48,0.8571428571428571,0.043544526902898666,0.056407335813264,0.0009751296020112932,0.0011376511538401246
+held_out_random,major_scratch,14,ep0012,scn0012,56,8,48,0.8571428571428571,0.043584208334645914,0.05603000914762833,0.0009755996870808303,0.0011381995864212513
+held_out_random,base,15,ep0013,scn0006,42,6,36,0.8571428571428571,1.0528031986044801,1.928247507626526,0.0691860020160675,0.08071700483560562
+held_out_random,improved,15,ep0013,scn0006,42,6,36,0.8571428571428571,0.7062211120906083,1.1961615534431878,0.05525721609592438,0.06446675211191177
+held_out_random,major_scratch,15,ep0013,scn0006,42,6,36,0.8571428571428571,0.70628088194391,1.4833444094878159,0.055163633078336716,0.06435757130384445
+held_out_random,base,16,ep0013,scn0030,246,25,221,0.8983739837398373,0.5068618838344683,3.5691537342752495,0.0539030097424984,0.06000063195824623
+held_out_random,improved,16,ep0013,scn0030,246,25,221,0.8983739837398373,0.36776214294641046,3.063694201770908,0.046937815845012665,0.05224752426147461
+held_out_random,major_scratch,16,ep0013,scn0030,246,25,221,0.8983739837398373,0.3702207822629621,3.0626617883926284,0.04714196175336838,0.05247476324439049
+held_out_random,base,17,ep0014,scn0013,144,18,126,0.875,0.47981958626478444,1.0737745113666797,0.02674793265759945,0.030569061636924744
+held_out_random,improved,17,ep0014,scn0013,144,18,126,0.875,0.10502393814385869,0.4013542365644717,0.0062255277298390865,0.00711488863453269
+held_out_random,major_scratch,17,ep0014,scn0013,144,18,126,0.875,0.10501460667510401,0.29268335536349904,0.006211420055478811,0.00709876511245966
+held_out_random,base,18,ep0014,scn0042,92,10,82,0.8913043478260869,0.6414036637544632,1.1322970883338974,0.037189092487096786,0.041724346578121185
+held_out_random,improved,18,ep0014,scn0042,92,10,82,0.8913043478260869,0.5591748964786529,1.4461608268265045,0.034245047718286514,0.03842126950621605
+held_out_random,major_scratch,18,ep0014,scn0042,92,10,82,0.8913043478260869,0.5484445261955261,1.4425526238239552,0.033866290003061295,0.0379963219165802
+held_out_random,base,19,ep0017,scn0016,263,22,241,0.9163498098859315,0.03250250209767911,0.3963438106142985,0.0031489375978708267,0.003436392406001687
+held_out_random,improved,19,ep0017,scn0016,263,22,241,0.9163498098859315,0.03582255152636619,0.3453605549369698,0.0012538115261122584,0.0013682673452422023
+held_out_random,major_scratch,19,ep0017,scn0016,263,22,241,0.9163498098859315,0.03580366348841165,0.3407181414776427,0.0012540105963125825,0.0013684845762327313
+production,base,0,ep0014,scn0014,257,52,205,0.7976653696498055,0.4951585664714198,1.7835226986320352,0.022816505283117294,0.028604108840227127
+production,improved,0,ep0014,scn0014,257,52,205,0.7976653696498055,0.21193486279792817,0.7280051445525751,0.010775289498269558,0.01350853405892849
+production,major_scratch,0,ep0014,scn0014,257,52,205,0.7976653696498055,0.2024221179721701,0.7279966412056085,0.010699337348341942,0.013413315638899803
+production,base,1,ep002,scn00100010,185,19,166,0.8972972972972973,0.06740809645098035,0.30126742505993137,0.006345629226416349,0.0070719365030527115
+production,improved,1,ep002,scn00100010,185,19,166,0.8972972972972973,0.04339691952313527,0.8983463542637274,0.004287191666662693,0.004777893889695406
+production,major_scratch,1,ep002,scn00100010,185,19,166,0.8972972972972973,0.043169402913882,0.9016755866144783,0.004256824497133493,0.004744051489979029
+production,base,2,ep002,scn00100030,62,32,30,0.4838709677419355,0.23419040170214747,1.2155687503983243,0.01744561456143856,0.03605426847934723
+production,improved,2,ep002,scn00100030,62,32,30,0.4838709677419355,0.17631081831843956,0.942859279856956,0.012410001829266548,0.02564733847975731
+production,major_scratch,2,ep002,scn00100030,62,32,30,0.4838709677419355,0.17608541185441223,0.9411194629841492,0.012415774166584015,0.025659266859292984
+production,base,3,ep002,scn00100070,87,19,68,0.7816091954022989,0.8306600193564708,4.502644782666697,0.07537706941366196,0.09643830358982086
+production,improved,3,ep002,scn00100070,87,19,68,0.7816091954022989,0.5392945821468647,2.9172883961119185,0.0522209033370018,0.06681203842163086
+production,major_scratch,3,ep002,scn00100070,87,19,68,0.7816091954022989,0.508885258582058,2.7728359758413843,0.04900099337100983,0.06269244849681854
+production,base,4,ep002,scn00100080,45,7,38,0.8444444444444444,0.21774884939193725,0.014657328159825109,0.004454364534467459,0.005274905823171139
+production,improved,4,ep002,scn00100080,45,7,38,0.8444444444444444,0.15552082061767578,0.023066843814450944,0.005905765108764172,0.006993670016527176
+production,major_scratch,4,ep002,scn00100080,45,7,38,0.8444444444444444,0.1463403010368347,0.023282158390065474,0.005865052342414856,0.0069454568438231945
+production,base,5,ep002,scn00100190,196,21,175,0.8928571428571429,0.4720254450343376,2.8010031404020403,0.03625967726111412,0.04061083868145943
+production,improved,5,ep002,scn00100190,196,21,175,0.8928571428571429,0.39072509282376877,1.9833827012140532,0.014386503025889397,0.016112884506583214
+production,major_scratch,5,ep002,scn00100190,196,21,175,0.8928571428571429,0.359532762959946,1.9829151779116954,0.013832833617925644,0.015492773614823818
+production,base,6,ep002,scn00100300,55,11,44,0.8,0.28831238485872746,0.13445814402211523,0.027023985981941223,0.03377998247742653
+production,improved,6,ep002,scn00100300,55,11,44,0.8,0.25815278072695946,0.1054597426797504,0.01370830275118351,0.01713537611067295
+production,major_scratch,6,ep002,scn00100300,55,11,44,0.8,0.18142272252589464,0.08712444530848194,0.010627704672515392,0.013284631073474884
+production,base,7,ep002,scn00100340,51,26,25,0.49019607843137253,0.265630943775177,0.1763511635269991,0.01177276112139225,0.024016432464122772
+production,improved,7,ep002,scn00100340,51,26,25,0.49019607843137253,0.20311457872390748,0.14731786026705262,0.010104714892804623,0.020613618195056915
+production,major_scratch,7,ep002,scn00100340,51,26,25,0.49019607843137253,0.19944136699040732,0.1505691671430182,0.010128315538167953,0.020661763846874237
+production,base,8,ep002,scn00100350,42,21,21,0.5,0.2182253535653724,0.11151887337703703,0.00952469278126955,0.0190493855625391
+production,improved,8,ep002,scn00100350,42,21,21,0.5,0.16997232935467704,0.06999827403921964,0.00935874879360199,0.01871749758720398
+production,major_scratch,8,ep002,scn00100350,42,21,21,0.5,0.16567011739386886,0.07000930154501948,0.00940627884119749,0.01881255954504013
+production,base,9,ep002,scn00100380,65,6,59,0.9076923076923077,0.9064284079316733,1.7236938100144665,0.025909053161740303,0.028543872758746147
+production,improved,9,ep002,scn00100380,65,6,59,0.9076923076923077,0.047659172528031944,0.06702957653868698,0.0017583018634468317,0.0019371124217286706
+production,major_scratch,9,ep002,scn00100380,65,6,59,0.9076923076923077,0.04706593916036081,0.0695992381000338,0.0017518147360533476,0.0019299652194604278
+production,base,10,ep002,scn00100390,49,7,42,0.8571428571428571,0.10018453443491901,0.3240764402634693,0.014412873424589634,0.01681501977145672
+production,improved,10,ep002,scn00100390,49,7,42,0.8571428571428571,0.05947591267802097,0.16604064594323276,0.004006510600447655,0.004674262832850218
+production,major_scratch,10,ep002,scn00100390,49,7,42,0.8571428571428571,0.059157952976723514,0.1661371838714614,0.0040053557604551315,0.004672914743423462
+production,base,11,ep002,scn00100400,114,25,89,0.7807017543859649,0.44290371290849945,1.6780594099636783,0.04307902231812477,0.05517986789345741
+production,improved,11,ep002,scn00100400,114,25,89,0.7807017543859649,0.354228679444233,1.5989762406104011,0.02389349415898323,0.030605150386691093
+production,major_scratch,11,ep002,scn00100400,114,25,89,0.7807017543859649,0.3457295968584771,1.6449796528823089,0.02308531664311886,0.029569953680038452
+production,base,12,ep002,scn00100410,203,31,172,0.8472906403940886,0.8462221674974365,5.3722355677835045,0.0852850079536438,0.10065614432096481
+production,improved,12,ep002,scn00100410,203,31,172,0.8472906403940886,0.7391767569111082,4.81192150239714,0.07300300151109695,0.08616051822900772
+production,major_scratch,12,ep002,scn00100410,203,31,172,0.8472906403940886,0.673128255835638,5.293385912180635,0.0698753148317337,0.08246912062168121
+production,base,13,ep0022,scn0004,157,60,97,0.6178343949044586,0.5380967730699584,0.298528282398152,0.018896056339144707,0.030584340915083885
+production,improved,13,ep0022,scn0004,157,60,97,0.6178343949044586,0.46066084323927414,0.5321975426215608,0.01718219928443432,0.027810366824269295
+production,major_scratch,13,ep0022,scn0004,157,60,97,0.6178343949044586,0.473616954060488,0.6124845879585576,0.016853859648108482,0.027278928086161613
+production,base,14,ep003,scn00100010,218,81,137,0.6284403669724771,0.1780188717826047,0.6168539116404228,0.0053977821953594685,0.00858917273581028
+production,improved,14,ep003,scn00100010,218,81,137,0.6284403669724771,1.189038209240846,4.624786517650676,0.024173304438591003,0.03846554830670357
+production,major_scratch,14,ep003,scn00100010,218,81,137,0.6284403669724771,1.1889215848261259,4.624401680213764,0.024169929325580597,0.038460176438093185
+production,base,15,ep003,scn00100040,47,4,43,0.9148936170212766,0.11055794419074545,0.07300827805015102,0.0016376771964132786,0.0017900191014632583
+production,improved,15,ep003,scn00100040,47,4,43,0.9148936170212766,0.035327231397434156,0.017673077359442978,0.0018148067174479365,0.001983625814318657
+production,major_scratch,15,ep003,scn00100040,47,4,43,0.9148936170212766,0.03524780334258566,0.01848136715836764,0.0018099676817655563,0.001978336600586772
+production,base,16,ep003,scn00100060,68,8,60,0.8823529411764706,0.07260099415843552,0.06366218804165234,0.0011959221446886659,0.0013553781900554895
+production,improved,16,ep003,scn00100060,68,8,60,0.8823529411764706,0.06967062482962737,0.06439513499750936,0.0007773740799166262,0.0008810239960439503
+production,major_scratch,16,ep003,scn00100060,68,8,60,0.8823529411764706,0.06965467011606372,0.06441951606330652,0.0007857228629291058,0.0008904858841560781
+production,base,17,ep003,scn00100390,62,16,46,0.7419354838709677,0.032940277241562545,0.06493508692775445,0.003659884911030531,0.004932887852191925
+production,improved,17,ep003,scn00100390,62,16,46,0.7419354838709677,0.016824806991376375,0.15810934585233055,0.003928063903003931,0.005294346250593662
+production,major_scratch,17,ep003,scn00100390,62,16,46,0.7419354838709677,0.015422197352898749,0.158242738745648,0.003916640765964985,0.005278950557112694
+production,base,18,ep003,scn00100392,14,10,4,0.2857142857142857,0.04498832063241438,0.007005838286639704,0.0010747680207714438,0.003761688247323036
+production,improved,18,ep003,scn00100392,14,10,4,0.2857142857142857,0.029417092488570648,0.007419384742672508,0.00103379029314965,0.0036182659678161144
+production,major_scratch,18,ep003,scn00100392,14,10,4,0.2857142857142857,0.029841747981580822,0.007806080392289838,0.0010424256324768066,0.0036484894808381796
+production,base,19,ep003,scn00100394,35,15,20,0.5714285714285714,0.04450146590049068,0.018860487859456165,0.0005599937867373228,0.0009799889521673322
+production,improved,19,ep003,scn00100394,35,15,20,0.5714285714285714,0.01976123017569383,0.11719788888251535,0.0011223889887332916,0.001964180963113904
+production,major_scratch,19,ep003,scn00100394,35,15,20,0.5714285714285714,0.018319697429736454,0.11747303682306769,0.00112050399184227,0.0019608817528933287

official_protocol_summary_metrics.csv

@@ -0,0 +1,10 @@
+test_set,model,clips,frames,mean_mask_ratio,shifted_distance,shifted_distance_median,shifted_distance_std,npss,npss_median,npss_std,full_l1,full_l1_median,full_l1_std,missing_l1,missing_l1_median,missing_l1_std
+held_out_algorithmic,base,20,2761,0.9005592126546578,0.5635700441298277,0.3973293047348959,0.5789553313958574,1.6164449568404926,0.7376146166286656,2.2313882746102753,0.022524041222641244,0.013523494824767113,0.024741861034447702,0.02539981409208849,0.01724134013056755,0.02791094618888122
+held_out_algorithmic,improved,20,2761,0.9005592126546578,0.31293257113134654,0.10416547441939256,0.49493006900668024,0.4975234434345155,0.19858078204558233,0.5986477767547608,0.010504539695102722,0.004716081544756889,0.01642234100953195,0.012192885919648688,0.005016518756747246,0.018367657231855836
+held_out_algorithmic,major_scratch,20,2761,0.9005592126546578,0.3030916049633498,0.10337592970656015,0.49330662595866864,0.4943081554823948,0.2142453324106498,0.5705970864084018,0.011196957573702093,0.004716202616691589,0.016538662774409327,0.013189485580369364,0.005016664741560817,0.018767806271011808
+held_out_random,base,20,2761,0.8824006015763473,0.332714187904665,0.16745417321036601,0.32247388332645976,0.9621530503120319,0.5530451889773913,1.0722316053179077,0.017667744151549414,0.008342467714101076,0.01874697560355533,0.019904192874673755,0.009538945741951466,0.021340581803422742
+held_out_random,improved,20,2761,0.8824006015763473,0.2993338749895051,0.14788039218607119,0.2864001494693676,0.9063433045806659,0.5543289849114983,0.9381369842839581,0.014602789636410308,0.005932004889473319,0.01630367203772323,0.016428220353554936,0.006658609490841627,0.018502989742684876
+held_out_random,major_scratch,20,2761,0.8824006015763473,0.2966546864184923,0.14784876577822054,0.2824031458742327,0.9064781272101436,0.5700359829499448,0.9387977765841757,0.014577854810340796,0.005925189005210996,0.016287218175314574,0.016399642699980177,0.006650809198617935,0.0184832842062924
+production,base,20,2012,0.7261683858264012,0.3203401765177655,0.22620787763375994,0.27399511657853765,1.0640955803737175,0.29989785372904165,1.5086611587002994,0.020606417080853135,0.013092817272990942,0.02319715492552421,0.02720442715799436,0.021532909013330936,0.028064481908314405
+production,improved,20,2012,0.7261683858264012,0.2584831672478787,0.17314157383655832,0.28787366397231107,0.9990735727197935,0.16207499589778165,1.4526559209453267,0.014292532837134785,0.009731731843203306,0.017846792527678713,0.01968566266295966,0.014810709282755852,0.021957319492503215
+production,major_scratch,20,2012,0.7261683858264012,0.2469537931085076,0.17087776462414056,0.28174928594805565,1.0217469455666672,0.1621899613085547,1.5094304201873916,0.013732498313765973,0.009767297189682722,0.01698585724705308,0.0189922235236736,0.013348973356187344,0.020972357275608075

training_config.yaml

@@ -0,0 +1,62 @@
+seed: 1234
+model:
+  type: improved_ais_bilstm
+  pose_dim: 596
+  input_dim: 597
+  temporal_dim: 4
+  hidden_size: 512
+  num_layers: 2
+  dropout: 0.3003
+  synthesis_hidden: 1024
+  beta_groups: 64
+data:
+  clip_length: 224
+  stride: 56
+  val_stride: 224
+  train_controller_h5: artifacts/hf_dataset/in_house_dataset/vectorized_controller_values/ef25a8e5ecd2fa86420741e5646cd785aa025c44c06f4118aab77f558c8f6981/train.h5
+  train_block_keyframes_h5: artifacts/hf_dataset/in_house_dataset/vectorized_block_keyframes/train.h5
+  val_controller_h5: artifacts/hf_dataset/in_house_dataset/vectorized_controller_values/ef25a8e5ecd2fa86420741e5646cd785aa025c44c06f4118aab77f558c8f6981/test.h5
+  val_block_keyframes_h5: artifacts/hf_dataset/in_house_dataset/vectorized_block_keyframes/test.h5
+training:
+  output_dir: runs/ais_major_scratch_10k
+  device: auto
+  batch_size: 8
+  eval_batch_size: 8
+  num_workers: 0
+  max_steps: 10000
+  gradient_accumulation_steps: 8
+  progress: true
+  progress_bar: true
+  progress_every_steps: 25
+  dashboard: true
+  train_log_every_steps: 25
+  val_every_steps: 1000
+  save_every_steps: 1000
+  grad_clip_norm: 1.0
+  lr: 0.0001
+  weight_decay: 0.001
+  hard_copy_keyposes_in_forward: false
+  eval_only: false
+  checkpoint:
+    best_metric: val_loss
+    save_periodic: true
+    keep_every_n: 5
+loss:
+  velocity_weight: 0.05
+  acceleration_weight: 0.01
+  spectral_weight: 0.001
+  gate_tv_weight: 0.001
+  keypose_weight: 10.0
+eval:
+  write_report: false
+  baselines:
+  - linear
+  - previous
+  max_report_clips: 20
+  max_report_plots: 12
+  sets:
+  - name: prod_test
+    controller_h5: artifacts/hf_dataset/prod_test_dataset/vectorized_controller_values/ef25a8e5ecd2fa86420741e5646cd785aa025c44c06f4118aab77f558c8f6981/test.h5
+    block_keyframes_h5: artifacts/hf_dataset/prod_test_dataset/vectorized_block_keyframes/test.h5
+    stride: 224
+    max_clips: 20