Hugging Face's logo

dreamdifferent
/
vam-cross-level2-panda-widowx-widowx-texture-teleopaligned-videolora200-action-decoder-iter1800

Robotics
mimic-video
action-prediction
Model card Files
xet
 Community
figurek1m commited on
Commit
ff72360
·
verified ·
1 Parent(s): 711ae72

Publish World2Action decoder iter_000001800.pt

Browse files
Files changed (7) hide show
  1. README.md +33 -0
  2. checkpoints/model/iter_000001800.pt +3 -0
  3. config.yaml +2112 -0
  4. latest_checkpoint.txt +1 -0
  5. vam_cross_action_decoder_manifest.json +348 -0
  6. vam_cross_video2world_config.json +650 -0
  7. vam_cross_world2action_config.json +311 -0
README.md ADDED
@@ -0,0 +1,33 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ tags:
3
+ - mimic-video
4
+ - robotics
5
+ - action-prediction
6
+ ---
7
+
8
+ # VAM-Cross MimicVideo World2Action decoder
9
+
10
+ This public repository contains the World2Action decoder checkpoint from
11
+ iteration 1800 of `w2a_panda_widowx_level2_widowx_texture_2cam_hstack_action_iter2374_videolora_iter200_widowx_teleop_recording_frame_v1`. The run stopped
12
+ because of `unknown`. The newest complete model,
13
+ optimizer, scheduler, and trainer checkpoint set was verified before selecting
14
+ the uploaded model weight.
15
+
16
+ ## Required frozen inputs
17
+
18
+ - MimicVideo commit: `e3355dbc93132b576c02f920a59b4fc18a4f5906`
19
+ - Initial Video2World backbone: `dreamdifferent/widowx250-video-fused@f0cea76b62c5dd66b06b9f965932ddea32a7b546`
20
+ - Initial action decoder: `dreamdifferent/vam-cross-target-widowx250-native-2cam-action-decoder@93750cccda01620e3c028477e4c49bc5c996a68d`
21
+ - Frozen Video LoRA: `dreamdifferent/vam-cross-level2-panda-widowx-widowx-texture-video-lora-iter200@5a22b23f6f4bf80b249d6215d15260b7b59acbcf`
22
+
23
+ ## Action and data contract
24
+
25
+ - Dataset: `dreamdifferent/vam-cross-level2-panda-widowx-widowx-texture@994be7f8952807008c316db7943ec732bf70b978`
26
+ - Episodes/frames: 256 / 54349
27
+ - Cameras: `observation.images.corner_cam, observation.images.front_cam`
28
+ - Target: 15 achieved-EE/gripper actions at 5 Hz
29
+ - Pose target: `relative_to_current_achieved_pose` in `widowx_reference_base/teleop_aligned_tool`
30
+ - Rotation: `rotation_6d`
31
+
32
+ The dataset and frozen inputs are not included. Use the pinned JSON and
33
+ effective `config.yaml` included in this repository.
checkpoints/model/iter_000001800.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e9cf5ae373416faa966d04c9707080f815a3710fd29b53c4d837380b9b3bfa47
3
+ size 998172132
config.yaml ADDED
@@ -0,0 +1,2112 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ action_pipe:
2
+ _target_: cosmos_predict2.configs.config_world2action.World2ActionPipelineConfig
3
+ ema:
4
+ enabled: false
5
+ iteration_shift: 0
6
+ rate: 0.1
7
+ net:
8
+ _target_: <class 'cosmos_predict2.models.world2action_dit.World2ActionDIT'>
9
+ adaln_lora_dim: '128'
10
+ atten_backend: flash_attn_no_cp
11
+ crossattn_emb_channels: '2048'
12
+ in_channels: '10'
13
+ max_horizon: '16'
14
+ mlp_ratio: '4.0'
15
+ model_channels: '1024'
16
+ num_blocks: '24'
17
+ num_heads: '8'
18
+ out_channels: '10'
19
+ pair_timestep_feature_rank: '1024'
20
+ sac_config:
21
+ every_n_blocks: 1
22
+ mode: none
23
+ use_adaln_lora: 'True'
24
+ precision: bfloat16
25
+ scheduler:
26
+ alpha: 1.0
27
+ beta: 1.0
28
+ num_denoising_steps: 10
29
+ checkpoint:
30
+ broadcast_via_filesystem: 'False'
31
+ dcp_allow_mismatched_size: 'False'
32
+ dcp_async_mode_enabled: 'False'
33
+ jit:
34
+ device: cuda
35
+ dtype: bfloat16
36
+ enabled: 'False'
37
+ input_shape: null
38
+ strict: 'True'
39
+ keys_not_to_resume: []
40
+ load_ema_to_reg: 'False'
41
+ load_path: ''
42
+ load_training_state: 'False'
43
+ only_load_scheduler_state: 'False'
44
+ save_iter: '100'
45
+ strict_resume: 'False'
46
+ type:
47
+ _target_: <class 'mimicvideo_runtime.checkpointer.KeepLatestCheckpointer'>
48
+ callbacks: null
49
+ keep_latest_only: 'False'
50
+ verbose: 'True'
51
+ data_config:
52
+ dataset:
53
+ data_components:
54
+ eef_pose_lowdim:
55
+ obs_type: ObsType.POSE_MATRIX
56
+ repr: LieRepr.ABSOLUTE
57
+ gripper_action_lowdim:
58
+ obs_type: ObsType.JOINT_POS
59
+ repr: LieRepr.ABSOLUTE
60
+ gripper_state_lowdim:
61
+ obs_type: ObsType.JOINT_POS
62
+ repr: LieRepr.ABSOLUTE
63
+ language_embedding:
64
+ obs_type: ObsType.LANGUAGE_EMBEDDING
65
+ repr: null
66
+ language_instruction:
67
+ obs_type: ObsType.LANGUAGE
68
+ repr: null
69
+ num_conditional_frames:
70
+ obs_type: ObsType.METADATA_LABEL
71
+ repr: null
72
+ workspace_rgb:
73
+ obs_type: ObsType.RGB
74
+ repr: null
75
+ workspace_rgb_embedding:
76
+ obs_type: ObsType.VIDEO_EMBEDDING
77
+ repr: null
78
+ data_transforms:
79
+ - name: CosmosProcessImage
80
+ resize_sizes:
81
+ - '480'
82
+ - '640'
83
+ targets:
84
+ - workspace_rgb
85
+ - additional_data:
86
+ relative_base_value: obs/eef_pose_lowdim
87
+ name: ConvertLowdimRepr
88
+ relative_base_value_idx: '-1'
89
+ targets:
90
+ - action/eef_pose_lowdim
91
+ - name: DeconstructPoseMat
92
+ out_key_prefix: action/eef
93
+ targets:
94
+ - action/eef_pose_lowdim
95
+ - name: DeconstructPoseMat
96
+ out_key_prefix: obs/eef
97
+ targets:
98
+ - obs/eef_pose_lowdim
99
+ - name: RotationMatrixTo6D
100
+ targets:
101
+ - action/eef_rot_lowdim
102
+ - obs/eef_rot_lowdim
103
+ - name: Flatten
104
+ targets:
105
+ - lowdim
106
+ - name: Concat
107
+ out_key: action/lowdim_concat
108
+ targets:
109
+ - action/eef_pos_lowdim
110
+ - action/eef_rot_lowdim
111
+ - action/gripper_action_lowdim
112
+ - name: Concat
113
+ out_key: obs/lowdim_concat
114
+ targets:
115
+ - obs/eef_pos_lowdim
116
+ - obs/eef_rot_lowdim
117
+ - obs/gripper_state_lowdim
118
+ dataset:
119
+ _convert_: partial
120
+ _target_: mimicvideo_runtime.action_dataset.WidowXActionDataset
121
+ data_components:
122
+ eef_pose_lowdim:
123
+ obs_type: ObsType.POSE_MATRIX
124
+ repr: LieRepr.ABSOLUTE
125
+ gripper_action_lowdim:
126
+ obs_type: ObsType.JOINT_POS
127
+ repr: LieRepr.ABSOLUTE
128
+ gripper_state_lowdim:
129
+ obs_type: ObsType.JOINT_POS
130
+ repr: LieRepr.ABSOLUTE
131
+ language_embedding:
132
+ obs_type: ObsType.LANGUAGE_EMBEDDING
133
+ repr: null
134
+ language_instruction:
135
+ obs_type: ObsType.LANGUAGE
136
+ repr: null
137
+ num_conditional_frames:
138
+ obs_type: ObsType.METADATA_LABEL
139
+ repr: null
140
+ workspace_rgb:
141
+ obs_type: ObsType.RGB
142
+ repr: null
143
+ workspace_rgb_embedding:
144
+ obs_type: ObsType.VIDEO_EMBEDDING
145
+ repr: null
146
+ data_dir: /cluster/scratch/ehalicki/vam_cross_mimicvideo/datasets/mimicvideo/panda_widowx_level2_widowx_texture_2cam_hstack_widowx_teleop_recording_frame_v1_action/action_dataset
147
+ data_transforms:
148
+ - name: CosmosProcessImage
149
+ resize_sizes:
150
+ - '480'
151
+ - '640'
152
+ targets:
153
+ - workspace_rgb
154
+ - additional_data:
155
+ relative_base_value: obs/eef_pose_lowdim
156
+ name: ConvertLowdimRepr
157
+ relative_base_value_idx: '-1'
158
+ targets:
159
+ - action/eef_pose_lowdim
160
+ - name: DeconstructPoseMat
161
+ out_key_prefix: action/eef
162
+ targets:
163
+ - action/eef_pose_lowdim
164
+ - name: DeconstructPoseMat
165
+ out_key_prefix: obs/eef
166
+ targets:
167
+ - obs/eef_pose_lowdim
168
+ - name: RotationMatrixTo6D
169
+ targets:
170
+ - action/eef_rot_lowdim
171
+ - obs/eef_rot_lowdim
172
+ - name: Flatten
173
+ targets:
174
+ - lowdim
175
+ - name: Concat
176
+ out_key: action/lowdim_concat
177
+ targets:
178
+ - action/eef_pos_lowdim
179
+ - action/eef_rot_lowdim
180
+ - action/gripper_action_lowdim
181
+ - name: Concat
182
+ out_key: obs/lowdim_concat
183
+ targets:
184
+ - obs/eef_pos_lowdim
185
+ - obs/eef_rot_lowdim
186
+ - obs/gripper_state_lowdim
187
+ num_val_episodes: '10'
188
+ policy_io:
189
+ action:
190
+ eef_pos_lowdim:
191
+ horizon: '15'
192
+ normalization_type: NormalizationType.VARIANCE
193
+ shift_right_by: '0.2'
194
+ target_frequency: '5'
195
+ target_repr: LieRepr.RELATIVE
196
+ eef_rot_lowdim:
197
+ horizon: '15'
198
+ normalization_type: NormalizationType.IDENTITY
199
+ shift_right_by: '0.2'
200
+ target_frequency: '5'
201
+ target_repr: LieRepr.RELATIVE
202
+ gripper_action_lowdim:
203
+ horizon: '15'
204
+ normalization_type: NormalizationType.VARIANCE
205
+ shift_right_by: '0.2'
206
+ target_frequency: '5'
207
+ target_repr: LieRepr.ABSOLUTE
208
+ workspace_rgb:
209
+ horizon: '56'
210
+ normalization_type: NormalizationType.NONE
211
+ shift_right_by: '0.2'
212
+ target_frequency: '5'
213
+ target_repr: null
214
+ obs:
215
+ eef_pos_lowdim:
216
+ horizon: '1'
217
+ normalization_type: NormalizationType.VARIANCE
218
+ shift_right_by: '0'
219
+ target_frequency: '5'
220
+ target_repr: LieRepr.ABSOLUTE
221
+ eef_rot_lowdim:
222
+ horizon: '1'
223
+ normalization_type: NormalizationType.IDENTITY
224
+ shift_right_by: '0'
225
+ target_frequency: '5'
226
+ target_repr: LieRepr.ABSOLUTE
227
+ gripper_state_lowdim:
228
+ horizon: '1'
229
+ normalization_type: NormalizationType.VARIANCE
230
+ shift_right_by: '0'
231
+ target_frequency: '5'
232
+ target_repr: LieRepr.ABSOLUTE
233
+ language_embedding:
234
+ horizon: '1'
235
+ normalization_type: NormalizationType.NONE
236
+ shift_right_by: '0'
237
+ target_frequency: null
238
+ target_repr: null
239
+ workspace_rgb:
240
+ horizon: '5'
241
+ normalization_type: NormalizationType.NONE
242
+ shift_right_by: '0'
243
+ target_frequency: '5'
244
+ target_repr: null
245
+ seed: '42'
246
+ should_include_padded_tails: 'True'
247
+ source_component_names:
248
+ action/eef_pos_lowdim: eef_pose_lowdim
249
+ action/eef_rot_lowdim: eef_pose_lowdim
250
+ obs/eef_pos_lowdim: eef_pose_lowdim
251
+ obs/eef_rot_lowdim: eef_pose_lowdim
252
+ timestep_anchor: workspace_rgb
253
+ source_component_names:
254
+ action/eef_pos_lowdim: eef_pose_lowdim
255
+ action/eef_rot_lowdim: eef_pose_lowdim
256
+ obs/eef_pos_lowdim: eef_pose_lowdim
257
+ obs/eef_rot_lowdim: eef_pose_lowdim
258
+ policy_io:
259
+ concat_groups:
260
+ action/lowdim_concat:
261
+ - action/eef_pos_lowdim
262
+ - action/eef_rot_lowdim
263
+ - action/gripper_action_lowdim
264
+ obs/lowdim_concat:
265
+ - obs/eef_pos_lowdim
266
+ - obs/eef_rot_lowdim
267
+ - obs/gripper_state_lowdim
268
+ img_resize_sizes:
269
+ - '480'
270
+ - '640'
271
+ policy_io:
272
+ action:
273
+ eef_pos_lowdim:
274
+ horizon: '15'
275
+ normalization_type: NormalizationType.VARIANCE
276
+ shift_right_by: '0.2'
277
+ target_frequency: '5'
278
+ target_repr: LieRepr.RELATIVE
279
+ eef_rot_lowdim:
280
+ horizon: '15'
281
+ normalization_type: NormalizationType.IDENTITY
282
+ shift_right_by: '0.2'
283
+ target_frequency: '5'
284
+ target_repr: LieRepr.RELATIVE
285
+ gripper_action_lowdim:
286
+ horizon: '15'
287
+ normalization_type: NormalizationType.VARIANCE
288
+ shift_right_by: '0.2'
289
+ target_frequency: '5'
290
+ target_repr: LieRepr.ABSOLUTE
291
+ workspace_rgb:
292
+ horizon: '56'
293
+ normalization_type: NormalizationType.NONE
294
+ shift_right_by: '0.2'
295
+ target_frequency: '5'
296
+ target_repr: null
297
+ obs:
298
+ eef_pos_lowdim:
299
+ horizon: '1'
300
+ normalization_type: NormalizationType.VARIANCE
301
+ shift_right_by: '0'
302
+ target_frequency: '5'
303
+ target_repr: LieRepr.ABSOLUTE
304
+ eef_rot_lowdim:
305
+ horizon: '1'
306
+ normalization_type: NormalizationType.IDENTITY
307
+ shift_right_by: '0'
308
+ target_frequency: '5'
309
+ target_repr: LieRepr.ABSOLUTE
310
+ gripper_state_lowdim:
311
+ horizon: '1'
312
+ normalization_type: NormalizationType.VARIANCE
313
+ shift_right_by: '0'
314
+ target_frequency: '5'
315
+ target_repr: LieRepr.ABSOLUTE
316
+ language_embedding:
317
+ horizon: '1'
318
+ normalization_type: NormalizationType.NONE
319
+ shift_right_by: '0'
320
+ target_frequency: null
321
+ target_repr: null
322
+ workspace_rgb:
323
+ horizon: '5'
324
+ normalization_type: NormalizationType.NONE
325
+ shift_right_by: '0'
326
+ target_frequency: '5'
327
+ target_repr: null
328
+ dataloader_train:
329
+ _target_: <class 'torch.utils.data.dataloader.DataLoader'>
330
+ batch_sampler: null
331
+ batch_size: '2'
332
+ collate_fn: null
333
+ dataset:
334
+ _target_: <function _instantiate_dataset at 0x14afb0f1edd0>
335
+ data_config:
336
+ dataset:
337
+ data_components:
338
+ eef_pose_lowdim:
339
+ obs_type: ObsType.POSE_MATRIX
340
+ repr: LieRepr.ABSOLUTE
341
+ gripper_action_lowdim:
342
+ obs_type: ObsType.JOINT_POS
343
+ repr: LieRepr.ABSOLUTE
344
+ gripper_state_lowdim:
345
+ obs_type: ObsType.JOINT_POS
346
+ repr: LieRepr.ABSOLUTE
347
+ language_embedding:
348
+ obs_type: ObsType.LANGUAGE_EMBEDDING
349
+ repr: null
350
+ language_instruction:
351
+ obs_type: ObsType.LANGUAGE
352
+ repr: null
353
+ num_conditional_frames:
354
+ obs_type: ObsType.METADATA_LABEL
355
+ repr: null
356
+ workspace_rgb:
357
+ obs_type: ObsType.RGB
358
+ repr: null
359
+ workspace_rgb_embedding:
360
+ obs_type: ObsType.VIDEO_EMBEDDING
361
+ repr: null
362
+ data_transforms:
363
+ - name: CosmosProcessImage
364
+ resize_sizes:
365
+ - '480'
366
+ - '640'
367
+ targets:
368
+ - workspace_rgb
369
+ - additional_data:
370
+ relative_base_value: obs/eef_pose_lowdim
371
+ name: ConvertLowdimRepr
372
+ relative_base_value_idx: '-1'
373
+ targets:
374
+ - action/eef_pose_lowdim
375
+ - name: DeconstructPoseMat
376
+ out_key_prefix: action/eef
377
+ targets:
378
+ - action/eef_pose_lowdim
379
+ - name: DeconstructPoseMat
380
+ out_key_prefix: obs/eef
381
+ targets:
382
+ - obs/eef_pose_lowdim
383
+ - name: RotationMatrixTo6D
384
+ targets:
385
+ - action/eef_rot_lowdim
386
+ - obs/eef_rot_lowdim
387
+ - name: Flatten
388
+ targets:
389
+ - lowdim
390
+ - name: Concat
391
+ out_key: action/lowdim_concat
392
+ targets:
393
+ - action/eef_pos_lowdim
394
+ - action/eef_rot_lowdim
395
+ - action/gripper_action_lowdim
396
+ - name: Concat
397
+ out_key: obs/lowdim_concat
398
+ targets:
399
+ - obs/eef_pos_lowdim
400
+ - obs/eef_rot_lowdim
401
+ - obs/gripper_state_lowdim
402
+ dataset:
403
+ _convert_: partial
404
+ _target_: mimicvideo_runtime.action_dataset.WidowXActionDataset
405
+ data_components:
406
+ eef_pose_lowdim:
407
+ obs_type: ObsType.POSE_MATRIX
408
+ repr: LieRepr.ABSOLUTE
409
+ gripper_action_lowdim:
410
+ obs_type: ObsType.JOINT_POS
411
+ repr: LieRepr.ABSOLUTE
412
+ gripper_state_lowdim:
413
+ obs_type: ObsType.JOINT_POS
414
+ repr: LieRepr.ABSOLUTE
415
+ language_embedding:
416
+ obs_type: ObsType.LANGUAGE_EMBEDDING
417
+ repr: null
418
+ language_instruction:
419
+ obs_type: ObsType.LANGUAGE
420
+ repr: null
421
+ num_conditional_frames:
422
+ obs_type: ObsType.METADATA_LABEL
423
+ repr: null
424
+ workspace_rgb:
425
+ obs_type: ObsType.RGB
426
+ repr: null
427
+ workspace_rgb_embedding:
428
+ obs_type: ObsType.VIDEO_EMBEDDING
429
+ repr: null
430
+ data_dir: /cluster/scratch/ehalicki/vam_cross_mimicvideo/datasets/mimicvideo/panda_widowx_level2_widowx_texture_2cam_hstack_widowx_teleop_recording_frame_v1_action/action_dataset
431
+ data_transforms:
432
+ - name: CosmosProcessImage
433
+ resize_sizes:
434
+ - '480'
435
+ - '640'
436
+ targets:
437
+ - workspace_rgb
438
+ - additional_data:
439
+ relative_base_value: obs/eef_pose_lowdim
440
+ name: ConvertLowdimRepr
441
+ relative_base_value_idx: '-1'
442
+ targets:
443
+ - action/eef_pose_lowdim
444
+ - name: DeconstructPoseMat
445
+ out_key_prefix: action/eef
446
+ targets:
447
+ - action/eef_pose_lowdim
448
+ - name: DeconstructPoseMat
449
+ out_key_prefix: obs/eef
450
+ targets:
451
+ - obs/eef_pose_lowdim
452
+ - name: RotationMatrixTo6D
453
+ targets:
454
+ - action/eef_rot_lowdim
455
+ - obs/eef_rot_lowdim
456
+ - name: Flatten
457
+ targets:
458
+ - lowdim
459
+ - name: Concat
460
+ out_key: action/lowdim_concat
461
+ targets:
462
+ - action/eef_pos_lowdim
463
+ - action/eef_rot_lowdim
464
+ - action/gripper_action_lowdim
465
+ - name: Concat
466
+ out_key: obs/lowdim_concat
467
+ targets:
468
+ - obs/eef_pos_lowdim
469
+ - obs/eef_rot_lowdim
470
+ - obs/gripper_state_lowdim
471
+ num_val_episodes: '10'
472
+ policy_io:
473
+ action:
474
+ eef_pos_lowdim:
475
+ horizon: '15'
476
+ normalization_type: NormalizationType.VARIANCE
477
+ shift_right_by: '0.2'
478
+ target_frequency: '5'
479
+ target_repr: LieRepr.RELATIVE
480
+ eef_rot_lowdim:
481
+ horizon: '15'
482
+ normalization_type: NormalizationType.IDENTITY
483
+ shift_right_by: '0.2'
484
+ target_frequency: '5'
485
+ target_repr: LieRepr.RELATIVE
486
+ gripper_action_lowdim:
487
+ horizon: '15'
488
+ normalization_type: NormalizationType.VARIANCE
489
+ shift_right_by: '0.2'
490
+ target_frequency: '5'
491
+ target_repr: LieRepr.ABSOLUTE
492
+ workspace_rgb:
493
+ horizon: '56'
494
+ normalization_type: NormalizationType.NONE
495
+ shift_right_by: '0.2'
496
+ target_frequency: '5'
497
+ target_repr: null
498
+ obs:
499
+ eef_pos_lowdim:
500
+ horizon: '1'
501
+ normalization_type: NormalizationType.VARIANCE
502
+ shift_right_by: '0'
503
+ target_frequency: '5'
504
+ target_repr: LieRepr.ABSOLUTE
505
+ eef_rot_lowdim:
506
+ horizon: '1'
507
+ normalization_type: NormalizationType.IDENTITY
508
+ shift_right_by: '0'
509
+ target_frequency: '5'
510
+ target_repr: LieRepr.ABSOLUTE
511
+ gripper_state_lowdim:
512
+ horizon: '1'
513
+ normalization_type: NormalizationType.VARIANCE
514
+ shift_right_by: '0'
515
+ target_frequency: '5'
516
+ target_repr: LieRepr.ABSOLUTE
517
+ language_embedding:
518
+ horizon: '1'
519
+ normalization_type: NormalizationType.NONE
520
+ shift_right_by: '0'
521
+ target_frequency: null
522
+ target_repr: null
523
+ workspace_rgb:
524
+ horizon: '5'
525
+ normalization_type: NormalizationType.NONE
526
+ shift_right_by: '0'
527
+ target_frequency: '5'
528
+ target_repr: null
529
+ seed: '42'
530
+ should_include_padded_tails: 'True'
531
+ source_component_names:
532
+ action/eef_pos_lowdim: eef_pose_lowdim
533
+ action/eef_rot_lowdim: eef_pose_lowdim
534
+ obs/eef_pos_lowdim: eef_pose_lowdim
535
+ obs/eef_rot_lowdim: eef_pose_lowdim
536
+ timestep_anchor: workspace_rgb
537
+ source_component_names:
538
+ action/eef_pos_lowdim: eef_pose_lowdim
539
+ action/eef_rot_lowdim: eef_pose_lowdim
540
+ obs/eef_pos_lowdim: eef_pose_lowdim
541
+ obs/eef_rot_lowdim: eef_pose_lowdim
542
+ policy_io:
543
+ concat_groups:
544
+ action/lowdim_concat:
545
+ - action/eef_pos_lowdim
546
+ - action/eef_rot_lowdim
547
+ - action/gripper_action_lowdim
548
+ obs/lowdim_concat:
549
+ - obs/eef_pos_lowdim
550
+ - obs/eef_rot_lowdim
551
+ - obs/gripper_state_lowdim
552
+ img_resize_sizes:
553
+ - '480'
554
+ - '640'
555
+ policy_io:
556
+ action:
557
+ eef_pos_lowdim:
558
+ horizon: '15'
559
+ normalization_type: NormalizationType.VARIANCE
560
+ shift_right_by: '0.2'
561
+ target_frequency: '5'
562
+ target_repr: LieRepr.RELATIVE
563
+ eef_rot_lowdim:
564
+ horizon: '15'
565
+ normalization_type: NormalizationType.IDENTITY
566
+ shift_right_by: '0.2'
567
+ target_frequency: '5'
568
+ target_repr: LieRepr.RELATIVE
569
+ gripper_action_lowdim:
570
+ horizon: '15'
571
+ normalization_type: NormalizationType.VARIANCE
572
+ shift_right_by: '0.2'
573
+ target_frequency: '5'
574
+ target_repr: LieRepr.ABSOLUTE
575
+ workspace_rgb:
576
+ horizon: '56'
577
+ normalization_type: NormalizationType.NONE
578
+ shift_right_by: '0.2'
579
+ target_frequency: '5'
580
+ target_repr: null
581
+ obs:
582
+ eef_pos_lowdim:
583
+ horizon: '1'
584
+ normalization_type: NormalizationType.VARIANCE
585
+ shift_right_by: '0'
586
+ target_frequency: '5'
587
+ target_repr: LieRepr.ABSOLUTE
588
+ eef_rot_lowdim:
589
+ horizon: '1'
590
+ normalization_type: NormalizationType.IDENTITY
591
+ shift_right_by: '0'
592
+ target_frequency: '5'
593
+ target_repr: LieRepr.ABSOLUTE
594
+ gripper_state_lowdim:
595
+ horizon: '1'
596
+ normalization_type: NormalizationType.VARIANCE
597
+ shift_right_by: '0'
598
+ target_frequency: '5'
599
+ target_repr: LieRepr.ABSOLUTE
600
+ language_embedding:
601
+ horizon: '1'
602
+ normalization_type: NormalizationType.NONE
603
+ shift_right_by: '0'
604
+ target_frequency: null
605
+ target_repr: null
606
+ workspace_rgb:
607
+ horizon: '5'
608
+ normalization_type: NormalizationType.NONE
609
+ shift_right_by: '0'
610
+ target_frequency: '5'
611
+ target_repr: null
612
+ is_train: 'True'
613
+ drop_last: 'True'
614
+ generator: null
615
+ in_order: 'True'
616
+ multiprocessing_context: null
617
+ num_workers: '1'
618
+ persistent_workers: 'True'
619
+ pin_memory: 'True'
620
+ pin_memory_device: ''
621
+ prefetch_factor: '2'
622
+ sampler:
623
+ _target_: <class 'cosmos_predict2.data.resumable_sampler.ResumableDistributedSampler'>
624
+ dataset:
625
+ _target_: <function _instantiate_dataset at 0x14afb0f1edd0>
626
+ data_config:
627
+ dataset:
628
+ data_components:
629
+ eef_pose_lowdim:
630
+ obs_type: ObsType.POSE_MATRIX
631
+ repr: LieRepr.ABSOLUTE
632
+ gripper_action_lowdim:
633
+ obs_type: ObsType.JOINT_POS
634
+ repr: LieRepr.ABSOLUTE
635
+ gripper_state_lowdim:
636
+ obs_type: ObsType.JOINT_POS
637
+ repr: LieRepr.ABSOLUTE
638
+ language_embedding:
639
+ obs_type: ObsType.LANGUAGE_EMBEDDING
640
+ repr: null
641
+ language_instruction:
642
+ obs_type: ObsType.LANGUAGE
643
+ repr: null
644
+ num_conditional_frames:
645
+ obs_type: ObsType.METADATA_LABEL
646
+ repr: null
647
+ workspace_rgb:
648
+ obs_type: ObsType.RGB
649
+ repr: null
650
+ workspace_rgb_embedding:
651
+ obs_type: ObsType.VIDEO_EMBEDDING
652
+ repr: null
653
+ data_transforms:
654
+ - name: CosmosProcessImage
655
+ resize_sizes:
656
+ - '480'
657
+ - '640'
658
+ targets:
659
+ - workspace_rgb
660
+ - additional_data:
661
+ relative_base_value: obs/eef_pose_lowdim
662
+ name: ConvertLowdimRepr
663
+ relative_base_value_idx: '-1'
664
+ targets:
665
+ - action/eef_pose_lowdim
666
+ - name: DeconstructPoseMat
667
+ out_key_prefix: action/eef
668
+ targets:
669
+ - action/eef_pose_lowdim
670
+ - name: DeconstructPoseMat
671
+ out_key_prefix: obs/eef
672
+ targets:
673
+ - obs/eef_pose_lowdim
674
+ - name: RotationMatrixTo6D
675
+ targets:
676
+ - action/eef_rot_lowdim
677
+ - obs/eef_rot_lowdim
678
+ - name: Flatten
679
+ targets:
680
+ - lowdim
681
+ - name: Concat
682
+ out_key: action/lowdim_concat
683
+ targets:
684
+ - action/eef_pos_lowdim
685
+ - action/eef_rot_lowdim
686
+ - action/gripper_action_lowdim
687
+ - name: Concat
688
+ out_key: obs/lowdim_concat
689
+ targets:
690
+ - obs/eef_pos_lowdim
691
+ - obs/eef_rot_lowdim
692
+ - obs/gripper_state_lowdim
693
+ dataset:
694
+ _convert_: partial
695
+ _target_: mimicvideo_runtime.action_dataset.WidowXActionDataset
696
+ data_components:
697
+ eef_pose_lowdim:
698
+ obs_type: ObsType.POSE_MATRIX
699
+ repr: LieRepr.ABSOLUTE
700
+ gripper_action_lowdim:
701
+ obs_type: ObsType.JOINT_POS
702
+ repr: LieRepr.ABSOLUTE
703
+ gripper_state_lowdim:
704
+ obs_type: ObsType.JOINT_POS
705
+ repr: LieRepr.ABSOLUTE
706
+ language_embedding:
707
+ obs_type: ObsType.LANGUAGE_EMBEDDING
708
+ repr: null
709
+ language_instruction:
710
+ obs_type: ObsType.LANGUAGE
711
+ repr: null
712
+ num_conditional_frames:
713
+ obs_type: ObsType.METADATA_LABEL
714
+ repr: null
715
+ workspace_rgb:
716
+ obs_type: ObsType.RGB
717
+ repr: null
718
+ workspace_rgb_embedding:
719
+ obs_type: ObsType.VIDEO_EMBEDDING
720
+ repr: null
721
+ data_dir: /cluster/scratch/ehalicki/vam_cross_mimicvideo/datasets/mimicvideo/panda_widowx_level2_widowx_texture_2cam_hstack_widowx_teleop_recording_frame_v1_action/action_dataset
722
+ data_transforms:
723
+ - name: CosmosProcessImage
724
+ resize_sizes:
725
+ - '480'
726
+ - '640'
727
+ targets:
728
+ - workspace_rgb
729
+ - additional_data:
730
+ relative_base_value: obs/eef_pose_lowdim
731
+ name: ConvertLowdimRepr
732
+ relative_base_value_idx: '-1'
733
+ targets:
734
+ - action/eef_pose_lowdim
735
+ - name: DeconstructPoseMat
736
+ out_key_prefix: action/eef
737
+ targets:
738
+ - action/eef_pose_lowdim
739
+ - name: DeconstructPoseMat
740
+ out_key_prefix: obs/eef
741
+ targets:
742
+ - obs/eef_pose_lowdim
743
+ - name: RotationMatrixTo6D
744
+ targets:
745
+ - action/eef_rot_lowdim
746
+ - obs/eef_rot_lowdim
747
+ - name: Flatten
748
+ targets:
749
+ - lowdim
750
+ - name: Concat
751
+ out_key: action/lowdim_concat
752
+ targets:
753
+ - action/eef_pos_lowdim
754
+ - action/eef_rot_lowdim
755
+ - action/gripper_action_lowdim
756
+ - name: Concat
757
+ out_key: obs/lowdim_concat
758
+ targets:
759
+ - obs/eef_pos_lowdim
760
+ - obs/eef_rot_lowdim
761
+ - obs/gripper_state_lowdim
762
+ num_val_episodes: '10'
763
+ policy_io:
764
+ action:
765
+ eef_pos_lowdim:
766
+ horizon: '15'
767
+ normalization_type: NormalizationType.VARIANCE
768
+ shift_right_by: '0.2'
769
+ target_frequency: '5'
770
+ target_repr: LieRepr.RELATIVE
771
+ eef_rot_lowdim:
772
+ horizon: '15'
773
+ normalization_type: NormalizationType.IDENTITY
774
+ shift_right_by: '0.2'
775
+ target_frequency: '5'
776
+ target_repr: LieRepr.RELATIVE
777
+ gripper_action_lowdim:
778
+ horizon: '15'
779
+ normalization_type: NormalizationType.VARIANCE
780
+ shift_right_by: '0.2'
781
+ target_frequency: '5'
782
+ target_repr: LieRepr.ABSOLUTE
783
+ workspace_rgb:
784
+ horizon: '56'
785
+ normalization_type: NormalizationType.NONE
786
+ shift_right_by: '0.2'
787
+ target_frequency: '5'
788
+ target_repr: null
789
+ obs:
790
+ eef_pos_lowdim:
791
+ horizon: '1'
792
+ normalization_type: NormalizationType.VARIANCE
793
+ shift_right_by: '0'
794
+ target_frequency: '5'
795
+ target_repr: LieRepr.ABSOLUTE
796
+ eef_rot_lowdim:
797
+ horizon: '1'
798
+ normalization_type: NormalizationType.IDENTITY
799
+ shift_right_by: '0'
800
+ target_frequency: '5'
801
+ target_repr: LieRepr.ABSOLUTE
802
+ gripper_state_lowdim:
803
+ horizon: '1'
804
+ normalization_type: NormalizationType.VARIANCE
805
+ shift_right_by: '0'
806
+ target_frequency: '5'
807
+ target_repr: LieRepr.ABSOLUTE
808
+ language_embedding:
809
+ horizon: '1'
810
+ normalization_type: NormalizationType.NONE
811
+ shift_right_by: '0'
812
+ target_frequency: null
813
+ target_repr: null
814
+ workspace_rgb:
815
+ horizon: '5'
816
+ normalization_type: NormalizationType.NONE
817
+ shift_right_by: '0'
818
+ target_frequency: '5'
819
+ target_repr: null
820
+ seed: '42'
821
+ should_include_padded_tails: 'True'
822
+ source_component_names:
823
+ action/eef_pos_lowdim: eef_pose_lowdim
824
+ action/eef_rot_lowdim: eef_pose_lowdim
825
+ obs/eef_pos_lowdim: eef_pose_lowdim
826
+ obs/eef_rot_lowdim: eef_pose_lowdim
827
+ timestep_anchor: workspace_rgb
828
+ source_component_names:
829
+ action/eef_pos_lowdim: eef_pose_lowdim
830
+ action/eef_rot_lowdim: eef_pose_lowdim
831
+ obs/eef_pos_lowdim: eef_pose_lowdim
832
+ obs/eef_rot_lowdim: eef_pose_lowdim
833
+ policy_io:
834
+ concat_groups:
835
+ action/lowdim_concat:
836
+ - action/eef_pos_lowdim
837
+ - action/eef_rot_lowdim
838
+ - action/gripper_action_lowdim
839
+ obs/lowdim_concat:
840
+ - obs/eef_pos_lowdim
841
+ - obs/eef_rot_lowdim
842
+ - obs/gripper_state_lowdim
843
+ img_resize_sizes:
844
+ - '480'
845
+ - '640'
846
+ policy_io:
847
+ action:
848
+ eef_pos_lowdim:
849
+ horizon: '15'
850
+ normalization_type: NormalizationType.VARIANCE
851
+ shift_right_by: '0.2'
852
+ target_frequency: '5'
853
+ target_repr: LieRepr.RELATIVE
854
+ eef_rot_lowdim:
855
+ horizon: '15'
856
+ normalization_type: NormalizationType.IDENTITY
857
+ shift_right_by: '0.2'
858
+ target_frequency: '5'
859
+ target_repr: LieRepr.RELATIVE
860
+ gripper_action_lowdim:
861
+ horizon: '15'
862
+ normalization_type: NormalizationType.VARIANCE
863
+ shift_right_by: '0.2'
864
+ target_frequency: '5'
865
+ target_repr: LieRepr.ABSOLUTE
866
+ workspace_rgb:
867
+ horizon: '56'
868
+ normalization_type: NormalizationType.NONE
869
+ shift_right_by: '0.2'
870
+ target_frequency: '5'
871
+ target_repr: null
872
+ obs:
873
+ eef_pos_lowdim:
874
+ horizon: '1'
875
+ normalization_type: NormalizationType.VARIANCE
876
+ shift_right_by: '0'
877
+ target_frequency: '5'
878
+ target_repr: LieRepr.ABSOLUTE
879
+ eef_rot_lowdim:
880
+ horizon: '1'
881
+ normalization_type: NormalizationType.IDENTITY
882
+ shift_right_by: '0'
883
+ target_frequency: '5'
884
+ target_repr: LieRepr.ABSOLUTE
885
+ gripper_state_lowdim:
886
+ horizon: '1'
887
+ normalization_type: NormalizationType.VARIANCE
888
+ shift_right_by: '0'
889
+ target_frequency: '5'
890
+ target_repr: LieRepr.ABSOLUTE
891
+ language_embedding:
892
+ horizon: '1'
893
+ normalization_type: NormalizationType.NONE
894
+ shift_right_by: '0'
895
+ target_frequency: null
896
+ target_repr: null
897
+ workspace_rgb:
898
+ horizon: '5'
899
+ normalization_type: NormalizationType.NONE
900
+ shift_right_by: '0'
901
+ target_frequency: '5'
902
+ target_repr: null
903
+ is_train: 'True'
904
+ num_replicas:
905
+ _target_: <function get_data_parallel_world_size at 0x14b0798f4700>
906
+ partial_data_parallel: 'False'
907
+ with_context_parallel: 'False'
908
+ rank:
909
+ _target_: <function get_data_parallel_rank at 0x14b0798f4820>
910
+ partial_data_parallel: 'False'
911
+ with_context_parallel: 'False'
912
+ seed: '0'
913
+ shuffle: 'True'
914
+ shuffle: null
915
+ timeout: '0'
916
+ worker_init_fn: null
917
+ dataloader_val:
918
+ _target_: <class 'torch.utils.data.dataloader.DataLoader'>
919
+ batch_sampler: null
920
+ batch_size: '1'
921
+ collate_fn: null
922
+ dataset:
923
+ _target_: <function get_dataset at 0x14b01ac7ef80>
924
+ data_config:
925
+ dataset:
926
+ data_components:
927
+ eef_pose_lowdim:
928
+ obs_type: ObsType.POSE_MATRIX
929
+ repr: LieRepr.ABSOLUTE
930
+ gripper_action_lowdim:
931
+ obs_type: ObsType.JOINT_POS
932
+ repr: LieRepr.ABSOLUTE
933
+ gripper_state_lowdim:
934
+ obs_type: ObsType.JOINT_POS
935
+ repr: LieRepr.ABSOLUTE
936
+ language_embedding:
937
+ obs_type: ObsType.LANGUAGE_EMBEDDING
938
+ repr: null
939
+ language_instruction:
940
+ obs_type: ObsType.LANGUAGE
941
+ repr: null
942
+ num_conditional_frames:
943
+ obs_type: ObsType.METADATA_LABEL
944
+ repr: null
945
+ workspace_rgb:
946
+ obs_type: ObsType.RGB
947
+ repr: null
948
+ workspace_rgb_embedding:
949
+ obs_type: ObsType.VIDEO_EMBEDDING
950
+ repr: null
951
+ data_transforms:
952
+ - name: CosmosProcessImage
953
+ resize_sizes:
954
+ - '480'
955
+ - '640'
956
+ targets:
957
+ - workspace_rgb
958
+ - additional_data:
959
+ relative_base_value: obs/eef_pose_lowdim
960
+ name: ConvertLowdimRepr
961
+ relative_base_value_idx: '-1'
962
+ targets:
963
+ - action/eef_pose_lowdim
964
+ - name: DeconstructPoseMat
965
+ out_key_prefix: action/eef
966
+ targets:
967
+ - action/eef_pose_lowdim
968
+ - name: DeconstructPoseMat
969
+ out_key_prefix: obs/eef
970
+ targets:
971
+ - obs/eef_pose_lowdim
972
+ - name: RotationMatrixTo6D
973
+ targets:
974
+ - action/eef_rot_lowdim
975
+ - obs/eef_rot_lowdim
976
+ - name: Flatten
977
+ targets:
978
+ - lowdim
979
+ - name: Concat
980
+ out_key: action/lowdim_concat
981
+ targets:
982
+ - action/eef_pos_lowdim
983
+ - action/eef_rot_lowdim
984
+ - action/gripper_action_lowdim
985
+ - name: Concat
986
+ out_key: obs/lowdim_concat
987
+ targets:
988
+ - obs/eef_pos_lowdim
989
+ - obs/eef_rot_lowdim
990
+ - obs/gripper_state_lowdim
991
+ dataset:
992
+ _convert_: partial
993
+ _target_: mimicvideo_runtime.action_dataset.WidowXActionDataset
994
+ data_components:
995
+ eef_pose_lowdim:
996
+ obs_type: ObsType.POSE_MATRIX
997
+ repr: LieRepr.ABSOLUTE
998
+ gripper_action_lowdim:
999
+ obs_type: ObsType.JOINT_POS
1000
+ repr: LieRepr.ABSOLUTE
1001
+ gripper_state_lowdim:
1002
+ obs_type: ObsType.JOINT_POS
1003
+ repr: LieRepr.ABSOLUTE
1004
+ language_embedding:
1005
+ obs_type: ObsType.LANGUAGE_EMBEDDING
1006
+ repr: null
1007
+ language_instruction:
1008
+ obs_type: ObsType.LANGUAGE
1009
+ repr: null
1010
+ num_conditional_frames:
1011
+ obs_type: ObsType.METADATA_LABEL
1012
+ repr: null
1013
+ workspace_rgb:
1014
+ obs_type: ObsType.RGB
1015
+ repr: null
1016
+ workspace_rgb_embedding:
1017
+ obs_type: ObsType.VIDEO_EMBEDDING
1018
+ repr: null
1019
+ data_dir: /cluster/scratch/ehalicki/vam_cross_mimicvideo/datasets/mimicvideo/panda_widowx_level2_widowx_texture_2cam_hstack_widowx_teleop_recording_frame_v1_action/action_dataset
1020
+ data_transforms:
1021
+ - name: CosmosProcessImage
1022
+ resize_sizes:
1023
+ - '480'
1024
+ - '640'
1025
+ targets:
1026
+ - workspace_rgb
1027
+ - additional_data:
1028
+ relative_base_value: obs/eef_pose_lowdim
1029
+ name: ConvertLowdimRepr
1030
+ relative_base_value_idx: '-1'
1031
+ targets:
1032
+ - action/eef_pose_lowdim
1033
+ - name: DeconstructPoseMat
1034
+ out_key_prefix: action/eef
1035
+ targets:
1036
+ - action/eef_pose_lowdim
1037
+ - name: DeconstructPoseMat
1038
+ out_key_prefix: obs/eef
1039
+ targets:
1040
+ - obs/eef_pose_lowdim
1041
+ - name: RotationMatrixTo6D
1042
+ targets:
1043
+ - action/eef_rot_lowdim
1044
+ - obs/eef_rot_lowdim
1045
+ - name: Flatten
1046
+ targets:
1047
+ - lowdim
1048
+ - name: Concat
1049
+ out_key: action/lowdim_concat
1050
+ targets:
1051
+ - action/eef_pos_lowdim
1052
+ - action/eef_rot_lowdim
1053
+ - action/gripper_action_lowdim
1054
+ - name: Concat
1055
+ out_key: obs/lowdim_concat
1056
+ targets:
1057
+ - obs/eef_pos_lowdim
1058
+ - obs/eef_rot_lowdim
1059
+ - obs/gripper_state_lowdim
1060
+ num_val_episodes: '10'
1061
+ policy_io:
1062
+ action:
1063
+ eef_pos_lowdim:
1064
+ horizon: '15'
1065
+ normalization_type: NormalizationType.VARIANCE
1066
+ shift_right_by: '0.2'
1067
+ target_frequency: '5'
1068
+ target_repr: LieRepr.RELATIVE
1069
+ eef_rot_lowdim:
1070
+ horizon: '15'
1071
+ normalization_type: NormalizationType.IDENTITY
1072
+ shift_right_by: '0.2'
1073
+ target_frequency: '5'
1074
+ target_repr: LieRepr.RELATIVE
1075
+ gripper_action_lowdim:
1076
+ horizon: '15'
1077
+ normalization_type: NormalizationType.VARIANCE
1078
+ shift_right_by: '0.2'
1079
+ target_frequency: '5'
1080
+ target_repr: LieRepr.ABSOLUTE
1081
+ workspace_rgb:
1082
+ horizon: '56'
1083
+ normalization_type: NormalizationType.NONE
1084
+ shift_right_by: '0.2'
1085
+ target_frequency: '5'
1086
+ target_repr: null
1087
+ obs:
1088
+ eef_pos_lowdim:
1089
+ horizon: '1'
1090
+ normalization_type: NormalizationType.VARIANCE
1091
+ shift_right_by: '0'
1092
+ target_frequency: '5'
1093
+ target_repr: LieRepr.ABSOLUTE
1094
+ eef_rot_lowdim:
1095
+ horizon: '1'
1096
+ normalization_type: NormalizationType.IDENTITY
1097
+ shift_right_by: '0'
1098
+ target_frequency: '5'
1099
+ target_repr: LieRepr.ABSOLUTE
1100
+ gripper_state_lowdim:
1101
+ horizon: '1'
1102
+ normalization_type: NormalizationType.VARIANCE
1103
+ shift_right_by: '0'
1104
+ target_frequency: '5'
1105
+ target_repr: LieRepr.ABSOLUTE
1106
+ language_embedding:
1107
+ horizon: '1'
1108
+ normalization_type: NormalizationType.NONE
1109
+ shift_right_by: '0'
1110
+ target_frequency: null
1111
+ target_repr: null
1112
+ workspace_rgb:
1113
+ horizon: '5'
1114
+ normalization_type: NormalizationType.NONE
1115
+ shift_right_by: '0'
1116
+ target_frequency: '5'
1117
+ target_repr: null
1118
+ seed: '42'
1119
+ should_include_padded_tails: 'True'
1120
+ source_component_names:
1121
+ action/eef_pos_lowdim: eef_pose_lowdim
1122
+ action/eef_rot_lowdim: eef_pose_lowdim
1123
+ obs/eef_pos_lowdim: eef_pose_lowdim
1124
+ obs/eef_rot_lowdim: eef_pose_lowdim
1125
+ timestep_anchor: workspace_rgb
1126
+ source_component_names:
1127
+ action/eef_pos_lowdim: eef_pose_lowdim
1128
+ action/eef_rot_lowdim: eef_pose_lowdim
1129
+ obs/eef_pos_lowdim: eef_pose_lowdim
1130
+ obs/eef_rot_lowdim: eef_pose_lowdim
1131
+ policy_io:
1132
+ concat_groups:
1133
+ action/lowdim_concat:
1134
+ - action/eef_pos_lowdim
1135
+ - action/eef_rot_lowdim
1136
+ - action/gripper_action_lowdim
1137
+ obs/lowdim_concat:
1138
+ - obs/eef_pos_lowdim
1139
+ - obs/eef_rot_lowdim
1140
+ - obs/gripper_state_lowdim
1141
+ img_resize_sizes:
1142
+ - '480'
1143
+ - '640'
1144
+ policy_io:
1145
+ action:
1146
+ eef_pos_lowdim:
1147
+ horizon: '15'
1148
+ normalization_type: NormalizationType.VARIANCE
1149
+ shift_right_by: '0.2'
1150
+ target_frequency: '5'
1151
+ target_repr: LieRepr.RELATIVE
1152
+ eef_rot_lowdim:
1153
+ horizon: '15'
1154
+ normalization_type: NormalizationType.IDENTITY
1155
+ shift_right_by: '0.2'
1156
+ target_frequency: '5'
1157
+ target_repr: LieRepr.RELATIVE
1158
+ gripper_action_lowdim:
1159
+ horizon: '15'
1160
+ normalization_type: NormalizationType.VARIANCE
1161
+ shift_right_by: '0.2'
1162
+ target_frequency: '5'
1163
+ target_repr: LieRepr.ABSOLUTE
1164
+ workspace_rgb:
1165
+ horizon: '56'
1166
+ normalization_type: NormalizationType.NONE
1167
+ shift_right_by: '0.2'
1168
+ target_frequency: '5'
1169
+ target_repr: null
1170
+ obs:
1171
+ eef_pos_lowdim:
1172
+ horizon: '1'
1173
+ normalization_type: NormalizationType.VARIANCE
1174
+ shift_right_by: '0'
1175
+ target_frequency: '5'
1176
+ target_repr: LieRepr.ABSOLUTE
1177
+ eef_rot_lowdim:
1178
+ horizon: '1'
1179
+ normalization_type: NormalizationType.IDENTITY
1180
+ shift_right_by: '0'
1181
+ target_frequency: '5'
1182
+ target_repr: LieRepr.ABSOLUTE
1183
+ gripper_state_lowdim:
1184
+ horizon: '1'
1185
+ normalization_type: NormalizationType.VARIANCE
1186
+ shift_right_by: '0'
1187
+ target_frequency: '5'
1188
+ target_repr: LieRepr.ABSOLUTE
1189
+ language_embedding:
1190
+ horizon: '1'
1191
+ normalization_type: NormalizationType.NONE
1192
+ shift_right_by: '0'
1193
+ target_frequency: null
1194
+ target_repr: null
1195
+ workspace_rgb:
1196
+ horizon: '5'
1197
+ normalization_type: NormalizationType.NONE
1198
+ shift_right_by: '0'
1199
+ target_frequency: '5'
1200
+ target_repr: null
1201
+ is_train: 'False'
1202
+ drop_last: 'False'
1203
+ generator: null
1204
+ in_order: 'True'
1205
+ multiprocessing_context: null
1206
+ num_workers: '0'
1207
+ persistent_workers: 'False'
1208
+ pin_memory: 'False'
1209
+ pin_memory_device: ''
1210
+ prefetch_factor: null
1211
+ sampler:
1212
+ _target_: <class 'cosmos_predict2.data.resumable_sampler.ResumableDistributedSampler'>
1213
+ dataset:
1214
+ _target_: <function get_dataset at 0x14b01ac7ef80>
1215
+ data_config:
1216
+ dataset:
1217
+ data_components:
1218
+ eef_pose_lowdim:
1219
+ obs_type: ObsType.POSE_MATRIX
1220
+ repr: LieRepr.ABSOLUTE
1221
+ gripper_action_lowdim:
1222
+ obs_type: ObsType.JOINT_POS
1223
+ repr: LieRepr.ABSOLUTE
1224
+ gripper_state_lowdim:
1225
+ obs_type: ObsType.JOINT_POS
1226
+ repr: LieRepr.ABSOLUTE
1227
+ language_embedding:
1228
+ obs_type: ObsType.LANGUAGE_EMBEDDING
1229
+ repr: null
1230
+ language_instruction:
1231
+ obs_type: ObsType.LANGUAGE
1232
+ repr: null
1233
+ num_conditional_frames:
1234
+ obs_type: ObsType.METADATA_LABEL
1235
+ repr: null
1236
+ workspace_rgb:
1237
+ obs_type: ObsType.RGB
1238
+ repr: null
1239
+ workspace_rgb_embedding:
1240
+ obs_type: ObsType.VIDEO_EMBEDDING
1241
+ repr: null
1242
+ data_transforms:
1243
+ - name: CosmosProcessImage
1244
+ resize_sizes:
1245
+ - '480'
1246
+ - '640'
1247
+ targets:
1248
+ - workspace_rgb
1249
+ - additional_data:
1250
+ relative_base_value: obs/eef_pose_lowdim
1251
+ name: ConvertLowdimRepr
1252
+ relative_base_value_idx: '-1'
1253
+ targets:
1254
+ - action/eef_pose_lowdim
1255
+ - name: DeconstructPoseMat
1256
+ out_key_prefix: action/eef
1257
+ targets:
1258
+ - action/eef_pose_lowdim
1259
+ - name: DeconstructPoseMat
1260
+ out_key_prefix: obs/eef
1261
+ targets:
1262
+ - obs/eef_pose_lowdim
1263
+ - name: RotationMatrixTo6D
1264
+ targets:
1265
+ - action/eef_rot_lowdim
1266
+ - obs/eef_rot_lowdim
1267
+ - name: Flatten
1268
+ targets:
1269
+ - lowdim
1270
+ - name: Concat
1271
+ out_key: action/lowdim_concat
1272
+ targets:
1273
+ - action/eef_pos_lowdim
1274
+ - action/eef_rot_lowdim
1275
+ - action/gripper_action_lowdim
1276
+ - name: Concat
1277
+ out_key: obs/lowdim_concat
1278
+ targets:
1279
+ - obs/eef_pos_lowdim
1280
+ - obs/eef_rot_lowdim
1281
+ - obs/gripper_state_lowdim
1282
+ dataset:
1283
+ _convert_: partial
1284
+ _target_: mimicvideo_runtime.action_dataset.WidowXActionDataset
1285
+ data_components:
1286
+ eef_pose_lowdim:
1287
+ obs_type: ObsType.POSE_MATRIX
1288
+ repr: LieRepr.ABSOLUTE
1289
+ gripper_action_lowdim:
1290
+ obs_type: ObsType.JOINT_POS
1291
+ repr: LieRepr.ABSOLUTE
1292
+ gripper_state_lowdim:
1293
+ obs_type: ObsType.JOINT_POS
1294
+ repr: LieRepr.ABSOLUTE
1295
+ language_embedding:
1296
+ obs_type: ObsType.LANGUAGE_EMBEDDING
1297
+ repr: null
1298
+ language_instruction:
1299
+ obs_type: ObsType.LANGUAGE
1300
+ repr: null
1301
+ num_conditional_frames:
1302
+ obs_type: ObsType.METADATA_LABEL
1303
+ repr: null
1304
+ workspace_rgb:
1305
+ obs_type: ObsType.RGB
1306
+ repr: null
1307
+ workspace_rgb_embedding:
1308
+ obs_type: ObsType.VIDEO_EMBEDDING
1309
+ repr: null
1310
+ data_dir: /cluster/scratch/ehalicki/vam_cross_mimicvideo/datasets/mimicvideo/panda_widowx_level2_widowx_texture_2cam_hstack_widowx_teleop_recording_frame_v1_action/action_dataset
1311
+ data_transforms:
1312
+ - name: CosmosProcessImage
1313
+ resize_sizes:
1314
+ - '480'
1315
+ - '640'
1316
+ targets:
1317
+ - workspace_rgb
1318
+ - additional_data:
1319
+ relative_base_value: obs/eef_pose_lowdim
1320
+ name: ConvertLowdimRepr
1321
+ relative_base_value_idx: '-1'
1322
+ targets:
1323
+ - action/eef_pose_lowdim
1324
+ - name: DeconstructPoseMat
1325
+ out_key_prefix: action/eef
1326
+ targets:
1327
+ - action/eef_pose_lowdim
1328
+ - name: DeconstructPoseMat
1329
+ out_key_prefix: obs/eef
1330
+ targets:
1331
+ - obs/eef_pose_lowdim
1332
+ - name: RotationMatrixTo6D
1333
+ targets:
1334
+ - action/eef_rot_lowdim
1335
+ - obs/eef_rot_lowdim
1336
+ - name: Flatten
1337
+ targets:
1338
+ - lowdim
1339
+ - name: Concat
1340
+ out_key: action/lowdim_concat
1341
+ targets:
1342
+ - action/eef_pos_lowdim
1343
+ - action/eef_rot_lowdim
1344
+ - action/gripper_action_lowdim
1345
+ - name: Concat
1346
+ out_key: obs/lowdim_concat
1347
+ targets:
1348
+ - obs/eef_pos_lowdim
1349
+ - obs/eef_rot_lowdim
1350
+ - obs/gripper_state_lowdim
1351
+ num_val_episodes: '10'
1352
+ policy_io:
1353
+ action:
1354
+ eef_pos_lowdim:
1355
+ horizon: '15'
1356
+ normalization_type: NormalizationType.VARIANCE
1357
+ shift_right_by: '0.2'
1358
+ target_frequency: '5'
1359
+ target_repr: LieRepr.RELATIVE
1360
+ eef_rot_lowdim:
1361
+ horizon: '15'
1362
+ normalization_type: NormalizationType.IDENTITY
1363
+ shift_right_by: '0.2'
1364
+ target_frequency: '5'
1365
+ target_repr: LieRepr.RELATIVE
1366
+ gripper_action_lowdim:
1367
+ horizon: '15'
1368
+ normalization_type: NormalizationType.VARIANCE
1369
+ shift_right_by: '0.2'
1370
+ target_frequency: '5'
1371
+ target_repr: LieRepr.ABSOLUTE
1372
+ workspace_rgb:
1373
+ horizon: '56'
1374
+ normalization_type: NormalizationType.NONE
1375
+ shift_right_by: '0.2'
1376
+ target_frequency: '5'
1377
+ target_repr: null
1378
+ obs:
1379
+ eef_pos_lowdim:
1380
+ horizon: '1'
1381
+ normalization_type: NormalizationType.VARIANCE
1382
+ shift_right_by: '0'
1383
+ target_frequency: '5'
1384
+ target_repr: LieRepr.ABSOLUTE
1385
+ eef_rot_lowdim:
1386
+ horizon: '1'
1387
+ normalization_type: NormalizationType.IDENTITY
1388
+ shift_right_by: '0'
1389
+ target_frequency: '5'
1390
+ target_repr: LieRepr.ABSOLUTE
1391
+ gripper_state_lowdim:
1392
+ horizon: '1'
1393
+ normalization_type: NormalizationType.VARIANCE
1394
+ shift_right_by: '0'
1395
+ target_frequency: '5'
1396
+ target_repr: LieRepr.ABSOLUTE
1397
+ language_embedding:
1398
+ horizon: '1'
1399
+ normalization_type: NormalizationType.NONE
1400
+ shift_right_by: '0'
1401
+ target_frequency: null
1402
+ target_repr: null
1403
+ workspace_rgb:
1404
+ horizon: '5'
1405
+ normalization_type: NormalizationType.NONE
1406
+ shift_right_by: '0'
1407
+ target_frequency: '5'
1408
+ target_repr: null
1409
+ seed: '42'
1410
+ should_include_padded_tails: 'True'
1411
+ source_component_names:
1412
+ action/eef_pos_lowdim: eef_pose_lowdim
1413
+ action/eef_rot_lowdim: eef_pose_lowdim
1414
+ obs/eef_pos_lowdim: eef_pose_lowdim
1415
+ obs/eef_rot_lowdim: eef_pose_lowdim
1416
+ timestep_anchor: workspace_rgb
1417
+ source_component_names:
1418
+ action/eef_pos_lowdim: eef_pose_lowdim
1419
+ action/eef_rot_lowdim: eef_pose_lowdim
1420
+ obs/eef_pos_lowdim: eef_pose_lowdim
1421
+ obs/eef_rot_lowdim: eef_pose_lowdim
1422
+ policy_io:
1423
+ concat_groups:
1424
+ action/lowdim_concat:
1425
+ - action/eef_pos_lowdim
1426
+ - action/eef_rot_lowdim
1427
+ - action/gripper_action_lowdim
1428
+ obs/lowdim_concat:
1429
+ - obs/eef_pos_lowdim
1430
+ - obs/eef_rot_lowdim
1431
+ - obs/gripper_state_lowdim
1432
+ img_resize_sizes:
1433
+ - '480'
1434
+ - '640'
1435
+ policy_io:
1436
+ action:
1437
+ eef_pos_lowdim:
1438
+ horizon: '15'
1439
+ normalization_type: NormalizationType.VARIANCE
1440
+ shift_right_by: '0.2'
1441
+ target_frequency: '5'
1442
+ target_repr: LieRepr.RELATIVE
1443
+ eef_rot_lowdim:
1444
+ horizon: '15'
1445
+ normalization_type: NormalizationType.IDENTITY
1446
+ shift_right_by: '0.2'
1447
+ target_frequency: '5'
1448
+ target_repr: LieRepr.RELATIVE
1449
+ gripper_action_lowdim:
1450
+ horizon: '15'
1451
+ normalization_type: NormalizationType.VARIANCE
1452
+ shift_right_by: '0.2'
1453
+ target_frequency: '5'
1454
+ target_repr: LieRepr.ABSOLUTE
1455
+ workspace_rgb:
1456
+ horizon: '56'
1457
+ normalization_type: NormalizationType.NONE
1458
+ shift_right_by: '0.2'
1459
+ target_frequency: '5'
1460
+ target_repr: null
1461
+ obs:
1462
+ eef_pos_lowdim:
1463
+ horizon: '1'
1464
+ normalization_type: NormalizationType.VARIANCE
1465
+ shift_right_by: '0'
1466
+ target_frequency: '5'
1467
+ target_repr: LieRepr.ABSOLUTE
1468
+ eef_rot_lowdim:
1469
+ horizon: '1'
1470
+ normalization_type: NormalizationType.IDENTITY
1471
+ shift_right_by: '0'
1472
+ target_frequency: '5'
1473
+ target_repr: LieRepr.ABSOLUTE
1474
+ gripper_state_lowdim:
1475
+ horizon: '1'
1476
+ normalization_type: NormalizationType.VARIANCE
1477
+ shift_right_by: '0'
1478
+ target_frequency: '5'
1479
+ target_repr: LieRepr.ABSOLUTE
1480
+ language_embedding:
1481
+ horizon: '1'
1482
+ normalization_type: NormalizationType.NONE
1483
+ shift_right_by: '0'
1484
+ target_frequency: null
1485
+ target_repr: null
1486
+ workspace_rgb:
1487
+ horizon: '5'
1488
+ normalization_type: NormalizationType.NONE
1489
+ shift_right_by: '0'
1490
+ target_frequency: '5'
1491
+ target_repr: null
1492
+ is_train: 'False'
1493
+ num_replicas:
1494
+ _target_: <function get_data_parallel_world_size at 0x14b0798f4700>
1495
+ partial_data_parallel: 'False'
1496
+ with_context_parallel: 'False'
1497
+ rank:
1498
+ _target_: <function get_data_parallel_rank at 0x14b0798f4820>
1499
+ partial_data_parallel: 'False'
1500
+ with_context_parallel: 'False'
1501
+ seed: '0'
1502
+ shuffle: 'False'
1503
+ shuffle: null
1504
+ timeout: '0'
1505
+ worker_init_fn: null
1506
+ defaults:
1507
+ - _self_
1508
+ - data_config: null
1509
+ - video_dataset_train: null
1510
+ - video_dataset_val: null
1511
+ - dataloader_train: null
1512
+ - dataloader_val: null
1513
+ - action_pipe: null
1514
+ - optimizer: fusedadamw
1515
+ - scheduler: constant
1516
+ - model: null
1517
+ - callbacks:
1518
+ - basic
1519
+ - net: null
1520
+ - ema: null
1521
+ - checkpoint: null
1522
+ - ckpt_type: null
1523
+ - experiment: null
1524
+ job:
1525
+ group: world2action
1526
+ name: w2a_panda_widowx_level2_widowx_texture_2cam_hstack_action_iter2374_videolora_iter200_widowx_teleop_recording_frame_v1_full
1527
+ project: vam_cross
1528
+ model:
1529
+ _recursive_: 'False'
1530
+ _target_: <class 'mimicvideo_runtime.action_model.WidowXWorld2ActionModel'>
1531
+ config:
1532
+ _target_: mimicvideo_runtime.action_model.WidowXWorld2ActionModelConfig
1533
+ action_dit_path: /cluster/scratch/ehalicki/vam_cross_mimicvideo/checkpoints/initial_action_decoders/dreamdifferent--vam-cross-target-widowx250-native-2cam-action-decoder-93750cc/checkpoints/model/iter_000002374.pt
1534
+ data_config:
1535
+ dataset:
1536
+ data_components:
1537
+ eef_pose_lowdim:
1538
+ obs_type: ObsType.POSE_MATRIX
1539
+ repr: LieRepr.ABSOLUTE
1540
+ gripper_action_lowdim:
1541
+ obs_type: ObsType.JOINT_POS
1542
+ repr: LieRepr.ABSOLUTE
1543
+ gripper_state_lowdim:
1544
+ obs_type: ObsType.JOINT_POS
1545
+ repr: LieRepr.ABSOLUTE
1546
+ language_embedding:
1547
+ obs_type: ObsType.LANGUAGE_EMBEDDING
1548
+ repr: null
1549
+ language_instruction:
1550
+ obs_type: ObsType.LANGUAGE
1551
+ repr: null
1552
+ num_conditional_frames:
1553
+ obs_type: ObsType.METADATA_LABEL
1554
+ repr: null
1555
+ workspace_rgb:
1556
+ obs_type: ObsType.RGB
1557
+ repr: null
1558
+ workspace_rgb_embedding:
1559
+ obs_type: ObsType.VIDEO_EMBEDDING
1560
+ repr: null
1561
+ data_transforms:
1562
+ - name: CosmosProcessImage
1563
+ resize_sizes:
1564
+ - '480'
1565
+ - '640'
1566
+ targets:
1567
+ - workspace_rgb
1568
+ - additional_data:
1569
+ relative_base_value: obs/eef_pose_lowdim
1570
+ name: ConvertLowdimRepr
1571
+ relative_base_value_idx: '-1'
1572
+ targets:
1573
+ - action/eef_pose_lowdim
1574
+ - name: DeconstructPoseMat
1575
+ out_key_prefix: action/eef
1576
+ targets:
1577
+ - action/eef_pose_lowdim
1578
+ - name: DeconstructPoseMat
1579
+ out_key_prefix: obs/eef
1580
+ targets:
1581
+ - obs/eef_pose_lowdim
1582
+ - name: RotationMatrixTo6D
1583
+ targets:
1584
+ - action/eef_rot_lowdim
1585
+ - obs/eef_rot_lowdim
1586
+ - name: Flatten
1587
+ targets:
1588
+ - lowdim
1589
+ - name: Concat
1590
+ out_key: action/lowdim_concat
1591
+ targets:
1592
+ - action/eef_pos_lowdim
1593
+ - action/eef_rot_lowdim
1594
+ - action/gripper_action_lowdim
1595
+ - name: Concat
1596
+ out_key: obs/lowdim_concat
1597
+ targets:
1598
+ - obs/eef_pos_lowdim
1599
+ - obs/eef_rot_lowdim
1600
+ - obs/gripper_state_lowdim
1601
+ dataset:
1602
+ _convert_: partial
1603
+ _target_: mimicvideo_runtime.action_dataset.WidowXActionDataset
1604
+ data_components:
1605
+ eef_pose_lowdim:
1606
+ obs_type: ObsType.POSE_MATRIX
1607
+ repr: LieRepr.ABSOLUTE
1608
+ gripper_action_lowdim:
1609
+ obs_type: ObsType.JOINT_POS
1610
+ repr: LieRepr.ABSOLUTE
1611
+ gripper_state_lowdim:
1612
+ obs_type: ObsType.JOINT_POS
1613
+ repr: LieRepr.ABSOLUTE
1614
+ language_embedding:
1615
+ obs_type: ObsType.LANGUAGE_EMBEDDING
1616
+ repr: null
1617
+ language_instruction:
1618
+ obs_type: ObsType.LANGUAGE
1619
+ repr: null
1620
+ num_conditional_frames:
1621
+ obs_type: ObsType.METADATA_LABEL
1622
+ repr: null
1623
+ workspace_rgb:
1624
+ obs_type: ObsType.RGB
1625
+ repr: null
1626
+ workspace_rgb_embedding:
1627
+ obs_type: ObsType.VIDEO_EMBEDDING
1628
+ repr: null
1629
+ data_dir: /cluster/scratch/ehalicki/vam_cross_mimicvideo/datasets/mimicvideo/panda_widowx_level2_widowx_texture_2cam_hstack_widowx_teleop_recording_frame_v1_action/action_dataset
1630
+ data_transforms:
1631
+ - name: CosmosProcessImage
1632
+ resize_sizes:
1633
+ - '480'
1634
+ - '640'
1635
+ targets:
1636
+ - workspace_rgb
1637
+ - additional_data:
1638
+ relative_base_value: obs/eef_pose_lowdim
1639
+ name: ConvertLowdimRepr
1640
+ relative_base_value_idx: '-1'
1641
+ targets:
1642
+ - action/eef_pose_lowdim
1643
+ - name: DeconstructPoseMat
1644
+ out_key_prefix: action/eef
1645
+ targets:
1646
+ - action/eef_pose_lowdim
1647
+ - name: DeconstructPoseMat
1648
+ out_key_prefix: obs/eef
1649
+ targets:
1650
+ - obs/eef_pose_lowdim
1651
+ - name: RotationMatrixTo6D
1652
+ targets:
1653
+ - action/eef_rot_lowdim
1654
+ - obs/eef_rot_lowdim
1655
+ - name: Flatten
1656
+ targets:
1657
+ - lowdim
1658
+ - name: Concat
1659
+ out_key: action/lowdim_concat
1660
+ targets:
1661
+ - action/eef_pos_lowdim
1662
+ - action/eef_rot_lowdim
1663
+ - action/gripper_action_lowdim
1664
+ - name: Concat
1665
+ out_key: obs/lowdim_concat
1666
+ targets:
1667
+ - obs/eef_pos_lowdim
1668
+ - obs/eef_rot_lowdim
1669
+ - obs/gripper_state_lowdim
1670
+ num_val_episodes: '10'
1671
+ policy_io:
1672
+ action:
1673
+ eef_pos_lowdim:
1674
+ horizon: '15'
1675
+ normalization_type: NormalizationType.VARIANCE
1676
+ shift_right_by: '0.2'
1677
+ target_frequency: '5'
1678
+ target_repr: LieRepr.RELATIVE
1679
+ eef_rot_lowdim:
1680
+ horizon: '15'
1681
+ normalization_type: NormalizationType.IDENTITY
1682
+ shift_right_by: '0.2'
1683
+ target_frequency: '5'
1684
+ target_repr: LieRepr.RELATIVE
1685
+ gripper_action_lowdim:
1686
+ horizon: '15'
1687
+ normalization_type: NormalizationType.VARIANCE
1688
+ shift_right_by: '0.2'
1689
+ target_frequency: '5'
1690
+ target_repr: LieRepr.ABSOLUTE
1691
+ workspace_rgb:
1692
+ horizon: '56'
1693
+ normalization_type: NormalizationType.NONE
1694
+ shift_right_by: '0.2'
1695
+ target_frequency: '5'
1696
+ target_repr: null
1697
+ obs:
1698
+ eef_pos_lowdim:
1699
+ horizon: '1'
1700
+ normalization_type: NormalizationType.VARIANCE
1701
+ shift_right_by: '0'
1702
+ target_frequency: '5'
1703
+ target_repr: LieRepr.ABSOLUTE
1704
+ eef_rot_lowdim:
1705
+ horizon: '1'
1706
+ normalization_type: NormalizationType.IDENTITY
1707
+ shift_right_by: '0'
1708
+ target_frequency: '5'
1709
+ target_repr: LieRepr.ABSOLUTE
1710
+ gripper_state_lowdim:
1711
+ horizon: '1'
1712
+ normalization_type: NormalizationType.VARIANCE
1713
+ shift_right_by: '0'
1714
+ target_frequency: '5'
1715
+ target_repr: LieRepr.ABSOLUTE
1716
+ language_embedding:
1717
+ horizon: '1'
1718
+ normalization_type: NormalizationType.NONE
1719
+ shift_right_by: '0'
1720
+ target_frequency: null
1721
+ target_repr: null
1722
+ workspace_rgb:
1723
+ horizon: '5'
1724
+ normalization_type: NormalizationType.NONE
1725
+ shift_right_by: '0'
1726
+ target_frequency: '5'
1727
+ target_repr: null
1728
+ seed: '42'
1729
+ should_include_padded_tails: 'True'
1730
+ source_component_names:
1731
+ action/eef_pos_lowdim: eef_pose_lowdim
1732
+ action/eef_rot_lowdim: eef_pose_lowdim
1733
+ obs/eef_pos_lowdim: eef_pose_lowdim
1734
+ obs/eef_rot_lowdim: eef_pose_lowdim
1735
+ timestep_anchor: workspace_rgb
1736
+ source_component_names:
1737
+ action/eef_pos_lowdim: eef_pose_lowdim
1738
+ action/eef_rot_lowdim: eef_pose_lowdim
1739
+ obs/eef_pos_lowdim: eef_pose_lowdim
1740
+ obs/eef_rot_lowdim: eef_pose_lowdim
1741
+ policy_io:
1742
+ concat_groups:
1743
+ action/lowdim_concat:
1744
+ - action/eef_pos_lowdim
1745
+ - action/eef_rot_lowdim
1746
+ - action/gripper_action_lowdim
1747
+ obs/lowdim_concat:
1748
+ - obs/eef_pos_lowdim
1749
+ - obs/eef_rot_lowdim
1750
+ - obs/gripper_state_lowdim
1751
+ img_resize_sizes:
1752
+ - '480'
1753
+ - '640'
1754
+ policy_io:
1755
+ action:
1756
+ eef_pos_lowdim:
1757
+ horizon: '15'
1758
+ normalization_type: NormalizationType.VARIANCE
1759
+ shift_right_by: '0.2'
1760
+ target_frequency: '5'
1761
+ target_repr: LieRepr.RELATIVE
1762
+ eef_rot_lowdim:
1763
+ horizon: '15'
1764
+ normalization_type: NormalizationType.IDENTITY
1765
+ shift_right_by: '0.2'
1766
+ target_frequency: '5'
1767
+ target_repr: LieRepr.RELATIVE
1768
+ gripper_action_lowdim:
1769
+ horizon: '15'
1770
+ normalization_type: NormalizationType.VARIANCE
1771
+ shift_right_by: '0.2'
1772
+ target_frequency: '5'
1773
+ target_repr: LieRepr.ABSOLUTE
1774
+ workspace_rgb:
1775
+ horizon: '56'
1776
+ normalization_type: NormalizationType.NONE
1777
+ shift_right_by: '0.2'
1778
+ target_frequency: '5'
1779
+ target_repr: null
1780
+ obs:
1781
+ eef_pos_lowdim:
1782
+ horizon: '1'
1783
+ normalization_type: NormalizationType.VARIANCE
1784
+ shift_right_by: '0'
1785
+ target_frequency: '5'
1786
+ target_repr: LieRepr.ABSOLUTE
1787
+ eef_rot_lowdim:
1788
+ horizon: '1'
1789
+ normalization_type: NormalizationType.IDENTITY
1790
+ shift_right_by: '0'
1791
+ target_frequency: '5'
1792
+ target_repr: LieRepr.ABSOLUTE
1793
+ gripper_state_lowdim:
1794
+ horizon: '1'
1795
+ normalization_type: NormalizationType.VARIANCE
1796
+ shift_right_by: '0'
1797
+ target_frequency: '5'
1798
+ target_repr: LieRepr.ABSOLUTE
1799
+ language_embedding:
1800
+ horizon: '1'
1801
+ normalization_type: NormalizationType.NONE
1802
+ shift_right_by: '0'
1803
+ target_frequency: null
1804
+ target_repr: null
1805
+ workspace_rgb:
1806
+ horizon: '5'
1807
+ normalization_type: NormalizationType.NONE
1808
+ shift_right_by: '0'
1809
+ target_frequency: '5'
1810
+ target_repr: null
1811
+ ema:
1812
+ enabled: false
1813
+ iteration_shift: 0
1814
+ rate: 0.1
1815
+ fsdp_shard_size: '0'
1816
+ init_lora_weights: 'True'
1817
+ lora_alpha: '16'
1818
+ lora_rank: '16'
1819
+ lora_target_modules: q_proj,k_proj,v_proj,output_proj,mlp.layer1,mlp.layer2
1820
+ loss_reduce: mean
1821
+ loss_scale: '10.0'
1822
+ pipe_config:
1823
+ _target_: cosmos_predict2.configs.config_world2action.World2ActionPipelineConfig
1824
+ ema:
1825
+ enabled: false
1826
+ iteration_shift: 0
1827
+ rate: 0.1
1828
+ net:
1829
+ _target_: <class 'cosmos_predict2.models.world2action_dit.World2ActionDIT'>
1830
+ adaln_lora_dim: '128'
1831
+ atten_backend: flash_attn_no_cp
1832
+ crossattn_emb_channels: '2048'
1833
+ in_channels: '10'
1834
+ max_horizon: '16'
1835
+ mlp_ratio: '4.0'
1836
+ model_channels: '1024'
1837
+ num_blocks: '24'
1838
+ num_heads: '8'
1839
+ out_channels: '10'
1840
+ pair_timestep_feature_rank: '1024'
1841
+ sac_config:
1842
+ every_n_blocks: 1
1843
+ mode: none
1844
+ use_adaln_lora: 'True'
1845
+ precision: bfloat16
1846
+ scheduler:
1847
+ alpha: 1.0
1848
+ beta: 1.0
1849
+ num_denoising_steps: 10
1850
+ xattn_layer_idx: '20'
1851
+ precision: bfloat16
1852
+ train_architecture: base
1853
+ video_dit_path: /cluster/scratch/ehalicki/vam_cross_mimicvideo/initial_backbones/dreamdifferent--widowx250-video-fused-f0cea76/checkpoints/video_backbone/iter_000001060_fused.pt
1854
+ video_lora_alpha: '32'
1855
+ video_lora_dit_path: /cluster/scratch/ehalicki/vam_cross_mimicvideo/checkpoints/selected_video_loras/dreamdifferent--vam-cross-level2-panda-widowx-widowx-texture-video-lora-iter200-5a22b23/checkpoints/model/iter_000000200.pt
1856
+ video_lora_rank: '256'
1857
+ video_lora_target_modules: q_proj,k_proj,v_proj,output_proj,x_embedder.proj.1,linear_1,linear_2,mlp.layer1,mlp.layer2
1858
+ video_pipe_config:
1859
+ adjust_video_noise: true
1860
+ conditioner:
1861
+ _target_: <class 'cosmos_predict2.conditioner.VideoConditioner'>
1862
+ fps:
1863
+ _target_: <class 'cosmos_predict2.conditioner.ReMapkey'>
1864
+ dropout_rate: '0.0'
1865
+ dtype: null
1866
+ input_key: fps
1867
+ output_key: fps
1868
+ padding_mask:
1869
+ _target_: <class 'cosmos_predict2.conditioner.ReMapkey'>
1870
+ dropout_rate: '0.0'
1871
+ dtype: null
1872
+ input_key: padding_mask
1873
+ output_key: padding_mask
1874
+ text:
1875
+ _target_: <class 'cosmos_predict2.conditioner.TextAttr'>
1876
+ dropout_rate: '0.0'
1877
+ input_key:
1878
+ - obs/language_embedding
1879
+ use_video_condition:
1880
+ _target_: <class 'cosmos_predict2.conditioner.BooleanFlag'>
1881
+ dropout_rate: '0.0'
1882
+ input_key: fps
1883
+ output_key: use_video_condition
1884
+ conditioning_strategy: frame_replace
1885
+ ema:
1886
+ _target_: cosmos_predict2.configs.defaults.ema.EMAConfig
1887
+ enabled: 'False'
1888
+ iteration_shift: '0'
1889
+ rate: '0.1'
1890
+ guardrail_config:
1891
+ checkpoint_dir: /cluster/scratch/ehalicki/vam_cross_mimicvideo/checkpoints
1892
+ enabled: false
1893
+ offload_model_to_cpu: true
1894
+ input_image_key: images
1895
+ input_video_key: video
1896
+ max_num_conditional_frames: 2
1897
+ min_num_conditional_frames: 1
1898
+ net:
1899
+ _target_: <class 'cosmos_predict2.models.video2world_dit.MinimalV1LVGDiT'>
1900
+ adaln_lora_dim: '256'
1901
+ atten_backend: minimal_a2a
1902
+ concat_padding_mask: 'True'
1903
+ extra_per_block_abs_pos_emb: 'False'
1904
+ in_channels: '16'
1905
+ max_frames: '128'
1906
+ max_img_h: '240'
1907
+ max_img_w: '240'
1908
+ model_channels: '2048'
1909
+ num_blocks: '28'
1910
+ num_heads: '16'
1911
+ out_channels: '16'
1912
+ patch_spatial: '2'
1913
+ patch_temporal: '1'
1914
+ pos_emb_cls: rope3d
1915
+ pos_emb_interpolation: crop
1916
+ pos_emb_learnable: 'True'
1917
+ rope_enable_fps_modulation: 'False'
1918
+ rope_h_extrapolation_ratio: '3.0'
1919
+ rope_t_extrapolation_ratio: '1.0'
1920
+ rope_w_extrapolation_ratio: '3.0'
1921
+ sac_config:
1922
+ _target_: cosmos_predict2.models.text2image_dit.SACConfig
1923
+ every_n_blocks: '1'
1924
+ mode: predict2_2b_720
1925
+ use_adaln_lora: 'True'
1926
+ precision: bfloat16
1927
+ rectified_flow_loss_weight_uniform: true
1928
+ rectified_flow_t_scaling_factor: 1.0
1929
+ resize_online: false
1930
+ resolution: '480'
1931
+ sigma_conditional: 0.0001
1932
+ sigma_data: 1.0
1933
+ state_ch: 16
1934
+ state_t: 16
1935
+ text_encoder:
1936
+ cls: !!python/object/apply:imaginaire.constants.TextEncoderClass
1937
+ - t5
1938
+ t5:
1939
+ ckpt_path: /cluster/scratch/ehalicki/vam_cross_mimicvideo/checkpoints/text_encoder/t5-11b
1940
+ embed_dim: 1024
1941
+ num_tokens: 512
1942
+ timestamps:
1943
+ is_forward: false
1944
+ nfe: 35
1945
+ order: 7.0
1946
+ t_max: 80.0
1947
+ t_min: 0.002
1948
+ tokenizer:
1949
+ _target_: <class 'cosmos_predict2.tokenizers.tokenizer.TokenizerInterface'>
1950
+ chunk_duration: '81'
1951
+ load_mean_std: 'False'
1952
+ name: tokenizer
1953
+ temporal_window: '16'
1954
+ vae_pth: /cluster/scratch/ehalicki/vam_cross_mimicvideo/checkpoints/video_backbone//tokenizer/tokenizer.pth
1955
+ model_parallel:
1956
+ _cpu_offloading_context: null
1957
+ async_tensor_model_parallel_allreduce: false
1958
+ autocast_dtype: torch.float32
1959
+ barrier_with_L1_time: true
1960
+ batch_p2p_comm: true
1961
+ batch_p2p_sync: true
1962
+ bf16: false
1963
+ context_parallel_size: 1
1964
+ cpu_offloading: false
1965
+ cpu_offloading_activations: false
1966
+ cpu_offloading_num_layers: 0
1967
+ cpu_offloading_weights: false
1968
+ cross_entropy_fusion_impl: native
1969
+ cross_entropy_loss_fusion: false
1970
+ deallocate_pipeline_outputs: false
1971
+ defer_embedding_wgrad_compute: false
1972
+ deterministic_mode: false
1973
+ enable_autocast: false
1974
+ expert_model_parallel_size: 1
1975
+ expert_tensor_parallel_size: 1
1976
+ finalize_model_grads_func: null
1977
+ fp16: false
1978
+ grad_scale_func: null
1979
+ grad_sync_func: null
1980
+ gradient_accumulation_fusion: false
1981
+ hierarchical_context_parallel_sizes: null
1982
+ microbatch_group_size_per_vp_stage: 1
1983
+ moe_extended_tp: false
1984
+ no_sync_func: null
1985
+ num_microbatches_with_partial_activation_checkpoints: null
1986
+ overlap_p2p_comm: false
1987
+ overlap_p2p_comm_warmup_flush: false
1988
+ param_sync_func: null
1989
+ params_dtype: torch.float32
1990
+ perform_initialization: true
1991
+ pipeline_dtype: null
1992
+ pipeline_model_parallel_comm_backend: null
1993
+ pipeline_model_parallel_size: 1
1994
+ pipeline_model_parallel_split_rank: null
1995
+ sequence_parallel: false
1996
+ tensor_model_parallel_size: 1
1997
+ timers: null
1998
+ tp_comm_atomic_ag: false
1999
+ tp_comm_atomic_rs: false
2000
+ tp_comm_bootstrap_backend: nccl
2001
+ tp_comm_bulk_dgrad: true
2002
+ tp_comm_bulk_wgrad: true
2003
+ tp_comm_overlap: false
2004
+ tp_comm_overlap_ag: true
2005
+ tp_comm_overlap_disable_fc1: false
2006
+ tp_comm_overlap_disable_qkv: false
2007
+ tp_comm_overlap_rs: true
2008
+ tp_comm_overlap_rs_dgrad: false
2009
+ tp_comm_split_ag: true
2010
+ tp_comm_split_rs: true
2011
+ use_cpu_initialization: false
2012
+ use_ring_exchange_p2p: false
2013
+ use_te_rng_tracker: false
2014
+ variable_seq_lengths: false
2015
+ virtual_pipeline_model_parallel_size: null
2016
+ wgrad_deferral_limit: 0
2017
+ optimizer:
2018
+ _target_: <function get_base_optimizer at 0x14afbd39e710>
2019
+ betas:
2020
+ - '0.9'
2021
+ - '0.99'
2022
+ capturable: 'True'
2023
+ eps: 1e-08
2024
+ lr: '0.0001'
2025
+ master_weights: 'True'
2026
+ model: null
2027
+ optim_type: fusedadam
2028
+ weight_decay: '0.1'
2029
+ scheduler:
2030
+ _target_: <class 'cosmos_predict2.functional.lr_scheduler.LambdaLinearScheduler'>
2031
+ cycle_lengths:
2032
+ - '10000000000000'
2033
+ f_max:
2034
+ - '1.0'
2035
+ f_min:
2036
+ - '1.0'
2037
+ f_start:
2038
+ - 1e-06
2039
+ verbosity_interval: '0'
2040
+ warm_up_steps:
2041
+ - '1000'
2042
+ trainer:
2043
+ callbacks:
2044
+ device_monitor:
2045
+ _target_: <class 'cosmos_predict2.callbacks.device_monitor.DeviceMonitor'>
2046
+ every_n: '10'
2047
+ log_memory_detail: 'True'
2048
+ step_size: '1'
2049
+ ema:
2050
+ _target_: <class 'imaginaire.utils.callback.EMAModelCallback'>
2051
+ config: null
2052
+ trainer: null
2053
+ grad_clip:
2054
+ _target_: <class 'cosmos_predict2.callbacks.grad_clip.GradClip'>
2055
+ clip_norm: '10.0'
2056
+ force_finite: 'True'
2057
+ log_wandb: 'False'
2058
+ iter_speed:
2059
+ _target_: <class 'cosmos_predict2.callbacks.iter_speed.IterSpeed'>
2060
+ every_n: '10'
2061
+ hit_thres: '5'
2062
+ low_prec:
2063
+ _target_: <class 'imaginaire.utils.callback.LowPrecisionCallback'>
2064
+ config: null
2065
+ trainer: null
2066
+ update_iter: '1'
2067
+ manual_gc:
2068
+ _target_: <class 'imaginaire.callbacks.manual_gc.ManualGarbageCollection'>
2069
+ every_n: '5'
2070
+ warm_up: '5'
2071
+ progress_bar:
2072
+ _target_: <class 'imaginaire.utils.callback.ProgressBarCallback'>
2073
+ config: null
2074
+ trainer: null
2075
+ wandb:
2076
+ _target_: <class 'cosmos_predict2.callbacks.wandb.WandBCallback'>
2077
+ entity_name: ''
2078
+ mode: offline
2079
+ project_name: vam_cross_mimicvideo_action
2080
+ run_name: w2a_panda_widowx_level2_widowx_texture_2cam_hstack_action_iter2374_videolora_iter200_widowx_teleop_recording_frame_v1_full
2081
+ cudnn:
2082
+ benchmark: 'True'
2083
+ deterministic: 'False'
2084
+ ddp:
2085
+ broadcast_buffers: 'True'
2086
+ find_unused_parameters: 'False'
2087
+ static_graph: 'True'
2088
+ distributed_parallelism: ddp
2089
+ epoch_checkpoint_throttling_min_period_minutes: '30'
2090
+ grad_accum_iter: '8'
2091
+ grad_scaler_args:
2092
+ enabled: 'False'
2093
+ logging_iter: '10'
2094
+ max_iter: '3000'
2095
+ max_val_iter: null
2096
+ memory_format: torch.preserve_format
2097
+ profiling:
2098
+ enable_memory_snapshot: 'False'
2099
+ enable_profiling: 'False'
2100
+ first_n_rank: '4'
2101
+ profile_freq: '1'
2102
+ profile_memory: 'True'
2103
+ record_shape: 'True'
2104
+ with_modules: 'True'
2105
+ with_stack: 'True'
2106
+ run_validation: 'False'
2107
+ seed: '0'
2108
+ timeout_period: '999999999'
2109
+ type: <class 'imaginaire.trainer.ImaginaireTrainer'>
2110
+ validation_iter: '3000'
2111
+ video_dataset_train: null
2112
+ video_dataset_val: null
latest_checkpoint.txt ADDED
@@ -0,0 +1 @@
 
 
1
+ iter_000001800.pt
vam_cross_action_decoder_manifest.json ADDED
@@ -0,0 +1,348 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "action_contract": {
3
+ "action_dimensions": 10,
4
+ "action_horizon": 15,
5
+ "action_vector_order": [
6
+ "relative_ee_xyz_3",
7
+ "relative_ee_rotation_6d_6",
8
+ "absolute_gripper_1"
9
+ ],
10
+ "coordinate_frame": "widowx_reference_base/teleop_aligned_tool",
11
+ "gripper_source": "action.gripper_position",
12
+ "gripper_target_representation": "absolute_normalized_0_1",
13
+ "observation_dimensions": 10,
14
+ "observation_horizon": 1,
15
+ "pose_rotation_representation": "rotation_6d",
16
+ "pose_source": "observation.ee_pose",
17
+ "pose_source_semantics": "future_achieved_pose",
18
+ "pose_target_representation": "relative_to_current_achieved_pose",
19
+ "source_fps": 30,
20
+ "target_fps": 5,
21
+ "tool_frame_definition": {
22
+ "jaw_symmetry_correction": [
23
+ [
24
+ 1.0,
25
+ 0.0,
26
+ 0.0
27
+ ],
28
+ [
29
+ 0.0,
30
+ 1.0,
31
+ 0.0
32
+ ],
33
+ [
34
+ 0.0,
35
+ 0.0,
36
+ 1.0
37
+ ]
38
+ ],
39
+ "paired_teleop_alignment": "derived_identity_from_panda_robotiq_mapping_and_widowx_jaw_axes",
40
+ "recording_frame_id": "widowx_teleop_recording_frame_v1",
41
+ "recording_raw_from_canonical": [
42
+ [
43
+ 0.0,
44
+ 0.0,
45
+ -1.0
46
+ ],
47
+ [
48
+ 0.0,
49
+ 1.0,
50
+ 0.0
51
+ ],
52
+ [
53
+ 1.0,
54
+ 0.0,
55
+ 0.0
56
+ ]
57
+ ],
58
+ "reference_from_robot_base": [
59
+ [
60
+ 1.0,
61
+ 0.0,
62
+ 0.0,
63
+ -0.1
64
+ ],
65
+ [
66
+ 0.0,
67
+ 1.0,
68
+ 0.0,
69
+ 0.0
70
+ ],
71
+ [
72
+ 0.0,
73
+ 0.0,
74
+ 1.0,
75
+ 0.0
76
+ ],
77
+ [
78
+ 0.0,
79
+ 0.0,
80
+ 0.0,
81
+ 1.0
82
+ ]
83
+ ],
84
+ "reference_robot": "widowx250",
85
+ "reference_tool_frame_definition_sha256": "f3aef5e783001a5718e4a0c42020b06d137fb429c04e544a67934dac7decf4ba",
86
+ "robot_name": "panda-widowx",
87
+ "schema_version": 1,
88
+ "tool_frame_definition": {
89
+ "axis_semantics": [
90
+ "forward",
91
+ "left_finger",
92
+ "palm_up"
93
+ ],
94
+ "canonical_reference_robot": "widowx250",
95
+ "canonical_tool_frame_id": "widowx_tool_canonical_v1",
96
+ "evidence": [
97
+ "compiled wx250s/gripper_link +X finger-forward equals grip_site raw +Z",
98
+ "compiled left_finger positive slide opens along grip_site raw +Y",
99
+ "grip_site is Ry(+90deg) relative to the transplanted WidowX gripper body"
100
+ ],
101
+ "raw_forward_axis": [
102
+ 0.0,
103
+ 0.0,
104
+ 1.0
105
+ ],
106
+ "raw_from_canonical": [
107
+ [
108
+ 0.0,
109
+ 0.0,
110
+ -1.0
111
+ ],
112
+ [
113
+ 0.0,
114
+ 1.0,
115
+ 0.0
116
+ ],
117
+ [
118
+ 1.0,
119
+ 0.0,
120
+ 0.0
121
+ ]
122
+ ],
123
+ "raw_left_axis": [
124
+ 0.0,
125
+ 1.0,
126
+ 0.0
127
+ ],
128
+ "raw_palm_up_axis": [
129
+ -1.0,
130
+ 0.0,
131
+ 0.0
132
+ ],
133
+ "robot_name": "panda-widowx",
134
+ "robot_xml_sha256": "5f8b242ead3600740065e135c609c34ed40bfdc29201d50a5b366abf0b13da3c",
135
+ "schema_version": 1
136
+ },
137
+ "tool_frame_definition_sha256": "360ba397b682c1e042beebb3dd4f5cf47561601cef9c3d14c6558184ad37ecdd",
138
+ "transform_semantics": "T_widowxbase_recordingtool=inv(T_world_widowxbase)@T_world_robotbase@T_robotbase_raw@T_raw_recordingtool",
139
+ "world_from_reference_base": [
140
+ [
141
+ 1.0,
142
+ 0.0,
143
+ 0.0,
144
+ 0.1
145
+ ],
146
+ [
147
+ 0.0,
148
+ 1.0,
149
+ 0.0,
150
+ 0.0
151
+ ],
152
+ [
153
+ 0.0,
154
+ 0.0,
155
+ 1.0,
156
+ 0.42
157
+ ],
158
+ [
159
+ 0.0,
160
+ 0.0,
161
+ 0.0,
162
+ 1.0
163
+ ]
164
+ ],
165
+ "world_from_robot_base": [
166
+ [
167
+ 1.0,
168
+ 0.0,
169
+ 0.0,
170
+ 0.0
171
+ ],
172
+ [
173
+ 0.0,
174
+ 1.0,
175
+ 0.0,
176
+ 0.0
177
+ ],
178
+ [
179
+ 0.0,
180
+ 0.0,
181
+ 1.0,
182
+ 0.42
183
+ ],
184
+ [
185
+ 0.0,
186
+ 0.0,
187
+ 0.0,
188
+ 1.0
189
+ ]
190
+ ]
191
+ },
192
+ "tool_frame_definition_sha256": "0cf1ec052e9add9d54e479106a5617f30717a9a82da9c437505a1e6480ff6956",
193
+ "tool_frame_id": "widowx_teleop_recording_frame_v1",
194
+ "unused_command_pose": "action.ee_pose"
195
+ },
196
+ "artifact_type": "mimicvideo_world2action_decoder",
197
+ "base_artifacts": {
198
+ "action_frame_initialization": {
199
+ "numeric_transform": "identity",
200
+ "reason": "widowx250 raw grip_site defines the canonical reference",
201
+ "reference_robot": "widowx250",
202
+ "schema_version": 1,
203
+ "source_dataset_repo_id": "dreamdifferent/vam-cross-target-widowx250-native",
204
+ "source_tool_frame_id": "native_grip_site_v1",
205
+ "target_tool_frame_id": "widowx_teleop_recording_frame_v1"
206
+ },
207
+ "initial_action_decoder": {
208
+ "bytes": 998172132,
209
+ "delivery": "standalone_hub",
210
+ "iteration": 2374,
211
+ "kind": "pretrained_world2action_decoder",
212
+ "local_path": "initial_action_decoders/dreamdifferent--vam-cross-target-widowx250-native-2cam-action-decoder-93750cc/checkpoints/model/iter_000002374.pt",
213
+ "manifest_path": "vam_cross_action_decoder_manifest.json",
214
+ "path": "checkpoints/model/iter_000002374.pt",
215
+ "path_in_repo": "checkpoints/model/iter_000002374.pt",
216
+ "receipt_path": "initial_action_decoders/dreamdifferent--vam-cross-target-widowx250-native-2cam-action-decoder-93750cc/VAM_CROSS_INITIAL_ACTION_DECODER.json",
217
+ "repo_id": "dreamdifferent/vam-cross-target-widowx250-native-2cam-action-decoder",
218
+ "revision": "93750cccda01620e3c028477e4c49bc5c996a68d",
219
+ "sha256": "3a9a75686164351468ef9b99098ae9152a1f07082b869e1ad03c9e1e1cf6885c",
220
+ "source_iteration": 2374,
221
+ "training_dataset_repo_id": "dreamdifferent/vam-cross-target-widowx250-native",
222
+ "training_robot_name": "widowx250"
223
+ },
224
+ "initial_video_backbone": {
225
+ "bytes": 3913057284,
226
+ "delivery": "standalone_hub",
227
+ "kind": "fused_video2world_dit",
228
+ "local_path": "initial_backbones/dreamdifferent--widowx250-video-fused-f0cea76/checkpoints/video_backbone/iter_000001060_fused.pt",
229
+ "manifest_path": "vam_cross_fused_video_lora_manifest.json",
230
+ "path_in_repo": "checkpoints/video_backbone/iter_000001060_fused.pt",
231
+ "receipt_path": "initial_backbones/dreamdifferent--widowx250-video-fused-f0cea76/VAM_CROSS_INITIAL_VIDEO_BACKBONE.json",
232
+ "repo_id": "dreamdifferent/widowx250-video-fused",
233
+ "revision": "f0cea76b62c5dd66b06b9f965932ddea32a7b546",
234
+ "sha256": "d0f24c049bee63b03d3b62747b240a2d1822ddd5f83a52fcd866a882e80122b1",
235
+ "source_iteration": 1060
236
+ },
237
+ "mimicvideo_commit": "e3355dbc93132b576c02f920a59b4fc18a4f5906",
238
+ "video_lora": {
239
+ "bytes": 741588124,
240
+ "delivery": "standalone_hub",
241
+ "iteration": 200,
242
+ "kind": "video2world_lora",
243
+ "local_path": "selected_video_loras/dreamdifferent--vam-cross-level2-panda-widowx-widowx-texture-video-lora-iter200-5a22b23/checkpoints/model/iter_000000200.pt",
244
+ "manifest_path": "vam_cross_video_lora_manifest.json",
245
+ "path": "checkpoints/model/iter_000000200.pt",
246
+ "path_in_repo": "checkpoints/model/iter_000000200.pt",
247
+ "receipt_path": "selected_video_loras/dreamdifferent--vam-cross-level2-panda-widowx-widowx-texture-video-lora-iter200-5a22b23/VAM_CROSS_SELECTED_VIDEO_LORA.json",
248
+ "repo_id": "dreamdifferent/vam-cross-level2-panda-widowx-widowx-texture-video-lora-iter200",
249
+ "revision": "5a22b23f6f4bf80b249d6215d15260b7b59acbcf",
250
+ "sha256": "45ff819e0044fc5a77d894e8f5d36caef9c28d68fb470ada0fe89724922e9b13",
251
+ "source_iteration": 200
252
+ }
253
+ },
254
+ "checkpoint": {
255
+ "complete_components": [
256
+ "model",
257
+ "optim",
258
+ "scheduler",
259
+ "trainer"
260
+ ],
261
+ "filename": "iter_000001800.pt",
262
+ "iteration": 1800,
263
+ "reached_requested_max_iterations": false,
264
+ "requested_max_iterations": 3000,
265
+ "selection": "explicit",
266
+ "source_job_id": null,
267
+ "termination_reason": "unknown",
268
+ "uploaded_training_state": false
269
+ },
270
+ "code": {
271
+ "vam_cross_commit": "673f60cfc089a23dc920afa21526c10450a4dbbf"
272
+ },
273
+ "created_at_utc": "2026-09-27T15:20:49.227697+00:00",
274
+ "dataset": {
275
+ "camera_keys": [
276
+ "observation.images.corner_cam",
277
+ "observation.images.front_cam"
278
+ ],
279
+ "commit": "994be7f8952807008c316db7943ec732bf70b978",
280
+ "episodes": 256,
281
+ "frames": 54349,
282
+ "instruction": null,
283
+ "repo_id": "dreamdifferent/vam-cross-level2-panda-widowx-widowx-texture",
284
+ "revision": "994be7f8952807008c316db7943ec732bf70b978",
285
+ "robot_type": "panda-widowx",
286
+ "source_fps": 30,
287
+ "target_fps": 5,
288
+ "task_instructions": [
289
+ "reach toward the bowl and stop immediately before contact",
290
+ "reach toward the pot and stop immediately before contact",
291
+ "reach toward the tomato and stop immediately before contact",
292
+ "touch the pan without attempting to move it",
293
+ "touch the dish sponge without grasping it",
294
+ "touch the egg without grasping it",
295
+ "grasp the apple without lifting it",
296
+ "grasp the candle without lifting it",
297
+ "release the grasped pepper shaker while it rests on the table",
298
+ "release the held cup while keeping it at its current position",
299
+ "lift the apple vertically and hold it without horizontal movement",
300
+ "lift the bar of soap vertically and hold it without horizontal movement",
301
+ "lower the held egg while maintaining the grasp",
302
+ "lower the held potato while maintaining the grasp",
303
+ "move the held cup horizontally a short distance",
304
+ "move the held mug horizontally a short distance",
305
+ "push the dish sponge forward without grasping it",
306
+ "push the bar of soap forward without grasping it",
307
+ "pull the potato backward toward the robot",
308
+ "pull the bar of soap backward toward the robot",
309
+ "rotate the potato in place around the vertical axis without translation",
310
+ "rotate the dish sponge in place around the vertical axis without translation",
311
+ "release the held tomato from a low height onto the table",
312
+ "release the held candle from a low height onto the table"
313
+ ]
314
+ },
315
+ "files": {
316
+ "checkpoints/model/iter_000001800.pt": {
317
+ "bytes": 998172132,
318
+ "sha256": "e9cf5ae373416faa966d04c9707080f815a3710fd29b53c4d837380b9b3bfa47"
319
+ },
320
+ "config.yaml": {
321
+ "bytes": 72384,
322
+ "sha256": "0044315d397aac5cd66a86f1d8d5a48a5db9e1da22545373785493ea513cd219"
323
+ },
324
+ "latest_checkpoint.txt": {
325
+ "bytes": 18,
326
+ "sha256": "9d7680d3739b134ff8ac9e2eee33528b7d4381b4c646db6384bc14d2e65c4469"
327
+ },
328
+ "vam_cross_video2world_config.json": {
329
+ "bytes": 17838,
330
+ "sha256": "5ff1fe0e2d9b095a0db73757275980464fbcd25616630798f5b2eecc152210b3"
331
+ },
332
+ "vam_cross_world2action_config.json": {
333
+ "bytes": 10218,
334
+ "sha256": "b84772ea506143d0ac5c747941203217daa5b5a014599d65b699f8b9a515725b"
335
+ }
336
+ },
337
+ "hub_repo_id": "dreamdifferent/vam-cross-level2-panda-widowx-widowx-texture-teleopaligned-videolora200-action-decoder-iter1800",
338
+ "model": {
339
+ "data_config": "bridge",
340
+ "experiment": "w2a_panda_widowx_level2_widowx_texture_2cam_hstack_action_iter2374_videolora_iter200_widowx_teleop_recording_frame_v1",
341
+ "initialization": "pretrained_action_decoder",
342
+ "learning_rate": 0.0001,
343
+ "stage": "world2action",
344
+ "xattn_layer": 20
345
+ },
346
+ "private": false,
347
+ "schema_version": 1
348
+ }
vam_cross_video2world_config.json ADDED
@@ -0,0 +1,650 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "schema_version": 1,
3
+ "project": {
4
+ "mimicvideo_repo": "external/mimic-video",
5
+ "expected_mimicvideo_commit": "e3355dbc93132b576c02f920a59b4fc18a4f5906",
6
+ "artifact_root": "/cluster/scratch/${USER}/vam_cross_mimicvideo"
7
+ },
8
+ "paths": {
9
+ "lerobot_dataset": "datasets/lerobot/vam-cross-level2-panda-widowx-widowx-texture-994be7f",
10
+ "video_smoke_export_root": "datasets/mimicvideo/panda_widowx_level2_widowx_texture_2cam_hstack_video_smoke_ep000000",
11
+ "video_export_root": "datasets/mimicvideo/panda_widowx_level2_widowx_texture_2cam_hstack_video",
12
+ "checkpoint_root": "checkpoints",
13
+ "run_root": "runs",
14
+ "log_root": "logs",
15
+ "tmp_root": "tmp",
16
+ "environment_root": "/cluster/home/${USER}/.vam_cross_envs",
17
+ "uv_python_root": "/cluster/home/${USER}/.vam_cross_uv_python",
18
+ "uv_cache_root": "/cluster/home/${USER}/.vam_cross_uv_cache",
19
+ "container_root": "containers",
20
+ "cache_root": "cache"
21
+ },
22
+ "dataset": {
23
+ "hf_repo_id": "dreamdifferent/vam-cross-level2-panda-widowx-widowx-texture",
24
+ "hf_revision": "994be7f8952807008c316db7943ec732bf70b978",
25
+ "hf_expected_commit": "994be7f8952807008c316db7943ec732bf70b978",
26
+ "private": true,
27
+ "manifest_kind": "vam_cross",
28
+ "manifest_repo_id": "local/vam_cross_level2_panda-widowx",
29
+ "collection_name": "level2",
30
+ "codebase_version": "v3.0",
31
+ "robot_type": "panda-widowx",
32
+ "robot_appearance": "widowx-texture",
33
+ "instruction": null,
34
+ "task_instructions": [
35
+ "reach toward the bowl and stop immediately before contact",
36
+ "reach toward the pot and stop immediately before contact",
37
+ "reach toward the tomato and stop immediately before contact",
38
+ "touch the pan without attempting to move it",
39
+ "touch the dish sponge without grasping it",
40
+ "touch the egg without grasping it",
41
+ "grasp the apple without lifting it",
42
+ "grasp the candle without lifting it",
43
+ "release the grasped pepper shaker while it rests on the table",
44
+ "release the held cup while keeping it at its current position",
45
+ "lift the apple vertically and hold it without horizontal movement",
46
+ "lift the bar of soap vertically and hold it without horizontal movement",
47
+ "lower the held egg while maintaining the grasp",
48
+ "lower the held potato while maintaining the grasp",
49
+ "move the held cup horizontally a short distance",
50
+ "move the held mug horizontally a short distance",
51
+ "push the dish sponge forward without grasping it",
52
+ "push the bar of soap forward without grasping it",
53
+ "pull the potato backward toward the robot",
54
+ "pull the bar of soap backward toward the robot",
55
+ "rotate the potato in place around the vertical axis without translation",
56
+ "rotate the dish sponge in place around the vertical axis without translation",
57
+ "release the held tomato from a low height onto the table",
58
+ "release the held candle from a low height onto the table"
59
+ ],
60
+ "require_all_success": true,
61
+ "camera_keys": [
62
+ "observation.images.corner_cam",
63
+ "observation.images.front_cam"
64
+ ],
65
+ "source_fps": 30,
66
+ "video_fps": 5,
67
+ "height": 480,
68
+ "width": 640,
69
+ "total_episodes": 256,
70
+ "total_frames": 54349,
71
+ "total_tasks": 24,
72
+ "splits": {
73
+ "train": "0:256"
74
+ },
75
+ "video_codec": "h264",
76
+ "video_pixel_format": "yuv420p",
77
+ "feature_contract": {
78
+ "observation.state": {
79
+ "dtype": "float32",
80
+ "shape": [
81
+ 9
82
+ ],
83
+ "names": {
84
+ "axes": [
85
+ "joint_0",
86
+ "joint_1",
87
+ "joint_2",
88
+ "joint_3",
89
+ "joint_4",
90
+ "joint_5",
91
+ "joint_6",
92
+ "gripper_0",
93
+ "gripper_1"
94
+ ]
95
+ }
96
+ },
97
+ "observation.joint_position": {
98
+ "dtype": "float32",
99
+ "shape": [
100
+ 7
101
+ ],
102
+ "names": {
103
+ "axes": [
104
+ "joint_0",
105
+ "joint_1",
106
+ "joint_2",
107
+ "joint_3",
108
+ "joint_4",
109
+ "joint_5",
110
+ "joint_6"
111
+ ]
112
+ }
113
+ },
114
+ "observation.joint_velocity": {
115
+ "dtype": "float32",
116
+ "shape": [
117
+ 7
118
+ ],
119
+ "names": {
120
+ "axes": [
121
+ "joint_0",
122
+ "joint_1",
123
+ "joint_2",
124
+ "joint_3",
125
+ "joint_4",
126
+ "joint_5",
127
+ "joint_6"
128
+ ]
129
+ }
130
+ },
131
+ "observation.gripper_position": {
132
+ "dtype": "float32",
133
+ "shape": [
134
+ 2
135
+ ],
136
+ "names": {
137
+ "axes": [
138
+ "gripper_0",
139
+ "gripper_1"
140
+ ]
141
+ }
142
+ },
143
+ "observation.ee_pose": {
144
+ "dtype": "float32",
145
+ "shape": [
146
+ 16
147
+ ],
148
+ "names": {
149
+ "axes": [
150
+ "m00",
151
+ "m01",
152
+ "m02",
153
+ "m03",
154
+ "m10",
155
+ "m11",
156
+ "m12",
157
+ "m13",
158
+ "m20",
159
+ "m21",
160
+ "m22",
161
+ "m23",
162
+ "m30",
163
+ "m31",
164
+ "m32",
165
+ "m33"
166
+ ]
167
+ }
168
+ },
169
+ "observation.object_position": {
170
+ "dtype": "float32",
171
+ "shape": [
172
+ 3
173
+ ],
174
+ "names": {
175
+ "axes": [
176
+ "x",
177
+ "y",
178
+ "z"
179
+ ]
180
+ }
181
+ },
182
+ "observation.target_position": {
183
+ "dtype": "float32",
184
+ "shape": [
185
+ 3
186
+ ],
187
+ "names": {
188
+ "axes": [
189
+ "x",
190
+ "y",
191
+ "z"
192
+ ]
193
+ }
194
+ },
195
+ "action.ee_pose": {
196
+ "dtype": "float32",
197
+ "shape": [
198
+ 16
199
+ ],
200
+ "names": {
201
+ "axes": [
202
+ "m00",
203
+ "m01",
204
+ "m02",
205
+ "m03",
206
+ "m10",
207
+ "m11",
208
+ "m12",
209
+ "m13",
210
+ "m20",
211
+ "m21",
212
+ "m22",
213
+ "m23",
214
+ "m30",
215
+ "m31",
216
+ "m32",
217
+ "m33"
218
+ ]
219
+ }
220
+ },
221
+ "action.gripper_position": {
222
+ "dtype": "float32",
223
+ "shape": [
224
+ 1
225
+ ],
226
+ "names": {
227
+ "axes": [
228
+ "gripper"
229
+ ]
230
+ }
231
+ },
232
+ "legacy.realized_joint_position": {
233
+ "dtype": "float32",
234
+ "shape": [
235
+ 7
236
+ ],
237
+ "names": {
238
+ "axes": [
239
+ "joint_0",
240
+ "joint_1",
241
+ "joint_2",
242
+ "joint_3",
243
+ "joint_4",
244
+ "joint_5",
245
+ "joint_6"
246
+ ]
247
+ }
248
+ },
249
+ "source.timestamp_s": {
250
+ "dtype": "float64",
251
+ "shape": [
252
+ 1
253
+ ],
254
+ "names": {
255
+ "axes": [
256
+ "seconds"
257
+ ]
258
+ }
259
+ },
260
+ "source.frame_index": {
261
+ "dtype": "int64",
262
+ "shape": [
263
+ 1
264
+ ],
265
+ "names": {
266
+ "axes": [
267
+ "frame_index"
268
+ ]
269
+ }
270
+ },
271
+ "episode.success": {
272
+ "dtype": "bool",
273
+ "shape": [
274
+ 1
275
+ ],
276
+ "names": {
277
+ "axes": [
278
+ "success"
279
+ ]
280
+ }
281
+ },
282
+ "replay.robot_joint_position": {
283
+ "dtype": "float64",
284
+ "shape": [
285
+ 9
286
+ ],
287
+ "names": {
288
+ "axes": [
289
+ "joint1",
290
+ "joint2",
291
+ "joint3",
292
+ "joint4",
293
+ "joint5",
294
+ "joint6",
295
+ "joint7",
296
+ "left_finger",
297
+ "right_finger"
298
+ ]
299
+ }
300
+ },
301
+ "replay.manip_object_pose": {
302
+ "dtype": "float64",
303
+ "shape": [
304
+ 7
305
+ ],
306
+ "names": {
307
+ "axes": [
308
+ "x",
309
+ "y",
310
+ "z",
311
+ "qw",
312
+ "qx",
313
+ "qy",
314
+ "qz"
315
+ ]
316
+ }
317
+ },
318
+ "replay.target_object_pose": {
319
+ "dtype": "float64",
320
+ "shape": [
321
+ 7
322
+ ],
323
+ "names": {
324
+ "axes": [
325
+ "x",
326
+ "y",
327
+ "z",
328
+ "qw",
329
+ "qx",
330
+ "qy",
331
+ "qz"
332
+ ]
333
+ }
334
+ },
335
+ "replay.object_presence": {
336
+ "dtype": "bool",
337
+ "shape": [
338
+ 2
339
+ ],
340
+ "names": {
341
+ "axes": [
342
+ "manip_obj",
343
+ "target_obj"
344
+ ]
345
+ }
346
+ },
347
+ "replay.sim_time_s": {
348
+ "dtype": "float64",
349
+ "shape": [
350
+ 1
351
+ ],
352
+ "names": {
353
+ "axes": [
354
+ "seconds"
355
+ ]
356
+ }
357
+ },
358
+ "observation.images.corner_cam": {
359
+ "dtype": "video",
360
+ "shape": [
361
+ 480,
362
+ 640,
363
+ 3
364
+ ],
365
+ "names": [
366
+ "height",
367
+ "width",
368
+ "channels"
369
+ ]
370
+ },
371
+ "observation.images.front_cam": {
372
+ "dtype": "video",
373
+ "shape": [
374
+ 480,
375
+ 640,
376
+ 3
377
+ ],
378
+ "names": [
379
+ "height",
380
+ "width",
381
+ "channels"
382
+ ]
383
+ },
384
+ "timestamp": {
385
+ "dtype": "float32",
386
+ "shape": [
387
+ 1
388
+ ]
389
+ },
390
+ "frame_index": {
391
+ "dtype": "int64",
392
+ "shape": [
393
+ 1
394
+ ]
395
+ },
396
+ "episode_index": {
397
+ "dtype": "int64",
398
+ "shape": [
399
+ 1
400
+ ]
401
+ },
402
+ "index": {
403
+ "dtype": "int64",
404
+ "shape": [
405
+ 1
406
+ ]
407
+ },
408
+ "task_index": {
409
+ "dtype": "int64",
410
+ "shape": [
411
+ 1
412
+ ]
413
+ }
414
+ }
415
+ },
416
+ "export": {
417
+ "smoke_episode_index": 0,
418
+ "video_crf": 18,
419
+ "view_layout": "hstack",
420
+ "height": 480,
421
+ "width": 640
422
+ },
423
+ "preprocessing": {
424
+ "language_embedding_dirname": "language_embeddings",
425
+ "language_embedding_format": "safetensors",
426
+ "language_embedding_receipt_filename": "VAM_CROSS_T5_EMBEDDINGS.json",
427
+ "t5": {
428
+ "tensor_key": "encoded_text",
429
+ "max_length": 512,
430
+ "embedding_dim": 1024,
431
+ "compute_dtype": "bfloat16",
432
+ "storage_dtype": "float32",
433
+ "deduplicate_prompts": true,
434
+ "gpu_type": "a100",
435
+ "minimum_gpu_memory_gb": 40
436
+ },
437
+ "precompute_video_embeddings": false
438
+ },
439
+ "checkpoints": {
440
+ "huggingface_repo": "jonpai/mimic-video",
441
+ "hf_expected_commit": "f28339034831e3c2374be075e622e1ff38ebe0f8",
442
+ "download_model_selector": "finetuned_cosmos_bridge",
443
+ "receipt_filename": "VAM_CROSS_MIMICVIDEO_CHECKPOINTS.json",
444
+ "bridge_video_backbone": "video_backbone/v2w_bridge_lora_rank256_lr1.778e-04_bsz64_iter_000070043_fused.pt",
445
+ "video_tokenizer": "video_backbone/tokenizer/tokenizer.pth",
446
+ "t5_text_encoder": "text_encoder/t5-11b",
447
+ "expected_total_bytes": 49651509498,
448
+ "required_files": [
449
+ {
450
+ "path": "text_encoder/t5-11b/README.md",
451
+ "size": 8603,
452
+ "sha256": null
453
+ },
454
+ {
455
+ "path": "text_encoder/t5-11b/config.json",
456
+ "size": 1203,
457
+ "sha256": null
458
+ },
459
+ {
460
+ "path": "text_encoder/t5-11b/pytorch_model.bin",
461
+ "size": 45229452544,
462
+ "sha256": "5fdc64177b14b0f72437fea171a752c626733f67755842528c75b90cacd1c807"
463
+ },
464
+ {
465
+ "path": "text_encoder/t5-11b/tokenizer.json",
466
+ "size": 1389353,
467
+ "sha256": null
468
+ },
469
+ {
470
+ "path": "video_backbone/tokenizer/tokenizer.pth",
471
+ "size": 507609880,
472
+ "sha256": "38071ab59bd94681c686fa51d75a1968f64e470262043be31f7a094e442fd981"
473
+ },
474
+ {
475
+ "path": "video_backbone/v2w_bridge_lora_rank256_lr1.778e-04_bsz64_iter_000070043_fused.pt",
476
+ "size": 3913047915,
477
+ "sha256": "2723645edb158661af6d8e1c623ec21a63626d96e959799abb0b758397ac8c88"
478
+ }
479
+ ],
480
+ "estimated_download_gb_decimal": {
481
+ "t5_text_encoder": 45.2,
482
+ "video_tokenizer": 0.51,
483
+ "bridge_video_backbone": 3.91,
484
+ "total": 49.62
485
+ }
486
+ },
487
+ "initial_video_backbone": {
488
+ "kind": "fused_video2world_dit",
489
+ "delivery": "standalone_hub",
490
+ "repo_id": "dreamdifferent/widowx250-video-fused",
491
+ "revision": "f0cea76b62c5dd66b06b9f965932ddea32a7b546",
492
+ "path_in_repo": "checkpoints/video_backbone/iter_000001060_fused.pt",
493
+ "local_path": "initial_backbones/dreamdifferent--widowx250-video-fused-f0cea76/checkpoints/video_backbone/iter_000001060_fused.pt",
494
+ "bytes": 3913057284,
495
+ "sha256": "d0f24c049bee63b03d3b62747b240a2d1822ddd5f83a52fcd866a882e80122b1",
496
+ "manifest_path": "vam_cross_fused_video_lora_manifest.json",
497
+ "receipt_path": "initial_backbones/dreamdifferent--widowx250-video-fused-f0cea76/VAM_CROSS_INITIAL_VIDEO_BACKBONE.json",
498
+ "source_iteration": 1060
499
+ },
500
+ "environments": {
501
+ "data": {
502
+ "manager": "uv",
503
+ "prefix_name": "vam-cross-data",
504
+ "requirements": "configs/mimicvideo/requirements_data.in",
505
+ "python": "3.12",
506
+ "import_timeout_seconds": 300,
507
+ "torch_backend": "cpu",
508
+ "lock_exclude_newer": "2026-07-16",
509
+ "lock_filename": "vam-cross-data-py312-cpu.txt",
510
+ "lock_path": "configs/mimicvideo/locks/vam-cross-data-py312-cpu.txt",
511
+ "lock_sha256": "10bdc1165dd033d1459c0042c62e040019b1ebce909e68d0aa3db18073353b1c",
512
+ "packages": {
513
+ "lerobot": "0.6.0",
514
+ "scipy": "1.15.3"
515
+ }
516
+ },
517
+ "model": {
518
+ "manager": "apptainer",
519
+ "python": "3.10",
520
+ "import_timeout_seconds": 300,
521
+ "apptainer_supported_versions": [
522
+ "1.5.1",
523
+ "1.5.2",
524
+ "1.5.3"
525
+ ],
526
+ "base_image": "python@sha256:5cc3381b18ec240e4462510648ed7970917ac8eff4c3463320429fec285491be",
527
+ "definition": "containers/mimicvideo_e335_cu126.def",
528
+ "image_filename": "mimic-video-e335-cu126.sif",
529
+ "uv_version": "0.11.8",
530
+ "uv_extra": "cu126",
531
+ "uv_index": "https://nvidia-cosmos.github.io/cosmos-dependencies/cu126_torch260/simple",
532
+ "requirements_overrides": "configs/mimicvideo/requirements_model_cu126_overrides.txt",
533
+ "lock_filename": "mimic-video-e335-cu126.txt",
534
+ "lock_path": "configs/mimicvideo/locks/mimic-video-e335-cu126.txt",
535
+ "lock_sha256": "bb16106bea9361f7ccfa620ae0bef776f08b8a976fba17598ae83d9349f0e0d5",
536
+ "packages": {
537
+ "torch": "2.6.0",
538
+ "torchvision": "0.21.0",
539
+ "triton": "3.2.0",
540
+ "flash-attn": "2.6.3",
541
+ "transformer-engine": "1.13",
542
+ "apex": "0.1.0",
543
+ "numpy": "1.26.4",
544
+ "transformers": "4.51.3",
545
+ "safetensors": "0.8.0"
546
+ }
547
+ }
548
+ },
549
+ "training": {
550
+ "stage": "video2world_lora",
551
+ "initialization": "fused_video2world_dit",
552
+ "experiment": "v2w_panda_widowx_level2_widowx_texture_2cam_hstack_from_widowx250_video_fused_f0cea76_lora_r256",
553
+ "job_project": "vam_cross",
554
+ "job_group": "video2world_lora",
555
+ "model_size": "2B",
556
+ "resolution": "480p",
557
+ "num_frames": 61,
558
+ "observation_history": 5,
559
+ "lora": {
560
+ "rank": 256,
561
+ "alpha": 32,
562
+ "learning_rate": 0.00017782794100389227,
563
+ "target_modules": "q_proj,k_proj,v_proj,output_proj,x_embedder.proj.1,linear_1,linear_2,mlp.layer1,mlp.layer2",
564
+ "train_adaln_modulation": false,
565
+ "save_trainable_only": true
566
+ },
567
+ "profiles": {
568
+ "smoke": {
569
+ "run_name": "v2w_panda_widowx_level2_widowx_texture_2cam_hstack_from_widowx250_video_fused_f0cea76_lora_r256_smoke",
570
+ "gpus": 1,
571
+ "per_gpu_batch_size": 1,
572
+ "gradient_accumulation": 64,
573
+ "max_iterations": 10,
574
+ "save_every": 10,
575
+ "log_every": 1,
576
+ "num_workers": 1,
577
+ "run_validation": false,
578
+ "walltime": "04:00:00"
579
+ },
580
+ "full": {
581
+ "run_name": "v2w_panda_widowx_level2_widowx_texture_2cam_hstack_from_widowx250_video_fused_f0cea76_lora_r256_full",
582
+ "gpus": 2,
583
+ "gpu_type": "a100_80gb",
584
+ "cpus_per_task": 8,
585
+ "memory_per_cpu": "16G",
586
+ "per_gpu_batch_size": 2,
587
+ "gradient_accumulation": 8,
588
+ "max_iterations": 3000,
589
+ "save_every": 200,
590
+ "log_every": 10,
591
+ "num_workers": 1,
592
+ "run_validation": false,
593
+ "walltime": "24:00:00",
594
+ "require_smoke_receipt": false,
595
+ "keep_latest_only": false,
596
+ "wandb": {
597
+ "mode": "offline",
598
+ "project": "vam_cross_mimicvideo"
599
+ },
600
+ "retention_probe": {
601
+ "enabled": true,
602
+ "every": 200,
603
+ "training_dataset": {
604
+ "enabled": true,
605
+ "episode_index": 10
606
+ },
607
+ "output_dirname": "video_probes",
608
+ "manifest_filename": "VAM_CROSS_RETENTION_PROBE.json",
609
+ "failure_dirname": "failures"
610
+ }
611
+ }
612
+ }
613
+ },
614
+ "slurm": {
615
+ "cpus_per_task": 4,
616
+ "memory_per_cpu": "16G",
617
+ "minimum_gpu_memory_gb": 80,
618
+ "modules": [
619
+ "eth_proxy"
620
+ ],
621
+ "partition": null,
622
+ "account": null,
623
+ "qos": null,
624
+ "gpu_type": null
625
+ },
626
+ "storage": {
627
+ "filesystem": "scratch",
628
+ "recommended_free_space_gb": 100,
629
+ "note": "Video2World uses the 5 Hz MP4 export and language embeddings; the large raw-RGB action SafeTensors export is not required for this stage.",
630
+ "expected_video_frames_5hz": 9166,
631
+ "expected_raw_rgb_bytes_5hz": 8447385600,
632
+ "expected_initial_video_backbone_bytes": 3913057284
633
+ },
634
+ "scaffold": {
635
+ "schema_version": 1,
636
+ "experiment_slug": "panda_widowx_level2_widowx_texture_2cam_hstack",
637
+ "dataset_url": "https://huggingface.co/datasets/dreamdifferent/vam-cross-level2-panda-widowx-widowx-texture",
638
+ "requested_revision": "main",
639
+ "resolved_commit": "994be7f8952807008c316db7943ec732bf70b978",
640
+ "camera_order_source": "vam_manifest",
641
+ "metadata_files": [
642
+ "meta/info.json",
643
+ "meta/vam_cross_manifest.json"
644
+ ],
645
+ "initial_backbone": {
646
+ "repo_id": "dreamdifferent/widowx250-video-fused",
647
+ "resolved_commit": "f0cea76b62c5dd66b06b9f965932ddea32a7b546"
648
+ }
649
+ }
650
+ }
vam_cross_world2action_config.json ADDED
@@ -0,0 +1,311 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "schema_version": 1,
3
+ "base_config": "configs/mimicvideo/experiments/panda_widowx_level2_widowx_texture_2cam_hstack_v2w.json",
4
+ "paths": {
5
+ "action_smoke_export_root": "datasets/mimicvideo/panda_widowx_level2_widowx_texture_2cam_hstack_widowx_teleop_recording_frame_v1_action_smoke_ep000000",
6
+ "action_export_root": "datasets/mimicvideo/panda_widowx_level2_widowx_texture_2cam_hstack_widowx_teleop_recording_frame_v1_action",
7
+ "action_checkpoint_receipt": "checkpoints/experiments/w2a_panda_widowx_level2_widowx_texture_2cam_hstack_action_iter2374_videolora_iter200_widowx_teleop_recording_frame_v1/VAM_CROSS_MIMICVIDEO_ACTION_CHECKPOINT.json",
8
+ "action_input_receipt": "runs/vam_cross/world2action/w2a_panda_widowx_level2_widowx_texture_2cam_hstack_action_iter2374_videolora_iter200_widowx_teleop_recording_frame_v1/VAM_CROSS_ACTION_INPUTS.json",
9
+ "action_statistics_receipt": "runs/vam_cross/world2action/w2a_panda_widowx_level2_widowx_texture_2cam_hstack_action_iter2374_videolora_iter200_widowx_teleop_recording_frame_v1/VAM_CROSS_ACTION_STATISTICS.json"
10
+ },
11
+ "action_contract": {
12
+ "coordinate_frame": "widowx_reference_base/teleop_aligned_tool",
13
+ "pose_source": "observation.ee_pose",
14
+ "unused_command_pose": "action.ee_pose",
15
+ "pose_source_semantics": "future_achieved_pose",
16
+ "pose_target_representation": "relative_to_current_achieved_pose",
17
+ "pose_rotation_representation": "rotation_6d",
18
+ "gripper_source": "action.gripper_position",
19
+ "gripper_target_representation": "absolute_normalized_0_1",
20
+ "source_fps": 30,
21
+ "target_fps": 5,
22
+ "observation_horizon": 1,
23
+ "action_horizon": 15,
24
+ "observation_dimensions": 10,
25
+ "action_dimensions": 10,
26
+ "action_vector_order": [
27
+ "relative_ee_xyz_3",
28
+ "relative_ee_rotation_6d_6",
29
+ "absolute_gripper_1"
30
+ ],
31
+ "tool_frame_id": "widowx_teleop_recording_frame_v1",
32
+ "tool_frame_definition_sha256": "0cf1ec052e9add9d54e479106a5617f30717a9a82da9c437505a1e6480ff6956",
33
+ "tool_frame_definition": {
34
+ "schema_version": 1,
35
+ "recording_frame_id": "widowx_teleop_recording_frame_v1",
36
+ "reference_robot": "widowx250",
37
+ "robot_name": "panda-widowx",
38
+ "transform_semantics": "T_widowxbase_recordingtool=inv(T_world_widowxbase)@T_world_robotbase@T_robotbase_raw@T_raw_recordingtool",
39
+ "world_from_reference_base": [
40
+ [
41
+ 1.0,
42
+ 0.0,
43
+ 0.0,
44
+ 0.1
45
+ ],
46
+ [
47
+ 0.0,
48
+ 1.0,
49
+ 0.0,
50
+ 0.0
51
+ ],
52
+ [
53
+ 0.0,
54
+ 0.0,
55
+ 1.0,
56
+ 0.42
57
+ ],
58
+ [
59
+ 0.0,
60
+ 0.0,
61
+ 0.0,
62
+ 1.0
63
+ ]
64
+ ],
65
+ "world_from_robot_base": [
66
+ [
67
+ 1.0,
68
+ 0.0,
69
+ 0.0,
70
+ 0.0
71
+ ],
72
+ [
73
+ 0.0,
74
+ 1.0,
75
+ 0.0,
76
+ 0.0
77
+ ],
78
+ [
79
+ 0.0,
80
+ 0.0,
81
+ 1.0,
82
+ 0.42
83
+ ],
84
+ [
85
+ 0.0,
86
+ 0.0,
87
+ 0.0,
88
+ 1.0
89
+ ]
90
+ ],
91
+ "reference_from_robot_base": [
92
+ [
93
+ 1.0,
94
+ 0.0,
95
+ 0.0,
96
+ -0.1
97
+ ],
98
+ [
99
+ 0.0,
100
+ 1.0,
101
+ 0.0,
102
+ 0.0
103
+ ],
104
+ [
105
+ 0.0,
106
+ 0.0,
107
+ 1.0,
108
+ 0.0
109
+ ],
110
+ [
111
+ 0.0,
112
+ 0.0,
113
+ 0.0,
114
+ 1.0
115
+ ]
116
+ ],
117
+ "tool_frame_definition": {
118
+ "schema_version": 1,
119
+ "canonical_tool_frame_id": "widowx_tool_canonical_v1",
120
+ "canonical_reference_robot": "widowx250",
121
+ "robot_name": "panda-widowx",
122
+ "axis_semantics": [
123
+ "forward",
124
+ "left_finger",
125
+ "palm_up"
126
+ ],
127
+ "raw_forward_axis": [
128
+ 0.0,
129
+ 0.0,
130
+ 1.0
131
+ ],
132
+ "raw_left_axis": [
133
+ 0.0,
134
+ 1.0,
135
+ 0.0
136
+ ],
137
+ "raw_palm_up_axis": [
138
+ -1.0,
139
+ 0.0,
140
+ 0.0
141
+ ],
142
+ "raw_from_canonical": [
143
+ [
144
+ 0.0,
145
+ 0.0,
146
+ -1.0
147
+ ],
148
+ [
149
+ 0.0,
150
+ 1.0,
151
+ 0.0
152
+ ],
153
+ [
154
+ 1.0,
155
+ 0.0,
156
+ 0.0
157
+ ]
158
+ ],
159
+ "robot_xml_sha256": "5f8b242ead3600740065e135c609c34ed40bfdc29201d50a5b366abf0b13da3c",
160
+ "evidence": [
161
+ "compiled wx250s/gripper_link +X finger-forward equals grip_site raw +Z",
162
+ "compiled left_finger positive slide opens along grip_site raw +Y",
163
+ "grip_site is Ry(+90deg) relative to the transplanted WidowX gripper body"
164
+ ]
165
+ },
166
+ "tool_frame_definition_sha256": "360ba397b682c1e042beebb3dd4f5cf47561601cef9c3d14c6558184ad37ecdd",
167
+ "reference_tool_frame_definition_sha256": "f3aef5e783001a5718e4a0c42020b06d137fb429c04e544a67934dac7decf4ba",
168
+ "recording_raw_from_canonical": [
169
+ [
170
+ 0.0,
171
+ 0.0,
172
+ -1.0
173
+ ],
174
+ [
175
+ 0.0,
176
+ 1.0,
177
+ 0.0
178
+ ],
179
+ [
180
+ 1.0,
181
+ 0.0,
182
+ 0.0
183
+ ]
184
+ ],
185
+ "jaw_symmetry_correction": [
186
+ [
187
+ 1.0,
188
+ 0.0,
189
+ 0.0
190
+ ],
191
+ [
192
+ 0.0,
193
+ 1.0,
194
+ 0.0
195
+ ],
196
+ [
197
+ 0.0,
198
+ 0.0,
199
+ 1.0
200
+ ]
201
+ ],
202
+ "paired_teleop_alignment": "derived_identity_from_panda_robotiq_mapping_and_widowx_jaw_axes"
203
+ }
204
+ },
205
+ "language_embedding": {
206
+ "reuse_video_embeddings": true,
207
+ "source_tensor_key": "encoded_text",
208
+ "output_tensor_key": "language_embedding",
209
+ "max_length": 512,
210
+ "embedding_dim": 1024,
211
+ "storage_dtype": "float32"
212
+ },
213
+ "checkpoints": {
214
+ "repo_id": "jonpai/mimic-video",
215
+ "revision": "f28339034831e3c2374be075e622e1ff38ebe0f8",
216
+ "bridge_action_decoder": "action_decoder/w2a_bridge_v2w_bridge_lora_rank256_lr1.778e-04_bsz64_iter_000070043_fused_lr1.000e-04_layer20_bsz256_iter_000014112.pt",
217
+ "bridge_action_decoder_bytes": 998172132,
218
+ "bridge_action_decoder_sha256": "ab36a48096f13e6b5fcf6c48f9d33a9bf902a20e3e37f47f4815920bf3ea6559",
219
+ "video_lora_selection": "latest_complete_after_video_job"
220
+ },
221
+ "published_video_lora": {
222
+ "delivery": "standalone_hub",
223
+ "repo_id": "dreamdifferent/vam-cross-level2-panda-widowx-widowx-texture-video-lora-iter200",
224
+ "revision": "5a22b23f6f4bf80b249d6215d15260b7b59acbcf",
225
+ "path": "checkpoints/model/iter_000000200.pt",
226
+ "path_in_repo": "checkpoints/model/iter_000000200.pt",
227
+ "local_path": "selected_video_loras/dreamdifferent--vam-cross-level2-panda-widowx-widowx-texture-video-lora-iter200-5a22b23/checkpoints/model/iter_000000200.pt",
228
+ "iteration": 200,
229
+ "source_iteration": 200,
230
+ "bytes": 741588124,
231
+ "sha256": "45ff819e0044fc5a77d894e8f5d36caef9c28d68fb470ada0fe89724922e9b13",
232
+ "manifest_path": "vam_cross_video_lora_manifest.json",
233
+ "receipt_path": "selected_video_loras/dreamdifferent--vam-cross-level2-panda-widowx-widowx-texture-video-lora-iter200-5a22b23/VAM_CROSS_SELECTED_VIDEO_LORA.json",
234
+ "kind": "video2world_lora"
235
+ },
236
+ "training": {
237
+ "stage": "world2action",
238
+ "initialization": "pretrained_action_decoder",
239
+ "data_config": "bridge",
240
+ "experiment": "w2a_panda_widowx_level2_widowx_texture_2cam_hstack_action_iter2374_videolora_iter200_widowx_teleop_recording_frame_v1",
241
+ "job_project": "vam_cross",
242
+ "job_group": "world2action",
243
+ "dataset_seed": 42,
244
+ "num_val_episodes": 10,
245
+ "statistics_num_workers": 0,
246
+ "statistics_batch_size": 4,
247
+ "xattn_layer": 20,
248
+ "learning_rate": 0.0001,
249
+ "run_validation": false,
250
+ "wandb_mode": "offline",
251
+ "profiles": {
252
+ "smoke": {
253
+ "run_name": "w2a_panda_widowx_level2_widowx_texture_2cam_hstack_action_iter2374_videolora_iter200_widowx_teleop_recording_frame_v1_smoke",
254
+ "gpus": 1,
255
+ "gpu_type": "a100_80gb",
256
+ "per_gpu_batch_size": 1,
257
+ "gradient_accumulation": 8,
258
+ "num_workers": 0,
259
+ "max_iterations": 10,
260
+ "save_every": 10,
261
+ "log_every": 1,
262
+ "walltime": "04:00:00"
263
+ },
264
+ "full": {
265
+ "run_name": "w2a_panda_widowx_level2_widowx_texture_2cam_hstack_action_iter2374_videolora_iter200_widowx_teleop_recording_frame_v1_full",
266
+ "gpus": 2,
267
+ "gpu_type": "a100_80gb",
268
+ "per_gpu_batch_size": 2,
269
+ "gradient_accumulation": 8,
270
+ "num_workers": 1,
271
+ "effective_batch_size": 32,
272
+ "max_iterations": 3000,
273
+ "save_every": 100,
274
+ "log_every": 10,
275
+ "keep_latest_only": false,
276
+ "require_smoke_receipt": false,
277
+ "walltime": "24:00:00"
278
+ }
279
+ }
280
+ },
281
+ "storage": {
282
+ "expected_raw_rgb_bytes": 8447385600,
283
+ "recommended_free_space_gb": 20
284
+ },
285
+ "action_frame_initialization": {
286
+ "schema_version": 1,
287
+ "source_tool_frame_id": "native_grip_site_v1",
288
+ "target_tool_frame_id": "widowx_teleop_recording_frame_v1",
289
+ "reference_robot": "widowx250",
290
+ "numeric_transform": "identity",
291
+ "source_dataset_repo_id": "dreamdifferent/vam-cross-target-widowx250-native",
292
+ "reason": "widowx250 raw grip_site defines the canonical reference"
293
+ },
294
+ "initial_action_decoder": {
295
+ "delivery": "standalone_hub",
296
+ "repo_id": "dreamdifferent/vam-cross-target-widowx250-native-2cam-action-decoder",
297
+ "revision": "93750cccda01620e3c028477e4c49bc5c996a68d",
298
+ "path": "checkpoints/model/iter_000002374.pt",
299
+ "path_in_repo": "checkpoints/model/iter_000002374.pt",
300
+ "local_path": "initial_action_decoders/dreamdifferent--vam-cross-target-widowx250-native-2cam-action-decoder-93750cc/checkpoints/model/iter_000002374.pt",
301
+ "iteration": 2374,
302
+ "source_iteration": 2374,
303
+ "bytes": 998172132,
304
+ "sha256": "3a9a75686164351468ef9b99098ae9152a1f07082b869e1ad03c9e1e1cf6885c",
305
+ "manifest_path": "vam_cross_action_decoder_manifest.json",
306
+ "receipt_path": "initial_action_decoders/dreamdifferent--vam-cross-target-widowx250-native-2cam-action-decoder-93750cc/VAM_CROSS_INITIAL_ACTION_DECODER.json",
307
+ "training_dataset_repo_id": "dreamdifferent/vam-cross-target-widowx250-native",
308
+ "training_robot_name": "widowx250",
309
+ "kind": "pretrained_world2action_decoder"
310
+ }
311
+ }