Ma7ee7 commited on
Commit
93b70dc
·
verified ·
1 Parent(s): e5e4582

Upload 5 files

Browse files
Files changed (6) hide show
  1. .gitattributes +1 -0
  2. README.md +102 -1
  3. bad_apple_nn.py +0 -0
  4. demo.mp4 +3 -0
  5. model.pt +3 -0
  6. requirements.txt +3 -0
.gitattributes CHANGED
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ demo.mp4 filter=lfs diff=lfs merge=lfs -text
README.md CHANGED
@@ -1,3 +1,104 @@
1
  ---
2
- license: apache-2.0
 
 
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ library_name: pytorch
3
+ tags:
4
+ - pytorch
5
+ - implicit-neural-representation
6
+ - coordinate-network
7
+ - multimodal
8
+ - video
9
+ - audio
10
  ---
11
+
12
+ # Bad Apple Unified Neural Representation
13
+
14
+ This is a compact coordinate-based neural representation of the complete
15
+ *Bad Apple!!* shadow video and its stereo audio.
16
+
17
+ The checkpoint does not store ordinary video frames or compressed audio. Given
18
+ a normalized time and pixel coordinate, the network predicts the image
19
+ brightness. Given a normalized time coordinate with audio conditioning, the
20
+ same network predicts the stereo waveform.
21
+
22
+ ## Generated Result
23
+
24
+ <!--
25
+ Upload your rendered MP4 as demo.mp4, then replace YOUR_USERNAME and
26
+ YOUR_REPOSITORY below.
27
+ -->
28
+
29
+ <video controls width="100%">
30
+ <source src="https://huggingface.co/YOUR_USERNAME/YOUR_REPOSITORY/resolve/main/demo.mp4" type="video/mp4">
31
+ Your browser does not support embedded video.
32
+ </video>
33
+
34
+ [Download or open the generated video](./demo.mp4)
35
+
36
+ ## Run
37
+
38
+ Install the Python requirements and make sure `ffmpeg` is available on your
39
+ system. Then render the checkpoint:
40
+
41
+ ```bash
42
+ python bad_apple_nn.py render model.pt --audio-source generated
43
+ ```
44
+
45
+ The generated MP4 is written to `outputs/`.
46
+
47
+ ## What The Model Is
48
+
49
+ This is **one unified multimodal model**, not two independently trained models.
50
+
51
+ Video and audio share:
52
+
53
+ - one normalized time representation;
54
+ - Fourier coordinate features;
55
+ - an interpolated learned time-memory;
56
+ - modality conditioning; and
57
+ - one residual neural-network trunk.
58
+
59
+ The shared trunk ends in two small task-specific output heads:
60
+
61
+ - a video head that predicts grayscale pixel brightness;
62
+ - an audio head that predicts two stereo waveform channels.
63
+
64
+ Separate output heads are necessary because pixels and audio samples have
65
+ different output shapes, but the representation and main network are shared
66
+ and trained together in one checkpoint.
67
+
68
+ ## Model Details
69
+
70
+ | Property | Value |
71
+ |---|---:|
72
+ | Architecture | Unified coordinate neural field (`unified-v3`) |
73
+ | Trainable parameters | 1,111,571 |
74
+ | Training video resolution | 192 x 144 |
75
+ | Rendered resolution at default 3x scale | 576 x 432 |
76
+ | Frames | 6,572 |
77
+ | Frame rate | 30 FPS |
78
+ | Duration | About 3 minutes 39 seconds |
79
+ | Generated audio | 16 kHz stereo |
80
+ | Best checkpoint step | 59,000 / 80,000 |
81
+ | Video pixel accuracy | 99.6% |
82
+ | Silhouette IoU | 0.991 |
83
+
84
+ The inference weights occupy approximately 4.45 MB in FP32. The uploaded
85
+ training checkpoint may be larger because it contains both regular and
86
+ exponential-moving-average weights. Rendering uses the EMA weights by default.
87
+
88
+ ## Files
89
+
90
+ | File | Purpose |
91
+ |---|---|
92
+ | `model.pt` | Unified video-and-audio checkpoint |
93
+ | `bad_apple_nn.py` | Model definition and renderer |
94
+ | `requirements.txt` | Python dependencies |
95
+ | `demo.mp4` | Example generated result |
96
+
97
+ ## Notes
98
+
99
+ - This is a custom PyTorch model, not a Transformers model.
100
+ - It is an implicit neural representation specialized to one video.
101
+ - The checkpoint generates both the frames and audio.
102
+ - `ffmpeg` is required to encode and combine the rendered streams.
103
+ - This repository is an experimental and educational demonstration. Rights to
104
+ the original music and animation remain with their respective owners.
bad_apple_nn.py ADDED
The diff for this file is too large to render. See raw diff
 
demo.mp4 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:710bdf4bfebdbb4e610a6201a73f7d0e20fb082d39794bbceed9ae7cde085828
3
+ size 27383988
model.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0eeb84cb010b6cd1722eb1a2e1050586cddf1af2c6b27f9ae99295f894f17889
3
+ size 8918518
requirements.txt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ numpy
2
+ opencv-python
3
+ torch