Spaces:
Running
Running
| <html> | |
| <head> | |
| <meta charset="utf-8"> | |
| <meta name="description" | |
| content="We introduce DissolveStereo, a novel framework for zero-shot stereo video generation that leverages video diffusion priors without requiring paired training data."> | |
| <meta name="keywords" content="StereoCrafter, Dissolving, Depth, Stereo, StereoCrafter-Zero, DissolveStereo"> | |
| <meta name="viewport" content="width=device-width, initial-scale=1"> | |
| <title>DissolveStereo</title> | |
| <link href="https://fonts.googleapis.com/css?family=Google+Sans|Noto+Sans|Castoro" | |
| rel="stylesheet"> | |
| <link rel="stylesheet" href="./static/css/bulma.min.css"> | |
| <link rel="stylesheet" href="./static/css/bulma-carousel.min.css"> | |
| <link rel="stylesheet" href="./static/css/bulma-slider.min.css"> | |
| <link rel="stylesheet" href="./static/css/fontawesome.all.min.css"> | |
| <link rel="stylesheet" | |
| href="https://cdn.jsdelivr.net/gh/jpswalsh/academicons@1/css/academicons.min.css"> | |
| <link rel="stylesheet" href="./static/css/index.css"> | |
| <link rel="icon" href="./static/images/favicon.svg"> | |
| <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.5.1/jquery.min.js"></script> | |
| <script defer src="./static/js/fontawesome.all.min.js"></script> | |
| <script src="./static/js/bulma-carousel.min.js"></script> | |
| <script src="./static/js/bulma-slider.min.js"></script> | |
| <script src="./static/js/index.js"></script> | |
| </head> | |
| <body> | |
| <section class="hero"> | |
| <div class="hero-body"> | |
| <div class="container is-max-desktop"> | |
| <div class="columns is-centered"> | |
| <div class="column has-text-centered"> | |
| <h1 class="title is-1 publication-title">DissolveStereo: Coarse Depth Injection for Zero-Shot Stereo Video Generation</h1> | |
| <div class="is-size-5 publication-authors"> | |
| <span class="author-block"> | |
| <a href="" target="_blank">Jian Shi</a>,</span> | |
| <span class="author-block"> | |
| <a href="" target="_blank">Qian Wang</a>,</span> | |
| <span class="author-block"> | |
| <a href="" target="_blank">Zhenyu Li</a>, | |
| </span> | |
| <span class="author-block"> | |
| <a href="" target="_blank">WenQing Cui</a>, | |
| </span> | |
| <span class="author-block"> | |
| <a href="" target="_blank">Ramzi Idoughi</a>, | |
| </span> | |
| <span class="author-block"> | |
| <a href="" target="_blank">Peter Wonka</a> | |
| </span> | |
| </div> | |
| <div class="is-size-5 publication-authors"> | |
| <span class="author-block">KAUST</span> | |
| </div> | |
| <div class="column has-text-centered"> | |
| <div class="publication-links"> | |
| <!-- PDF Link. --> | |
| <span class="link-block"> | |
| <a href="https://arxiv.org/pdf/2411.14295" target="_blank" | |
| class="external-link button is-normal is-rounded is-dark"> | |
| <span class="icon"> | |
| <i class="fas fa-file-pdf"></i> | |
| </span> | |
| <span>Paper</span> | |
| </a> | |
| </span> | |
| <span class="link-block"> | |
| <a href="https://arxiv.org/abs/2411.14295" target="_blank" | |
| class="external-link button is-normal is-rounded is-dark"> | |
| <span class="icon"> | |
| <i class="ai ai-arxiv"></i> | |
| </span> | |
| <span>arXiv</span> | |
| </a> | |
| </span> | |
| <!-- Video Link. --> | |
| <!-- <span class="link-block"> | |
| <a href="https://www.youtube.com/watch?v=MrKrnHhk8IA" target="_blank" | |
| class="external-link button is-normal is-rounded is-dark"> | |
| <span class="icon"> | |
| <i class="fab fa-youtube"></i> | |
| </span> | |
| <span>Video</span> | |
| </a> | |
| </span> --> | |
| <!-- Code Link. --> | |
| <span class="link-block"> | |
| <a href="https://github.com/shijianjian/DissolveStereo" target="_blank" | |
| class="external-link button is-normal is-rounded is-dark"> | |
| <span class="icon"> | |
| <i class="fab fa-github"></i> | |
| </span> | |
| <span>Code</span> | |
| </a> | |
| </span> | |
| <!-- Dataset Link. --> | |
| <!-- <span class="link-block"> | |
| <a href="https://github.com/google/nerfies/releases/tag/0.1" target="_blank" | |
| class="external-link button is-normal is-rounded is-dark"> | |
| <span class="icon"> | |
| <i class="far fa-images"></i> | |
| </span> | |
| <span>Data</span> | |
| </a> | |
| </span> --> | |
| </div> | |
| </div> | |
| </div> | |
| </div> | |
| </div> | |
| </div> | |
| </section> | |
| <section class="hero is-small is-light"> | |
| <div class="hero-body"> | |
| <div class="container is-max-widescreen"> | |
| <h2 class="title is-3">Side-By-Side 3D</h2> | |
| <p class="subtitle has-text-centered" style="margin-bottom: 1.5em;"> | |
| View with cross-eye or parallel viewing techniques, or use a VR headset for immersive 3D experience. | |
| </p> | |
| <div id="results-carousel-1" class="carousel results-carousel"> | |
| <div class="item item-video4"> | |
| <video poster="" id="video4" autoplay controls muted loop height="100%" width="100%">\ | |
| <!-- Your video file here --> | |
| <source src="videos/027_a_girl_holding_a_staff,_on_rocks,_lo-r.mp4" | |
| type="video/mp4"> | |
| </video> | |
| </div> | |
| <div class="item item-video1"> | |
| <video poster="" id="video1" autoplay controls muted loop height="100%" width="100%"> | |
| <!-- Your video file here --> | |
| <source src="videos/010_the_land_with_trees-r.mp4" | |
| type="video/mp4"> | |
| </video> | |
| </div> | |
| <div class="item item-video4"> | |
| <video poster="" id="video4" autoplay controls muted loop height="100%" width="100%">\ | |
| <!-- Your video file here --> | |
| <source src="videos/t05_in_a_room-r.mp4" | |
| type="video/mp4"> | |
| </video> | |
| </div> | |
| <div class="item item-video2"> | |
| <video poster="" id="video2" autoplay controls muted loop height="100%" width="100%"> | |
| <source src="videos/004_white_grass-r.mp4" | |
| type="video/mp4"> | |
| </video> | |
| </div> | |
| <div class="item item-video3"> | |
| <video poster="" id="video3" autoplay controls muted loop height="100%" width="100%">\ | |
| <!-- Your video file here --> | |
| <source src="videos/020_a_frozen_leaf_and_rock-r_cropped_ffmpeg.mp4" | |
| type="video/mp4"> | |
| </video> | |
| </div> | |
| <div class="item item-video4"> | |
| <video poster="" id="video4" autoplay controls muted loop height="100%" width="100%">\ | |
| <!-- Your video file here --> | |
| <source src="videos/043_sprout-r.mp4" | |
| type="video/mp4"> | |
| </video> | |
| </div> | |
| <div class="item item-video4"> | |
| <video poster="" id="video3" autoplay controls muted loop height="100%" width="100%">\ | |
| <!-- Your video file here --> | |
| <source src="videos/t01_one_guy-r.mp4" | |
| type="video/mp4"> | |
| </video> | |
| </div> | |
| <!-- <div class="item item-video4"> | |
| <video poster="" id="video4" autoplay controls muted loop height="100%" width="100%">\ | |
| <source src="videos/t03_objects_on_a_table-r.mp4" | |
| type="video/mp4"> | |
| </video> | |
| </div> --> | |
| </div> | |
| </div> | |
| </div> | |
| </div> | |
| </section> | |
| <!-- Video carousel --> | |
| <section class="hero is-small is-light"> | |
| <div class="hero-body"> | |
| <div class="container is-max-desktop"> | |
| <h2 class="title is-3">Anaglyph 3D</h2> | |
| <p class="subtitle has-text-centered" style="margin-bottom: 1.5em;"> | |
| Use red-cyan 3D glasses to view the stereoscopic depth effect. | |
| </p> | |
| <div id="results-carousel-2" class="carousel results-carousel"> | |
| <div class="item item-video4"> | |
| <video poster="" id="video4" autoplay controls muted loop height="100%" width="100%">\ | |
| <!-- Your video file here --> | |
| <source src="videos/027_a_girl_holding_a_staff,_on_rocks,_lo-r_overlay.mp4" | |
| type="video/mp4"> | |
| </video> | |
| </div> | |
| <div class="item item-video5"> | |
| <video poster="" id="video5" autoplay controls muted loop height="100%" width="100%"> | |
| <!-- Your video file here --> | |
| <source src="videos/010_the_land_with_trees-r_overlay.mp4" | |
| type="video/mp4"> | |
| </video> | |
| </div> | |
| <div class="item item-video7"> | |
| <video poster="" id="video7" autoplay controls muted loop height="100%" width="100%">\ | |
| <!-- Your video file here --> | |
| <source src="videos/t05_in_a_room-r_overlay.mp4" | |
| type="video/mp4"> | |
| </video> | |
| </div> | |
| <div class="item item-video6"> | |
| <video poster="" id="video6" autoplay controls muted loop height="100%" width="100%"> | |
| <source src="videos/004_white_grass-r_overlay.mp4" | |
| type="video/mp4"> | |
| </video> | |
| </div> | |
| <div class="item item-video3"> | |
| <video poster="" id="video3" autoplay controls muted loop height="100%" width="100%">\ | |
| <!-- Your video file here --> | |
| <source src="videos/020_a_frozen_leaf_and_rock-r_overlay_cropped_ffmpeg.mp4" | |
| type="video/mp4"> | |
| </video> | |
| </div> | |
| <div class="item item-video8"> | |
| <video poster="" id="video8" autoplay controls muted loop height="100%" width="100%">\ | |
| <!-- Your video file here --> | |
| <source src="videos/043_sprout-r_overlay.mp4" | |
| type="video/mp4"> | |
| </video> | |
| </div> | |
| <div class="item item-video8"> | |
| <video poster="" id="video8" autoplay controls muted loop height="100%" width="100%">\ | |
| <!-- Your video file here --> | |
| <source src="videos/t01_one_guy-r_overlay.mp4" | |
| type="video/mp4"> | |
| </video> | |
| </div> | |
| <!-- <div class="item item-video9"> | |
| <video poster="" id="video9" autoplay controls muted loop height="100%" width="100%">\ | |
| <source src="videos/t03_objects_on_a_table-r_overlay.mp4" | |
| type="video/mp4"> | |
| </video> | |
| </div> --> | |
| </div> | |
| </div> | |
| </div> | |
| </div> | |
| </section> | |
| <!-- End video carousel --> | |
| <!-- Paper abstract --> | |
| <section class="section hero"> | |
| <div class="container is-max-desktop"> | |
| <div class="columns is-centered has-text-centered"> | |
| <div class="column is-four-fifths"> | |
| <h2 class="title is-3">Abstract</h2> | |
| <div class="content has-text-justified"> | |
| <p> | |
| Generating high-quality stereo videos requires consistent depth perception and temporal coherence across frames. Despite advances in image and video synthesis using diffusion models, producing high-quality stereo videos remains a challenging task due to the difficulty of maintaining consistent temporal and spatial coherence between left and right views. | |
| We introduce DissolveStereo, a novel framework for zero-shot stereo video generation that leverages video diffusion priors without requiring paired training data. Our key innovations include a noisy restart strategy to initialize stereo-aware latent representations and an iterative refinement process that progressively harmonizes the latent space, addressing issues like temporal flickering and view inconsistencies. | |
| In addition, we propose the use of dissolved depth maps to streamline latent space operations by reducing high-frequency depth information. | |
| Our comprehensive evaluations, including quantitative metrics and user studies, demonstrate that DissolveStereo produces high-quality stereo videos with enhanced depth consistency and temporal smoothness. In terms of epipolar consistency, our method achieves an $11.7\%$ improvement in MEt3R score over the current state-of-the-art. Furthermore, user studies indicate strong perceptual gains over the previous arts, with an $8.0\%$ higher perceived frame quality and $10.9\%$ higher perceived temporal coherence. | |
| Our code is available on <a href="https://github.com/shijianjian/DissolveStereo">GitHub</a>. | |
| </p> | |
| </div> | |
| </div> | |
| </div> | |
| </div> | |
| </section> | |
| <!-- End paper abstract --> | |
| <!-- Paper abstract --> | |
| <section class="section hero is-light"> | |
| <div class="container is-max-widescreen"> | |
| <div class="columns is-centered has-text-centered"> | |
| <div class="column is-four-fifths"> | |
| <h2 class="title is-3">Our Pipeline</h2> | |
| <div class="content has-text-justified"> | |
| <img src="assets/main_figure.jpg" alt="Pipeline Diagram" style="max-width: 100%; height: auto;"> | |
| <p> | |
| An overview of the DissolveStereo pipeline. Top: Our method contains two main components: | |
| (1) <b>Noisy Restart</b> for a robust initial latent estimation and | |
| (2) <b>Iterative Refinement</b> for latent refinement. | |
| These components act on target view latents (blue) for temporal and inter-view consistency with the source view (orange). | |
| Bottom: Given an image and text prompt,our pipeline generates stereo videos with a strong stereoscopic effect. | |
| </p> | |
| </div> | |
| </div> | |
| </div> | |
| </div> | |
| </section> | |
| <!-- Teaser video--> | |
| <section class="hero teaser"> | |
| <div class="container" style="max-width: 1600px;"> | |
| <h3 class="title is-3" style="padding-top: 2em; padding-bottom: 1em;">Visual Comparison</h3> | |
| <p class="subtitle has-text-centered" style="margin-bottom: 2em;"> | |
| Interactive comparison with <i>ProPainter</i>, <i>RoDynRF</i>, <i>ImmersePro</i>, <i>TrajactoryCrafter</i>, <i>StereoCrafter</i>, and <i>StereoDiffusion</i>. Select tabs to compare each method. | |
| </p> | |
| <p class="subtitle has-text-centered" style="margin-bottom: 2em;"> | |
| <b>Artifacts are stronger with zoom-in videos.</b> Using ctrl+scroll to zoom in and out can help in better visualizing these artifacts. | |
| </p> | |
| <!-- Upper Comparison --> | |
| <div class="hero-body"> | |
| <div class="tabs is-centered is-boxed"> | |
| <ul> | |
| <li class="is-active" data-tab="comp-ours-upper"> | |
| <a>Ours</a> | |
| </li> | |
| <li data-tab="comp-stereo-upper"> | |
| <a>StereoDiffusion</a> | |
| </li> | |
| <li data-tab="comp-stereocraft-upper"> | |
| <a>StereoCrafter</a> | |
| </li> | |
| <!-- <li data-tab="comp-traj-upper"> | |
| <a>TrajactoryCrafter</a> | |
| </li> --> | |
| <li data-tab="comp-immerse-upper"> | |
| <a>ImmersePro</a> | |
| </li> | |
| <li data-tab="comp-painter-upper"> | |
| <a>ProPainter</a> | |
| </li> | |
| <li data-tab="comp-svg-upper"> | |
| <a>SVG</a> | |
| </li> | |
| </ul> | |
| <div class="has-text-centered" style="margin-bottom:1em; visibility: hidden;"> | |
| <button class="button is-primary">Play / Pause Both</button> | |
| </div> | |
| </div> | |
| <div style="display:flex; justify-content: space-between;"> | |
| <div style="display:flex; width: 50%;"></div> | |
| <div style="display:flex; width: 50%; justify-content: center;"> | |
| 🔻 Comparing View | |
| </div> | |
| </div> | |
| <div id="comp-ours-upper" class="tab-content is-active"> | |
| <video poster="" autoplay controls muted loop height="100%" width="100%"> | |
| <source src="videos/comparison/ours_out.mp4" type="video/mp4"> | |
| </video> | |
| </div> | |
| <div id="comp-stereo-upper" class="tab-content"> | |
| <video poster="" autoplay controls muted loop height="100%" width="100%"> | |
| <source src="videos/comparison/stereodiffusion_out_cropped_ffmpeg.mp4" type="video/mp4"> | |
| </video> | |
| <p style="float: right; color: red;">No temporal consistency</p> | |
| </div> | |
| <div id="comp-stereocraft-upper" class="tab-content"> | |
| <video poster="" autoplay controls muted loop height="100%" width="100%"> | |
| <source src="videos/comparison/stereocrafter_out_cropped_ffmpeg.mp4" type="video/mp4"> | |
| </video> | |
| <p style="float: right; color: red;">Oversmoothed texture</p> | |
| </div> | |
| <!-- <div id="comp-traj-upper" class="tab-content"> | |
| <video poster="" autoplay controls muted loop height="100%" width="100%"> | |
| <source src="videos/comparison/_output_aligned.mp4" type="video/mp4"> | |
| </video> | |
| <div style="float: right;"> | |
| <p>TrajctoryCrafter contains less frames since it uses several frames for camera pose estimation.</p> | |
| <p>Video frames were trippled and downsampled the to meet their computation requirements.</p> | |
| </div> | |
| </div> --> | |
| <div id="comp-immerse-upper" class="tab-content"> | |
| <video poster="" autoplay controls muted loop height="100%" width="100%"> | |
| <source src="videos/comparison/immersepro_out.mp4" type="video/mp4"> | |
| </video> | |
| <p style="float: right; color: red;">Whitened overall coloring; less stereoscopic effect.</p> | |
| </div> | |
| <div id="comp-painter-upper" class="tab-content"> | |
| <video poster="" autoplay controls muted loop height="100%" width="100%"> | |
| <source src="videos/comparison/propainter_out.mp4" type="video/mp4"> | |
| </video> | |
| <p style="float: right; color: red;">Significant background noise.</p> | |
| </div> | |
| <div id="comp-svg-upper" class="tab-content"> | |
| <video poster="" autoplay controls muted loop height="100%" width="100%"> | |
| <source src="videos/comparison/svg_out.mp4" type="video/mp4"> | |
| </video> | |
| <p style="float: right; color: red;">Significant artifacts.</p> | |
| </div> | |
| </div> | |
| <!-- Lower Comparison --> | |
| <div class="hero-body"> | |
| <div class="tabs is-centered is-boxed"> | |
| <ul> | |
| <li data-tab="comp-ours-lower"> | |
| <a>Ours</a> | |
| </li> | |
| <li class="is-active" data-tab="comp-stereo-lower"> | |
| <a>StereoDiffusion</a> | |
| </li> | |
| <li data-tab="comp-stereocraft-lower"> | |
| <a>StereoCrafter</a> | |
| </li> | |
| <!-- <li data-tab="comp-traj-lower"> | |
| <a>TrajactoryCrafter</a> | |
| </li> --> | |
| <li data-tab="comp-immerse-lower"> | |
| <a>ImmersePro</a> | |
| </li> | |
| <li data-tab="comp-painter-lower"> | |
| <a>ProPainter</a> | |
| </li> | |
| <li data-tab="comp-svg-lower"> | |
| <a>SVG</a> | |
| </li> | |
| </ul> | |
| <div class="has-text-centered" style="margin-bottom:1em;"> | |
| <button id="sync-playpause" class="button is-primary">Play / Pause Both</button> | |
| </div> | |
| </div> | |
| <div style="display:flex; justify-content: space-between;"> | |
| <div style="display:flex; width: 50%;"></div> | |
| <div style="display:flex; width: 50%; justify-content: center;"> | |
| 🔻 Comparing View | |
| </div> | |
| </div> | |
| <div id="comp-ours-lower" class="tab-content"> | |
| <p style="float: right; color: red;"> </p> | |
| <video poster="" autoplay controls muted loop height="100%" width="100%"> | |
| <source src="videos/comparison/ours_out.mp4" type="video/mp4"> | |
| </video> | |
| </div> | |
| <div id="comp-stereo-lower" class="tab-content is-active"> | |
| <p style="float: right; color: red;">No temporal consistency</p> | |
| <video poster="" autoplay controls muted loop height="100%" width="100%"> | |
| <source src="videos/comparison/stereodiffusion_out_cropped_ffmpeg.mp4" type="video/mp4"> | |
| </video> | |
| </div> | |
| <div id="comp-stereocraft-lower" class="tab-content"> | |
| <p style="float: right; color: red;">Oversmoothed texture</p> | |
| <video poster="" autoplay controls muted loop height="100%" width="100%"> | |
| <source src="videos/comparison/stereocrafter_out_cropped_ffmpeg.mp4" type="video/mp4"> | |
| </video> | |
| </div> | |
| <!-- <div id="comp-traj-lower" class="tab-content"> | |
| <video poster="" autoplay controls muted loop height="100%" width="100%"> | |
| <source src="videos/comparison/_output_aligned.mp4" type="video/mp4"> | |
| </video> | |
| <div style="float: right;"> | |
| <p>TrajctoryCrafter contains less frames since it uses several frames for camera pose estimation.</p> | |
| <p>Video frames were trippled and downsampled the to meet their computation requirements.</p> | |
| </div> | |
| </div> --> | |
| <div id="comp-immerse-lower" class="tab-content"> | |
| <p style="float: right; color: red;">Whitened overall coloring; less stereoscopic effect.</p> | |
| <video poster="" autoplay controls muted loop height="100%" width="100%"> | |
| <source src="videos/comparison/immersepro_out.mp4" type="video/mp4"> | |
| </video> | |
| </div> | |
| <div id="comp-painter-lower" class="tab-content"> | |
| <p style="float: right; color: red;">Significant background noise.</p> | |
| <video poster="" autoplay controls muted loop height="100%" width="100%"> | |
| <source src="videos/comparison/propainter_out.mp4" type="video/mp4"> | |
| </video> | |
| </div> | |
| <div id="comp-svg-lower" class="tab-content"> | |
| <p style="float: right; color: red;">Significant artifacts.</p> | |
| <video poster="" autoplay controls muted loop height="100%" width="100%"> | |
| <source src="videos/comparison/svg_out.mp4" type="video/mp4"> | |
| </video> | |
| </div> | |
| </div> | |
| </div> | |
| </section> | |
| <section class="hero teaser"> | |
| <div class="container is-max-widescreen"> | |
| <h3 class="title is-3" style="padding-top: 2em; padding-bottom: 1em;">Diverse Capabilities</h3> | |
| </div> | |
| </section> | |
| <section class="hero teaser is-light"> | |
| <div class="container is-max-widescreen"> | |
| <h3 class="title is-3">Multi-Resolution Support</h3> | |
| <h3 class="title is-5">576x1024</h3> | |
| <table class="center"> | |
| <!-- <tr> | |
| <td colspan="1">"fireworks display"</td> | |
| <td colspan="1">"a robot is walking through a destroyed city"</td> | |
| </tr> --> | |
| <tr> | |
| <td> | |
| <img src=assets/showcase/1024/fireworks_display-r.gif height="160"> | |
| </td> | |
| <td style="width: 1em;"></td> | |
| <td> | |
| <img src=assets/showcase/1024/fireworks_display-r_overlay.gif height="160"> | |
| </td> | |
| </tr> | |
| <tr> | |
| <td> | |
| <img src=assets/showcase/1024/a_robot_is_walking_through_a_destroyed_c-r.gif height="160"> | |
| </td> | |
| <td></td> | |
| <td> | |
| <img src=assets/showcase/1024/a_robot_is_walking_through_a_destroyed_c-r_overlay.gif height="160"> | |
| </td> | |
| </tr> | |
| </table> | |
| <h3 class="title is-5">320x512</h3> | |
| <table class="center"> | |
| <tr> | |
| <td> | |
| <img src=assets/showcase/512/a_sailboat_sailing_in_rough_seas_with_a_-r.gif height="160"> | |
| </td> | |
| <td></td> | |
| <td> | |
| <img src=assets/showcase/512/a_sailboat_sailing_in_rough_seas_with_a_-r_overlay.gif height="160"> | |
| </td> | |
| </tr> | |
| <tr> | |
| <td> | |
| <img src=assets/showcase/512/a_woman_looking_out_in_the_rain-r.gif height="160"> | |
| </td> | |
| <td></td> | |
| <td> | |
| <img src=assets/showcase/512/a_woman_looking_out_in_the_rain-r_overlay.gif height="160"> | |
| </td> | |
| </tr> | |
| </table> | |
| <h3 class="title is-5">256x256</h3> | |
| <div style="display: flex; justify-content: center;"> | |
| <table class="center"> | |
| <tr> | |
| <td> | |
| <img src=assets/showcase/256/a_campfire_on_the_beach_and_the_ocean_wa-r.gif height="140"> | |
| </td> | |
| <td style="width: 1em;"></td> | |
| <td> | |
| <img src=assets/showcase/256/a_campfire_on_the_beach_and_the_ocean_wa-r_overlay.gif height="140"> | |
| </td> | |
| </tr> | |
| <tr> | |
| <td> | |
| <img src=assets/showcase/256/bear_playing_guitar_happily,_snowing-r.gif height="140"> | |
| </td> | |
| <td></td> | |
| <td> | |
| <img src=assets/showcase/256/bear_playing_guitar_happily,_snowing-r_overlay.gif height="140"> | |
| </td> | |
| </tr> | |
| </table> | |
| </div> | |
| </div> | |
| </section> | |
| <section class="hero teaser is-light"> | |
| <div class="container is-max-widescreen"> | |
| <h3 class="title is-3">Generative frame interpolation</h3> | |
| <h3 class="title is-5">We input the starting and ending frames as our input and generate stereo content with our method.</h3> | |
| <p style="display: flex;justify-content: center;"> | |
| <div style="display: block; max-width: 50%;"> | |
| <table class="center is-max-desktop"> | |
| <tr style="font-weight: bolder;text-align:center;"> | |
| <td>Input starting frame</td> | |
| <td></td> | |
| <td>Input ending frame</td> | |
| </tr> | |
| <tr> | |
| <td> | |
| <img src=assets/application/smile_start.png> | |
| </td> | |
| <td style="font-size: 80px; font-weight: bolder;margin: 2em;"> | |
| <span class="big-ellipsis">…</span> | |
| </td> | |
| <td> | |
| <img src=assets/application/smile_end.png> | |
| </td> | |
| </tr> | |
| </table> | |
| </div> | |
| <div style="display: block; max-width: 100%;"> | |
| <table class="center"> | |
| <tr> | |
| <td> | |
| <img src=assets/showcase/interp/a_smiling_girl-r.gif height="80"> | |
| </td> | |
| <td style="width: 1em;"></td> | |
| <td> | |
| <img src=assets/showcase/interp/a_smiling_girl-r_overlay.gif height="80"> | |
| </td> | |
| </tr> | |
| </table> | |
| </div> | |
| </p> | |
| <p style="display: flex;justify-content: center;"> | |
| <div style="display: block; max-width: 50%;"> | |
| <table class="center is-max-desktop"> | |
| <tr style="font-weight: bolder;text-align:center;"> | |
| <td>Input starting frame</td> | |
| <td></td> | |
| <td>Input ending frame</td> | |
| </tr> | |
| <tr> | |
| <td> | |
| <img src=assets/application/stone01_start.png> | |
| </td> | |
| <td style="font-size: 80px; font-weight: bolder;margin: 2em;"> | |
| <span class="big-ellipsis">…</span> | |
| </td> | |
| <td> | |
| <img src=assets/application/stone01_end.png> | |
| </td> | |
| </tr> | |
| </table> | |
| </div> | |
| <div style="display: block; max-width: 100%;"> | |
| <table class="center"> | |
| <tr> | |
| <td> | |
| <img src=assets/showcase/interp/rotating_view-r.gif height="80"> | |
| </td> | |
| <td style="width: 1em;"></td> | |
| <td> | |
| <img src=assets/showcase/interp/rotating_view-r_overlay.gif height="80"> | |
| </td> | |
| </tr> | |
| </table> | |
| </div> | |
| </p> | |
| </div> | |
| </section> | |
| <section class="hero teaser"> | |
| <div class="container is-max-widescreen"> | |
| <h2 class="title">Depth Dissolving</h2> | |
| <h3 class="title is-5"> | |
| We apply depth dissolving to smooth out high-frequency artifacts in the depth map. | |
| This technique progressively reduces fine details in the depth representation, yielding a smoother and more stable depth representation for stereo video generation | |
| without introducing sharp transitions or discontinuities. | |
| Eventually, resulting in smoother and more stable stereo video generation. | |
| The parameter <b>t</b> controls the dissolving strength, with higher values producing more aggressive smoothing. | |
| </h3> | |
| <!-- Lower Comparison --> | |
| <div class="hero-body"> | |
| <div class="tabs is-centered is-boxed"> | |
| <ul> | |
| <li data-tab="comp-tnone"> | |
| <a>No Dissolving</a> | |
| </li> | |
| <li data-tab="comp-t20"> | |
| <a>t=20</a> | |
| </li> | |
| <li data-tab="comp-t30"> | |
| <a>t=30</a> | |
| </li> | |
| <li class="is-active" data-tab="comp-t40"> | |
| <a>t=40</a> | |
| </li> | |
| </ul> | |
| </div> | |
| <div id="comp-tnone" class="tab-content"> | |
| <video poster="" autoplay controls muted loop height="100%" width="50%"> | |
| <source src="videos/dissolving/sonic_dslv_none/running_sonic_the_hedgehog_vis.mp4" type="video/mp4"> | |
| </video> | |
| <video poster="" autoplay controls muted loop height="100%" width="100%"> | |
| <source src="videos/dissolving/sonic_dslv_none/running_sonic_the_hedgehog-r.mp4" type="video/mp4"> | |
| </video> | |
| </div> | |
| <div id="comp-t20" class="tab-content"> | |
| <video poster="" autoplay controls muted loop height="100%" width="50%"> | |
| <source src="videos/dissolving/sonic_dslv_20/running_sonic_the_hedgehog_vis.mp4" type="video/mp4"> | |
| </video> | |
| <video poster="" autoplay controls muted loop height="100%" width="100%"> | |
| <source src="videos/dissolving/sonic_dslv_20/running_sonic_the_hedgehog-r.mp4" type="video/mp4"> | |
| </video> | |
| </div> | |
| <div id="comp-t30" class="tab-content"> | |
| <video poster="" autoplay controls muted loop height="100%" width="50%"> | |
| <source src="videos/dissolving/sonic_dslv_30/running_sonic_the_hedgehog_vis.mp4" type="video/mp4"> | |
| </video> | |
| <video poster="" autoplay controls muted loop height="100%" width="100%"> | |
| <source src="videos/dissolving/sonic_dslv_30/running_sonic_the_hedgehog-r.mp4" type="video/mp4"> | |
| </video> | |
| </div> | |
| <div id="comp-t40" class="tab-content is-active"> | |
| <video poster="" autoplay controls muted loop height="100%" width="50%"> | |
| <source src="videos/dissolving/sonic_dslv_40/running_sonic_the_hedgehog_vis.mp4" type="video/mp4"> | |
| </video> | |
| <video poster="" autoplay controls muted loop height="100%" width="100%"> | |
| <source src="videos/dissolving/sonic_dslv_40/running_sonic_the_hedgehog-r.mp4" type="video/mp4"> | |
| </video> | |
| </div> | |
| </div> | |
| </div> | |
| </div> | |
| </section> | |
| <!-- | |
| <section class="hero teaser"> | |
| <div class="container is-max-widescreen"> | |
| <h4 class="title is-5" style="padding-top: 2em; padding-bottom: 1em;"> | |
| Looping video generation: <b> Starting with a single reference image, we produce a stereo video clip that seamlessly loops, offering visually continuous playback. </b> | |
| </h4> | |
| <table class="center"> | |
| <tr> | |
| <td> | |
| <img src=assets/showcase/interp/clothes_swaying_in_the_wind-r.gif height="140"> | |
| </td> | |
| <td style="width: 1em;"></td> | |
| <td> | |
| <img src=assets/showcase/interp/clothes_swaying_in_the_wind-r_overlay.gif height="140"> | |
| </td> | |
| </tr> | |
| <tr> | |
| <td> | |
| <img src=assets/showcase/interp/flowers_swaying_in_the_wind-r.gif height="140"> | |
| </td> | |
| <td></td> | |
| <td> | |
| <img src=assets/showcase/interp/flowers_swaying_in_the_wind-r_overlay.gif height="140"> | |
| </td> | |
| </tr> | |
| </table> | |
| </div> | |
| </section> --> | |
| <!-- Paper poster | |
| <section class="hero is-small is-light"> | |
| <div class="hero-body"> | |
| <div class="container"> | |
| <h2 class="title">Poster</h2> | |
| <iframe src="static/pdfs/sample.pdf" width="100%" height="550"> | |
| </iframe> | |
| </div> | |
| </div> | |
| </section> | |
| End paper poster --> | |
| <!--BibTex citation --> | |
| <!-- <section class="section" id="BibTeX"> | |
| <div class="container is-max-desktop content"> | |
| <h2 class="title">BibTeX</h2> | |
| <pre><code>BibTex Code Here</code></pre> | |
| </div> | |
| </section> --> | |
| <!--End BibTex citation --> | |
| <!-- Statcounter tracking code --> | |
| <!-- You can add a tracker to track page visits by creating an account at statcounter.com --> | |
| <!-- End of Statcounter Code --> | |
| <script> | |
| // Tab switching functionality — change only the clicked panel's tab, | |
| // but reset/play the other panel's currently active video so both play in sync. | |
| function showTabInContainer(tabGroup, targetName) { | |
| tabGroup.querySelectorAll('li').forEach(x => x.classList.remove('is-active')); | |
| const li = tabGroup.querySelector(`[data-tab="${targetName}"]`); | |
| if (li) li.classList.add('is-active'); | |
| const container = tabGroup.closest('.hero-body'); | |
| container.querySelectorAll('.tab-content').forEach(c => { | |
| c.classList.remove('is-active'); | |
| c.style.display = 'none'; | |
| }); | |
| const contentDiv = container.querySelector('#' + targetName); | |
| if (contentDiv) { | |
| contentDiv.classList.add('is-active'); | |
| contentDiv.style.display = 'block'; | |
| return contentDiv.querySelector('video'); | |
| } | |
| return container.querySelector('video'); | |
| } | |
| document.querySelectorAll('[data-tab]').forEach(tab => { | |
| tab.addEventListener('click', function(e) { | |
| e.preventDefault(); | |
| const tabName = this.getAttribute('data-tab'); // e.g. comp-ours-upper | |
| const tabGroup = this.closest('.tabs'); | |
| const container = tabGroup.closest('.hero-body'); | |
| // Activate only this panel's tab and get its video element | |
| const clickedVideo = showTabInContainer(tabGroup, tabName); | |
| // Find the sibling panel (the other hero-body inside the same section) | |
| const sectionRoot = this.closest('section'); | |
| if (!sectionRoot) return; | |
| const heroBodies = Array.from(sectionRoot.querySelectorAll('.hero-body')); | |
| const otherBody = heroBodies.find(b => b !== container); | |
| // Get the currently active video's element in the other panel (do NOT change its tab) | |
| let otherVideo = null; | |
| if (otherBody) { | |
| const activeContent = otherBody.querySelector('.tab-content.is-active'); | |
| otherVideo = activeContent ? activeContent.querySelector('video') : otherBody.querySelector('video'); | |
| } | |
| // Reset both videos to start and play them together | |
| const videos = [clickedVideo, otherVideo].filter(Boolean); | |
| videos.forEach(v => { try { v.pause(); v.currentTime = 0; } catch (e) {} }); | |
| setTimeout(() => { videos.forEach(v => { try { v.play(); } catch (e) {} }); }, 50); | |
| }); | |
| }); | |
| </script> | |
| <style> | |
| .tab-content { | |
| display: none; | |
| } | |
| .tab-content.is-active { | |
| display: block; | |
| } | |
| .tabs a::after { | |
| top: 90%; | |
| } | |
| section .title { | |
| padding-top: 1em; | |
| } | |
| </style> | |
| <!-- End teaser video --> | |
| <script> | |
| // Synchronized play/pause controls for the Visual Comparison section | |
| function findComparisonSection() { | |
| const sections = Array.from(document.querySelectorAll('section.hero')); | |
| return sections.find(s => { | |
| const h = s.querySelector('h3'); | |
| return h && h.innerText && h.innerText.trim().startsWith('Visual Comparison'); | |
| }); | |
| } | |
| function getActiveVideos() { | |
| const section = findComparisonSection(); | |
| if (!section) return []; | |
| const bodies = Array.from(section.querySelectorAll('.hero-body')); | |
| const videos = bodies.map(body => { | |
| const active = body.querySelector('.tab-content.is-active'); | |
| const v = active ? active.querySelector('video') : body.querySelector('video'); | |
| return v || null; | |
| }).filter(Boolean); | |
| return videos; | |
| } | |
| document.addEventListener('DOMContentLoaded', () => { | |
| const btn = document.getElementById('sync-playpause'); | |
| if (!btn) return; | |
| btn.addEventListener('click', () => { | |
| const videos = getActiveVideos(); | |
| if (videos.length < 2) return; | |
| // if any is playing, pause both and align to minimum currentTime | |
| const anyPlaying = videos.some(v => !v.paused); | |
| if (anyPlaying) { | |
| const t = Math.min(...videos.map(v => v.currentTime)); | |
| videos.forEach(v => { try { v.pause(); v.currentTime = t; } catch (e) {} }); | |
| } else { | |
| // start both from the minimum timestamp | |
| const t = Math.min(...videos.map(v => v.currentTime)); | |
| videos.forEach(v => { try { v.currentTime = t; } catch (e) {} }); | |
| setTimeout(() => { videos.forEach(v => { try { v.play(); } catch (e) {} }); }, 50); | |
| } | |
| }); | |
| // No separate align button — Play/Pause Both uses the minimum timestamp when pausing/playing | |
| }); | |
| </script> | |
| <!--/ Animation. --> | |
| <!-- Concurrent Work. --> | |
| <!-- <div class="columns is-centered"> | |
| <div class="column is-full-width"> | |
| <h2 class="title is-3">Related Links</h2> | |
| <div class="content has-text-justified"> | |
| <p> | |
| There's a lot of excellent work that was introduced around the same time as ours. | |
| </p> | |
| <p> | |
| <a href="https://arxiv.org/abs/2104.09125" target="_blank">Progressive Encoding for Neural Optimization</a> introduces an idea similar to our windowed position encoding for coarse-to-fine optimization. | |
| </p> | |
| <p> | |
| <a href="https://www.albertpumarola.com/research/D-NeRF/index.html" target="_blank">D-NeRF</a> and <a href="https://gvv.mpi-inf.mpg.de/projects/nonrigid_nerf/" target="_blank">NR-NeRF</a> | |
| both use deformation fields to model non-rigid scenes. | |
| </p> | |
| <p> | |
| Some works model videos with a NeRF by directly modulating the density, such as <a href="https://video-nerf.github.io/" target="_blank">Video-NeRF</a>, <a href="https://www.cs.cornell.edu/~zl548/NSFF/" target="_blank">NSFF</a>, and <a href="https://neural-3d-video.github.io/" target="_blank">DyNeRF</a> | |
| </p> | |
| <p> | |
| There are probably many more by the time you are reading this. Check out <a href="https://dellaert.github.io/NeRF/" target="_blank">Frank Dellart's survey on recent NeRF papers</a>, and <a href="https://github.com/yenchenlin/awesome-NeRF" target="_blank">Yen-Chen Lin's curated list of NeRF papers</a>. | |
| </p> | |
| </div> | |
| </div> | |
| </div> | |
| </div> | |
| </section> --> | |
| <section class="section" id="BibTeX"> | |
| <div class="container is-max-desktop content"> | |
| <h2 class="title">BibTeX</h2> | |
| <pre><code>@article{shi2024stereocrafter, | |
| title={Stereocrafter-zero: Zero-shot stereo video generation with noisy restart}, | |
| author={Shi, Jian and Wang, Qian and Li, Zhenyu and Idoughi, Ramzi and Wonka, Peter}, | |
| journal={arXiv preprint arXiv:2411.14295}, | |
| year={2024} | |
| }</code></pre> | |
| </div> | |
| </section> | |
| <footer class="footer"> | |
| <div class="container"> | |
| <!-- <div class="content has-text-centered"> | |
| <a class="icon-link" target="_blank" | |
| href="./static/videos/nerfies_paper.pdf"> | |
| <i class="fas fa-file-pdf"></i> | |
| </a> | |
| <a class="icon-link" href="https://github.com/keunhong" target="_blank" class="external-link" disabled> | |
| <i class="fab fa-github"></i> | |
| </a> | |
| </div> --> | |
| <div class="columns is-centered"> | |
| <div class="column is-8"> | |
| <div class="content"> | |
| <p> | |
| This website is licensed under a <a rel="license" target="_blank" | |
| href="http://creativecommons.org/licenses/by-sa/4.0/">Creative | |
| Commons Attribution-ShareAlike 4.0 International License</a>. | |
| </p> | |
| <p> | |
| This means you are free to borrow the <a target="_blank" | |
| href="https://github.com/nerfies/nerfies.github.io">source code</a> of this website, | |
| we just ask that you link back to this page in the footer. | |
| Please remember to remove the analytics code included in the header of the website which | |
| you do not want on your website. | |
| </p> | |
| </div> | |
| </div> | |
| </div> | |
| </div> | |
| </footer> | |
| </body> | |
| </html> | |