google lumiere ai

Google Research: LUMIERE is A Space-Time Diffusion Model for Video Generation.

Google introduces Lumiere -- a text-to-video diffusion model designed for synthesizing videos that portray realistic, diverse and coherent motion -- a pivotal challenge in video synthesis. To this end, Google introduces a Space-Time U-Net architecture that generates the entire temporal duration of the video at once, through a single pass in the model. This is in contrast to existing video models which synthesize distant keyframes followed by temporal super-resolution -- an approach that inherently makes global temporal consistency difficult to achieve. By deploying both spatial and (importantly) temporal down- and up-sampling and leveraging a pre-trained text-to-image diffusion model, This Google AI model learns to directly generate a full-frame-rate, low-resolution video by processing it in multiple space-time scales. Google demonstrate state-of-the-art text-to-video generation results, and show that our design easily facilitates a wide range of content creation tasks and video editing applications, including image-to-video, video inpainting, and stylized generation.

* Hover over the video to see the input prompt

* Hover over the video to see the input image and prompt.

Using a single reference image, Lumiere can generate videos in the target style by utilizing fine-tuned text-to-image model weights.
* Hover over the video to see the prompt.

With Lumiere, off-the-shelf text-based image editing methods can be used for consistent video editing.

The Lumiere model is able to animate the content of an image within a specific user-provided region.

Source: https://lumiere-video.gith...o/