Google may be running late in the AI game, but don’t count them out just yet. In fact, some of the products they’re working on will leave the competition scared.
The company has just announced a new AI video generation model, dubbed Lumiere. They claim that this new model achieves incredible realism while also maintaining consistency.
This is achieved through “a Space-Time U-Net architecture that generates the entire temporal duration of the video at once, through a single pass in the model.”
Other video models “synthesize distant keyframes followed by temporal super-resolution — an approach that inherently makes global temporal consistency difficult to achieve.”
Google Lumiere Capabilities
Google’s Lumiere is a diffusion model designed for synthesizing videos that portray realistic, diverse and coherent motion.
It works from Text-to-video, Image-to-video, video inpainting and also for stylized generations (using reference images to generate videos in the same style).
From the examples provided, it appears that Lumiere is also great at cinemagraphs, i.e. animating a specific region of an image.
Why this matters
In the last 1 year, AI video generation has come a long way. However, it still lags behind image generation in many ways.
Frame constistency is one of the key roadblocks in generating videos longer than a few seconds.
Achieving consistency will bring about a paradigm shift in how we make and consume video content, from YouTube to Hollywood.
