Today, even hobbyists can produce impressive videos thanks to AI-driven platforms like Veo 3 and Sora, which are integrated into popular apps such as ChatGPT and Gemini for premium users. These tools showcase the remarkable progress in democratizing video creation.

However, the rise of AI-generated content also brings challenges. The proliferation of low-quality or misleading videos floods social media, complicating content authenticity. Moreover, video synthesis demands significantly more computational power and energy compared to generating text or images, raising environmental concerns.

Decoding the Process: How AI Crafts Videos

For most users, video generation is as simple as typing a prompt like, “Create a video of a dragon soaring over mountains.” The AI then produces a series of clips, often requiring multiple attempts to perfect the output. Behind this simplicity lies a complex interplay of advanced machine learning models.

What Are Diffusion Models and How Do They Work?

At the core of many AI video generators is a class of models known as Latent Diffusion Transformers. To understand this, let’s first focus on diffusion models.

Imagine starting with a clear image and progressively adding random noise until it becomes indistinguishable from static on a television screen. A diffusion model is trained to reverse this process-transforming noisy, random pixels back into coherent images. During training, the model studies millions of images at various noise levels, learning how to reconstruct the original visuals from corrupted versions.

When generating new content, the model begins with random noise and iteratively refines it into an image that resembles those it has learned from, guided by the input prompt.

From Words to Moving Pictures: Linking Text and Video

To ensure the generated visuals match the user’s description, diffusion models are paired with large language models (LLMs) trained to associate text with images or videos. These LLMs steer the diffusion process, nudging it toward outputs that align with the prompt.

It’s important to note that these models learn from vast datasets containing billions of text-image or text-video pairs scraped from the internet, a practice that has sparked debate over copyright and content representation. Consequently, the AI’s output reflects a condensed and sometimes distorted version of online media.

While diffusion models are often illustrated with images, the same principles apply to audio and video, where sequences of frames or sound waves are progressively refined.

Efficiency Through Latent Diffusion

Generating videos frame-by-frame at full resolution is computationally expensive. To address this, most video AI models employ latent diffusion, which compresses video frames and text prompts into a condensed mathematical representation called latent space. This abstraction captures essential features without processing every pixel directly.

This approach is akin to how streaming services compress videos for faster delivery, decompressing them on your device for viewing. After the latent diffusion model generates the compressed video, it is decoded back into a standard format once the LLM confirms it matches the user’s request.

Latent diffusion significantly reduces the computational load compared to traditional diffusion, making video generation more feasible, though it remains energy-intensive relative to text or image generation.

Ensuring Consistency: The Power of Transformers

One challenge in AI video creation is maintaining coherence across frames-ensuring objects, lighting, and motion remain consistent. This is where transformers come into play. Originally designed to handle long sequences of data like text, transformers excel at preserving context over extended inputs.

In video generation, transformers treat segments of video as “tokens” by slicing frames both spatially and temporally into cubes. This method allows the model to process video data similarly to how it processes language, maintaining continuity throughout the clip.

By combining transformers with diffusion models, AI systems can produce videos where elements don’t randomly appear or vanish, and can handle diverse formats-from vertical smartphone clips to widescreen films. This versatility marks a significant improvement over earlier models.

Integrating Sound: The New Frontier in AI Video

Recent breakthroughs have enabled AI to generate synchronized audio alongside video, including lip-synced dialogue, ambient sounds, and effects. Veo 3, for example, pioneers this integration by compressing audio and video into a unified data stream within the diffusion framework, ensuring perfect alignment between sound and visuals.

Google DeepMind’s CEO Demis Hassabis highlighted this milestone, declaring the end of the “silent era” in AI video generation at a recent tech conference.

Looking Ahead: The Convergence of Diffusion Models and Language Models

While diffusion models currently excel at creating images, videos, and audio, transformers dominate text generation in large language models. However, the boundaries are blurring. Google DeepMind recently announced experimental LLMs that generate text using diffusion techniques instead of transformers, potentially offering greater efficiency.

This innovation could reduce the substantial energy consumption associated with current transformer-based models, signaling a future where diffusion models play a larger role across multiple AI domains.