Can AI Create Better VFX? A Dive into Paul Trillo’s Experimental Short “Notes To My Future Self”

July 5, 2024

The question of whether AI can create better visual effects (VFX) has been at the forefront of industry discussions, with many filmmakers and VFX artists exploring the potential of AI tools to enhance their creative processes. One of the most compelling explorations of this question is Paul Trillo’s short film “Notes To My Future Self.” Through this film, Trillo provides a detailed breakdown of how he integrated AI into the VFX pipeline, demonstrating both the capabilities and the current limitations of AI in film production.

The Concept and Creation

In “Notes To My Future Self,” Trillo harnessed the power of various AI tools to streamline the transition from pre-production to post-production. The film showcases how AI can be employed to generate backgrounds, animate elements, and create lighting effects that respond dynamically to the environment. The original film can be viewed here, and the behind-the-scenes process is discussed in-depth here.

Trillo’s approach involved taking each actor’s line of dialogue and inputting it into AI prompts, combined with other keywords, to generate the film’s backgrounds. These backgrounds were initially created using Stable Diffusion in Automatic1111, providing a foundation that was further refined using Photoshop’s Generative Fill and Expand tools. This process allowed Trillo to craft high-resolution (up to 8K) images, essential for maintaining visual fidelity in a film setting.

The AI Workflow

The AI workflow Trillo employed can be broken down into several key steps:

Generating Backgrounds: Stable Diffusion was used to create initial background images based on the dialogue and keywords. This step ensured that the visual environment was directly tied to the narrative content.
Editing and Refinement: Using Photoshop Generative Fill and Expand, Trillo edited these backgrounds to meet the specific aesthetic and narrative requirements of the film. This allowed for a level of customization and detail that automated processes alone could not achieve.
Upscaling: The backgrounds were then upscaled using tools like Magnific, Krea, and Topaz. This step was crucial for achieving the high resolution needed for cinematic quality, allowing Trillo to crop specific sections and enhance them further.
Animating Elements: For dynamic elements such as moving bushes, branches, and fog, Trillo utilized Runway’s Gen-2. This tool enabled the creation of high-quality moving elements that added depth and realism to the scenes.
Dynamic Lighting: Perhaps one of the most innovative aspects of Trillo’s process was the use of animated luma mattes generated by Gen-2. These mattes created the illusion that the lighting in the scene was dynamically reacting to the background, adding a layer of realism that is often challenging to achieve in traditional VFX workflows.

The Impact on VFX

Trillo’s experiment highlights several significant impacts of AI on the VFX industry:

Efficiency: AI tools can significantly reduce the time required to produce high-quality visual effects. Trillo completed the shooting, editing, and compositing of “Notes To My Future Self” in just ten days, a timeline that would be challenging without the aid of AI.
Creativity: By automating routine tasks, AI allows VFX artists to focus more on creative decision-making. The ability to generate backgrounds and dynamic elements quickly means more time can be spent on refining the artistic vision of the project.
Accessibility: AI tools can democratize the VFX process, making it accessible to filmmakers who may not have the budget for large VFX teams. This opens up new possibilities for independent creators to produce high-quality visual content.

Conclusion

While AI is not yet a complete replacement for traditional VFX techniques, Trillo’s “Notes To My Future Self” demonstrates its potential to revolutionize the industry. By seamlessly integrating AI into the VFX pipeline, Trillo was able to create a visually stunning film in a fraction of the time it would typically take. As AI technology continues to evolve, its role in film production is likely to expand, offering new tools and techniques that will redefine the boundaries of visual storytelling. The question “Can AI create better VFX?” may not have a definitive answer yet, but Trillo’s work shows that AI is undoubtedly a powerful ally in the quest for cinematic excellence.