The Human Face of Deepfaked Film
When AI and Scorcese Rendered a Masterpiece
Deepfakes and other synthetic media are widespread across social media. They can be wildly entertaining, odd, uncanny, or can contribute to disinformation.
Deepfakes can be created by artists, comedians, graphic designers, and, in the case of disinformation, by politically motivated trolls. And generating deepfakes and other synthetic media is altogether not a difficult process, given enough time and computing power, often employing free and open source software, like the popular library DeepFaceLab.
The field has also found new applications in Hollywood. Millions have watched the Netflix original The Irishman, released in 2019, about the legendary mafia hitman Frank Sheeran. Directed by Martin Scorcese, the film stars Robert DeNiro, Al Pacino, and Joe Pesci, who are now in their 70s. They played both older and younger versions of their characters, as the film narratively jumped between the 1950s and 2000s. Well received by critics, the film was the product of a star director, an ensemble cast, and a veteran film crew, but it took considerable technical expertise to produce, a process that started in 2015.
Many Hollywood films had previously made use of computer-generated imagery (CGI), but The Irishman posed a unique challenge. For its filmmakers, the challenge was making the senior ensemble cast look like their characters’ ages. Scorcese wanted the actors to look not like their younger selves, but to the character’s younger selves. Joe Pesci, for example, was made to look like a thinner version of Russell Bufalino than how the actor Pesci looked at that age. How did they make a 70 year old actor move and look like a 30 or 50 year old mobster?
The solution was to invest millions of dollars on CGI to “de-age” The Irishman’s cast. During pre-production, DeNiro reportedly told Scorcese and the visual effects crew that he did not want to wear the helmet often used to capture faces for CGI:
…you know, I’m not that kind of an actor, I want you to develop an approach that doesn’t have any markers on my face. I don’t want to wear a helmet or anything like that. And I want to be onset with the correct lighting, working with other actors.
(Source: fxguide)
Furthermore, Al Pacino had previously experienced an allergic reaction to special effects makeup, so Scorcese and the producers set out to develop an innovative solution to “de-age” the actors.
“De-aging” The Irishman’s cast required cutting edge technology. Generating younger faces required collecting data on human faces and movements during production and a tremendous amount of subtlety in post-production editing.
So Scorcese hired Pablo Helman of Industrial Light & Magic, veteran visual effects supervisors. While filming, Helman developed the technology necessary to de-age the actors’ faces. ILM’s thirty person team created the software pipeline that processed film footage before shooting completed, while footage was being created. The data necessary for generating de-aged footage were captured during film shooting without interfering with Scorcese’s direction or the actors’ performance. While actors performed scenes, the camera crew would film with a specialized rig designed by ILM. The rig contained a standard director’s camera flanked by two infrared cameras, which overcame the issue of SFX headgear and makeup, capturing details affected by lighting, shadows, volume, and texture.
We came up with this three-camera rig [picking up on infrared], but if we were going to be basing our solution on lighting and texture, then changes between the level of the key light and shadow side were going to be very important.
— Pablo Helman (Source: fxguide)
Film production needed specialized software to process the camera footage. ILM’s team developed software called Flux to transform captured hours of footage and gigabytes of data into the CGI film. Flux combined the infrared and principal camera footage to generate masks on each actor’s face, using catalogued images of noses, eyes, and mouths to render the de-aged faces in each scene. These models were trained with “an incredible library” built with thousands of extracted frames from the actors’ previous movie appearances, from Goodfellas to Casino.
“We collected clips and images of the three main actors at all their targeted ages. We then created a library, which we divided up so you could look for say the nearest eyes to what you needed… We broke it down into all these different components of performance and lighting that we were going to need.”
— Pablo Helman (Source: fxguide)
In parallel, between 2015 and 2017, ILM developed an artificial intelligence Face Finder program. Once the ILM crew rendered a scene in 3D, they would de-age the actors’ faces using Flux, using the Face Finder AI to match the Flux-rendered masks with similar features in the library, checking for suitable angles, ages, and lighting.
“The AI frames were not offered up as replacements but visual reference from a vast library of previous De Niro performances over decades of films.”
— Pablo Helman (Source: fxguide)
Despite the advanced software, manual work rendering scenes was often necessary. Flux and Face Finder worked well to generate de-aged faces, but Helman found that others body parts aged differently, such as hands. And if a character smoked a cigarette during a scene, its heat could interfere with signals to the infrared cameras. So Helman and his crew found themselves manually retouching scenes, sometimes to correct for heat and lighting, sometimes to make the lead actors look thinner and younger. The manual interventions throughout editing undoubtedly made the process more costly, both financially and time-wise.
All in all, the resulting three and a half hour film was impressive, but costly. Estimates of the film’s budget range from well over $100 million to about $250 million. ILM did receive recognition for their groundbreaking work on The Irishman, nominated for the 2020 Oscar for best visual effects. Even then, many viewers commented that DeNiro and his colleagues had the faces of 40 year olds but moved like they were in their 70s.
About a month after the film was released on Netflix, a YouTube creator usernamed iFake released a video comparing ILM’s de-aging with deepfakes generated with open-source software. Using DeepFaceLab, iFake claims that it “took 7 days to make the footage in the video, [and] the results could turn out even better with more training…” See for yourself which process generated the more convincing de-aged footage of The Irishman’s ensemble cast.
Follow BloomAI on Medium for the latest on artificial intelligence, neural networks, and the future of synthetic media:
And visit our website if you would like to see our own neural networks in action:
