Since its inception, Industrial Light & Magic has been a pioneer in visual effects, from the breakthrough work on Star Wars: Episode IV - A New Hope to creating the first digital character in Young Sherlock Holmes, to building the first fully digital film pipeline in Star Wars: Episode I - The Phantom Menace. Today, ILM continues to innovate in Martin Scorsese’s The Irishman, where ILM reinvented facial capture with a new markerless technology, in what is another pioneering leap for VFX.

The new markerless facial capture technology was dependent on a new camera system, which shot infrared light at different angles. “The special camera rig gave us information about the performance without harsh extremes between light and shadow created by the on-set lighting, showing us an evenly lit version of minute facial movement,” said Stéphane Grabli, R&D Facial Capture Supervisor at Industrial Light & Magic.

The Irishman © Netflix

The innovative camera rig can capture infrared light

There was no traditional animation used on The Irishman, thanks to an intricate retargeting solution that allowed them to capture incredibly subtle nuances in performance. “We then ran this data through our new in-house software called Flux, which would analyze pixel values on principal photography footage as well as infra-red footage, to capture the performance of the actors and transfer it onto a shape-driven facial rig to produce animated 3D geometry. The shape library was produced by the modeling department through the referencing of a large library of past performances from the actors and with the help of an AI engine, dubbed ILM Facefinder, to efficiently sort through the data,” explained Stéphane.

“Motion too subtle to be represented as shapes was captured by Flux as per-vertex deformation. Ultimately, this performance would be retargeted onto a younger-looking version of the facial rig for final rendering in RenderMan,” added Stéphane.

Although this data automation was crucial for the de-aging process, Scorsese didn’t want the performances to mimic that of their past characters, but to bring youth to the character’s performance in The Irishman. “This subtlety marks an important difference between The Irishman and Star Wars for example, where we needed to match the exact performance of Carrie Fisher as Leia. For The Irishman we’re not trying to make Robert De Niro’s character look or act like any of his past characters, for example, Ray Lamotta from Raging Bull - instead we’re aiming to match De Niro’s performance to a younger version of Frank Sheeran. Capturing the performance with the highest accuracy was the priority and our modeling and compositing artists had to be mindful of this when using Facefinder for reference,” explained Stéphane.

The stakes for visual fidelity were higher than ever due to incredibly up close and personal camera work. “Out of over 1750 VFX shots, we had to deliver 500 with full digital head replacements. Many of the shots were extreme closeups, so we had to make sure the tech was invisible to the actors in order to make sure they could focus on their performance,” said Stéphane Grabli.

ILM’s shading pipeline also had to go through a bit of an upgrade. “Tarkin and Leia were the first digital doubles to use path traced subsurface scattering for Rogue One: A Star Wars Story, but the challenges of photorealism in The Irishman needed a more customized solution, so we developed a new subsurface mode for our internal shading framework called Lama (for Layered Material), which was based on RenderMan’s new path traced subsurface algorithm,” said Stéphane. “We also had an experimental anisotropic treatment that modified bump mapping and displacement values based on how much the skin stretched in uv space,” added Kevin Reuter, Look Development Supervisor at ILM.

Matching surface detail for photorealistic skin is a challenge which ILM was excited to crack. “We made extensive use of regional face controls at the lookdev and shot levels to tweak spec and scatter in various areas of the face. This has been around for a few shows now but I think we were the first to use this so extensively at the shot level to match variations in the skin. This allowed for a perfect match with the footage and data we were getting" said Stéphane. “After this was set up, we had to tweak very little and the shots flowed really well. Sometimes we had to adjust the light rig manually in RenderMan before feeding it to Flux, especially in moving shots where stationary HDRI data was not enough to accurately represent the lighting in the scene, but overall the system was incredibly well suited to match the scene’s lighting data,” explained Stéphane.

This process was made more transparent thanks to RenderMan’s robust API...and a couple of neighborly visits from Pixar’s RenderMan team. “Thankfully, we were able to collaborate very closely with the RenderMan engineers and this gave us unique insight into the underlying implementation of Pixar’s subsurface scattering. Having this level of access to Pixar’s engineering team is immensely helpful,” said André Mazzone, Senior R&D Engineer at ILM.

Close collaboration with the world’s top VFX studios drives innovation in RenderMan, and Pixar’s unique historical relationship with Industrial Light & Magic continues to make sure versatility remains at the forefront of RenderMan, so that new ideas can flourish and drive the industry forward.