De- aging ft: the technology and the work behind it:

Well, no ones loves aging in fact the biggest challenge faced by film makers was how to make the actor look young for period flashback scenes. While prosthetics and makeup did the job, a new technology made them take the back seat. Join me as i show you in today's article on how De-aging is done and the work behind it.

What is De-aging:

De-aging is used to make an actor look younger than his/her present age mainly for flashback scenes sometimes , for nostalgia sake it involves touch ups such as removal of wrinkles or the use of CGI( computer generated imagery) overlays to sometimes, creating face models from scratch and overlaying it onto a person with similar facial proportions, in the case of wanting to showcase a drastic age difference eg: childhood/mid teens or deceased actors. With that small explanation let us dive into the interesting part the technology and work involved.

How De-aging is achieved

Collecting reference materials:

the first step involves collecting reference materials of the actors older pics and film clips. This is used to study the actors unique facial features and how it has progressively aged over the years in order to provide data for the AI model . Archived footage, TV shows interviews, candid photos are ideal as more diverse footages can help AI generate possibly accurate results under different lighting conditions. Depending on inputs ,the training of AI models can take up to weeks possibly months.

It involves deep Learning AI models which study unique features of an actors face to their skin texture, wrinkles etc. With a goal of providing a realistic younger version of the actor while maintaining his/her unique features .

Construction of 3D Model:

The next step involves a complete 3d scan of the actor's face and capturing expressions. For this , small markers are placed on the actors face and are recorded with high resolution cameras . The markers help the computer track movements of his/her facial expressions thus, helping in creating a 3d model of an actor's face. The 3d model is crucial as it helps in studying the underlying bone structure which does not change with age as the skin or wrinkles. This helps in providing a accurate manipulation of the actor's face and providing a realistic de- aged output of the actor.

The Mova Contour Camera System:

Developed by former Apple computer Engineer Steve Perlman, this technology consists of high resolution cameras in a light sealed room used to scan faces for a 3d model. The Contour system involves the covering of the body and the face with phosphorescent powder which is not visible under normal lightning. facing two video cameras which are synchronized accordingly to record their appearance and shape. Scenes are lit by rapidly flashing fluorescent lights and the cameras capture light from the glowing powder during the intervals of darkness . This image is then transferred into an system of computers which use these data to create a 3d model which can be manipulated and edited accordingly. The Mova Contour System can recreate facial images up to 2,00,000 pixels.

Press photos of the captured face in Mova Contour

Performance Capture:

Will Smith in GEMINI MAN(2019)

Performance capture is an advanced form of motion capture which involves capturing real time performances of the actors face. Not to be confused with motion capture which focuses on capturing the body movements, when it includes face and fingers voice or capture subtle expressions it is termed as performance capture. This method involves capturing of actor's expressions in real time and use this data to drive the de-aged face. Performance capture uses markers or markerless techniques in order to capture the most subtle expressions which is then matched onto the de-aged face. This step is important in order to provide the perfect sync with the actors movements and deliver a believable performance.

Markerless Performance capture in The Irishman(2019):

After years of development ,the makers met with a major problem. Lead actors Robert De Niro, Al Pacino, Joe Pesci were too old to portray their respective characters . The team came up with two solutions either use stunt doubles , or use motion tracking markers and cameras, flatly refusing to work with either, the ILM(Industrial light and Magic) team took a two year project to develop the camera rig consisting of three camera's and the Flux software. After a shot test shoot in 2015 with De Niro recreating the Christmas party scene from Goodfellas(1989). De Niro was de -aged to look like this then 46 year old self. Post this the project got a green light.

The Three Headed Monster:

This setup consisted of the primary (RED Helium) for the director and two witness cameras (Alexa Minis) each side two record the visual effects. The offset cameras focused on the actors face were fit with infrared light rings and filters. These cameras did the same job which a head rig with markers would have done but not being present on the actor. The Flux software then analyzed the infrared lighting and compared it to the 3D model which was made during prep. For the typical de-aging of the on set conversation between the following was the process.

• ILM combined the HDRI and the LIDAR and solved for the location and the movement of the cameras this provided intensity and color and temperature of lighting on the set .( All the film sets were LiDAR and captured HDRI ‘s . )

• This was then combined with infrared solutions.

• This was then processed and the flux output is compared against what was shot for alignment.

• The digital recreation of actors were then added and were deformed on a per frame to frame basis to match the Flux solution of the actors face.

Although some criticized the output stating De Niro didn’t look that young enough. Nevertheless, the film received critical acclaim for the innovative technology and the overall film.

The Process

The final output

Feature Mapping:

Landmark Detection:

As the name suggests, Landmark detection involves identifying key areas of the face such as the eyes, nose and the edges of the mouth. A human face consists of 68 landmarks. Land mark detection plays a crucial role in de-aging in accurately mapping out the actor's face and making sure any alterations carried out align with their Facial features. Advanced algorithms can detect these landmarks in different angles and expressions, ensuring that the tracking remains robust. Ensuring that the De-aged face moves in the same way and maintains his/her unique expressions again, for the believability of the de-aged face ensuring that it remains recognizably the same person.

The following are the methods for Landmark detection:

Traditional Methods:

Active Shape Models:(ASM)

• Uses statistical models of facial shape to fit a predefined shape to face.

• this technique has been widely used to analyze images of face in 2d or 3d

• This is commonly done by what is called a profile model which looks for sharp edges to match model templates for the point.

Active Appearance Model(AAM):

• Combines shape and texture models for more robust feature detection.

• It resolves the issue faced with ASM is that it only uses the shape constraints( with some image structure near the landmarks) and not taking advantage about all the available information. This can be modelled using AAM.

• Haar Cascades:

• A machine learning approach for identifying facial regions and landmarks

  Haar cascades scan images or videos at different scales, looking for regions that match pre-defined examples .These patterns are learned from matching a set of positive and negative examples.

• Haar cascades can work in real-time in low power devices. this method is used for face detection, eye detection ,mouth detection, and full/partial body scan.

• Machine based Learning Methods:

• Convolutional Neural Network: Is a type of artificial network that uses convolutional layers to learn and recognize patterns in images . it uses convolutional layers to filter inputs for useful information, and adjusts automatically to find the best feature on the task.

• Uses relatively little pre-processing compared to other image classification algorithms and learns to optimize filters through automation.

• Dlib Library:

• Widely used for 68 point-facial landmark detection ,offering high accuracy and efficiency.

  
  
  
  

  

  An example of 68 points in a human face

Morphing Algorithms:

Morphing algorithms are used to adjust facial features based on learned age progression or regression patterns. These algorithms can smoothen out wrinkles, adjust facial features in order to make the actor look younger. This involves complex mathematical transformation that preserve their features while altering their age- related characteristics. The goal is to provide realistic de-aged output while maintaining their unique features.

The goal of Morphing :

• Feature detection:

  Identifying key points on the face(eyes ,nose ,mouth, jawline)

• Mesh creation: This involves generating a mesh grid over the face for precise transformations.

• Warping: Gradually transforming pne mesh configuration to another to achieve smoother transitions.

• Blending:

• Combining textures, colors and shading to ensure realistic results.

The Fine Touches:

Texture Mapping:

Texture mapping involves generating realistic skin features that match the actor's appearance. This includes details like pores ,wrinkles and skin tone variations. Advanced texture synthesis are used to create these textures to ensure that they seamlessly blend with the actor's face. The skin texture is often enhanced using a technique called Sub surface Scattering(SSS) which simulates how light penetrates and diffuses under the skin. This is key to making the skin look natural and youthful.

Adding the fine details:

This step involves checking making sure all the fine details such as such as the shape of the eyes, the pointiness of the nose ,mouth ,sharpness of jawline and facial hair growth. The goal is to make the actor look younger without losing his/hers unique features . Overdoing such as smoothening the skin more than needed and too much reshaping may result in what is called the uncanny valley. Where the character's appearance is almost human but not right. Leading to disturbing and unnatural expressions.

An example of how over smoothness and reduction of jawline can entirely change the face and generate criticism.

Post processing and consistency checks:

Post AI processing, VFX artist's make manual adjustments to refine the finished product. This involves making sure all the fine details are present and checking continuity of lighting and color in various scenes across the film. Also, while checking for uncanny valley and expression continuity. This step is important in making sure that the final product meets the director's vision and this step often involves close collaboration between the director, actor and the VFX artists.