The art of compositing, an essential part of filmmaking, has often been a challenging process. However, Netflix is revolutionizing this practice with a novel machine learning technique that simplifies compositing but requires actors to be illuminated in a striking magenta hue.
Traditional compositing involved chroma keying, where actors were positioned against a vividly colored background, typically green or blue, which could then be replaced with any desired setting or visual effects.
You May Also Like: Even With The Growth Of Threads, Elon Musk’s Net Worth Still Surpasses Mark Zuckerberg’s By $137 Billion
While this method was affordable and easy, it presented limitations when it came to transparent objects, intricate details like hair, and anything sharing a similar color with the background.
Despite its shortcomings, attempts to replace chroma keying with more advanced and costly methods, such as light field cameras, have faced challenges.
Netflix researchers have ventured into this domain, combining old and new techniques to achieve seamless and straightforward compositing at the expense of an unconventional on-set lighting setup.
The “Magenta Green Screen,” as detailed in a recently published paper, involves placing actors in a unique lighting arrangement.
Bright green illumination is provided behind them, while a combination of red and blue lighting creates a visually striking contrast in the foreground.
The resulting on-set appearance may initially startle even the most experienced post-production artists. Typically, actors are illuminated with natural light, requiring minimal adjustments in post-production to achieve a normal look.
However, exclusively employing red and blue light significantly distorts the actors’ appearance since natural light does not possess such a restricted spectrum.
Nonetheless, the technique’s cleverness lies in simplifying the separation of foreground and background by using exclusively red/blue lighting for the former and green lighting for the latter.
Instead of capturing the full spectrum, a regular camera captures red, blue, and alpha channels, resulting in highly accurate mattes without the artifacts associated with separating full-spectrum inputs from limited-spectrum key backgrounds.
However, this substitution of difficulties introduces a new challenge: restoring the green channel to the magenta-lit subjects.
Accomplishing this task systematically and adaptively, considering the variation in subjects and compositions, requires an automated approach. This is where artificial intelligence (AI) comes to the rescue.
The research team trained a convolutional neural network (CNN) on their own training data, consisting of rehearsal takes of similar scenes but lit under normal conditions.
By providing the CNN with patches of the full-spectrum image for comparison with the magenta-lit versions, the model learned to restore the missing green channel in a more intelligent manner than a simple algorithm.
Remarkably, the post-processing restoration achieves color accuracy virtually indistinguishable from the ground truth captured in-camera.
However, the challenge remains regarding the discomfort caused to actors and the set by the harsh lighting conditions. Many actors already express dissatisfaction with working in front of a green screen, and the addition of intense, unnatural lighting exacerbates the issue.
To address this concern, the paper proposes “time-multiplexing” the lighting, which involves rapidly switching between magenta and green lighting multiple times per second.
While attempting this at the standard film and TV shooting frame rate of 24 frames per second would be distracting and potentially hazardous, increasing the switching speed to 144 times per second results in an appearance of near-constant lighting.
However, achieving this necessitates intricate synchronization between the camera and lighting, ensuring that the camera only captures light during the brief magenta moments. Moreover, the system must account for missing frames caused by motion.
You May Also Like: EA Announces Development Of Single-Player Black Panther Video Game
As evident, this approach is still highly experimental. Nevertheless, it represents an exciting way to tackle a long-standing challenge in media production using cutting-edge technology.
Unfeasible just five years ago, this technique is worth exploring further, although its adoption on sets remains uncertain.
