Born out of a long-term collaboration with the Meta Reality Labs hardware and software engineering teams, my team was tasked with overseeing the front-end development of Smart Camera, Portal’s hero feature.

Sitting directly at the intersection of art, craft, and code, the system was trained to perfectly keep all video call participants in frame, no matter where they moved in a room. The system was also designed to zoom in or pull out depending on the number of people on a call at any one time, meaning additional participants could seamlessly enter or leave a call with no affect to the overall framing.

To create Smart Camera, we first listed hundreds of potential permutations of participants and their actions during video calls - adjusting for variables such as number of people on a call, different types of movements around a room during a call, differing movements amongst various members of the group, and the entrance and exits of people mid-call. We then filmed actors performing each of these permutations with three cameras simultaneously, allowing us to test different angles, movements, and adjustments that might work best for the final product.

After individually selecting the best camera movement for every permutation, the selects were overlayed on a back-end system that could accurately map human presence and attention intent based on skeletal structure and eye-gaze, the system was trained to anticipate and react to these actions, and Smart Camera was complete.

The team received a patent for this first-of-a-kind video calling system, as well as a patent for a secondary feature modifying the display of an overlay on video data based on locations of interests within that video. Today, this innovative combination of hardware and software is quickly becoming the standard for more immersive video calling industry-wide, with Apple, Google, and Microsoft all creating versions of their own.