Facial Performance Sensing Head-Mounted Display

作者：Li Hao^*; Trutoiu Laura; Olszewski Kyle; Wei Lingyu; Trutna Tristan; Hsieh Pei Lun; Nicholls Aaron; Ma Chongyang

来源：ACM Transactions on Graphics, 2015, 34(4): 47.

DOI：10.1145/2766939

• real-time facial performance capture
• virtual reality
• depth camera
• strain gauge
• head-mounted display
• wearable sensors

There are currently no solutions for enabling direct face-to-face interaction between virtual reality (VR) users wearing head-mounted displays (HMDs). The main challenge is that the headset obstructs a significant portion of a user's face, preventing effective facial capture with traditional techniques. To advance virtual reality as a next-generation communication platform, we develop a novel HMD that enables 3D facial performance-driven animation in real-time. Our wearable system uses ultra-thin flexible electronic materials that are mounted on the foam liner of the headset to measure surface strain signals corresponding to upper face expressions. These strain signals are combined with a head-mounted RGB-D camera to enhance the tracking in the mouth region and to account for inaccurate HMD placement. To map the input signals to a 3D face model, we perform a single-instance offline training session for each person. For reusable and accurate online operation, we propose a short calibration step to readjust the Gaussian mixture distribution of the mapping before each use. The resulting animations are visually on par with cutting-edge depth sensor-driven facial performance capture systems and hence, are suitable for social interactions in virtual worlds.

• 出版日期2015-8