HACK: Learning a Parametric Head and Neck Model for High-fidelity Animation
Longwen Zhang1,2 Zijun Zhao1,2 Xinzhou Cong1,2 Qixuan Zhang1,2 Shuqi Gu1 Yuchong Gao1 Rui Zheng1 Wei Yang3 Lan Xu1 Jingyi Yu1
1ShanghaiTech University 2Deemos Technology 3Huazhong University of Science and Technology
Abstract
Significant advancements have been made in developing parametric models for digital humans, with various approaches concentrating on parts such as the human body, hand, or face. Nevertheless, connectors such as the neck have been overlooked in these models, with rich anatomical priors often unutilized. In this paper, we introduce HACK (Head-And-neCK), a novel parametric model for constructing the head and cervical region of digital humans. Our model seeks to disentangle the full spectrum of neck and larynx motions, facial expressions, and appearance variations, providing personalized and anatomically consistent controls, particularly for the neck regions. To build our HACK model, we acquire a comprehensive multi-modal dataset of the head and neck under various facial expressions. We employ a 3D ultrasound imaging scheme to extract the inner biomechanical structures, namely the precise 3D rotation information of the seven vertebrae of the cervical spine. We then adopt a multi-view photometric approach to capture the geometry and physically-based textures of diverse subjects, who exhibit a diverse range of static expressions as well as sequential head-and-neck movements. Using the multi-modal dataset, we train the parametric HACK model by separating the 3D head and neck depiction into various shape, pose, expression, and larynx blendshapes from the neutral expression and the rest skeletal pose. We adopt an anatomically-consistent skeletal design for the cervical region, and the expression is linked to facial action units for artist-friendly controls. We also propose to optimize the mapping from the identical shape space to the PCA spaces of personalized blendshapes to augment the pose and expression blendshapes, providing personalized properties within the framework of the generic model. Furthermore, we use larynx blendshapes to accurately control the larynx deformation and force the larynx slicing motions along the vertical direction in the UV-space for precise modeling of the larynx beneath the neck skin. HACK addresses the head and neck as a unified entity, offering more accurate and expressive controls, with a new level of realism, particularly for the neck regions. This approach has significant benefits for numerous applications, including geometric fitting and animation, and enables inter-correlation analysis between head and neck for fine-grained motion synthesis and transfer.
Overview
Utilizing both the ultrasound imaging system and multi-view photometric capture system, we further process these multi-modal data for model learning. The processed data includes registered cervical spine joints, registered neutral meshes, solved personalized expressions, dynamic performance sequences, larynx motion sequences, and physically-based textures.
Gallery
Samples using HACK, of various identities, head and neck poses, expressions, and appearance. Our model demonstrates its strong ability for realistic modeling, animation and rendering.
Registration results of HACK on our testing dataset and masterpiece statues. We can fit HACK on different data, ranging from our captured unseen scans to masterpiece statues. In the left part, we first demonstrate the registration of an old man and a young lady from our captured data, then, in the right part, we show the registration of David, a masterpiece of Renaissance sculpture, and Venus de Milo, an ancient Greek sculpture that was created during the Hellenistic period, both with an elegant neck. HACK successfully models across identities, poses, and expressions, with details restored and can be realistically rendered with appearance applied.
Novel pose and inference results of previous HACK registrations. Note that for David, our driving result provides the realistic bowstring effect where the platysma is contracting, thanks to our joint modeling of the head and neck. As shown in the figure, HACK provides realistic driven and rendering results.
Registration results on various released dataset, including samples from FaceScape (upper left), 3D Scan Store (upper right), MultiFace (lower left), and VOCA (lower right).
Video (Coming soon ...)
Paper link
Citation
@article{10.1145/3592093,
author = {Zhang, Longwen and Zhao, Zijun and Cong, Xinzhou and Zhang, Qixuan and Gu, Shuqi and Gao, Yuchong and Zheng, Rui and Yang, Wei and Xu, Lan and Yu, Jingyi},
title = {HACK: Learning a Parametric Head and Neck Model for High-fidelity Animation},
year = {2023},
issue_date = {August 2023},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
volume = {42},
number = {4},
issn = {0730-0301},
url = {https://doi.org/10.1145/3592093},
doi = {10.1145/3592093},
journal = {ACM Trans. Graph.},
month = {jul},
articleno = {41},
numpages = {20},
keywords = {head and neck modeling, anatomical model, facial expressions, neck animation, parametric learning}
}
Notice: You may not copy, reproduce, distribute, publish, display, perform, modify, create derivative works, transmit, or in any way exploit any such content, nor may you distribute any part of this content over any network, including a local area network, sell or offer it for sale, or use such content to construct any kind of database.
