Although PIFu-based 3D human reconstruction methods are popular, the quality of recovered details is still unsatisfactory. In a sparse (e.g., 3 RGBD sensors) capture setting, the depth noise is typically amplified in the PIFu representation, resulting in flat facial surfaces and geometry-fallible bodies. In this paper, we propose a novel geometry-aware two-scale PIFu for 3D human reconstruction from sparse, noisy inputs. Our key idea is to exploit the complementary properties of depth denoising and 3D reconstruction, for learning a two-scale PIFu representation to reconstruct high-frequency facial details and consistent bodies separately. To this end, we first formulate depth denoising and 3D reconstruction as a multi-task learning problem. The depth denoising process enriches the local geometry information of the reconstruction features, while the reconstruction process enhances depth denoising with global topology information. We then propose to learn the two-scale PIFu representation using two MLPs based on the denoised depth and geometry-aware features. Extensive experiments demonstrate the effectiveness of our approach in reconstructing facial details and bodies of different poses and its superiority over state-of-the-art methods.

Proposed method overview. Given sparse and noisy RGBDs as inputs, Geometry-aware PIFu-Body performs depth denoising and predicts the body occupancy field. High-resolution PIFu-Face predicts the face occupancy field with fine-grained details. The body and face occupancy fields are fused to produce final results via the Face-to-body Fusion scheme.

Figure E : Qualitative comparisons between three face reconstruction methods and our PIFu-Face model.  Input facial RGB image (a). Predicted facial normal maps of Abrevava et. al [1] (b). 3D reconstructed facial model of DF2Net [2] (c), FaceVerse [3] (d). Our reconstructed facial results (e). Zoom in to see the details.

[1] Abrevaya V F, Boukhayma A, Torr P H S, et al. Cross-modal deep face normals with deactivable skip connections[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020: 4979-4989.

[2] Zeng X, Peng X, Qiao Y. Df2net: A dense-fine-finer network for detailed 3d face reconstruction[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019: 2315-2324.

[3] Wang L, Chen Z, Yu T, et al. FaceVerse: a Fine-grained and Detail-controllable 3D Face Morphable Model from a Hybrid Dataset[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 20333-20342.