MrNeRF (@janusch_patas)
2026-01-13 | โค๏ธ 44 | ๐ 5
Mon3tr: Monocular 3D Telepresence with Pre-built Gaussian Avatars as Amortization
lepresence: โข A High-fidelity Human Mesh Template: We propose a new parametric human model, SPMM3 (skinned person model for Mon3tr), to obtain a personalized animatable mesh template that captures both coarse and fine-grained details on the human body. For body modeling, we reconstruct a detailed human body mesh that can represent more details of clothes and hair from multi-view images. By combining the captured body mesh with dedicated hand and face models, the expressiveness of the human model is further extended with vivid expressions and flexible gestures.
โข An Animatable 3DGS-based Avatar Representation: We build a 3DGS-based avatar by binding Gaussians to the proposed mesh template SPMM3. To capture realistic non-rigid deformations, we introduce a lightweight mesh deformation network to deform the mesh according to motion parameters, which is then mapped to the coordinates of Gaussians to adjust the associated Gaussian primitives. Moreover, we develop a learnable attribute deformation network with multiple local attribute controllers to refine appearance dynamics during inference. This design achieves photorealistic rendering quality (over 28 dB PSNR for novel poses and 32 dB PSNR for novel views) while remaining lightweight for mobile deployment.
โข An End-to-end Monocular Telepresence System with SOTA Performance: We develop a real-time system for monocular telepresence, as illustrated in Fig. 1. The sender-side module extracts human model parameters from a single monocular RGB camera for efficient transmission, while the receiver uses these parameters to drive the pre-built avatar, enabling high-fidelity rendering with low end-to-end latency of โผ 80 ms.
โข Real-world Validation on Consumer Hardware: We demonstrate the practicality of our system on a PC and a Meta Quest 3 [24]. Our implementation supports real-time 3D teleconference on a single device and runs smoothly at a rate of โผ 60 FPS, enabling real-time on-device immersive experience.
๋ฏธ๋์ด
๐ Related
Auto-generated - needs manual review