MonoFVV Homepage

Real-time Geometry, Albedo and Motion Reconstruction Using
a Single RGBD Camera
Kaiwen Guo1, Feng Xu1, Tao Yu1,2, Xiaoyang Liu1, Qionghai Dai1, Yebin Liu1
Tsinghua University1
Beihang University2
This paper proposes a real-time method that uses a single-view RGBD input to simultaneously reconstruct a casual scene with a detailed geometry model, surface albedo, per-frame non-rigid motion and per-frame low-frequency lighting, without requiring any template or motion priors. The key observation is that accurate scene motion can be used to integrate temporal information to recover the precise appearance, whereas the intrinsic appearance can help to establish true correspondence in the temporal domain to recover motion. Based on this observation, we first propose a shading-based scheme to leverage appearance information for motion estimation. Then, using the reconstructed motion, a volumetric albedo fusing scheme is proposed to complete and refine the intrinsic appearance of the scene by incorporating information from multiple frames. Since the two schemes are iteratively applied during recording, the reconstructed appearance and motion become increasingly more accurate. In addition to the reconstruction results, our experiments also show that additional applications can be achieved, such as relighting, albedo editing and free-viewpoint rendering of a dynamic scene, since geometry, appearance and motion are all reconstructed by our technique.
Figure. 1: Our system can capture fast and natural motions, geometry and surface albedo and simultaneously render them in new lighting environments in real-time.
Abstract
Figure. 2: Overview of our proposed pipeline. The green box represents the optimization of the motion field and the environmental lighting, and the red box represents the updating of the geometry and albedo of the canoniacl model.
Results
Figure. 3: Images of the results of our method, which are shown as rendered appearances and shaded geometries.
Free-viewpoint Video
Figure. 4: Our system can render high quality free-viewpoint videos based on a single RGBD sensor. The 1st line represents input depth and color; the 2nd line demonstrates the reconstructed results rendered at 3 virual views.
Video Results
Primary Video
Secondary Video
Presentation Slides
This work will be presented at SIGGRAPH 2017.
The presentation slides are coming soon.
[primary video]
[secondary video]
[primary video]
[secondary video]
Technical Paper
Citation
@article{guo2017real,
  title={Real-time Geometry, Albedo and Motion Reconstruction Using a Single RGBD Camera},
  author={Guo, Kaiwen and Xu, Feng and Yu, Tao and Liu, Xiaoyang and Dai, Qionghai and Liu, Yebin},
  journal={ACM Transactions on Graphics (TOG)},
  year={2017},
  publisher={ACM}
}
Kaiwen Guo, Feng Xu, Tao Yu, Xiaoyang Liu, Qionghai Dai and Yebin Liu, "Real-time Geometry, Albedo and Motion Reconstruction Using a Single RGBD Camera". ACM Transaction on Graphics 2017.