3D reconstruction from real-world input data is essential for many downstream applications such as large scale scene reconstruction, augmented reality, and medical imaging. However, real-world data poses significant challenges, it is often messy -- noisy, incomplete, obscured, or corrupted. This makes precise reconstruction challenging. In addition, several methods offer single solution to the problem, failing to accommodate the variability and uncertainty present in these imperfect data, often leading to unreliable results. By exploring multiple hypotheses, it allows for a more robust approach, addressing the inconsistency and ambiguity present in the data. This becomes particularly crucial in scenarios lacking absolute ground truth, such as the fabrication industry, where simplified representations are needed for effective design, planning, and fabrication to cope with cutting and manufacturing cost constraints. Therefore, in this thesis, we present works that implement multi-hypothesis reconstruction to effectively tackle the challenges encountered in downstream applications, thereby enhancing the accuracy and reliability of 3D reconstructions.
In our initial thesis project, we tackle the task of reconstructing from point cloud data, targeting applications in the fabrication industry for urban planning and large-scale scene simulation and reconstruction. Particularly for these applications, the emphasis lies on approximating the structure to reduce cost and computational overhead. Therefore, in this work we focus on reconstructing these surfaces as piecewise developable surfaces, serving as approximations. As these representations are without absolute ground truth, our method explores various hypothetical surface abstractions using different regularization techniques based on the zero Gaussian curvature property. In this approach, the patches appear automatically based upon the weight of the regularizer, which might blend irrelevant parts such as building parts in urban planning, revealing the necessity for guidance in this process. There is a shortage of large-scale part ``annotated'' dataset for 3D building models to aid in this guidance reconstruction. To bridge this gap, in the second part of this thesis, we introduce BuildingNet, a publicly accessible dataset of annotated 3D building models with labeled exteriors and surroundings.
Within the realm of real-world data reconstruction applications, it's crucial not to solely concentrate on tasks dealing with rigid structures. Equally paramount are the challenges associated with reconstructing human faces and are sought after in various domains, including medical imaging, animations, and avatar creation. Thus, in the second chapter of the thesis, we shift our focus to human face reconstruction, particularly from occluded face images. Addressing this occlusion challenge, the thesis introduces OFER, a generative modeling framework designed to reconstruct expressive 3D faces from partially occluded 'in-the-wild' images. It also provides a ranking method to select low reconstruction error samples for minimal to no obscured images. While OFER produces high-quality reconstruction samples, the DDPM's iterative denoising process is notorious for long sampling times. The ranking method depends on these samples for training leading to longer training times, hindering the ability to quickly produce results and test other techniques for improving the ranking mechanism.
In order to overcome this limitation, this proposal outlines a method aimed at efficient inference sampling without requiring to re-train the model.
Advisor: Erik Learned-Miller