Abstract

Learning surfaces from neural radiance field (NeRF) became a rising topic in Multi-View Stereo (MVS). Recent Signed Distance Function (SDF)-based methods demonstrated their ability to reconstruct accurate 3D shapes of Lambertian scenes. However, their results on reflective scenes are unsatisfactory due to the entanglement of specular radiance and complicated geometry. To address the challenges, we propose a Gaussian-based representation of normals in SDF fields. Supervised by polarization priors, this representation guides the learning of geometry behind the specular reflection and captures more details than existing methods. Moreover, we propose a reweighting strategy in the optimization process to alleviate the noise issue of polarization priors. To validate the effectiveness of our design, we capture polarimetric information, and ground truth meshes in additional reflective scenes with various geometry. We also evaluated our framework on the PANDORA dataset. Comparisons prove our method outperforms existing neural 3D reconstruction methods in reflective scenes by a large margin.

Polarization and Surface Normals

Polarimetry describes the vibration status of light waves. Since light is a type of transverse wave that only oscillates in the plane perpendicular to the light path. The full polarimetric cues of rays are always represented by planar ellipses. The magnitude of vectors inside these ellipses alludes to the amplitude of the light wave vibration along the vectors. During reflection, the vibration in each direction is absorbed unequally, and unpolarized incident light turns into partially polarized reflected light captured by polarization cameras. The shift of polarization status is functionally related to projected surface normals at the points of reflection.

Gaussian-based NeuRecon

Our key idea is to extend the geometry representation from scalar SDFs to Gaussian fields of normals supervised by polar ization priors. Given a surface point, the normals within its neighborhood are approximated by a 3D Gaussian. Its mean shows the overall (low-frequency) orientation of the surface, while the covariance captures high-frequency details. The representation can be splatted into the image plane as 2D Gaussians. It skips the disentangled specular radiance. Learning of the 2D Gaussians can be directly supervised by the polarization information about surface normals.

Visual Results

Additional Results

Citation

Acknowledgements

We would like to thank Wenhang Ge for providing the code for evaluation and the results for comparison.
The website template was borrowed from Dor Verbin.