Visualize 3D Gaussian splats

[ssheorey]

3D Gaussian splatting is a novel 3D representation that enables fast scene reconstruction and real time photorealistic rendering. We are adding support for rendering 3DGS point clouds in Open3D.

Steps [filament renderer]:

Study the Open3D filament based visualizer cpp/open3d/visualization/rendering/filament and the filament material format.

1. Study the 3DGS paper, reference implementation of the 3DGS viewer gsplat/rendering.py and read its docs.
2. We want to port the Python code to C++17 and the CUDA kernels to GLSL vertex and fragment shaders. At the moment, we will only port the Python code and use mock-ups for the GLSL shaders. To enable the mock-ups to be functional, we will provide sample returned data. We will load this data to the GPU as buffer objects and the mock-up shaders will simply return them. The result should be a correct rendering of the 3DGS scene, but from a fixed viewpoint.
3. Create a new filament material "3dgs" with mock vertex and fragment shaders in ESSL 3.0 (OpenGL ES variant of GLSL) to render 3DGS point clouds. ESSL 3.0 is a filament limitation.
4. Add the new material to Open3D, update O3DVisualizer to use the 3dgs material. Provide all required point cloud attributes to the GPU when 3DGS attributes are detected. See the folder cpp/open3d/visualization/rendering/ for the rendering implementation. For now, also create buffers for the intermediate data returned by the shaders.
5. Use it to render 3DGS point clouds.

Tests:

• Load these Numpy files and display scene from a few different viewpoints. Compare with provided reference images.

Open3D Tensor point cloud representation:

• pcd.point[‘positions’] (N,3) – (x,y,z) for each splat.
• pcd.point[‘opacity’] – (N,) opacity
• pcd.point[‘rot’] (N, 4) - quaternion rotation of Gaussian.
• pcd.point[‘scale'] (N, 3) - x, y, z scales for Gaussian.
• pcd.point[‘f_dc’] (N, 3) – DC components for RGB colors.
• pcd.point[‘f_rest’] (N, Nc, 3) – SH coeffs for RGB colors. (Nc = 3, 8 or 15)