ProjectGaussians

Given 3D gaussians parametrized by means \(μ\), covariances \(Σ\), colors \(c\), and opacities \(o\), the gsplat.project_gaussians() function computes the projected 2D gaussians in the camera frame with means \(μ'\), covariances \(Σ'\), and depths \(z\) as well as their maximum radii in screen space and conic parameters.

Note, covariances are reparametrized by the eigen decomposition:

\[Σ = RSS^{T}R^{T}\]

Where rotation matrices \(R\) are obtained from four dimensional quaternions.

The projection of 3D Gaussians is approximated with the Jacobian of the perspective projection equation as shown in [1]:

\[\begin{split}J = \begin{bmatrix} f_{x}/t_{z} & 0 & -f_{x} t_{x}/t_{z}^{2} \\ 0 & f_{y}/t_{z} & -f_{y} t_{y}/t_{z}^{2} \\ 0 & 0 & 0 \end{bmatrix}\end{split}\]

Where \(t\) is the center of a gaussian in camera frame \(t = Wμ+p\). The projected 2D covarience is then given by:

\[Σ' = J W Σ W^{⊤} J^{⊤}\]

Citations

[1]

M. Zwicker, H. Pfister, J. van Baar, and M. Gross. Ewa splatting. IEEE Transactions on Visualization and Computer Graphics, 8(3):223–238, 07/2002-09/2002 2002.

gsplat.project_gaussians(means3d, scales, glob_scale, quats, viewmat, fx, fy, cx, cy, img_height, img_width, block_width, clip_thresh=0.01)

This function projects 3D gaussians to 2D using the EWA splatting method for gaussian splatting.

Note

This function is differentiable w.r.t the means3d, scales and quats inputs.

Parameters:

  • means3d (Tensor) – xyzs of gaussians.

  • scales (Tensor) – scales of the gaussians.

  • glob_scale (float) – A global scaling factor applied to the scene.

  • quats (Tensor) – rotations in normalized quaternion [w,x,y,z] format.

  • viewmat (Tensor) – view matrix for rendering.

  • fx (float) – focal length x.

  • fy (float) – focal length y.

  • cx (float) – principal point x.

  • cy (float) – principal point y.

  • img_height (int) – height of the rendered image.

  • img_width (int) – width of the rendered image.

  • block_width (int) – side length of tiles inside projection/rasterization in pixels (always square). 16 is a good default value, must be between 2 and 16 inclusive.

  • clip_thresh (float) – minimum z depth threshold.

Returns:

  • xys (Tensor): x,y locations of 2D gaussian projections.

  • depths (Tensor): z depth of gaussians.

  • radii (Tensor): radii of 2D gaussian projections.

  • conics (Tensor): conic parameters for 2D gaussian.

  • compensation (Tensor): the density compensation for blurring 2D kernel

  • num_tiles_hit (Tensor): number of tiles hit per gaussian.

  • cov3d (Tensor): 3D covariances.

Return type:

A tuple of {Tensor, Tensor, Tensor, Tensor, Tensor, Tensor, Tensor}