Evaluation
===================================

3DGS
----------------------------------------------

.. table:: Performance on `Mip-NeRF 360 Captures <https://jonbarron.info/mipnerf360/>`_ (Averaged Over 7 Scenes)

    +---------------------+-------+-------+-------+------------------+------------+
    |                     | PSNR  | SSIM  | LPIPS | Train Mem        | Train Time |
    +=====================+=======+=======+=======+==================+============+
    | inria-7k            | 27.23 | 0.829 | 0.204 | 7.7 GB           | 6m05s      |
    +---------------------+-------+-------+-------+------------------+------------+
    | gsplat-7k           | 27.21 | 0.831 | 0.202 | **4.3GB**        | **5m35s**  |
    +---------------------+-------+-------+-------+------------------+------------+
    | inria-30k           | 28.95 | 0.870 | 0.138 | 9.0 GB           | 37m13s     |
    +---------------------+-------+-------+-------+------------------+------------+
    | gsplat-30k (1 GPU)  | 28.95 | 0.870 | 0.135 | **5.7 GB**       | **35m49s** |
    +---------------------+-------+-------+-------+------------------+------------+
    | gsplat-30k (4 GPUs) | 28.91 | 0.871 | 0.135 | **2.0 GB**       | **11m28s** |
    +---------------------+-------+-------+-------+------------------+------------+

This repo comes with a standalone script (:code:`examples/simple_trainer.py default`) that reproduces 
the `Gaussian Splatting <https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/>`_ with
exactly the same performance on PSNR, SSIM, LPIPS, and converged number of Gaussians. 
Powered by `gsplat`'s efficient CUDA implementation, the training takes up to 
**4x less GPU memory** with up to **15% less time** to finish than the official implementation.

Feature Ablation
----------------------------------------------
Evaluation of features provided in `gsplat` on Mip-NeRF (averaged over 7 scenes). We ablate `gsplat` with default settings, with absgrad and mcmc densification strategies, and antialiased mode.
Absgrad method uses `--grow_grad2d 0.0006` config. These results are obtained with an A100.

+-----------------------------+-------+-------+-------+----------+---------+------------+
|                             | PSNR  | SSIM  | LPIPS | Num GSs  | Mem (GB)| Time (min) |
+=============================+=======+=======+=======+==========+=========+============+
| gsplat (default settings)   | 29.00 | 0.87  | 0.14  | 3237318  | 5.62    | 19.39      |
+-----------------------------+-------+-------+-------+----------+---------+------------+
| absgrad                     | 29.11 | 0.88  | 0.12  | 2465986  | 4.40    | 18.10      |
+-----------------------------+-------+-------+-------+----------+---------+------------+
| antialiased                 | 29.03 | 0.87  | 0.14  | 3377807  | 5.87    | 19.52      |
+-----------------------------+-------+-------+-------+----------+---------+------------+
| mcmc  (1 mill)              | 29.18 | 0.87  | 0.14  | 1000000  | 1.98    | 15.42      |
+-----------------------------+-------+-------+-------+----------+---------+------------+
| mcmc  (2 mill)              | 29.53 | 0.88  | 0.13  | 2000000  | 3.43    | 21.79      |
+-----------------------------+-------+-------+-------+----------+---------+------------+
| mcmc  (3 mill)              | 29.65 | 0.89  | 0.12  | 3000000  | 4.99    | 27.63      |
+-----------------------------+-------+-------+-------+----------+---------+------------+
| absgrad & antialiased       | 29.14 | 0.88  | 0.13  | 2563156  | 4.57    | 18.43      |
+-----------------------------+-------+-------+-------+----------+---------+------------+
| mcmc & antialiased          | 29.23 | 0.87  | 0.14  | 1000000  | 2.00    | 15.75      |
+-----------------------------+-------+-------+-------+----------+---------+------------+


Trains Faster with Less GPU Memory
----------------------------------------------

+-----------------+---------+--------+---------+--------+---------+--------+--------+
| Train Mem (GB)  | Bicycle | Bonsai | Counter | Garden | Kitchen |  Room  | Stump  |
+=================+=========+========+=========+========+=========+========+========+
| inria-7k        |    7.86 |   7.61 |    6.47 |   8.99 |    8.08 |  7.88  |  7.23  |
+-----------------+---------+--------+---------+--------+---------+--------+--------+
| gsplat-7k       | **6.10**|**2.20**|**1.93** |**7.57**|**2.89** |**2.04**|**6.25**|
+-----------------+---------+--------+---------+--------+---------+--------+--------+
| inria-30k       |   11.56 |   7.70 |    6.73 |  11.04 |    8.33 |  8.50  |  8.82  |
+-----------------+---------+--------+---------+--------+---------+--------+--------+
| gsplat-30k      |**10.58**|**2.29**| **2.23**|**9.88**| **3.17**|**2.79**|**8.10**|
+-----------------+---------+--------+---------+--------+---------+--------+--------+

+-----------------+---------+--------+---------+--------+---------+--------+--------+
| Train Time (s)  | Bicycle | Bonsai | Counter | Garden | Kitchen |  Room  | Stump  |
+=================+=========+========+=========+========+=========+========+========+
| inria-7k        |    336  |   340  |    364  |    427 |    436  |  336   |  321   |
+-----------------+---------+--------+---------+--------+---------+--------+--------+
| gsplat-7k       |  **319**| **299**|  **318**| **415**|  **389**|**301** |**304** |
+-----------------+---------+--------+---------+--------+---------+--------+--------+
| inria-30k       |   2980  |   1552 |    1725 |   3092 |    2144 |  1773  |  2366  |
+-----------------+---------+--------+---------+--------+---------+--------+--------+
| gsplat-30k      | **2964**|**1422**| **1621**|**3013**|**2020** |**1708**|**2299**|
+-----------------+---------+--------+---------+--------+---------+--------+--------+

Reproduced Metrics
-------------------

+------------+---------+--------+---------+--------+---------+-------+-------+
| PSNR       | Bicycle | Bonsai | Counter | Garden | Kitchen |  Room | Stump |
+============+=========+========+=========+========+=========+=======+=======+
| inria-7k   |   23.59 |  29.75 |   27.21 |  26.13 |   29.02 | 29.26 | 25.64 |
+------------+---------+--------+---------+--------+---------+-------+-------+
| gsplat-7k  |   23.71 |  29.66 |   27.14 |  26.30 |   28.86 | 29.21 | 25.62 |
+------------+---------+--------+---------+--------+---------+-------+-------+
| inria-30k  |   25.19 |  32.21 |   29.02 |  27.29 |   31.07 | 31.31 | 26.56 |
+------------+---------+--------+---------+--------+---------+-------+-------+
| gsplat-30k |   25.22 |  32.06 |   29.02 |  27.32 |   31.16 | 31.36 | 26.53 |
+------------+---------+--------+---------+--------+---------+-------+-------+

+------------+---------+--------+---------+--------+---------+-------+-------+
| SSIM       | Bicycle | Bonsai | Counter | Garden | Kitchen |  Room | Stump |
+============+=========+========+=========+========+=========+=======+=======+
| inria-7k   | 0.662   | 0.921  | 0.877   | 0.824  | 0.902   | 0.893 | 0.721 |
+------------+---------+--------+---------+--------+---------+-------+-------+
| gsplat-7k  | 0.668   | 0.922  | 0.878   | 0.833  | 0.902   | 0.893 | 0.720 |
+------------+---------+--------+---------+--------+---------+-------+-------+
| inria-30k  | 0.763   | 0.941  | 0.906   | 0.863  | 0.925   | 0.918 | 0.771 |
+------------+---------+--------+---------+--------+---------+-------+-------+
| gsplat-30k | 0.764   | 0.941  | 0.907   | 0.865  | 0.926   | 0.918 | 0.768 |
+------------+---------+--------+---------+--------+---------+-------+-------+

+------------+---------+--------+---------+--------+---------+-------+-------+
| LPIPS      | Bicycle | Bonsai | Counter | Garden | Kitchen |  Room | Stump |
+============+=========+========+=========+========+=========+=======+=======+
| inria-7k   | 0.329   | 0.164  | 0.207   | 0.130  | 0.125   | 0.219 | 0.254 |
+------------+---------+--------+---------+--------+---------+-------+-------+
| gsplat-7k  | 0.324   | 0.162  | 0.206   | 0.123  | 0.127   | 0.217 | 0.253 |
+------------+---------+--------+---------+--------+---------+-------+-------+
| inria-30k  | 0.177   | 0.133  | 0.157   | 0.078  | 0.096   | 0.168 | 0.155 |
+------------+---------+--------+---------+--------+---------+-------+-------+
| gsplat-30k | 0.172   | 0.132  | 0.154   | 0.075  | 0.094   | 0.164 | 0.153 |
+------------+---------+--------+---------+--------+---------+-------+-------+

+-----------------+---------+--------+---------+--------+---------+-------+-------+
| Number of GSs   | Bicycle | Bonsai | Counter | Garden | Kitchen |  Room | Stump |
+=================+=========+========+=========+========+=========+=======+=======+
| inria-7k        |   3.57M |  1.16M |   1.01M |  4.33M |   1.63M | 1.11M | 3.75M |
+-----------------+---------+--------+---------+--------+---------+-------+-------+
| gsplat-7k       |   3.62M |  1.17M |   1.02M |  4.48M |   1.63M | 1.11M | 3.71M |
+-----------------+---------+--------+---------+--------+---------+-------+-------+
| inria-30k       |   6.06M |  1.24M |   1.19M |  5.71M |   1.78M | 1.55M | 4.82M |
+-----------------+---------+--------+---------+--------+---------+-------+-------+
| gsplat-30k      |   6.26M |  1.25M |   1.21M |  5.84M |   1.79M | 1.59M | 4.81M |
+-----------------+---------+--------+---------+--------+---------+-------+-------+

Note: Evaluations are conducted on a NVIDIA TITAN RTX GPU. The LPIPS metric is evaluated
using :code:`from torchmetrics.image.lpip import LearnedPerceptualImagePatchSimilarity`, which
is different from what's reported in the original paper that uses 
:code:`from lpipsPyTorch import lpips`.

The evaluation of `gsplat-X` can be reproduced with the command 
:code:`cd examples; bash benchmarks/basic.sh` 
within the gsplat repo (commit 6acdce4). 

The evaluation of `inria-X` can be 
reproduced with our forked wersion of the official implementation at 
`here <https://github.com/liruilong940607/gaussian-splatting/tree/benchmark>`_, 
with the command :code:`python full_eval_m360.py` (commit 36546ce).

2DGS
----------------------------------------------

No Regularization
----------------------------------------------

.. table:: Performance on `Mip-NeRF 360 Captures <https://jonbarron.info/mipnerf360/>`_ (Averaged Over 7 Scenes)

+---------------------+-------+-------+-------+------------------+------------+
|                     | PSNR  | SSIM  | LPIPS | Train Mem        | Train Time |
+=====================+=======+=======+=======+==================+============+
| inria-30k           | 28.73 | 0.860 | 0.148 | 3.73 GB          | 22m16s     |
+---------------------+-------+-------+-------+------------------+------------+
| gsplat-30k          | 28.76 | 0.867 | 0.145 | **3.70 GB**      | **15m44s** |
+---------------------+-------+-------+-------+------------------+------------+

With Normal Consistency and Distortion Regularization
------------------------------------------------------

+---------------------+-------+-------+-------+------------------+------------+
|                     | PSNR  | SSIM  | LPIPS | Train Mem        | Train Time |
+=====================+=======+=======+=======+==================+============+
| inria-30k           | 28.05 | 0.848 | 0.186 | 3.76 GB          | 22m06s     |
+---------------------+-------+-------+-------+------------------+------------+
| gsplat-30k          | 27.80 | 0.842 | 0.169 | **3.61 GB**      | **16m44s** |
+---------------------+-------+-------+-------+------------------+------------+

Runtime and GPU Memory
----------------------------------------------

+-----------------+---------+--------+---------+--------+---------+--------+--------+
| Train Mem (GB)  | Bicycle | Bonsai | Counter | Garden | Kitchen |  Room  | Stump  |
+=================+=========+========+=========+========+=========+========+========+
| inria-30k       |**6.74** |   2.27 |    2.06 |  4.79  |    2.25 |  2.40  |**5.58**|
+-----------------+---------+--------+---------+--------+---------+--------+--------+
| gsplat-30k      |   6.89  |**2.19**| **1.93**|**4.48**| **2.14**|**2.30**|  6.00  |
+-----------------+---------+--------+---------+--------+---------+--------+--------+

+-----------------+---------+--------+---------+--------+---------+--------+--------+
| Train Time (s)  | Bicycle | Bonsai | Counter | Garden | Kitchen |  Room  | Stump  |
+=================+=========+========+=========+========+=========+========+========+
| inria-30k       |  1463   |   1237 |   1318  |   1298 |    1422 |  1314  |  1252  |
+-----------------+---------+--------+---------+--------+---------+--------+--------+
| gsplat-30k      |**1231** |**788** |  **803**| **985**|  **828**| **789**|**1057**|
+-----------------+---------+--------+---------+--------+---------+--------+--------+


Reproduced Metrics
----------------------------------------------

+------------+---------+--------+---------+--------+---------+-------+-------+
| PSNR       | Bicycle | Bonsai | Counter | Garden | Kitchen |  Room | Stump |
+============+=========+========+=========+========+=========+=======+=======+
| inria-30k  |   24.92 |  31.87 |   28.78 |  26.88 |   31.08 | 31.21 | 26.36 |
+------------+---------+--------+---------+--------+---------+-------+-------+
| gsplat-30k |   24.97 |  31.94 |   28.76 |  26.95 |   31.08 | 31.27 | 26.37 |
+------------+---------+--------+---------+--------+---------+-------+-------+

+------------+---------+--------+---------+--------+---------+-------+-------+
| SSIM       | Bicycle | Bonsai | Counter | Garden | Kitchen |  Room | Stump |
+============+=========+========+=========+========+=========+=======+=======+
| inria-30k  | 0.741   | 0.937  | 0.899   | 0.847  | 0.921   | 0.914 | 0.760 |
+------------+---------+--------+---------+--------+---------+-------+-------+
| gsplat-30k | 0.764   | 0.937  | 0.899   | 0.849  | 0.921   | 0.915 | 0.761 |
+------------+---------+--------+---------+--------+---------+-------+-------+

+------------+---------+--------+---------+--------+---------+-------+-------+
| LPIPS      | Bicycle | Bonsai | Counter | Garden | Kitchen |  Room | Stump |
+============+=========+========+=========+========+=========+=======+=======+
| inria-30k  | 0.199   | 0.136  | 0.164   | 0.093  | 0.101   | 0.172 | 0.168 |
+------------+---------+--------+---------+--------+---------+-------+-------+
| gsplat-30k | 0.189   | 0.134  | 0.162   | 0.091  | 0.101   | 0.169 | 0.166 |
+------------+---------+--------+---------+--------+---------+-------+-------+

+-----------------+---------+--------+---------+--------+---------+-------+-------+
| Number of GSs   | Bicycle | Bonsai | Counter | Garden | Kitchen |  Room | Stump |
+=================+=========+========+=========+========+=========+=======+=======+
| inria-30k       |   3.97M |  0.91M |   0.72M |  2.79M |   0.85M | 1.01M | 3.27M |
+-----------------+---------+--------+---------+--------+---------+-------+-------+
| gsplat-30k      |   3.88M |  0.92M |   0.73M |  2.49M |   0.87M | 1.03M | 3.40M |
+-----------------+---------+--------+---------+--------+---------+-------+-------+

Note: Evaulations for 2DGS are conducted on a NVIDIA RTX 4090 GPU. The LPIPS metric is evaluated
using :code:`from torchmetrics.image.lpip import LearnedPerceptualImagePatchSimilarity`, which
is different from what's reported in the original paper that uses 
:code:`from lpipsPyTorch import lpips`.

The evaluation of `gsplat-X` can be reproduced with the command 
:code:`cd examples; bash benchmarks/basic_2dgs.sh` 
within the gsplat repo (commit 48abf70). 

The evaluation of `inria-X` can be 
reproduced with our forked wersion of the official implementation at 
`here <https://github.com/hbb1/diff-surfel-rasterization>`_;
you need to change the :code:`--model_type 2dgs` to :code:`--model_type 2dgs-inria` in
:code:`benchmars/basic_2dgs` and run command :code:`cd examples; bash benchmarks/basic_2dgs.sh` (commit 28c928a).