Evaluation¶
3DGS¶
PSNR |
SSIM |
LPIPS |
Train Mem |
Train Time |
|
|---|---|---|---|---|---|
inria-7k |
27.23 |
0.829 |
0.204 |
7.7 GB |
6m05s |
gsplat-7k |
27.21 |
0.831 |
0.202 |
4.3GB |
5m35s |
inria-30k |
28.95 |
0.870 |
0.138 |
9.0 GB |
37m13s |
gsplat-30k (1 GPU) |
28.95 |
0.870 |
0.135 |
5.7 GB |
35m49s |
gsplat-30k (4 GPUs) |
28.91 |
0.871 |
0.135 |
2.0 GB |
11m28s |
This repo comes with a standalone script (examples/simple_trainer.py default) that reproduces
the Gaussian Splatting with
exactly the same performance on PSNR, SSIM, LPIPS, and converged number of Gaussians.
Powered by gsplat’s efficient CUDA implementation, the training takes up to
4x less GPU memory with up to 15% less time to finish than the official implementation.
Feature Ablation¶
Evaluation of features provided in gsplat on Mip-NeRF (averaged over 7 scenes). We ablate gsplat with default settings, with absgrad and mcmc densification strategies, and antialiased mode. Absgrad method uses –grow_grad2d 0.0006 config. These results are obtained with an A100.
PSNR |
SSIM |
LPIPS |
Num GSs |
Mem (GB) |
Time (min) |
|
|---|---|---|---|---|---|---|
gsplat (default settings) |
29.00 |
0.87 |
0.14 |
3237318 |
5.62 |
19.39 |
absgrad |
29.11 |
0.88 |
0.12 |
2465986 |
4.40 |
18.10 |
antialiased |
29.03 |
0.87 |
0.14 |
3377807 |
5.87 |
19.52 |
mcmc (1 mill) |
29.18 |
0.87 |
0.14 |
1000000 |
1.98 |
15.42 |
mcmc (2 mill) |
29.53 |
0.88 |
0.13 |
2000000 |
3.43 |
21.79 |
mcmc (3 mill) |
29.65 |
0.89 |
0.12 |
3000000 |
4.99 |
27.63 |
absgrad & antialiased |
29.14 |
0.88 |
0.13 |
2563156 |
4.57 |
18.43 |
mcmc & antialiased |
29.23 |
0.87 |
0.14 |
1000000 |
2.00 |
15.75 |
Trains Faster with Less GPU Memory¶
Train Mem (GB) |
Bicycle |
Bonsai |
Counter |
Garden |
Kitchen |
Room |
Stump |
|---|---|---|---|---|---|---|---|
inria-7k |
7.86 |
7.61 |
6.47 |
8.99 |
8.08 |
7.88 |
7.23 |
gsplat-7k |
6.10 |
2.20 |
1.93 |
7.57 |
2.89 |
2.04 |
6.25 |
inria-30k |
11.56 |
7.70 |
6.73 |
11.04 |
8.33 |
8.50 |
8.82 |
gsplat-30k |
10.58 |
2.29 |
2.23 |
9.88 |
3.17 |
2.79 |
8.10 |
Train Time (s) |
Bicycle |
Bonsai |
Counter |
Garden |
Kitchen |
Room |
Stump |
|---|---|---|---|---|---|---|---|
inria-7k |
336 |
340 |
364 |
427 |
436 |
336 |
321 |
gsplat-7k |
319 |
299 |
318 |
415 |
389 |
301 |
304 |
inria-30k |
2980 |
1552 |
1725 |
3092 |
2144 |
1773 |
2366 |
gsplat-30k |
2964 |
1422 |
1621 |
3013 |
2020 |
1708 |
2299 |
Reproduced Metrics¶
PSNR |
Bicycle |
Bonsai |
Counter |
Garden |
Kitchen |
Room |
Stump |
|---|---|---|---|---|---|---|---|
inria-7k |
23.59 |
29.75 |
27.21 |
26.13 |
29.02 |
29.26 |
25.64 |
gsplat-7k |
23.71 |
29.66 |
27.14 |
26.30 |
28.86 |
29.21 |
25.62 |
inria-30k |
25.19 |
32.21 |
29.02 |
27.29 |
31.07 |
31.31 |
26.56 |
gsplat-30k |
25.22 |
32.06 |
29.02 |
27.32 |
31.16 |
31.36 |
26.53 |
SSIM |
Bicycle |
Bonsai |
Counter |
Garden |
Kitchen |
Room |
Stump |
|---|---|---|---|---|---|---|---|
inria-7k |
0.662 |
0.921 |
0.877 |
0.824 |
0.902 |
0.893 |
0.721 |
gsplat-7k |
0.668 |
0.922 |
0.878 |
0.833 |
0.902 |
0.893 |
0.720 |
inria-30k |
0.763 |
0.941 |
0.906 |
0.863 |
0.925 |
0.918 |
0.771 |
gsplat-30k |
0.764 |
0.941 |
0.907 |
0.865 |
0.926 |
0.918 |
0.768 |
LPIPS |
Bicycle |
Bonsai |
Counter |
Garden |
Kitchen |
Room |
Stump |
|---|---|---|---|---|---|---|---|
inria-7k |
0.329 |
0.164 |
0.207 |
0.130 |
0.125 |
0.219 |
0.254 |
gsplat-7k |
0.324 |
0.162 |
0.206 |
0.123 |
0.127 |
0.217 |
0.253 |
inria-30k |
0.177 |
0.133 |
0.157 |
0.078 |
0.096 |
0.168 |
0.155 |
gsplat-30k |
0.172 |
0.132 |
0.154 |
0.075 |
0.094 |
0.164 |
0.153 |
Number of GSs |
Bicycle |
Bonsai |
Counter |
Garden |
Kitchen |
Room |
Stump |
|---|---|---|---|---|---|---|---|
inria-7k |
3.57M |
1.16M |
1.01M |
4.33M |
1.63M |
1.11M |
3.75M |
gsplat-7k |
3.62M |
1.17M |
1.02M |
4.48M |
1.63M |
1.11M |
3.71M |
inria-30k |
6.06M |
1.24M |
1.19M |
5.71M |
1.78M |
1.55M |
4.82M |
gsplat-30k |
6.26M |
1.25M |
1.21M |
5.84M |
1.79M |
1.59M |
4.81M |
Note: Evaluations are conducted on a NVIDIA TITAN RTX GPU. The LPIPS metric is evaluated
using from torchmetrics.image.lpip import LearnedPerceptualImagePatchSimilarity, which
is different from what’s reported in the original paper that uses
from lpipsPyTorch import lpips.
The evaluation of gsplat-X can be reproduced with the command
cd examples; bash benchmarks/basic.sh
within the gsplat repo (commit 6acdce4).
The evaluation of inria-X can be
reproduced with our forked wersion of the official implementation at
here,
with the command python full_eval_m360.py (commit 36546ce).
2DGS¶
No Regularization¶
PSNR |
SSIM |
LPIPS |
Train Mem |
Train Time |
|
|---|---|---|---|---|---|
inria-30k |
28.73 |
0.860 |
0.148 |
3.73 GB |
22m16s |
gsplat-30k |
28.76 |
0.867 |
0.145 |
3.70 GB |
15m44s |
With Normal Consistency and Distortion Regularization¶
PSNR |
SSIM |
LPIPS |
Train Mem |
Train Time |
|
|---|---|---|---|---|---|
inria-30k |
28.05 |
0.848 |
0.186 |
3.76 GB |
22m06s |
gsplat-30k |
27.80 |
0.842 |
0.169 |
3.61 GB |
16m44s |
Runtime and GPU Memory¶
Train Mem (GB) |
Bicycle |
Bonsai |
Counter |
Garden |
Kitchen |
Room |
Stump |
|---|---|---|---|---|---|---|---|
inria-30k |
6.74 |
2.27 |
2.06 |
4.79 |
2.25 |
2.40 |
5.58 |
gsplat-30k |
6.89 |
2.19 |
1.93 |
4.48 |
2.14 |
2.30 |
6.00 |
Train Time (s) |
Bicycle |
Bonsai |
Counter |
Garden |
Kitchen |
Room |
Stump |
|---|---|---|---|---|---|---|---|
inria-30k |
1463 |
1237 |
1318 |
1298 |
1422 |
1314 |
1252 |
gsplat-30k |
1231 |
788 |
803 |
985 |
828 |
789 |
1057 |
Reproduced Metrics¶
PSNR |
Bicycle |
Bonsai |
Counter |
Garden |
Kitchen |
Room |
Stump |
|---|---|---|---|---|---|---|---|
inria-30k |
24.92 |
31.87 |
28.78 |
26.88 |
31.08 |
31.21 |
26.36 |
gsplat-30k |
24.97 |
31.94 |
28.76 |
26.95 |
31.08 |
31.27 |
26.37 |
SSIM |
Bicycle |
Bonsai |
Counter |
Garden |
Kitchen |
Room |
Stump |
|---|---|---|---|---|---|---|---|
inria-30k |
0.741 |
0.937 |
0.899 |
0.847 |
0.921 |
0.914 |
0.760 |
gsplat-30k |
0.764 |
0.937 |
0.899 |
0.849 |
0.921 |
0.915 |
0.761 |
LPIPS |
Bicycle |
Bonsai |
Counter |
Garden |
Kitchen |
Room |
Stump |
|---|---|---|---|---|---|---|---|
inria-30k |
0.199 |
0.136 |
0.164 |
0.093 |
0.101 |
0.172 |
0.168 |
gsplat-30k |
0.189 |
0.134 |
0.162 |
0.091 |
0.101 |
0.169 |
0.166 |
Number of GSs |
Bicycle |
Bonsai |
Counter |
Garden |
Kitchen |
Room |
Stump |
|---|---|---|---|---|---|---|---|
inria-30k |
3.97M |
0.91M |
0.72M |
2.79M |
0.85M |
1.01M |
3.27M |
gsplat-30k |
3.88M |
0.92M |
0.73M |
2.49M |
0.87M |
1.03M |
3.40M |
Note: Evaulations for 2DGS are conducted on a NVIDIA RTX 4090 GPU. The LPIPS metric is evaluated
using from torchmetrics.image.lpip import LearnedPerceptualImagePatchSimilarity, which
is different from what’s reported in the original paper that uses
from lpipsPyTorch import lpips.
The evaluation of gsplat-X can be reproduced with the command
cd examples; bash benchmarks/basic_2dgs.sh
within the gsplat repo (commit 48abf70).
The evaluation of inria-X can be
reproduced with our forked wersion of the official implementation at
here;
you need to change the --model_type 2dgs to --model_type 2dgs-inria in
benchmars/basic_2dgs and run command cd examples; bash benchmarks/basic_2dgs.sh (commit 28c928a).