G-SHARP v0.2 Integration Proposal

Status:

Implemented on branch vnath_gsharp (under gsplat.contrib.dynamic); experimental, APIs may change

Source:

holohub/applications/surgical_scene_recon/ (G-SHARP v0.2)

Destination:

gsplat/ (branch vnath_gsharp)

Goals

  • Port all training-time algorithmic contributions of G-SHARP v0.2 natively into gsplat.

  • Keep the deformable / 4D Gaussian machinery behind a clearly experimental gsplat/contrib/ namespace.

  • Ship test-first: every ported unit has positive + negative pytest cases.

  • Provide a runnable end-to-end tutorial on the EndoNeRF sample.

Non-goals

  • No Holoscan / HoloHub imports anywhere.

  • No Depth Anything V2 / MedSAM3 / VGGT code inside the gsplat/ package. These are external pretrained nets; the tutorial only references them as optional upstream preprocessing.

  • No compression / rate-distortion work (not present in G-SHARP v0.2).

  • No camera-pose optimization (G-SHARP v0.2 doesn’t do it either).

Component map

Core (stable) additions:

Component

New file

G-SHARP source

Binocular disparity L1 loss

gsplat/losses.py (added alongside existing depth losses)

EndoRunner.compute_depth_loss, training/gsplat_train.py 906–960

Pearson depth loss (mono)

gsplat/losses.py (added alongside existing depth losses)

same

Masked L1 / SSIM wrappers

gsplat/losses.py (added alongside existing depth losses)

training/gsplat_train.py 1059–1101

Occlusion-TV regularizer

gsplat/regularizers.py

compute_tv_loss_targeted, 1925–2028

Torch mask dilation

gsplat/regularizers.py

OpenCV dilation replaced by max-pool

Invisible-mask builder

gsplat/regularizers.py

create_invisible_mask_from_paths

Multi-frame depth unprojection

gsplat/init_utils.py

accumulate_multiframe_pointcloud, 2036–2159

KNN scale init

gsplat/init_utils.py

create_splats_with_optimizers, 1796–1906

Two-stage scheduler

gsplat/training/schedulers.py

EndoRunner._train_stage, 1007–1050

Experimental contrib additions (gsplat.contrib.dynamic):

Component

New file

G-SHARP source

HexPlane field

gsplat/contrib/dynamic/hexplane.py

training/scene/hexplane.py

Deform MLP + per-Gaussian deformation table

gsplat/contrib/dynamic/deformation.py

training/scene/deformation.py, gsplat_train.py 1376–1664

Plane / time smoothness regularizers

gsplat/contrib/dynamic/regulation.py

training/scene/regulation.py, loop 1233–1272

DynamicStrategy

gsplat/contrib/dynamic/strategy.py

rasterize_splats, 864–896

Examples / docs:

Deliverable

Path

EndoNeRF/SCARED dataset loader

examples/datasets/endonerf.py

Dynamic-scene trainer recipe

examples/dynamic_surgical_trainer.py

Tutorial

docs/source/examples/dynamic_surgical.rst

Contrib API docs

docs/source/apis/contrib.rst

Why this layout

  • The core library stays pure splatting math. Depth losses / occlusion-TV / multi-frame init / two-stage schedule are generic and low-risk, so they live alongside existing losses.py, rendering.py, strategy/.

  • 4D / deformable pieces add new public surfaces (HexPlane, DeformNet, DynamicStrategy) that are still research-grade → gsplat/contrib/dynamic/ signals “ships with the wheel, API may change”. Pattern follows the well-known torchvision.prototype convention.

  • EndoNeRF/SCARED loader is domain-specific dataset glue → belongs with colmap.py / ncore.py under examples/datasets/, not in the library.

  • DA2 / MedSAM3 / VGGT do not enter the tree; the tutorial explains “get poses_bounds.npy, masks/, depth/ some way” and points at the (unported) holohub folder for reference.

Test plan

Every new module gets a tests/test_*.py with both positive and negative cases. Tests follow the flat layout and pytest conventions already in gsplat/tests/ (conftest.py, seed 42, CUDA-aware).

See the component × test-case matrix in planning/gsharp_gsplat_plan.md. (A shareable HTML preview can be rendered from that markdown locally; it is intentionally not tracked in the repo.)

Risks

  • Deformation + ``rasterization()`` coupling. gsplat.rasterization has no time axis. Solution: DynamicStrategy applies the deform-net to (means, quats, opacities) before rasterization(...) is called (same pattern as G-SHARP’s rasterize_splats).

  • CUDA-only tests. HexPlane/DeformNet tests run on CPU for determinism; GPU smoke tests are marked CUDA-only.

  • EndoNeRF sample license. Documented in the tutorial, not redistributed in-tree.

  • sklearn optional dep. Pure-torch fallback for KNN init keeps the required-dep list unchanged.