.gitignore

@@ -1,3 +1,4 @@
 dist
 build
 node_modules
+**/*.DS_Store

README.md

@@ -22,20 +22,40 @@ When I started, web-based viewers were already available -- A WebGL-based viewer
     - WASM splat sort: Implemented in C++ using WASM SIMD instructions
     - Partially GPU accelerated splat sort: Uses transform feedback to pre-calculate splat distances
 
+## Tips
+
+- Progressively loaded `.ply` and `.splat` files will not have certain optimizations such as cache-optimized splat ordering applied to them. For optimial performance, convert these file types to `.ksplat` or load them non-progressively.
+- Converting your scenes to `.ksplat` will result in the fastest loading times since its format matches the internal format for splat data.
+- Scenes with large dimensions or high splat density will cause issues with the default settings. For those scenes, you can try a couple of things:
+  - Set the viewer parameter `integerBasedSort` to `false` to force a slower, floating-point based splat sort.
+  - Experiment with a larger value for viewer parameter `splatSortDistanceMapPrecision`, to adjust the precision for the distance map in the splat sort. Larger precision values will result in reduced performance, but often can alleviate visual artifacts that arise when the precision is too low.
+
+
 ## Known issues
 
 - Splat sort runs on the CPU – would be great to figure out a GPU-based approach
 - Artifacts are visible when you move or rotate too fast (due to CPU-based splat sort)
 - Sub-optimal performance on mobile devices
 - Custom `.ksplat` file format still needs work, especially around compression
-- The default, integer based splat sort does not work well for larger scenes. In that case a value of `false` for the `integerBasedSort` viewer parameter can force a slower, floating-point based sort
+- Scenes with very large dimensions will probably crash (often with an `Index out of bounds` error from the splat sort). Changing `splatSortDistanceMapPrecision` or `integerBasedSort` will probably not help in those cases.
+
+## Limitations
+
+Currently there are limits on the number of splats that can be rendered, and those limits depend mainly on the degree of spherical harmonics desired. Those limits are:
+
+| Spherical harmonics degree | Max splat count
+| --- | ---
+| `0` | ~ 16,000,000
+| `1` | ~ 11,000,000
+| `2` | ~ 8,000,000
+
+Future work will include optimizing how splat data is packed into data textures, which will help increase these limits.
 
 ## Future work
 This is still very much a work in progress! There are several things that still need to be done:
-  - Improve the method by which splat data is stored in textures
+  - Improve the way splat data is packed into data textures
   - Continue optimizing CPU-based splat sort - maybe try an incremental sort of some kind?
-  - Add editing mode, allowing users to modify scene and export changes
-  - Support very large scenes
+  - Support very large scenes (streaming sections & LOD)
 
 ## Online demo
 [https://projects.markkellogg.org/threejs/demo_gaussian_splats_3d.php](https://projects.markkellogg.org/threejs/demo_gaussian_splats_3d.php)
@@ -147,12 +167,13 @@ Parameters for `addSplatScene()`
 
 | Parameter | Purpose
 | --- | ---
+| `format` | Force the loader to assume the specified file format when loading a splat scene. This is useful when loading from a URL where there is no file extension. Valid values are defined in the `SceneFormat` enum: `Ply`, `Splat`, and `KSplat`.
 | `splatAlphaRemovalThreshold` | Tells `addSplatScene()` to ignore any splats with an alpha less than the specified value (valid range: 0 - 255). Defaults to `1`.
 | `showLoadingUI` | Displays a loading spinner and/or loading progress bar while the scene is loading.  Defaults to `true`.
 | `position` | Position of the scene, acts as an offset from its default position. Defaults to `[0, 0, 0]`.
 | `rotation` | Rotation of the scene represented as a quaternion, defaults to `[0, 0, 0, 1]` (identity quaternion).
 | `scale` | Scene's scale, defaults to `[1, 1, 1]`.
-| `streamView` | Stream the scene's splat data and allow the scene to be rendered and viewed as the splats are loaded. Option is only valid for `addSplatScene()`, and not for `addSplatScenes()`.
+| `progressiveLoad` | Progressively load the scene's splat data and allow the scene to be rendered and viewed as the splats are loaded. Option is only valid for `addSplatScene()`, and not for `addSplatScenes()`.
 
 <br>
 
@@ -263,17 +284,21 @@ const viewer = new GaussianSplats3D.Viewer({
     'useBuiltInControls': false,
     'ignoreDevicePixelRatio': false,
     'gpuAcceleratedSort': true,
-    'halfPrecisionCovariancesOnGPU': true,
+    `enableSIMDInSort`: true,
     'sharedMemoryForWorkers': true,
     'integerBasedSort': true,
+    'halfPrecisionCovariancesOnGPU': true,
     'dynamicScene': false,
     'webXRMode': GaussianSplats3D.WebXRMode.None,
     'renderMode': GaussianSplats3D.RenderMode.OnChange,
     'sceneRevealMode': GaussianSplats3D.SceneRevealMode.Instant,
     'antialiased': false,
     'focalAdjustment': 1.0,
     'logLevel': GaussianSplats3D.LogLevel.None,
-    'sphericalHarmonicsDegree': 0
+    'sphericalHarmonicsDegree': 0,
+    `enableOptionalEffects`: false,
+    `inMemoryCompressionLevel`: 2
+    `freeIntermediateSplatData`: false
 });
 viewer.addSplatScene('<path to .ply, .ksplat, or .splat file>')
 .then(() => {
@@ -293,22 +318,32 @@ Advanced `Viewer` parameters
 | Parameter | Purpose
 | --- | ---
 | `selfDrivenMode` | If `false`, tells the viewer that you will manually call its `update()` and `render()` methods. Defaults to `true`.
-| `useBuiltInControls` | Tells the viewer to use its own camera controls. Defaults to `true`.
 | `renderer` | Pass an instance of a Three.js `Renderer` to the viewer, otherwise it will create its own. Defaults to `undefined`.
 | `camera` | Pass an instance of a Three.js `Camera` to the viewer, otherwise it will create its own. Defaults to `undefined`.
+| `useBuiltInControls` | Tells the viewer to use its own camera controls. Defaults to `true`.
 | `ignoreDevicePixelRatio` | Tells the viewer to pretend the device pixel ratio is 1, which can boost performance on devices where it is larger, at a small cost to visual quality. Defaults to `false`.
 | `gpuAcceleratedSort` | Tells the viewer to use a partially GPU-accelerated approach to sorting splats. Currently this means pre-computation of splat distances from the camera is performed on the GPU. It is recommended that this only be set to `true` when `sharedMemoryForWorkers` is also `true`. Defaults to `false` on mobile devices, `true` otherwise.
-| `halfPrecisionCovariancesOnGPU` | Tells the viewer to use 16-bit floating point values when storing splat covariance data in textures, instead of 32-bit. Defaults to `false`.
+| `enableSIMDInSort` | Enable the usage of SIMD WebAssembly instructions for the splat sort. Default is `true`.
 | `sharedMemoryForWorkers` | Tells the viewer to use shared memory via a `SharedArrayBuffer` to transfer data to and from the sorting web worker. If set to `false`, it is recommended that `gpuAcceleratedSort` be set to `false` as well. Defaults to `true`.
 | `integerBasedSort` | Tells the sorting web worker to use the integer versions of relevant data to compute the distance of splats from the camera. Since integer arithmetic is faster than floating point, this reduces sort time. However it can result in integer overflows in larger scenes so it should only be used for small scenes. Defaults to `true`.
+| `splatSortDistanceMapPrecision` | Specify the precision for the distance map used in the splat sort algorithm. Defaults to 16 (16-bit). A lower precision is faster, but may result in visual artifacts in larger or denser scenes.
+| `halfPrecisionCovariancesOnGPU` | Tells the viewer to use 16-bit floating point values when storing splat covariance data in textures, instead of 32-bit. Defaults to `false`.
 | `dynamicScene` | Tells the viewer to not make any optimizations that depend on the scene being static. Additionally all splat data retrieved from the viewer's splat mesh will not have their respective scene transform applied to them by default.
 | `webXRMode` | Tells the viewer whether or not to enable built-in Web VR or Web AR. Valid values are defined in the `WebXRMode` enum: `None`, `VR`, and `AR`. Defaults to `None`.
+| `webXRSessionInit` | Tells the viewer to build a WebXR session with some options. Defaults with {}. For more details : https://developer.mozilla.org/en-US/docs/Web/API/XRSystem/requestSession#options
 | `renderMode` | Controls when the viewer renders the scene. Valid values are defined in the `RenderMode` enum: `Always`, `OnChange`, and `Never`. Defaults to `Always`.
 | `sceneRevealMode` | Controls the fade-in effect used when the scene is loaded. Valid values are defined in the `SceneRevealMode` enum: `Default`, `Gradual`, and `Instant`. `Default` results in a nice, slow fade-in effect for progressively loaded scenes, and a fast fade-in for non progressively loaded scenes. `Gradual` will force a slow fade-in for all scenes. `Instant` will force all loaded scene data to be immediately visible.
 | `antialiased` |  When true, will perform additional steps during rendering to address artifacts caused by the rendering of gaussians at substantially different resolutions than that at which they were rendered during training. This will only work correctly for models that were trained using a process that utilizes this compensation calculation. For more details: https://github.com/nerfstudio-project/gsplat/pull/117, https://github.com/graphdeco-inria/gaussian-splatting/issues/294#issuecomment-1772688093
+| `kernel2DSize` | A constant added to the size of the 2D screen-space gaussian kernel used in rendering splats. Default value is 0.3.
 | `focalAdjustment` | Hacky, non-scientific parameter for tweaking focal length related calculations. For scenes with very small gaussians & small details, increasing this value can help improve visual quality. Default value is 1.0.
 | `logLevel` | Verbosity of the console logging. Defaults to `GaussianSplats3D.LogLevel.None`.
 | `sphericalHarmonicsDegree` | Degree of spherical harmonics to utilize in rendering splats (assuming the data is present in the splat scene). Valid values are 0, 1, or 2. Default value is 0.
+| `enableOptionalEffects` | When true, allows for usage of extra properties and attributes during rendering for effects such as opacity adjustment. Default is `false` for performance reasons. These properties are separate from transform properties (scale, rotation, position) that are enabled by the `dynamicScene` parameter.
+| `optimizeSplatData` | After loading is complete, reorder splat data in memory to optimize cache utilization as well as apply in-memory compression to reduce memory usage. Default is `true`. This option is automatically disabled if the scene is progressively loaded.
+| `inMemoryCompressionLevel` | Level to compress scenes when loading them for direct rendering (not exporting to `.ksplat`). Valid values are the same as `.ksplat` compression levels (0, 1, or 2). Default is 0 (uncompressed). If a scene is loaded progressively, or `optimizeSplatData` is set to `false`, `inMemoryCompressionLevel` will be 0.
+| `freeIntermediateSplatData` | When true, the intermediate splat data that is the result of decompressing splat bufffer(s) and used to populate data textures will be freed. This will reduces memory usage, but if that data needs to be modified it will need to be re-populated from the splat buffer(s). Defaults to `false`.
+| `splatRenderMode` | Determine which splat rendering mode to enable. Valid values are defined in the `SplatRenderMode` enum: `ThreeD` and `TwoD`. `ThreeD` is the original/traditional mode and `TwoD` is the new mode described here: https://surfsplatting.github.io/
+| `sceneFadeInRateMultiplier` | Customize the speed at which the scene is revealed. Default is 1.0.
 <br>
 
 ### Creating KSPLAT files
@@ -321,9 +356,14 @@ const compressionLevel = 1;
 const splatAlphaRemovalThreshold = 5; // out of 255
 const sphericalHarmonicsDegree = 1;
 GaussianSplats3D.PlyLoader.loadFromURL('<path to .ply or .splat file>',
+                                        onProgress,
+                                        progressiveLoad,
+                                        onProgressiveLoadSectionProgress,
+                                        minimumAlpha,
                                         compressionLevel,
-                                        splatAlphaRemovalThreshold,
-                                        sphericalHarmonicsDegree)
+                                        optimizeSplatData,
+                                        sphericalHarmonicsDegree,
+                                        headers)
 .then((splatBuffer) => {
     GaussianSplats3D.KSplatLoader.downloadFile(splatBuffer, 'converted_file.ksplat');
 });
@@ -333,7 +373,13 @@ Both of the above methods will prompt your browser to automatically start downlo
 The third option is to use the included nodejs script:
 
 ```
-node util/create-ksplat.js [path to .PLY or .SPLAT] [output file] [compression level = 0] [alpha removal threshold = 1]
+node util/create-ksplat.js [path to .PLY or .SPLAT] [output file] [compression level = 0] [alpha removal threshold = 1] [scene center = "0,0,0"] [block size = 5.0] [bucket size = 256] [spherical harmonics level = 0]
+```
+
+For the nodejs script, it may be necessary to increase the heap size for larger scenes. Use the parameter `--max-old-space-size=[heap size in MB]` to do so:
+
+```
+node util/create-ksplat.js --max-old-space-size=8192 [... remaining arguments]
 ```
 
 Currently supported values for `compressionLevel` are `0`, `1`, or `2`. `0` means no compression and `1` means compression of scale, rotation, position, and spherical harmonics coefficient values from 32-bit to 16-bit. `2` is similar to `1` except spherical harmonics coefficients are compressed to 8-bit.

demo/bonsai.html

@@ -43,7 +43,7 @@
     });
     let path = 'assets/data/bonsai/bonsai' + (mode ? '_high' : '') + '.ksplat';
     viewer.addSplatScene(path, {
-      'streamView': true
+      'progressiveLoad': false
     })
     .then(() => {
         viewer.start();