Notable Future Work Directions As of Summer 2026
More Physically Decomposed Appearance Models
A potentially useful enhancement of the appearance module would further decompose the learned appearance representation into physically meaningful factors. Associating parameters such as surface albedo, normals, roughness, material properties, and bidirectional reflectance distribution functions with each Gaussian may improve generalization to new lighting conditions and produce more physically consistent reflective effects for spacecraft materials such as solar panels and metallic components.
The current SplatfactoAppearanceField predicts illumination-dependent appearance from learned per-Gaussian features and Sun-direction encodings. These features can represent material response, shading, and reconstruction errors without distinguishing among them. The shadow refinement network may introduce similar ambiguity by compensating for errors in visibility, geometry, or appearance.
A reasonable first step would be to add per-Gaussian albedo and normal outputs, followed by a simple diffuse or diffuse-plus-specular shading model. Geometry and raw shadowing should initially remain fixed, and the refinement network should be disabled or restricted so that it does not absorb the appearance decomposition.
Relevant Research Works
- https://arxiv.org/abs/2311.16043
- https://arxiv.org/abs/2312.16215
- https://arxiv.org/abs/2312.05133
- https://arxiv.org/abs/2410.11419v1
- https://arxiv.org/abs/2407.12306
Relevant Repository Files
nerfstudio/models/splatfacto.pynerfstudio/fields/splatfacto_appearance_field.pynerfstudio/utils/splatfacto_field_utils.pynerfstudio/utils/shadow_splatting_helpers.pysubmodules/gsplat/gsplat/rendering.py
2D Gaussian Splatting for Surface-Consistent Relighting and Shadowing
Another promising direction would be to investigate 2D Gaussian Splatting as a more surface-consistent representation for spacecraft radiance fields. Unlike volumetric 3D Gaussians, 2DGS represents the scene using oriented planar Gaussian disks, reducing the volumetric thickness that makes surface depth and light-space visibility difficult to define. The original 2DGS method additionally introduces ray–splat intersection, depth-distortion regularization, and normal-consistency losses to improve geometric accuracy.
This direction is directly relevant to the receiver-depth and shadow-acne issues encountered in the existing framework. Firmer surface localization could provide more reliable receiver depths and normals, particularly on grazing surfaces where the current receiver-depth bias remains limited.
This would require more than reducing one Gaussian scale. The primitive parameterization, rasterizer, depth computation, densification, and regularization losses would need to follow the 2DGS formulation. A practical first step would be to establish a working 2DGS baseline within SmurfStudio before adapting the Sun-conditioned appearance field or either shadowing method.
Relevant Research Works
Relevant Repository Files
nerfstudio/models/splatfacto.pysubmodules/gsplat/gsplat/rendering.pysubmodules/gsplat/gsplat/cuda/csrc/bindings.hsubmodules/gsplat/gsplat/cuda/_wrapper.pysubmodules/gsplat/gsplat/cuda/csrc/fully_fused_projection_2dgs_fwd.cusubmodules/gsplat/gsplat/cuda/csrc/fully_fused_projection_2dgs_bwd.cusubmodules/gsplat/gsplat/cuda/csrc/rasterize_to_pixels_2dgs_fwd.cusubmodules/gsplat/gsplat/cuda/csrc/rasterize_to_pixels_2dgs_bwd.cu
Modeling of Multiple Illumination Sources
A further extension of the relighting framework would model multiple illumination sources rather than assuming that the Sun is the only dominant source. In orbital imagery, Earth albedo may provide secondary illumination that brightens nominally shadowed surfaces and produces softer appearance changes than direct sunlight.
The existing framework models the Sun as a distant directional source, with an orthographic light camera used for physical shadow estimation. Earth illumination should likely be represented separately because the Earth is an extended, spatially distributed source rather than another hard directional light. A simple initial approximation could condition the appearance field on the direction and intensity of Earth illumination while retaining the existing Sun shadowing path unchanged. Later work could approximate the Earth using a hemisphere, spherical harmonics, or several sampled directional sources.
Separate Sun and Earth inputs or network branches would help prevent the appearance and refinement networks from treating both effects as a single unconstrained lighting code.
Relevant Research Works
Relevant Repository Files
nerfstudio/models/splatfacto.pynerfstudio/fields/splatfacto_appearance_field.pynerfstudio/utils/splatfacto_field_utils.pynerfstudio/utils/shadow_splatting_helpers.pynerfstudio/utils/render_helpers.pynerfstudio/data/dataparsers/splatfacto_dataparser.py
Tone Mapping for High-Dynamic-Range Space Imagery
Another useful direction would incorporate tone mapping or high-dynamic-range intensity modeling into the training pipeline. Space imagery can contain saturated highlights, strong specular reflections, directly illuminated surfaces, and nearly black shadowed regions. Training only on normalized low-dynamic-range RGB values prevents the model from determining how much radiance existed above the sensor saturation limit.
A possible extension would allow the appearance field to predict an internal radiance value before applying a differentiable camera-response or tone-mapping function. The rendered low-dynamic-range output could then be compared with the training image using the existing photometric losses. Exposure time, gain, or per-image exposure corrections could later be added when suitable metadata are available. (Logan also has experience experimenting with tone mapping for NeRFs, so I highly recommend touching base with him before diving into this)
Relevant Research Works
- https://arxiv.org/abs/2111.13679
- https://arxiv.org/abs/2503.10143v1
- https://bmvc2024.org/proceedings/22/
Relevant Repository Files
nerfstudio/models/splatfacto.pynerfstudio/fields/splatfacto_appearance_field.pynerfstudio/data/dataparsers/splatfacto_dataparser.pynerfstudio/data/utils/tone_mapping.py- (and other Logan-recommended tone mapping experiments as examples)
Estimating Sun Direction for Datasets Without Illumination Labels
For datasets without ground-truth illumination labels, such as SPEED+, accurate reconstruction across multiple Sun angles remains challenging. Rather than assigning each image an unconstrained appearance embedding, each image could be associated with a constrained pseudo-Sun direction, a residual rotation from an initial estimate, or one of (k) shared lighting bins.
Preliminary work in this repository has already implemented and tested several parts of this problem. Learned per-image Sun angles, differentiable shadowing method gradients, and other avenues have been explored to a moderate extent. These experiments showed that useful physical gradients can exist, but that convergence is slow and joint optimization allows geometry, appearance, or refinement (especially) to compensate for incorrect illumination.
Working Notes: I’ve put some time into this future work direction, to less avail than I think should be possible, given how intuitive the problem seems. There is a medium backbone of code and tools written to attempt to estimate/optimize a sun angle for each image either during/after regular training that may be helpful as a starting point. Namely, there is plumbing through splatfacto.py that allows each training image to possess an optimizable sun angle that can be learned during training, with multiple “methods” of learning them. The “bins” method was an attempt to categorize each image into a sun-angle bin, given that you know a priori how many total sun angles there are, and the three others (per-image vector, per-image rotation, and sun rotation MLP) assign an individual sun angle to each image. “Per-image vector” learns a 3-dim vector for each image, “per-image rotation” starts each training image with a base sun angle and learns a rotation to apply to shift it to a correct sun angle, and “sun rotation MLP” does something similar to the other rotation, but uses a neural network to output a sun rotation for each image. A cool diagnostic tool I began playing around with (definitely ask an agent to read out how to use this its a little confusing and I vibe coded it :D) is the sun_loss_viewer; it lets you analyze the effect of using different sun angles on the image-space loss for a given training image, and can be useful for inspecting the loss landscape for different loss function, sun angle, and image configurations. Since doing normal training AND sun-angle learning simultaneously was a bit tricky, most of the attempts I’ve run have been with splatfacto_resume_train_example.py, which will let you resume training of an existing model, but on different data/with different aspects of the model frozen. In this case, I’d freeze the geometry and appearance, and attempt to only learn sun angles (without shadow refinement) to isolate the issues, but had not been able to get it working. This is a semi-interesting sun direction, but has been a bit of a time sink, and I think some of the other directions should be given higher priority. If your spacecraft is in orbit, you almost always know your direction to the sun (via sun sensors for both navigation and instrument safety purposes), and so this was more of a focused exploration stemming from the annoyance of not having GT sun labels for the SPEED+ dataset.
For a more in-depth understanding, I recommend exploring the code with an agent to summarize and connect some of the pieces that might be hard to trace here in plain language.
Some findings up to this point are: the shadow refinement network will learn faster than the per-image sun angles can be optimized, and will absorb the effect of an incorrect sun angle if you attempt to learn appearance + sun angles simultaneously (if this future work direction is explored in more depth, extended characterization of these effects may be useful); LPIPS seems like the most consistent metric to use as a sun-angle-learning loss function (as given by sun_loss_viewer diagnostic experiments), though I’ve not gotten it to work fully; masking over and only evaluated loss using pixels above a luminance threshold (0.1 used, same as for the shadow-specific metrics) tends to have the same trend as evaluating over all pixels, but loss values are generally ~an order of magnitude larger than unmasked;
Relevant Research Works
Relevant Repository Files
nerfstudio/data/dataparsers/splatfacto_dataparser.pynerfstudio/models/splatfacto.pynerfstudio/utils/splatfacto_utils.pynerfstudio/utils/shadow_splatting_helpers.pysubmodules/gsplat/gsplat/cuda/_wrapper.pysubmodules/gsplat/gsplat/cuda/csrc/rasterize_to_pixels_shadow_bwd.cu(for backwards shadow splatting gradients)- (FIX LOCATION)
specific_data_scripts/sun_loss_viewer.py(for diagnostics) /model_training/splatfacto_resume_train_example.py(for taking existing trained models and attempting to only learn sun angles)
Depth Sensor-Guided Learning
An interesting reconstruction-fidelity avenue is to combine visible-band imagery with an active depth-sensing modality. Sparse or dense depth measurements could guide Gaussian initialization, supervise rendered depth, or penalize Gaussian support in regions inconsistent with the measured surface. This may improve reconstruction stability and help separate geometric structure from illumination-dependent appearance.
A practical early experiment might use simulated ground-truth depth from an existing spacecraft dataset, followed by progressively sparse and noisy measurements. This would establish whether depth improves geometry and shadow accuracy before introducing real sensor calibration and registration issues.
Relevant Research Works
- https://arxiv.org/abs/2502.11801
- https://s3.amazonaws.com/amz.xcdsystem.com/A464D031-C624-C138-7D0E208E29BC4EDD_abstract_File25843/PreprintPaperUpload_901_0725095212.pdf
Relevant Repository Files
nerfstudio/data/dataparsers/splatfacto_dataparser.pynerfstudio/data/datamanagers/full_images_datamanager.pynerfstudio/models/splatfacto.pysubmodules/gsplat/gsplat/rendering.py