Completed Work During SMURFs II FA25-SU26
Thesis and Defense Recording
Below is a link to view my full written thesis and a recording of my defense. These may help with some more in-depth details on certain components of the work done.
Defense Presentation:
(Link + Passcode if the embedded video doesn't work) Defense Recording Passcode: s70%x1A%
Multi-Model Composition and Rendering
The project initially investigated the composition and joint rendering of multiple independently trained neural radiance field models. For NeRF-based models, this required a transformation, composition, and rendering pipeline capable of placing each model into a shared scene while preserving the local coordinate system in which it had been trained. Camera rays were transformed from the shared world frame into each model’s local frame, evaluated independently by the corresponding radiance fields, and then transformed back into a common representation. Samples from the individual models were subsequently merged and sorted along each camera ray so that density accumulation and color rendering could be performed across the complete composed scene.
A corresponding pipeline was later developed for multiple independently trained 3D Gaussian Splatting models. Because 3DGS represents the scene through an explicit collection of Gaussian primitives, model composition could be performed more directly by transforming the Gaussian parameters themselves. Gaussian means, orientations, scales, and related directional quantities were transformed into a shared coordinate frame, after which the primitive sets from the individual models could be concatenated and rendered together. This allowed independently reconstructed objects to be translated, rotated, and scaled into new multi-object configurations without retraining a combined model.
Development ultimately shifted fully from NeRF-based rendering to 3DGS. Although both representations supported multi-model composition, 3DGS provided substantially faster rendering and more direct access to the scene representation. Its explicit Gaussian parameters also made it better suited for the later physical interaction components of the project, including light-space transformations, per-Gaussian shadow receiving, and the transfer of shadow information between independently trained models. The final multi-model framework therefore uses parameter-space transformation and concatenation to support efficient scene construction, rendering, and relighting.
Dynamic Relighting and Shadowing
A sun-conditioned neural appearance field was adopted to produce illumination-dependent Gaussian appearance and enable dynamic relighting. Rather than storing a single fixed color for each Gaussian, the field predicts appearance as a function of the incident sun direction and a learned per-Gaussian embedding vector. These embeddings provide local latent appearance information and allow different regions of the object to respond differently to the same illumination. Per-image appearance embeddings were intentionally excluded because they can encourage the model to memorize individual training images and produce non-physically-consistent relighting.
Explicit shadowing was incorporated through two methods: shadow mapping and shadow splatting. Both render the scene from the perspective of the sun and transfer the resulting visibility information to the Gaussians during camera-view rendering. Shadow mapping uses contribution-weighted expected depth to construct an approximate light-space surface. Each Gaussian receiver is projected into this representation and classified as shadowed when it lies behind the estimated surface. Shadow splatting was adapted from the GS³ method and instead reuses the transmittance accumulation process of Gaussian rasterization. Light is progressively attenuated as the scene is traversed in light-space depth, allowing each Gaussian to collect shadow according to the remaining transmittance along its light ray.
Both methods can be paired with a neural shadow refinement network. The network uses the raw visibility estimate, Gaussian appearance features, illumination direction, and geometric context to correct artifacts and provide a final appearance pass. The raw shadowing method remains responsible for the primary structure of the cast shadows, while the network accounts for discretization errors and illumination effects that are not fully captured by direct visibility. Additionally, a receiver-depth bias shifts each Gaussian’s shadow-receiving position slightly along the incident light direction to reduce superficial self-shadowing, or shadow acne. The bias is computed separately for each Gaussian using its orientation and extent relative to the light source, and greatly reduces shadow acne without causing shadows to recede excessively (“peter-panning”).
An adaptive light-camera placement algorithm was also developed to maintain shadow resolution across multi-model scenes of arbitrary size. Rather than expanding a single orthographic light camera until it covers the complete scene, Gaussian receivers are grouped in light space and distributed across multiple fitted cameras. Regions are subdivided when their required coverage would otherwise reduce the effective shadow resolution. This allows the number and placement of light cameras to adjust automatically as independently trained models are transformed and arranged into new configurations.
Informed Training Strategies
Two alternative density-control strategies were developed to investigate whether additional scene information could improve Gaussian placement and retention. Mesh-Aware density control uses ground-truth or approximate mesh information to remove Gaussians that drift far outside the expected model region while protecting primitives near the known surface. Rendering-contribution information is also used to preserve Gaussians that remain useful to the reconstruction. Support-Aware density control removes the requirement for an external mesh by maintaining moving averages of how strongly each Gaussian contributes to rendered images. These support estimates augment low-opacity culling decisions so that Gaussians representing dark or weakly observed regions are not discarded solely because their opacity remains low. Although both strategies produced competitive results for some reconstruction metrics, they increased the number of retained Gaussians and did not consistently outperform standard density control.
A separate warm-start procedure was developed for models initialized from a random spatial distribution instead of a structure-from-motion point cloud. An approximate foreground mask is used to attract projected Gaussians toward the object and encourage increased opacity on the object during the beginning of training. Culling is temporarily made more conservative, and the additional objectives are gradually reduced as normal photometric training takes over. This procedure improves the concentration of the initial geometry and the final reconstruction when a reliable point-cloud initialization is unavailable.
Case Studies
The completed tools were evaluated across a progression of increasingly realistic space-imagery datasets. Initial development and controlled testing used simulated Hubble imagery provided by the Johns Hopkins University Applied Physics Laboratory. This data supported early evaluation of object reconstruction and multi-model rendering under known synthetic conditions. The SPEED+ synthetic spacecraft dataset was used to test reconstruction and illumination-aware rendering across a larger range of spacecraft viewpoints and lighting configurations. Its controlled imagery provided a useful intermediate case between the initial simulated object data and real inspection imagery. SPEED+ was also used for exploratory work involving illumination-direction estimation, although robust recovery of unknown per-image sun directions remains an open research direction rather than a completed component of the final pipeline. Finally, the developed reconstruction and rendering tools were applied to imagery from the real-world ADRAS-J debris-inspection mission. This case study introduced practical challenges that were less prominent in the synthetic datasets, including sparse camera coverage, a more limited range of viewpoints, imperfect initialization, real sensor characteristics, and uncontrolled background content. The ADRAS-J experiments were used to assess how the developed 3DGS methods transfer from controlled simulation to real on-orbit imagery, and reinforced the notion that 3DGS struggles to reproduce intense specular reflections and camera-space artifacts like lens flare.