Skip to content

NeRF Program

Spring 2026

1. Program Overview and Objectives

The Neural Radiance Fields and Scene Representations for Spaceflight program develops shared infrastructure for learning-based three-dimensional scene representations relevant to spaceflight imagery, proximity operations, and terrain-relative navigation. The program consolidates Neural Radiance Fields, Gaussian Splatting, and related scene-representation methods into a unified and extensible software stack that supports both research experimentation and downstream simulation and autonomy use cases.

The central objective of this program is to establish a single, well-maintained NeRFStudio-based repository that serves as the lab's canonical implementation. Within this repository, both volumetric NeRF-style models and point-based Gaussian Splatting representations are treated as first-class citizens, with consistent interfaces for training, evaluation, and visualization. A key emphasis is placed on standardizing how datasets are described, transformed, evaluated, and visualized across heterogeneous spaceflight imagery sources, including archival data, partner-provided datasets, and lab-generated imagery.

Another core objective is to enable tight coupling between learned scene representations and high-fidelity spacecraft simulation. Learned models should be usable not only for offline reconstruction and analysis, but also as queryable components within simulation environments, supporting realistic rendering under correct geometry, lighting, and eclipse conditions. The program is applied-methodological in nature, operating near the 6.1/6.2 boundary: it advances perception methods while explicitly supporting operational simulation, autonomy, and guidance research.

2. Scope and Non-Goals

The program scope includes consolidation and extension of NeRFStudio for spaceflight imagery, development of dataset description and transformation standards, implementation of model training, evaluation, and visualization pipelines, and creation of interfaces between learned scene representations and spacecraft simulation. Tooling is expected to support both advanced research experiments and undergraduate-facing workflows.

Explicitly out of scope are flight-qualified perception systems, one-off visualization scripts that do not generalize, mission-specific pipelines tightly coupled to a single dataset, and wholesale re-implementation of NeRFStudio core functionality unless strictly required for spaceflight-specific needs.

3. Core Deliverables for the First Semester

By the end of the semester, the program is expected to deliver a unified NeRFStudio-based repository that consolidates existing NeRF and Gaussian Splatting efforts. Divergent branches should be resolved, with clear ownership, documentation, and design rationale. The repository must explicitly support both volumetric NeRF models and Gaussian Splatting or related point-based representations through a consistent and extensible interface.

The repository should follow a clear internal structure that separates dataset handling, model implementations, evaluation metrics, and visualization utilities. Dataset infrastructure must support formal dataset descriptions, including provenance, coordinate frames, required transformations, sensor models, illumination conditions, and other relevant metadata. Preprocessing and normalization steps should be explicitly documented, reproducible, and queryable without requiring model training.

Model training and evaluation pipelines must support both qualitative and quantitative assessment. Quantitative evaluation should include standard image-space metrics as well as geometry-based metrics where ground truth or proxy geometry is available. Visualization tools should support rendering, inspection, debugging, and the generation of publication-quality figures and animations, with clear separation between model logic and visualization logic.

A critical deliverable is a clean and documented interface to spacecraft simulation. Given spacecraft state, attitude, Sun position, and eclipse conditions, learned scene representations should be queryable to render physically consistent images at configurable rates. Simulation logic and rendering logic must remain decoupled to ensure reuse and extensibility.

Comprehensive documentation is required. This includes a "getting started" guide for new students, tutorials covering dataset ingestion, model training, evaluation, visualization, and simulation interfacing, and design documentation explaining architectural decisions and extension points. The repository must also include basic software quality infrastructure, including unit tests for dataset handling, model interfaces, and rendering utilities, as well as a continuous integration pipeline and documented coding standards.

4. User Stories and Intended Usage

The tools developed by this program should support a variety of realistic research workflows. A researcher with spacecraft imagery and an existing STL model should be able to generate a Gaussian Splatting representation and evaluate it quantitatively using both image-space and geometry-based metrics. A researcher working with planetary or lunar imagery without an existing mesh should be able to reconstruct a scene representation and assess reconstruction quality. Within simulation, a researcher should be able to load a pre-trained scene representation and render images from the perspective of a spacecraft under correct lighting and eclipse conditions.

The infrastructure should also support migration of experiments from local systems to HPC resources using documented and reproducible workflows. Undergraduate researchers should be able to train and visualize scene representations using provided scripts and datasets without modifying core infrastructure. Senior developers and the PI should be able to assess reproducibility, documentation quality, and architectural soundness with minimal overhead.

5. Development Tasks and Execution Expectations

During the semester, the program is expected to audit and consolidate existing NeRF and Gaussian Splatting codebases, define a clear architectural roadmap for the unified repository, and design dataset schemas and preprocessing pipelines. At least one complete end-to-end reconstruction workflow should be implemented, from raw imagery through model training, evaluation, and visualization.

Additional effort should focus on developing evaluation scripts for both image-space and geometry-based metrics, implementing and validating the Basilisk rendering interface, developing visualization and debugging utilities, writing tutorials and onboarding documentation, and setting up testing and continuous integration. Program members should also identify and document interfaces with the autonomy, DRL, and astrodynamics programs to ensure alignment and reuse.

6. Collaboration, Meetings, and Workflow Expectations

Program members meet every two weeks for at least one hour on Fridays. These meetings are intended to review code, reconstructions, and evaluation results; discuss architectural decisions and trade-offs; identify gaps needed by downstream users; and read and discuss relevant literature.

All substantive work is expected to follow disciplined version-control practices. Program members are responsible for filing issues and pull requests, reviewing and commenting on others' code, and actively using shared tooling in their own research workflows to expose usability and design issues.

7. Faculty Interface and Code Review

At least once per semester, the program will conduct a structured design and code review with the PI. Program members should be prepared to present a high-level overview of the repository architecture, demonstrate representative datasets, trained models, and renderings, and discuss technical debt, limitations, and next steps. This review is intended to ensure architectural coherence and long-term sustainability.

8. Undergraduate Integration and Mentorship

Program members are responsible for supervising undergraduate researchers within the program. Undergraduate projects should focus on dataset preparation and exploration, training and evaluating existing models, and visualization and qualitative analysis. Graduate students are expected to provide well-scoped tasks with clear success criteria, ensure tools are usable with minimal setup, and treat undergraduate usage as a validation of usability, documentation quality, and robustness.

9. End-of-Semester Deliverables Checklist

By the end of the Spring 2026 semester, the following items should be complete and verifiable.

Repository and Architecture

  • A single unified NeRFStudio-based repository designated as the lab's canonical implementation

  • Consolidation of prior NeRF and Gaussian Splatting branches with documented ownership

  • Clear internal structure separating data, models, metrics, and visualization

Dataset Infrastructure

  • Standardized dataset description schema implemented and documented

  • Support for dataset provenance, coordinate frames, and transformations

  • Ability to inspect and visualize datasets without training a model

Models and Training

  • At least one NeRF-style model and one Gaussian Splatting model fully supported

  • End-to-end training workflow demonstrated on at least one spaceflight-relevant dataset

Evaluation and Visualization

  • Image-space evaluation metrics implemented and documented

  • Geometry-based evaluation metrics implemented where applicable

  • Ability to generate publication-quality figures or videos from the codebase

Simulation Interface

  • Documented interface for querying trained models from spacecraft simulation

  • Rendering demonstrated using spacecraft state, attitude, Sun position, and eclipse conditions

Documentation and Onboarding

  • A "getting started" guide that runs end-to-end

  • Tutorials for dataset ingestion, training, evaluation, and visualization

  • Design documentation describing architectural choices and extension points

Software Quality

  • Unit tests covering dataset handling, model interfaces, and rendering utilities

  • Continuous integration pipeline running automatically

  • Coding standards and contribution guidelines documented

Collaboration and Oversight

  • Regular biweekly program meetings held

  • Active use of issues and pull requests by multiple contributors

  • One structured code and design review conducted with the PI

Undergraduate Integration

  • At least one undergraduate successfully training or visualizing a scene representation

  • Undergraduate-facing scripts or documentation requiring minimal setup