Embedded Files
Research Team
Research Team
Trevor Chase
Trevor Chase
Geography and Sustainable Development
Daniela Inclezan
Daniela Inclezan
Computer Science and Software Engineering
Robbyn Abbitt
Robbyn Abbitt
Geography Department
Abstract/Introduction
Abstract/Introduction
Spatial data pipelines receive, store, transform, and analyze geographic information. Despite decades of building and researching such structures, few models are ‘general’ and which reduce the necessary functions to their minimal extent. Most implementations are, on the contrary, highly specialized, proprietary, and cloud-dependent, designed for specific enterprise use.
This paper presents a minimal implementation of a spatial data pipeline which can ingest heterogeneous spatial data (raster, vector, etc) using only free, open-source components. Its purpose is not to compete with larger commercial products, but to expose the architectural invariants common to all spatial pipelines. The model isolates the minimal set of operations which any spatial data stream must perform, while still successfully ingesting, storing, and transforming spatial data.
Methods
Methods
The design of the pipeline was guided by the need for generality, simplicity, and the ability to process heterogeneous spatial data in a variety of environments. The core requirements established were: (1) the pipeline must be able to receive and preserve any data that can be defined as spatial, maintaining the invariant properties of such data; (2) the ingestion layer must be capable of receiving data directly via TCP, supporting scenarios with or without standard internet connectivity; and (3) the system must be able to produce derivative spatial data products, such as transforming point clouds into rasters and digital elevation models (DEMs).
To meet these requirements, the software was implemented entirely with open-source tools and organized into four distinct layers: reception, ingestion, catalog, and analysis/transform. Each layer acts as a data silo, preserving data provenance as information is transformed. The reception layer uses a lightweight TCP receiver to accept spatial files in formats like GeoTIFF, Shapefile, and LAS/LAZ. The ingestion and cataloging layers employ a dispatcher to route files to the appropriate routines, using libraries such as Fiona and Rasterio, and record metadata in a SQLite manifest. The analysis/transform layer demonstrates the pipeline’s ability to convert LiDAR point clouds into DEMs and perform spatial analysis using SQL queries, validating the system’s capacity to handle and process diverse geospatial data streams.
Results/Findings and Discussion
Results/Findings and Discussion
The implemented prototype successfully demonstrated a fully operational, local spatial data pipeline capable of ingesting, cataloging, transforming, and analyzing heterogeneous geospatial datasets using only open-source tools. All files were transmitted, classified, and processed without error, and the pipeline’s LiDAR-to-DEM transformation produced results consistent with those generated in QGIS. The intersection analysis, leveraging SQL queries on the manifest, accurately detected spatial overlaps between raster and vector datasets. These results confirm that a general, open-source pipeline can effectively handle diverse spatial formats and workflows without reliance on proprietary or cloud-based infrastructure, though limitations remain in fine-scale topological analysis, concurrency, and support for certain remote sensing data types.
Conclusions and Future Study
Conclusions and Future Study
[Enter conclusions and future study. Adjust title as needed.]
This study presented and implemented a rapidly deployable, general, and open-source spatial data pipeline capable of receiving, cataloging, transforming, and analyzing georeferenced heterogeneous spatial datasets with few dependencies. Through a layered architecture, the pipeline demonstrates that spatial data workflows can be reduced to their logical essentials while remaining extensible and reproducible.
The prototype confirmed that LiDAR, raster, and vector data can coexist and interoperate within a single minimal framework, that transformation and analysis can be unified effectively, and that spatial invariants are preserved across processes. While simplified, the system establishes a replicable template for further experimentation and benchmarking of spatial infrastructures. Ultimately, it provides a foundation for both research and field deployment where accessibility, transparency, and modularity are more critical than scale.
Future Work
Expand manifest schema for lineage tracking, CRS transformations, and checksum verification.
Integrate a lightweight job scheduler for automation.
Extend analysis layer for topological joins and raster algebra.
Benchmark against distributed solutions like PostGIS or Databricks Lakehouse.
Acknowledgements
Acknowledgements
Thank you to Dr. Daniela Inclezan and Robbyn Abbit for their guidance.
References
References
Breunig, M., Bradley, P. E., Jahn, M., Kuper, P., Mazroob, N., Rösch, N., Al-Doori, M., Stefanakis, E., & Jadidi, M. (2020). Geospatial data management research: Progress and future directions. ISPRS International Journal of Geo-Information, 9(2), 95. https://doi.org/10.3390/ijgi9020095
Li, Z., Yang, C., Yu, M., Liu, K., & Sun, M. (2017). Cloud-based storage and computing for remote sensing data. Computers & Geosciences, 98, 81–90. https://doi.org/10.1016/j.cageo.2016.10.010
Wu, Y., Fan, H., Zhang, L., & Zhu, X. (2020). A processing pipeline for X3D Earth-based spatial data. International Journal of Digital Earth, 13(7), 802–820. https://doi.org/10.1080/17538947.2019.1705390
Zheng, J., Wang, S., & Cao, G. (2022). A novel spatial data pipeline for streaming smart-city data. Computers, Environment and Urban Systems, 95, 101841. https://doi.org/10.1016/j.compenvurbsys.2022.101841
Zhang, G., & Zheng, Y. (2015). Multiple auxiliary RPC products for the geometric processing of high-resolution spaceborne SAR datasets. Journal of Remote Sensing, 19(3), 409–430.
Biard, J., Tilmes, C., Kehoe, K., Wolfe, R., & Isaacman, A. (2013). The VIIRS Climate Raw Data Record: An Easy-to-use Raw Data Set (Level 1b) for the Suomi-NPP VIIRS. 93rd AMS Annual Meeting.
European Space Agency (ESA). (2016). Sentinel-1 Product Specification. ESA Standard Document.
NOAA/NASA. (2013). JPSS Common Data Format Control Book (CDFCB), Volume II: Raw Data Records (RDR) Formats. Joint Polar Satellite System.
Pan, H., Li, L., Zhang, Y., & Ding, M. (2019). Camera geolocation using digital elevation models in hilly area. Multimedia Tools and Applications, 78(7), 8711–8732. https://doi.org/10.1007/s11042-018-6230-8
USGS. (n.d.). 092—Handling raw data. U.S. Geological Survey Fundamental Science Practices. https://www.usgs.gov/office-of-science-quality-and-integrity/fundamental-science-practices/faq/092-handling-raw-data
NASA. (2023). Heliophysics data management handbook. National Aeronautics and Space Administration.
Foster, I., Vöckler, J., Wilde, M., & Zhao, Y. (2002). Chimera: A virtual data system for representing, querying, and automating data derivation. Argonne National Laboratory & University of Chicago.
GISGeography. (n.d.). Composite Bands: Black and White to Color Imagery. GISGeography. Retrieved December 7, 2025, from https://gisgeography.com/arcgis-composite-bands/
NACE Career Readiness Competencies
NACE Career Readiness Competencies
Career + Self-Development
This was my primary focus. I was able to earn a job based on the learning I was able to apply from this project.
Critical Thinking
Software Engineering is a science not just of fitting parts into a keyhole, but of optimization.
Professionalism
Open-Source designed to help, not profit
Technology
Function > Familiarity
Research Compliance Protocols
Research Compliance Protocols
There were no stated or enforced research compliance protocols.
Page updated
Report abuse