Research & Projects
(unfold drop-down for details)

Embed2Scale aims to research the potential of representation learning for federated geospatial applications across high-performance compute infrastructure. Self-supervised learning (cf. foundation models / Large-Language Models) transforms data such as a 1000x1000 = 1 million pixel image into a feature vector of, e.g., 1000 floating point values.
In a collaboration of the German Aerospace Center, IBM Research, Oxford University, Juelich Supercomputing Center, the European Union Satellite Center, Zurich University, Muenster University, and Sinergise/Planet, we explore the potential for compression by self-supervised learning on geospatial [DOI10.1109/MGRS.2022.3198244] data across data centers.

The broad spectrum of European satellite missions under the umbrella of the Copernicus Programme and the success of deep learning in computer vision call for a test of multi-sensor data fusion with artificial neural networks. As part of the EvoLand consortium with Vito, GAF AG, CESBIO/CNES, CLS, the German Aerospace Center, Joanneum Research, and Sinergise/Planet, we explore the potential and limitations of land use mapping with deep learning for real-world industry use cases. 

How may we exploit high-quality remote sensing data (such as LiDAR) and incomplete crowdsourcing annotations (such as OpenStreetMap), to automatically label geospatial, multi-spectral imagery? To answer that question, we study noise-robust training of deep neural networks for semantic segmentation of geospatial (urban) scenes, e.g.:

• automatic classification of local climate zones from LiDAR data [DOI:10.48550/arXiv.2405.13993]

• adaptively correct incomplete building footprint labels while training a deep neural network for semantic segmentation [DOI:10.1109/TGRS.2024.3373908]

• auto-generated urban labels from airborne LiDAR [DOI:10.1109/BigData52589.2021.9672060] 

• monitoring the impact of Hurricane Sandy from auto-generated vegetation labels [DOI:10.1109/IGARSS46834.2022.9884017] 

• early stopping strategy for training a semantic segmentation model on noisy labels generated by an airborne LiDAR survey [DOI:10.1109/BigData55660.2022.10020164] 

• a novel methodology for unsupervised generative modelling of remote sensor correlation with proof-of-concept for urban heat island analysis [DOI:10.1109/IGARSS52108.2023.10281573] 

• OpenStreetMap label-generation from style transfer of high-resolution overhead imagery to rasterized maps [DOI:10.1145/3394486.3403301], DOI:10.3390/ijgi9070427]

While remote sensing streams petabytes of optical, radar, and LiDAR (laser) data for various Earth observation (EO) applications, human labeling and efficient multi-modal data fusion to train deep neural networks poses a major challenge. With my students, we research self-supervised learning (SSL) methodologies for representation learning (feature vectors) of remote sensing information, e.g.:

• autoencoder-generated feature vectors from reconstructing classical computer vision features [DOI:10.48550/arXiv.2310.18653]

• interpretation of SSL feature vectors for radar-optical fusion [DOI:10.48550/arXiv.2309.05300]

• radar-optical neural networks fusion robust to missing a modality [DOI:10.1109/IGARSS46834.2022.9883983]

• label-assisted SSL for hyperspectral image segmentation [DOI:10.1109/IGARSS52108.2023.10282971] 

• benchmark dataset and models (convolutional neural networks and vision transformers) for radar-optical data fusion [DOI:10.1109/MGRS.2023.32816511] 

Remote sensing such as aerial imaging and airborne LiDAR surveys provide a versatile tool to scan large areas for ancient artifacts. However, a major technical challenge poses the low signal-to-noise ratio (artifact erosion) and the little amount of available labels (scarcity of artifacts). In close collaboration with IBM Research, I develop machine learning pipelines to guide archaeologists in their field work, e.g.:

• Nasca UNESCO World Heritage, Peru: protection of geoglyphs from flood events [DOI:10.48550/arXiv.2405.11814]

• Angamacu city, Mexico: identification of eroded buildings buried under vegetation [DOI:10.1109/BigData47090.2019.9005548] 

• Negev desert agriculture, Israel: identification of ancient agricultural terraces [DOI:10.1109/BigData55660.2022.10020329] 

Carbon sequestration through trees is a natural approach with additional benefit for local climate zones and biodiversity. On various scales, I collaborate in projects to estimate biomass from remote sensing data, e.g.:

• study the cooling potential of trees in various urban scenarios thermal band of Landsat 8 satellite and Local Climate Zones [FragileEarth@KDD2022] 

• carbon sequestration of urban forests [ClimateChangeAI@ICML2021] 

• estimate biomass from single tree height through Gaussian process [DOI:/10.1109/JSTARS.2023.3271186] 

• continent-scale biomass estimation with physics-informed deep neural network [ClimateChangeAI@NEURIPS2022], [DOI:10.1109/IGARSS52108.2023.10282838]

Complex geospatial analytics requires scalable infrastructure and distributed software to index, process, and fuse data spatio-temporal information. I contributed the following innovations:

• efficient cross-layer spatio-temporal queries through overviews [Patent:US11360970B2]

• vector-raster data fusion method [Patent:US11594004B2]

• spatio-temporal vector-raster data query system [Patent:US11204896B2]

Employing open-source technologies such as Apache HBase, Hadoop, and Spark, I co-developed, tested, and employed a platform for Earth observation data science, cf.:

• platform implementation, benchmark, and use case demonstration [DOI:10.1109/BigData.2015.7363884], [DOI:10.5194/isprs-archives-XLII-3-W12-2020-255-2020], [DOI:10.1109/BigData.2016.7840910]

• novel temporal clustering algorithm [DOI:10.1109/BigData.2017.825849f], [Patent:US10706080B2]

• application for agriculture [DOI:10.1147/JRD.2016.2591698], infrastructure risk assessment [DOI:10.1109/BigData47090.2019.9006600], and disaster management DOI:10.1147/JRD.2020.2970903 

• Phys.org blog post on IBM PAIRS

• open-source IBM PAIRS API on GitHub: https://github.com/IBM/ibmpairs 

• my 2021 presentation of IBM PAIRS on YouTube

• YouTube recordings of my 2022 presentation on open-source stacks for geospatial data platforms at the 2nd NASA workshop on Open Source Science for the Earth System Observatory Mission