MultiEarth 2023
CVPR 2023 Workshop on Multimodal Learning for Earth and Environment
Mission of the workshop
The Multimodal Learning for Earth and Environment Workshop (MultiEarth 2023) is the second annual CVPR workshop aimed at leveraging the significant amount of remote sensing data that is continuously being collected to aid in the monitoring and analysis of the health of Earth ecosystems. The goal of the workshop is to bring together the Earth and environmental science communities as well as the multimodal representation learning communities to examine new ways to leverage technological advances in support of environmental monitoring. In addition, through a series of public challenges, the MultiEarth Workshop hopes to provide a common benchmark for remote sensing multimodal information processing. These challenges are focused on the monitoring of the Amazon rainforest and include deforestation estimation, fire detection, cross-modal image translation, and environmental change projection.
Invited Speakers
Workshop Schedule
June 19 - MultiEarth 2023 Workshop @ CVPR (West 109 - 110)
13:00-13:15: Opening Remarks
13:15-13:45: Invited Talk (Sara Beery, Assistant Professor, MIT)
13:45-14:15: Invited Talk (Amir Zamir, Assistant Professor, EPFL)
14:15-14:25: Coffee Break
14:25-14:35: Evaluating Loss Functions and Learning Data Pre-Processing for Deep Climate Downscaling Models (Xingying Huang et al.)
14:35-14:45: Debiased Learning from Naturally Imbalanced Remote Sensing Data (Chun-Hsiao Yeh et al.)
14:45-14:55: General-Purpose Multimodal Transformer meets Remote Sensing Semantic Segmentation (Nhi Kieu et al.)
14:55-15:05: Multi-Scale Transformer-Based Hierarchical Network for Remote Sensing Image Dehazing (Tony Zhang et al.)
15:05-15:15: Upscaling Global Hourly GPP with Temporal Fusion Transformer (TFT) (Rumi Nakagawa et al.)
15:15-15:30: Challenge Overview
15:30-15:45: Deforestation Estimation (Seunghan Park et al.)
15:45-16:00: SAR-to-EO Image Translation (Jingi Ju et al.)
16:00-16:15: Award Announcement
16:15-16:30: Closing
Below is a subset of submitted challenge reports:
Deforestation Estimation (Park et al.) *Winning Team*
Deforestation Estimation (Chen et al.)
Deforestation Estimation (Arya et al.)
Deforestation Estimation (Fodor et al.)
SAR-to-EO Image Translation (Ju et al.) *Winning Team*
Dates and Deadlines
June 19 - MultiEarth 2023 Workshop @ CVPR
Paper Track:
May 29 - Paper submission deadline
June 06 - Author notification
June 13 - Camera-ready deadline
Challenge Track:
April 18 - Challenge training data released
May 29 May 31 - Evaluation server open for the test set, with leaderboard available
June 3 June 6 June 7* - Evaluation server closes (*extended due to server maintenance shutdown)
June 06 11:59PM PST June 13 11:59PM PST - Model and Paper submission deadline
Email multiearth2023@gmail.com a link to the model (e.g. google drive, dropbox, etc)
Call For Papers
We are soliciting papers that use machine learning to address problems in earth and environmental science and monitoring, including but not limited to the following topics:
Agriculture
Climate modeling
Disaster prediction, management, and relief
Ecosystems and natural systems
Carbon capture and sequestration
Land use applications
Land cover dynamics
Forest and vegetation: application and modeling
Forest and vegetation: biomass and carbon cycle
Topography, geology, and geomorphology
Soils and soil moisture
Wetlands
Inland waters
Multimodal remote sensing analysis
Change detection and multi-temporal analysis
Geographic information science
Multimodal generative modeling
Multimodal representation learning
Paper Submission Guidelines
We accept submissions of max 8 pages (excluding references) on the aforementioned and related topics. We encourage authors to submit 4-page works.
Submitted manuscripts should follow the CVPR 2023 paper template.
Accepted papers are not archival and will not be included in the proceedings of CVPR 2023
Submissions will be rejected without review if they:
Contain more than 8 pages (excluding references)
Violate the double-blind policy or violate the dual-submission policy
Paper submission must contain substantial original contents not submitted to any other conference, workshop, or journal
Papers will be peer-reviewed under a double-blind policy and need to be submitted online through the CMT submission website.
MultiEarth Challenge
A key component of this challenge is to monitor the Amazon rainforest in all weather and lighting conditions using our multimodal remote sensing dataset, which includes a time series of multispectral and synthetic aperture radar (SAR) images. This year’s challenge includes two focus areas: 1) rapid detection and 2) projection of environmental change in the Amazon. We propose to conduct the following challenges to support the interpretation and analysis of the rainforest at any time and any weather conditions:
Rapid detection of environmental change in the Amazon:
Deforestation segmentation: Given multi-modal images, segment deforested areas
Fire segmentation: Given multi-modal images, segment burned areas
SAR-to-EO Image Translation: Given a SAR image, predict a set of possible cloud-free corresponding electro-optical (EO) images.
Projection of environmental change in the Amazon:
Deforestation segmentation: Given multi-modal historical imagery prior to the truth date, estimate deforestation at multi-year projections.
Image generation: Given multi-modal historical imagery prior to the truth date, predict appearance at a novel (location, modality) query at multi-year projections.
Data
Participants can download the training dataset with azcopy (setup instructions here) as follows:
azcopy cp "https://rainforestchallenge.blob.core.windows.net/multiearth2023-dataset-final/" /local/directory --recursive
The data is stored in two different formats for convenience. They are NetCDF files and zip files holding TIFF images. The desired version can be downloaded with azcopy using appropriate wildcards (e.g., *.zip or *.nc).
Participants can also access the data directly at the following URLs
ZIP:
NetCDF:
Additional Data
Deforestation Estimation
Fire Detection
Environmental Change Prediction
Evaluation
Deforestation Estimation
Test Queries
TIFF File Users:
NetCDF File Users:
Submission Site:
Fire Detection
Test Queries
TIFF File Users:
NetCDF File Users:
Submission Site:
Environmental Change Prediction
Test Queries
TIFF File Users:
NetCDF File Users:
prediction_targets.zip (A zip of NetCDF files)
Submission Site:
Multimodal SAR-to-EO Image Translation
Test Queries
TIFF File Users:
NetCDF File Users:
Submission Site:
Starter Code
This code repository holds tools for working with the large quantity of remote sensing data provided for these challenges. It includes datasets related to each challenge to aid in loading and filtering data from the provided NetCDF files. In addition, there are some simple utilities to aid in retrieving spatially and temporally aligned TIFF images.
Submission Guidelines
Deforestation Estimation
NetCDF file user’s test targets
We will provide a NetCDF file that includes an array of 1,000 target test samples. Each sample will contain longitude, latitude, collection date, and a data source (‘Deforestation’) related to a test target.
Functionality for retrieving images related to these test targets is provided at the MultiEarth Challenge repository
TIFF file user’s test targets
We will provide a csv file that includes 1,000 test queries as a list of lists
E.g., [[lon_0, lat_0, date_0, modality_0], …, [lon_999, lat_999, date_999, modality_999]]
Each test query is in the format [lon, lat, date, modality]
For example, [-55.15, -4.11, 2021_08_01, deforestation] will represent image deforestation_-55.15_-4.11_2021_08_01.tiff
To have a consistent naming convention for the date (i.e. year_month_day), we add a nominal day label of “_01” to all deforestation estimation test queries
Functionality for retrieving TIFF images related to these queries is provided at the MultiEarth Challenge repository
Challenge Submission: Participants will submit in total 1,000 256x256 binary masks, one 256x256 binary mask for each input test query
0’s in the binary mask will represent forested/other areas and 1’s will represent deforested areas
The output files should have dtype of uint8 and should use the naming convention Deforestation_Lon_Lat_Date.tiff
For example, the expected output file name for test query [-55.15, -4.11, 2021_08_01, deforestation] is deforestation_-55.15_-4.11_2021_08_01.tiff
Imagery related to the input test queries will be made available for the purpose of generating the requested output
Fire Detection
NetCDF file user’s test targets
We will provide a NetCDF file that includes an array of 1,000 target test samples. Each sample will contain longitude, latitude, collection date, and a data source (‘Fire’) related to a test target.
Functionality for retrieving images related to these test targets is provided at the MultiEarth Challenge repository
TIFF file user’s test targets
We will provide a csv file that includes1,000 test queries as a list of lists
E.g., [[lon_0, lat_0, date_0, modality_0], …, [lon_999, lat_999, date_999, modality_999]].
Each test query is in the format [lon, lat, date, modality]
For example, [-55.15, -4.11, 2021_08_01, fire] will represent image fire_-55.15_-4.11_2021_08_01.tiff
To have a consistent naming convention for the date (i.e. year_month_day), we add a nominal day label of “_01” to all fire estimation test queries.
Functionality for retrieving TIFF images related to these queries is provided at the MultiEarth Challenge repository
Challenge Submission: Participants will submit in total 1,000 10x10 binary masks, one 10x10 binary mask for each input test query
0’s in the binary mask will represent unburned areas and 1’s will represent fire / burned areas
These masks will be compared to the FireCCI51 data’s ConfidenceLevel band with a threshold of 50. (The range of ConfidenceLevel: [1,100])
The output files should have dtype of uint8 and should use the naming convention Fire_Lon_Lat_Date.tiff
For example, the expected output file name for test query [-55.15, -4.11, 2021_08_01, fire] is fire_-55.15_-4.11_2021_08_01.tiff
Imagery related to the input test queries will be made available for the purpose of generating the requested output
SAR-to-EO Image Translation
NetCDF file user’s test targets
We will provide a NetCDF file that includes an array of 5,000 test samples. Each sample will contain longitude, latitude, collection date, data source (‘Sentinel-2’), and bands (B4, B3, B2) related to a test target.
Each target sample will have a collection date exactly corresponding to the corresponding Sentinel-1 SAR image that serves as an input
Functionality for retrieving images related to these test targets is provided at the MultiEarth Challenge repository
TIFF file user’s test targets
10,000 Sentinel-1 SAR images will be provided. The SAR images will have shape 256x256x1 with 5,000 having VV polarization and the other 5,000 the corresponding VH polarization.
Challenge Submission: Participants will submit in total 15,000 256x256x3 EO images. Three translated 256x256x3 EO images for each input Sentinel-1 SAR image.
The EO image channels correspond to the Sentinel-2 RGB channels (B4, B3, B2).
This is a multimodal image-to-image translation problem where participants will generate three possible EO images given an input SAR image.
The output files should have dtype of uint16 and should use the naming convention SAR2EO_Modality_Lon_Lat_Date_Sample#.tiff For example:
Input: SAR images Sentinel1_VH_-54.80_-4.01_2019_03_18.tiff and Sentinel1_VV_-54.80_-4.01_2019_03_18.tiff
Expected output: SAR2EO_Sentinel2_EO_-54.80_-4.01_2019_03_18_1.tiff, SAR2EO_Sentinel2_EO_-54.80_-4.01_2019_03_18_2.tiff, and SAR2EO_Sentinel2_EO_-54.80_-4.01_2019_03_18_3.tiff
Additional SAR imagery related to these tests will be made available for the purpose of generating the requested output
Environmental Change Prediction
NetCDF file user’s test targets
We will provide a NetCDF file that includes an array of 2,000 target test samples. Each sample will contain longitude, latitude, collection date, data source, and band related to a test target.
Functionality for retrieving images related to these test targets is provided at the MultiEarth Challenge repository
TIFF file user’s test targets
We will provide a csv file that includes 2,000 test queries as a list of lists
E.g., [[lon_0, lat_0, date_0, modality_0], …, [lon_1999, lat_1999, date_1999, modality_1999]]
Each test query is in the format [lon, lat, date, modality]
For example, [-55.15, -4.11, 2021_12_04, Landsat8_SR_B2] will represent image Landsat8_SR_B2_-55.15_-4.11_2021_12_04.tiff
Functionality for retrieving TIFF images related to these queries is provided at the MultiEarth Challenge repository
Challenge Submission: Participants will submit in total 2,000 256x256 images, one 256x256 image for each input test query
The output files should have the following dtypes:
float32 for Sentinel-1 bands
uint16 for Sentinel-2, Landsat 5 and Landsat 8 bands
The output files should use the naming convention Prediction_Modality_Lon_Lat_Date.tiff
For example, output file name for test query [-55.15, -4.11, 2021_12_04, Landsat8_SR_B2] should be Prediction_Landsat8_SR_B2_-55.15_-4.11_2021_12_04.tiff
Imagery related to the input test queries will be made available for the purpose of generating the requested output
Organizing Committee
Miriam Cha, Technical staff, MIT Lincoln Laboratory
Gregory Angelides, Technical staff, MIT Lincoln Laboratory
Mark Hamilton, PhD Student, EECS, MIT
Andy Soszynski, Assistant staff, MIT Lincoln Laboratory
Brandon M. Swenson, Technical lead, USAF
Nathaniel Maidel, Master Sergeant, DAF MIT AI Accelerator
Phillip Isola, Associate professor, EECS, MIT
Taylor Perron, Professor, EAPS, MIT
Bill Freeman, Professor, EECS, MIT
Whitepaper
