Embedded Files
CVPR 2022 Workshop on Multimodal Learning for Earth and Environment
9:00-9:15 (15 min): Opening remarks
9:15-9:45 (30 min): Invited talk (Phillip Isola, Associate Professor, MIT)
9:45-10:15 (30 min): Invited talk (Taylor Perron, Professor, MIT)
10:45-11:00 (15 min): Paper oral 1 (Xiaoxiang Zhu; virtual)
11:00-11:15 (15 min): Paper oral 2 (Colorado Reed)
11:15-11:30 (15 min): Paper oral 3 (Azin Asgarian)
11:30-11:45 (15 min): Challenge overview
11:45-12:00 (15 min): Challenge oral 1 (Team PAII-RS (Bo Peng))
12:00-12:15 (15 min): Challenge oral 2 (Team ForestGump (Yeonju Choi); virtual)
12:15-12:30 (15 min): Award announcement
12:30-12:45 (15 min): Closing
Here are the documentation about a subset of challenge submission:
Deforestation Estimation (Lee)
Deforestation Estimation (Prabhakar)
Image-to-Image Translation (Gou)
Image-to-Image Translation (Cheng)
Image-to-Image Translation (Prabhakar)
Mission of the workshop
Mission of the workshop
Despite international efforts to reduce deforestation, the world loses, so far, an area of forest that is equivalent to the size of 40 football fields every minute. Deforestation in the Amazon rainforest accounts for the largest share, contributing to reduced biodiversity, habitat loss, and climate change. Since much of the region is difficult to access, satellite remote sensing offers a powerful tool to track changes in the Amazon. However, obtaining a continuous time series of images is hindered by seasonal weather, clouds, smoke, and other inherent limitations of optical sensors. Synthetic aperture radar (SAR), which is insensitive to lighting and weather conditions, appears to be a well-suited tool for the task, but SAR images are more difficult for humans to interpret than optical images.
The 2022 Multimodal Learning for Earth and Environment Workshop (MultiEarth 2022) aims to gather a wide audience of researchers in academia, industry, and related fields to leverage remote sensing images collected by multiple sensors for positive environmental impact. The workshop comprises two tracks: the challenge track and the paper track. For the challenge track, we conduct the following three sub-challenges to support the interpretation and analysis of the rainforest at any time and under any weather conditions: 1) matrix completion, 2) deforestation estimation and 3) multimodal image-to-image translation sub-challenges. The paper track includes a broader call for papers that use machine learning to address problems in earth and environmental monitoring.
Dates and Deadlines
Dates and Deadlines
6/20 - Multi Earth 2022 Workshop @ CVPR
6/20 - Multi Earth 2022 Workshop @ CVPR
Paper Track:
Paper Track:
5/23 11:59PM PST - Paper submission deadline
5/23 11:59PM PST - Paper submission deadline
6/06 - Acceptance decisions
6/06 - Acceptance decisions
6/13 - Camera ready paper submission
6/13 - Camera ready paper submission
Challenge Track:
Challenge Track:
4/02 - Challenge training data released.
4/02 - Challenge training data released.
5/30 6/06 - Evaluation server open for the test set, with leaderboard available.
6/03 6/10 - Evaluation server closes
6/08 6/12 11:59PM PST - Paper submission deadline
6/20 - Challenge winners announced
6/20 - Challenge winners announced
Call For Papers
Call For Papers
We are soliciting papers that use machine learning to address problems in earth and environmental science and monitoring, including but not limited to the following topics:
Agriculture
Climate modeling
Disaster prediction, management, and relief
Ecosystems and natural systems
Carbon capture and sequestration
Land use applications
Land cover dynamics
Forest and vegetation: application and modeling
Forest and vegetation: biomass and carbon cycle
Topography, geology, and geomorphology
Soils and soil moisture
Wetlands
Inland waters
Multimodal remote sensing analysis
Change detection and multi-temporal analysis
Geographic information science
Multimodal generative modeling
Multimodal representation learning
Paper Submission Guidelines
Paper Submission Guidelines
We accept submissions of max 8 pages (excluding references) on the aforementioned and related topics. We encourage authors to submit 4-page works.
Submitted manuscripts should follow the CVPR 2022 paper template.
Accepted papers are not archival and will not be included in the proceedings of CVPR 2022
Submissions will be rejected without review if they:
Contain more than 8 pages (excluding references)
Violate the double-blind policy or violate the dual-submission policy
Paper submission must contain substantial original contents not submitted to any other conference, workshop, or journal
Papers will be peer-reviewed under a double-blind policy and need to be submitted online through the CMT submission website.
The Amazon Rainforest Challenge
The Amazon Rainforest Challenge
A key component of this event will be to monitor the Amazon rainforest in all weather and lighting conditions using our multimodal remote sensing dataset, which includes a time series of multispectral and synthetic aperture radar (SAR) images. We propose to conduct the following challenges to support the interpretation and analysis of the rainforest at any time and any weather conditions:
A key component of this event will be to monitor the Amazon rainforest in all weather and lighting conditions using our multimodal remote sensing dataset, which includes a time series of multispectral and synthetic aperture radar (SAR) images. We propose to conduct the following challenges to support the interpretation and analysis of the rainforest at any time and any weather conditions:
Image-to-Image Translation: Given a SAR image, predict a set of possible cloud-free corresponding electro-optical (EO) images
Matrix Completion: Given images taken at different locations and times, and in different modalities, predict appearance at a novel (time, location, modality) query
Downstream Task: Environment change estimation (e.g. deforestation, fire, water coverage) from the cloud-free view of the Amazon
Data
Data
Additional Data:
Challenge 1 - Matrix Completion
Challenge 2 - Deforestation Estimation
Test Queries:
Challenge 1 - Matrix Completion
Challenge 2 - Deforestation Estimation
Challenge 3 - Image-to-Image Translation
Submission sites:
*Please review the exact submission format in FAQ
Challenge 1 - Matrix Completion
Challenge 2 - Deforestation Estimation
Challenge 3 - Image-to-Image Translation
Organizing Committe
Organizing Committe
Miriam Cha, Technical staff, MIT Lincoln Laboratory
Sam Goldberg, Postdoc researcher, EAPS, MIT
Morgan Schmidt, Research affiliate, EAPS, MIT
Armando Cabrera, Flight chief, DAF MIT AI Accelerator
Taylor Perron, Professor, EAPS, MIT
Phillip Isola, Associate professor, EECS, MIT
Bill Freeman, Professor, EECS, MIT
Gregory Angelides, Technical staff, MIT Lincoln Laboratory
Kuan Wei Huang, Masters Student, EECS, MIT
Mark Hamilton, PhD Student, EECS, MIT
Yen-Chen Lin, PhD Student, EECS, MIT
Brandon Swenson, Technical Sergeant, DAF MIT AI Accelerator
Jean Piou, Technical Staff, MIT Lincoln Laboratory
Whitepaper
Whitepaper
FAQ
FAQ
Q: I am struggling with slow data download and loss of connection while downloading. Is there another way to download?
Q: I am struggling with slow data download and loss of connection while downloading. Is there another way to download?
A: Yes. You can download the dataset with azcopy (setup instructions here and here). Then download the images using:
azcopy cp “{dataset_url}” /local/directory --recursive
For example:
azcopy cp "https://rainforestchallenge.blob.core.windows.net/dataset/sent1_vv_train.zip” /cvpr/dataset --recursive
Q: Can we use some pre-trained models for the challenge (e.g. a model pre-trained on other remote sensing or natural images)?
Q: Can we use some pre-trained models for the challenge (e.g. a model pre-trained on other remote sensing or natural images)?
A: Yes. We encourage participants to leverage pre-trained models to make the results more impressive!
Q: Will the target images in the test set be cloud-free?
Q: Will the target images in the test set be cloud-free?
A: Yes. The target images in the test set will be mostly cloud-free. There may be a minor presence of clouds as in Figure 2 d) of our white paper.
Q: What is the exact input and output format?
Q: What is the exact input and output format?
A:
Matrix Completion
Input: 2000 test queries will be provided as a list of lists ie [[lon_0, lat_0, date_0, modality_0], …, [lon_1999, lat_1999, date_1999, modality_1999]].
Each test query is in the format [lon, lat, date, modality]. For example, [-55.15, -4.11, 2021_12_04, Landsat8_SR_B2] will represent Landsat8_SR_B2_-55.15_-4.11_2021_12_04.tiff.
Output: Participants will submit in total 2,000 256x256 images, one 256x256 image for each input test query. The output files should have the following dtypes: float32 for Sentinel-1 bands and uint16 for Sentinel-2, Landsat 5 and Landsat 8 bands. The output files should use the naming convention CH1_Band_Lon_Lat_Date.tiff. For example, the expected output file name for test query [-55.15, -4.11, 2021_12_04, Landsat8_SR_B2] should be CH1_Landsat8_SR_B2_-55.15_-4.11_2021_12_04.tiff.
Imagery related to the input test queries will be made available and can be used to help generate the requested output.
Deforestation Estimation
Input: 1000 test queries will be provided as a list of lists ie [[lon_0, lat_0, date_0, modality_0], …, [lon_999, lat_999, date_999, modality_999]].
Each test query is in the format [lon, lat, date, modality]. For example, [-55.15, -4.11, 2021_08_01, deforestation] will represent deforestation_-55.15_-4.11_2021_08.png.
To have a consistent naming convention for the date (i.e. year_month_day), we add a nominal day label of “_01” to all deforestation estimation test queries.
Output: Participants will submit in total 1,000 256x256 binary masks, one 256x256 binary mask for each input test query. 0’s in the binary mask will represent forested/other areas and 1’s will represent deforested areas. The output files should have dtype of uint8 and should use the naming convention CH2_Band_Lon_Lat_Date.png. For example, the expected output file name for test query [-55.15, -4.11, 2021_08_01, deforestation] should be CH2_deforestation_-55.15_-4.11_2021_08.png.
Imagery related to the input test queries will be made available and can be used to help generate the requested output.
Image-to-Image Translation
Input: 5000 Sentinel-1 SAR images, where each SAR image is 256x256x2 and the 2 channels correspond to the Sentinel-1 VV and VH bands, respectively.
Output: Participants will submit in total 15,000 256x256x3 EO images. Three translated 256x256x3 EO images for each input Sentinel-1 SAR image. The EO image channels correspond to the Sentinel-2 RGB channels (B4, B3, B2). This is a multimodal image-to-image translation problem where participants will generate three possible EO images given an input SAR image. The output files should have dtype of uint16 and should use the naming convention CH3_Band_Lon_Lat_Date_Sample#.tiff. For example, the expected output file names for input SAR images Sentinel1_VH_-54.80_-4.01_2019_03_18.tiff and Sentinel1_VV_-54.80_-4.01_2019_03_18.tiff should be CH3_Sentinel2_EO_-54.80_-4.01_2019_03_18_1.tiff, CH3_Sentinel2_EO_-54.80_-4.01_2019_03_18_2.tiff and CH3_Sentinel2_EO_-54.80_-4.01_2019_03_18_3.tiff.
For training, JSON files specifying which Sentinel-2 EO images (B4, B3, and B2) correspond to which SAR images have been provided. Two JSON files has been supplied, one for Sentinel1-VV band and one for Sentinel1-VH band. The mappings in both files are identical.
Citation
Citation
@misc{https://doi.org/10.48550/arxiv.2204.07649,
@misc{https://doi.org/10.48550/arxiv.2204.07649,
doi = {10.48550/ARXIV.2204.07649},
doi = {10.48550/ARXIV.2204.07649},
url = {https://arxiv.org/abs/2204.07649},
url = {https://arxiv.org/abs/2204.07649},
author = {Cha, Miriam and Huang, Kuan Wei and Schmidt, Morgan and Angelides, Gregory and Hamilton, Mark and Goldberg, Sam and Cabrera, Armando and Isola, Phillip and Perron, Taylor and Freeman, Bill and Lin, Yen-Chen and Swenson, Brandon and Piou, Jean},
author = {Cha, Miriam and Huang, Kuan Wei and Schmidt, Morgan and Angelides, Gregory and Hamilton, Mark and Goldberg, Sam and Cabrera, Armando and Isola, Phillip and Perron, Taylor and Freeman, Bill and Lin, Yen-Chen and Swenson, Brandon and Piou, Jean},
keywords = {Computer Vision and Pattern Recognition (cs.CV), FOS: Computer and information sciences, FOS: Computer and information sciences},
keywords = {Computer Vision and Pattern Recognition (cs.CV), FOS: Computer and information sciences, FOS: Computer and information sciences},
title = {MultiEarth 2022 -- Multimodal Learning for Earth and Environment Workshop and Challenge},
title = {MultiEarth 2022 -- Multimodal Learning for Earth and Environment Workshop and Challenge},
publisher = {arXiv},
publisher = {arXiv},
year = {2022},
year = {2022},
copyright = {Creative Commons Attribution 4.0 International}
copyright = {Creative Commons Attribution 4.0 International}
}
}
Page updated
Google Sites
Report abuse