Call for Papers

Submission link: https://openreview.net/group?id=ICLR.cc/2025/Workshop/MCDC

Questions can be directed to: mcdc-workshop@googlegroups.com

Key Dates

All deadlines are 23:59 AoE (Anywhere on Earth)

• Paper submission deadline: February 12, 2025

• Decision notifications: March 5, 2025

• Camera ready deadline: April 4, 2025

• Workshop: April 27, 2025

We invite researchers and practitioners to submit their work to the ICLR 2025 Workshop on Modular, Collaborative, and Decentralized Deep Learning (MCDC@ICLR25).   

The workshop aims to explore new paradigms in designing neural network architectures based on modularity, functional specialization, and model recycling [Pfeiffer et al., 2023] to enable more flexible and reusable architectures and unlock the collaborative development of large-scale models [Raffel, 2023].

A non-exhaustive list of topics of interest includes:

• Mixture-of-Experts (MoE) Architectures: advancements in MoE for sparsely activated models, including novel training methods, efficient routing algorithms, and applications in diverse domains and modalities [Jacobs et al., 1991, Eigen et al., 2014, Shazeer et al., 2017, Douillard et al., 2024];

• Routing of Specialized Experts (MoErging): Exploring techniques for effectively recycling and routing among pre-trained models or Parameter-Efficient Fine-Tuning (PEFT) modules as specialized experts [Zhou et al., 2022, Filippova et al., 2024, Yadav et al., 2024a, Ostapenko et al., 2024];

• Upcycling and MoE-fication: Exploring techniques for adapting existing dense models into modular frameworks, including converting monolithic architectures into MoE systems [Zhang et al., 2022, Zhang et al., 2024, Komatsuzaki et al., 2022, Sukhbaatar et al., 2024, Cai et al., 2024];

• Model Soups and Model Merging: Investigating methods for combining independently trained checkpoints to create better and multi-task models, and understanding the theoretical foundations of model merging [Wortsman et al 2022, Ramé et al., 2022, Ilharco et al., 2022, Yadav et al., 2023, Yadav et al., 2024, Tam et al., 2024];

• Applications of modularity: We encourage explorations of modular architectures to create more flexible and maintainable models, particularly in areas like lifelong/continual learning [Ostapenko et al., 2021], machine unlearning [Bourtoule et al., 2019 ], and compositional generalization [Pfeiffer et al., 2020, 2021, 2023]. 

• Decentralized and Collaborative Training: Developing novel algorithms and engineering solutions for extremely communication-efficient collaborative and distributed training of models, modular and otherwise. [McMahan et al., 2017, Diskin et al., 2021, Douillard et al., 2023, Jaghouar et al., 2024];

• Adaptive Architectures: Designing architectures that dynamically adjust their structure and computational at runtime to modulate computational capacity based on the input data, task demands, or available resources. This includes dynamic depth, dynamic width, and conditional computation [Devrit et al., 2023, Cai et al., 2024, Raposo et al. 2024];

Submission format: 

• Submissions should be anonymized (double-blind review) and follow the ICLR 2025 template and formatting guidelines.

• We encourage both short papers (2 pages) and long papers (6 pages).

• Submission to the workshop is non-archival (i.e. double submission is allowed, and accepted papers will be posted on the workshop website)

• We especially encourage submissions from individuals from underrepresented groups in the machine learning community.