Activities

Activities of the Benchmarking Network and Related Activities

2023

• Workshops

  • Good Benchmarking Practices for Evolutionary Computation (Benchmarking@GECCO; organized by Carola Doerr, Vanessa Volz, Pascal Kerschke, Olaf Mersmann, Boris Naujoks, and Mike Preuss; proposed and under revision)

2022

• Workshops

  • GECCO 2022 (July 2022, here)

    • Good Benchmarking Practices for Evolutionary Computation (Benchmarking@GECCO, organized by Carola Doerr, Tome Eftimov, Pascal Kerschke, Boris Naujoks, Vanessa Volz, Mike Preuss)

  • PPSN 2022 (September 2022, here)

    • Good Benchmarking Practices for Evolutionary Computation (Benchmarking@PPSN, organized by Benjamin Doerr, Carola Doerr, Pascal Kerschke, Boris Naujoks, Olaf Mersmann, Vanessa Volz, Mike Preuss)

• Invitation-Based Seminars / Workshops

  • Lorentz Center workshop on Benchmarked: Optimization Meets Machine Learning
    (by Carola Doerr, Mike Preuss, Marc Schoenauer, Thomas Stützle, and Joaquin Vanschoren)

2021

• Workshops

  • CEC 2021 (online conference, June 2021)

    • Good Benchmarking Practices for Evolutionary Computation (Benchmarking@CEC, organized by Carola Doerr, Tome Eftimov, Pascal Kerschke, Boris Naujoks, Mike Preuss)

  • GECCO 2021 (online conference, July 2021)

    • Good Benchmarking Practices for Evolutionary Computation (Benchmarking@GECCO, organized by Jürgen Branke, Carola Doerr, Tome Eftimov, Pascal Kerschke, Manuel López-Ibáñez, Boris Naujoks, Luís Paquete , Vanessa Volz, Thomas Weise )

    • Black-Box Optimization Benchmarking (BBOB workshop, organized by Anne Auger, Peter A. N. Bosman, Dimo Brockhoff, Tobias Glasmachers, Nikolaus Hansen, Petr Pošík and Tea Tušar)

• Special Sessions

  • CEC 2021 (online conference, June 2021)

    • Benchmarking of Computational Intelligence Algorithms (BOCIA'21)
      (by Aleš Zamuda, Thomas Weise and Markus Wagner)

    • Representation Learning Meets Meta-Heuristic Optimization (RepL4Opt)
      (by Tome Eftimov, Carola Doerr and Peter Korošec)

• Tutorials

  • CEC 2021 (online conference, June 2021)

    • Benchmarking and Experimentation: Pitfalls and Best Practices

(by Thomas Bartz-Beielstein, Boris Naujoks and Mike Preuss)

• • GECCO 2021 (online conference, July 2021)

    • Benchmarking: state-of-the-art and beyond (by Anne Auger and Nikolaus Hansen)

    • Benchmarking Multiobjective Optimizers 2.0 (by Dimo Brockhoff and Tea Tušar)

• Invitation-Based Seminars / Workshops

  • Dagstuhl Seminar on Challenges in Benchmarking Optimization Heuristics (unfortunately cancelled due to Covid 19)
    (by Anne Auger, Peter A. N. Bosman, Pascal Kerschke, and Darrell Whitley)

• Special Issues

  • IEEE Transactions on Evolutionary Computation

    • Benchmarking Sampling-Based Optimization Heuristics: Methodology and Software
      (Guest Editors: Thomas Bäck, Carola Doerr, Bernhard Sendhoff, and Thomas Stützle)

  • Evolutionary Computation Journal

    • Special Issue on Reproducibility
      (Guest Editors: Manuel López-Ibáñez, Luís Paquete, and Mike Preuss)

• Competitions

  • Open Optimization Competition 2021
    (by Thomas Bäck, Carola Doerr, Jeremy Rapin, and Olivier Teytaud)

2020

• Workshops

  • GECCO 2020 (Cancun, Mexico, July 2020; online event)

    • Good Benchmarking Practices for Evolutionary Computation

  • PPSN 2020 (Leiden, Netherlands, September 2020; hybrid on-site/online event)

    • Good Benchmarking Practices for Evolutionary Computation

    • (Multimodal) Multi-Objective Optimization: Challenges, Characteristics, and Peculiarities
      (by Christian Grimme, Pascal Kerschke, Heike Trautmann, Michael T. M. Emmerich and Hao Wang)

    • Understanding Machine Learning Optimization Problems
      (by Marcus Gallagher, Pascal Kerschke, Mike Preuss and Olivier Teytaud)

• Competitions

  • Game Benchmark Competition 2020
    (by Vanessa Volz, Boris Naujoks, Tea Tušar and Pascal Kerschke)

  • Open Optimization Competition 2020
    (by Thomas Bäck, Carola Doerr, Antoine Moreau, Jeremy Rapin, Baptiste Roziere, Ofer M. Shir and Olivier Teytaud)

• Tutorials

  • GECCO 2020 (Cancun, Mexico, July 2020; online event)

    • Benchmarking and Analyzing Iterative Optimization Heuristics with IOHprofiler
      (by Thomas Bäck, Carola Doerr, Ofer M. Shir and Hao Wang)

  • PPSN 2020 (Leiden, Netherlands, September 2020; hybrid on-site/online events)

    • Exploratory Landscape Analysis
      (by Pascal Kerschke and Mike Preuss)

    • Next Generation Statistics for Meta-heuristic Stochastic Optimization Algorithms
      (by Tome Eftimov and Peter Korošec)

• Summer Schools

  • 2020 Canberra Artificial Intelligence Summer School (CAISS)
    (by Markus Wagner, Saber Mohamed Elsayed and Hussein Abbass)

• Special Sessions

  • IEEE WCCI CEC 2020 (Glasgow, UK, 19 - 24th July, 2020; online event)

    • Benchmarking of Evolutionary Algorithms for Discrete Optimization (BEADO)
      (by Aleš Zamuda, Tome Eftimov and Stjepan Picek)

• Invitation-Based Seminars / Workshops

  • Lorentz Center workshop on Benchmarked: Optimization Meets Machine Learning (held as online event)
    (by Carola Doerr, Thomas Stützle, Mike Preuss, Marc Schoenauer and Joaquin Vanschoren)

Other Activities

• Surveys

  • Benchmarking in Optimization: Best Practice and Open Issues
    (main contact: Thomas Bartz-Beielstein)

  • Survey on Real-World Optimization Problems
    (main contact: MACODA working group)

• Networks / Working Groups

  • IEEE Task Force
    (main contacts: Markus Wagner, Mike Preuss and Aleš Zamuda)

  • COST Action 15140 ImAppNIO: Working Group 3 on Benchmarks
    (main contact: Pietro S. Oliveto)

• Research projects

• • Modern approaches for Benchmarking in Evolutionary Computation
    (PI: Tome Eftimov, Host institution: Jožef Stefan Institute)