Embedded Files
Wednesday, July 8, 2020 at 16:10 Cancun time
Wednesday, July 8, 2020 at 16:10 Cancun time
5:10 pm New York = 22:10 UK = 23:10 France/Spain/Germany = 12:10 Moscow = 5:10 am July 9 Beijing = 6:10 am July 9 Tokyo
5:10 pm New York = 22:10 UK = 23:10 France/Spain/Germany = 12:10 Moscow = 5:10 am July 9 Beijing = 6:10 am July 9 Tokyo
A platform to come together and to discuss recent progress and challenges in the area of benchmarking optimization heuristics.
A platform to come together and to discuss recent progress and challenges in the area of benchmarking optimization heuristics.
Links
Links
to the presentations:
- Carola Doerr, Thomas Bartz-Beielstein, Boris Naujoks: Common Goals of Benchmarking Studies
- William La Cava: What are good benchmarking practices?
- Vanessa Volz: What now? Open issues in benchmarking
to the mentioned paper
Schedule
Schedule
1. Discussion: What is the purpose of benchmarking? (30 min approx)
- Organizer: Boris Naujoks, joint work with Thomas Bartz-Beielstein and Carola Doerr (slides)
- Session Chair: Tome Eftimov
2. Discussion: What are good benchmarking practices / how to do benchmarking / what not to do? (30 min approx)
- Organizer: William La Cava (slides)
- Session Chair: Pietro S. Oliveto
3. Contributed Presentation: “A Benchmark with Facile Adjustment of Difficulty for Many-Objective Genetic Programming and Its Reference Set” (20 min: 15 presentation + 5 discussion)
- Speaker: Makoto Ohki
- Session Chair: Thomas Weise
4. Discussion: Open questions/issues in benchmarking / What to do next? (30 min approx)
- Organizer: Vanessa Volz (slides)
- Session Chair: Carola Doerr
Scope and Objectives
Scope and Objectives
Benchmarking aims to illuminate the strengths and weaknesses of algorithms regarding different problem characteristics. To this end, several benchmarking suites have been designed which target different types of characteristics.
Gaining insight into the behavior of algorithms on a wide array of problems has benefits for different stakeholders. It helps engineers new to the field of optimization find an algorithm suitable for their problem. It also allows experts in optimization to develop new algorithms and improve existing ones.
Even though benchmarking is a highly-researched topic within the evolutionary computation community, there are still a number of open questions and challenges that should be explored:
(i) most commonly-used benchmarks are small and do not cover the space of meaningful problems,
(ii) benchmarking suites lack the complexity of real-world problems,
(iii) proper statistical analysis techniques that can easily be applied depending on the nature of the data are lacking or seldom used, and
(iv) user-friendly, openly accessible benchmarking techniques and software need to be developed and spread.
We wish to enable a culture of sharing to ensure direct access to resources as well as reproducibility. This helps to avoid common pitfalls in benchmarking such as overfitting to specific test cases. We aim to establish new standards for benchmarking in evolutionary computation research so we can objectively compare novel algorithms and fully demonstrate where they excel and where they can be improved.
Workshop Description
Workshop Description
As the goal of the workshop is to discuss, develop and improve benchmarking practices in evolutionary computation, we particularly welcome informal position statements addressing or identifying open challenges in benchmarking, as well as all other suggestions and contributions for a discussion. Possible contributions include, but are not limited to:
- lists of open questions/issues in benchmarking
- examples of good benchmarking
- descriptions of common pitfalls in benchmarking and how to avoid them.
For all other information about the workshop, please contact Thomas Weise at tweise@ustc.edu.cn with CC to p.oliveto@sheffield.ac.uk, lacava@upenn.edu, boris.naujoks@th-koeln.de, and vanessa@modl.ai.
We also welcome the submission of workshop papers to be published in the GECCO companion proceedings. The Workshop Call for Participation (CfP) can also be downloaded in PDF format here.
Our goal for the WORKshop is to collaboratively produce output that improves the state-of-the-art of benchmarking in evolutionary computation, not to organize yet another mini-conference!
Topics
Topics
The topics of interest for this workshop include, but are not limited to:
- the selection of meaningful (real-world) benchmark problems,
- performance measures for comparing algorithm behavior,
- novel statistical approaches for analyzing empirical data,
- landscape analysis,
- data mining approaches for understanding algorithm behavior,
- transfer learning from benchmark experiences to real-world problems, and
- benchmarking tools for executing experiments and analysis of experimental results.
Important dates
Important dates
(all dates are strict, i.e., no extensions possible!):
Submission opening: February 27, 2020Submission deadline: April 17, 2020Notification of acceptance: May 1, 2020Camera-Ready Material: May 8, 2020Author registration deadline: April 27, 2020 (extension to May 8 likely, but we are waiting for a formal confirmation)
Please also note that, by GECCO rules, each accepted paper needs to have at least one author registered by the author registration deadline. If an author is presenting more than one paper at the conference, she/he does not pay any additional registration fees.
Send your suggestions for presentations and/or discussions by e-mail to all five main contacts listed below. Please indicate the format of your suggested contribution (talk, discussion, breakout, brainstorming, etc.) and how much time you suggest for this activity.
Submission Instructions
Submission Instructions
All relevant instructions regarding paper submission are available at https://gecco-2020.sigevo.org/index.html/tiki-index.php?page=Workshops.
Organizers
Organizers
- Thomas Bäck (Leiden University, The Netherlands)
- Bilel Derbel (University of Lille, Lille, France)
- Carola Doerr (CNRS researcher at Sorbonne University, Paris, France)
- Tome Eftimov (Jožef Stefan Institute, Ljubljana, Slovenia)
- Pascal Kerschke (University of Münster, Germany)
- William La Cava (University of Pennsylvania, USA)
- Manuel López-Ibáñez (University of Manchester, UK)
- Katherine Malan (University of South Africa)
- Boris Naujoks (TH Cologne, Germany)
- Pietro S. Oliveto (University of Sheffield, UK)
- Patryk Orzechowski (University of Pennsylvania, USA)
- Mike Preuss (Leiden University, The Netherlands)
- Jérémy Rapin (Facebook AI Research, Paris, France)
- Ofer M. Shir (Tel-Hai College and Migal Institute, Israel)
- Olivier Teytaud (Facebook AI Research, Paris, France)
- Heike Trautmann (University of Münster, Germany)
- Ryan J. Urbanowicz (University of Pennsylvania, USA)
- Vanessa Volz (modl.ai, Copenhagen, Denmark)
- Markus Wagner (The University of Adelaide, Australia)
- Hao Wang (LIACS, Leiden University, The Netherlands)
- Thomas Weise (Institute of Applied Optimization, Hefei University, Hefei, China)
- Borys Wróbel (Adam Mickiewicz University, Poland)
- Aleš Zamuda (University of Maribor, Slovenia)
Hosting Event
Hosting Event
The Genetic and Evolutionary Computation Conference (GECCO 2020), which is held as an online event this year! takes place is Cancun, Mexico, from July 8 -12, 2020
Related Event
Related Event
A similar benchmarking best practices workshop will be held at PPSN 2020, which takes place from September 5-9, 2020, in Leiden, The Netherlands: https://sites.google.com/view/benchmarking-network/home/activities/PPSN20. Contributions to this workshop are welcome any time before the conference dates!
This workshop is organized as part of ImAppNIO Cost Action 15140.
Page updated
Google Sites
Report abuse