CompBench 2026
1st Workshop on Comparative Evaluation and Benchmarking

Motivation

Practical and theoretical research in Automated Reasoning (AR) has enabled ground-breaking success in a variety of formal methods applications over the past decades. Automated reasoning techniques allow us to overcome the complexity of systems, guarantee that computer systems behave as specified, and contribute to explainable AI. At the core of AR techniques are sophisticated and complex pieces of software (so-called solvers), which tackle specific problems in sub-areas of AR.

Comparative evaluation is a necessary technique to measure the state of the art of AR techniques and solvers in an objective way. This is also related with recurring solver competitions and large events such as the FLoC Olympic Games that have played a significant role in the practical success of AR and made significant impact on the research community and their empirical methods.

Empirical evaluations in AR typically face the same issues and questions, such as: 

• evaluation of scientific and theoretical value of advancements;

• collection, archival, compilation, selection, and distribution of benchmark instances and solvers;

• construction of meaningful evaluation measure;

• repeatability and easy replicability of results; and

• design of experimental evaluations with a long-term perspective, as stable and easy-to-establish setups that use computational infrastructure efficiently.

This workshop aims to provide a platform for exploring and harnessing the potential of interactions between different AR communities from various perspectives. We aim at bringing together a broad range of leading researchers who organize competitions, develop AR tools, work on evaluation algorithms and techniques to understand/diversify

benchmarks and visualize data, as well as researchers with influence employing cluster/cloud resources.

Our workshop will be orthogonal to the FLoC Olympic Games. There will be no presentation of results of the various competitions/challenges during the workshop, but we are open to talk proposals on evaluation-related aspects (related also to competitions) going beyond competition overviews and results.

Topics

We invite the submission of talk proposals, including new and emerging ideas, work-in-progress reports, and mature (possibly already published) results within the scope of the workshop, including but not restricted to 

Benchmarks: Collection, generation, selection, and distribution of benchmark instances and solvers,

Metrics: Construction of meaningful evaluation measures and metrics,

Replicability: Ensuring repeatability and easy replicability of results, and

Reports from the field: Showcasing experience reports from evaluations and competitions.

Submissions

We invite talk proposals in the form of short abstracts (1-2 pages + references as appropriate) submitted electronically in PDF using the LIPIcs latex style. Please submit to HotCRP. Submissions will be assessed for suitability by workshop organizers. Accepted talks will be provided a 20-30 min presentation slot during the workshop day. The abstracts of accepted talks  will be made available on the workshop webpage. 

Important dates

Submission deadline: May 15, 2026 

Notifications: May 31, 2026

Workshop: July 25, 2026

Organizers

Dirk Beyer LMU München, Germany

Johannes Fichte Linköping University, Sweden

Matti Järvisalo University of Helsinki, Finland

Anna V. Kononova Leiden University, Netherlands

Olaf Mersmann Hochschule des Bundes für öffentliche Verwaltung, Germany

Aina Niemetz Stanford University, USA
Guido Tack Monash University, Australia

Contact

The organizers can be reached by email via floccompbench@gmail.com .