CompBench 2026
1st Workshop on Comparative Evaluation and Benchmarking
Affiliated with FLoC 2026
July 25, 2026 Lisbon, Portugal
Affiliated with FLoC 2026
July 25, 2026 Lisbon, Portugal
Practical and theoretical research in Automated Reasoning (AR) has enabled ground-breaking success in a variety of formal methods applications over the past decades. Automated reasoning techniques allow us to overcome the complexity of systems, guarantee that computer systems behave as specified, and contribute to explainable AI. At the core of AR techniques are sophisticated and complex pieces of software (so-called solvers), which tackle specific problems in sub-areas of AR.
Comparative evaluation is a necessary technique to measure the state of the art of AR techniques and solvers in an objective way. This is also related with recurring solver competitions and large events such as the FLoC Olympic Games that have played a significant role in the practical success of AR and made significant impact on the research community and their empirical methods.
Empirical evaluations in AR typically face the same issues and questions, such as:
evaluation of scientific and theoretical value of advancements;
collection, archival, compilation, selection, and distribution of benchmark instances and solvers;
construction of meaningful evaluation measure;
repeatability and easy replicability of results; and
design of experimental evaluations with a long-term perspective, as stable and easy-to-establish setups that use computational infrastructure efficiently.
This workshop aims to provide a platform for exploring and harnessing the potential of interactions between different AR communities from various perspectives. We aim at bringing together a broad range of leading researchers who organize competitions, develop AR tools, work on evaluation algorithms and techniques to understand/diversify
benchmarks and visualize data, as well as researchers with influence employing cluster/cloud resources.
Our workshop will be orthogonal to the FLoC Olympic Games. There will be no presentation of results of the various competitions/challenges during the workshop, but we are open to talk proposals on evaluation-related aspects (related also to competitions) going beyond competition overviews and results.
We invite the submission of talk proposals, including new and emerging ideas, work-in-progress reports, and mature (possibly already published) results within the scope of the workshop, including but not restricted to
Benchmarks: Collection, generation, selection, and distribution of benchmark instances and solvers,
Metrics: Construction of meaningful evaluation measures and metrics,
Replicability: Ensuring repeatability and easy replicability of results, and
Reports from the field: Showcasing experience reports from evaluations and competitions.
We invite talk proposals in the form of short abstracts (1-2 pages + references as appropriate) submitted electronically in PDF using the LIPIcs latex style. Please submit to HotCRP. Submissions will be assessed for suitability by workshop organizers. Accepted talks will be provided a 20-30 min presentation slot during the workshop day. The abstracts of accepted talks will be made available on the workshop webpage.
Submission deadline: May 15, 2026
Notifications: May 31, 2026
Workshop: July 25, 2026
Dirk Beyer LMU München, Germany
Johannes Fichte Linköping University, Sweden
Matti Järvisalo University of Helsinki, Finland
Anna V. Kononova Leiden University, Netherlands
Olaf Mersmann Hochschule des Bundes für öffentliche Verwaltung, Germany
Aina Niemetz Stanford University, USA
Guido Tack Monash University, Australia
The organizers can be reached by email via floccompbench@gmail.com .