2025 GAC 1

What Benchmarks are Useful for Cognitive Science?

Susanne Haridi, Helmholtz Munich

Marcel Binz, Helmholtz Munich

Rodrigo Carrasco-Davis, Gatsby Unit, UCL

Clementine Domine, Gatsby Unit, UCL

Erin Grant, Gatsby Unit & Sainsbury Wellcome Centre, UCL

Alireza Modirshanechi, Helmholtz Munich and MPI for Biological Cybernetics

Short description

Cognitive science aims to resolve theoretical disputes through empirical evidence, but as the field grows, it becomes harder to agree on what the key questions are and which data best address them. Benchmarks offer a way forward by providing shared datasets and evaluation criteria that help compare theories on common ground. They can also streamline research by reducing redundant data collection and encouraging cumulative progress. But adopting benchmarks as a standard tool would represent a significant shift in how the field operates. Accordingly, we ask which benchmarks are most useful in cognitive science and why. To address this, we will host structured debates around three key controversies: (1) What should benchmarks measure? (2) Should they aim to push theoretical boundaries or establish a shared empirical foundation? (3) How can benchmarks be designed to support the needs of a diverse research community? To ground the discussion, we’ve invited speakers with first-hand experience developing benchmarks across different subfields.

Please fill out this survey!

GAC kickoff workshop, August 13th at CCN

Schedule of Events

4:00 pm - 5:00 pm    Introduction into the debate (15 mins) by Susanne Haridi

[Room A0.01]

5:00 pm - 6:30 pm `   Main discussion
[Room C1.03]

5:00-5:30 pm    Debate 1: What should benchmarks measure?

With Greta Tuckute and Wilka Carvalho, moderated by Erin Grant 

5:30-6:00 pm    Debate 2: Should benchmarks push boundaries or establish a knowledge base?

    With Gemma Roig and Jacqueline Gottlieb, moderated by Clementine Domine 

6:00-6:30 pm    Panel Discussion: How can benchmarks be structured to benefit the community?

                             With Klaus Oberauer, Martin Schrimpf and Britta Westner, moderated by Alireza Modirshanechi

Each segment will have a 10 minute window for Q &A at the end.

Organizers

Speakers

Comment on this GAC proposal!

Instructions for CCN community members: 

1. Download the proposal PDF using the button above. 

2. Use the Disqus box below to leave comments and reviews, and discuss the submission. You'll need to create a Disqus account or log into Disqus using a google account or similar in order to leave a comment/review.