AI competitions and benchmarks: the science behind the contests

For authors: Submission instructions

Foreword

In the rapidly evolving landscape of artificial intelligence (AI), the significance of competitions and benchmarks cannot be overstated. This book provides a comprehensive exploration of the role, design, and impact of AI challenges and benchmarks across academic, industrial, and educational domains. From historical perspectives and design principles to hands-on tutorials, the book offers an invaluable analysis of the organization and execution of AI competitions.

This book compiles insights from experienced challenge organizers, providing guidelines for the effective design of data-driven scientific competitions. The authors represent various institutions from academia, industry, and non-profit.

The book offers critical insights for researchers, engineers, and organizers to develop high impact competitions, through an exploration of dataset development, evaluation metrics, competition platforms, incentives, execution, and practical aspects. By addressing both theoretical and real-world considerations, this book serves as an essential guide for anyone looking to understand, participate in, or organize AI challenges and benchmarks.

Over the last 15 years, challenges in machine learning, data science, and artificial intelligence have proven to be effective and cost-efficient methods for rapidly bringing research solutions into industry. They have also emerged as a means to direct academic research, advance the state-of-the-art, and explore entirely new domains. Additionally, these challenges, with their engaging and playful nature, naturally attract students, making them an excellent educational resource. Finally, challenges act as a catalyst for community engagement by offering a structured and stimulating environment for individuals to collectively work towards a common goal.

This book addresses the gap in the literature on the theoretical foundations and optimization of challenge protocols, which has persisted despite the remarkable successes and progress achieved in challenge organization. It assembles leading experts in challenge organization to provide insights and directions for future research. It also provides a deeper understanding of challenge design, and introduces new methods and application domains for designing and implementing high-impact challenges that advance the frontiers of innovation.

The table of contents is given below, along with the abstract for each chapter.

Chapter abstracts

Chapter 1: Introduction - The life cycle of challenges and benchmarks [Preprint]

Data Science research is undergoing a revolution fueled by the transformative power of technology, the Internet and an ever-increasing computational capacity. The massive amounts of data that available to researchers and the ever more sophisticated algorithms developed to analyze it is unprecedented. Yet there seems to be a bottleneck in our current ability to synthesize complex data into knowledge or actionable information. Here we argue for the need to creatively leverage the scientific research community as an axis of innovation. Engaging researchers in the scientific discovery enterprise by Crowdsourcing will multiply opportunities to make discoveries from big data, thus accelerating the generation of useful and actionable information. Crowdsourcing the analysis of highly complex and massive data has emerged as one possible way to find robust methodologies that best match the data. When the crowdsourcing is done in the form of competitions—or Challenges—the validation of the analytical methodology is automatically addressed. Challenges also foster open innovation creating communities that collaborate directly or indirectly to solve many and important problems in health, climate change and sustainability, to name a few, currently faced by society.

PART I: FUNDAMENTALS

Chapter 2: Challenge design roadmap [Preprint]

This chapter provides a structured roadmap for researchers and engineers aiming to design and implement AI challenges. It begins with the fundamental questions and reflections that should be addressed to ensure a challenge's relevance and feasibility. Then, we detail the critical elements to write a well-articulated challenge proposition, emphasizing clarity, scientific rigor, and goal alignment. To further aid comprehension and practical application, the chapter provides a sample successful proposal, presenting a real-world example that exemplifies the key principles discussed. By following this roadmap, readers will be equipped with a clear, methodical approach to crafting meaningful and impactful AI challenges.

Chapter 3: Dataset development [Preprint]

Machine learning is now used in many applications thanks to its ability to predict, generate or discover patterns from large quantities of data. However, the process of collecting and transforming data for practical use is complex. Even in today's digital era, where substantial data is generated daily, it is uncommon for it to be readily usable; most often, it necessitates meticulous manual preparation. The haste in developing new models can frequently result in various shortcomings, potentially posing risks when deployed in real-world scenarios (e.g., social discrimination, critical failures), leading to the failure or substantial escalation of costs in AI-based projects. This chapter provides a comprehensive overview of established methodological tools, enriched by our practical experience, in the development of datasets for machine learning. Initially, we develop the tasks involved in dataset development and offer insights into their effective management (including requirements, design, implementation, evaluation, distribution, and maintenance). Then, we provide more details about the implementation process which includes data collection, transformation, and quality evaluation. Finally, we address practical considerations regarding dataset distribution and maintenance.

Chapter 4: How to judge a competition: Fairly judging a competition or assessing benchmark results [Preprint]

This chapter outlines the critical considerations in evaluating AI competitions or benchmark results. We begin with a discussion on selecting appropriate metrics, categorizing them into performance, societal impact, resources consumption, and agent-centric metrics, highlighting their specific applications. The subsequent section on statistical evaluation emphasizes the importance of error bars, test set sizes, and phase splitting to prevent overfitting. We then explore the problem of combining multiple scores, introducing ranking functions and shedding light on the inherent difficulty of ranking. The chapter concludes with a brief note on human analysis, underlining its role in ensuring nuanced and comprehensive evaluations. Readers will garner a structured understanding of the various facets involved in judging competitions and benchmarks effectively and fairly.

Chapter 5: Towards impactful challenges : post-challenge paper, benchmarks and other dissemination actions [Preprint]

The lasting impact of an AI challenge extends far beyond its conclusion. This chapter analyses the activities and efforts that can maximize the influence and utility of a challenge in the post-competition phase. Initially, we discuss the underlying purpose of post-challenge activities, emphasizing their role in furthering research, fostering community engagement, and ensuring transparency. The section on ``Challenge raw output'' provides insights into the essential components like scores, fact sheets, and code, which constitute the tangible outcomes of the competition. We then move to the significance of ``Post-challenge workshops and discussion'', highlighting their role in collaborative learning, feedback collection, and direction setting for future challenges. ``Post-challenge paper template'' presents a structured framework to document and disseminate the challenge's findings and methodologies effectively. The chapter concludes with an exploration of ``Post-challenge benchmark'', offering guidance on how to transform challenge results into lasting benchmarks that can guide and inspire subsequent AI research. Through this chapter, organizers will be equipped to navigate the post-challenge landscape effectively, ensuring their competition's insights and contributions resonate long after its completion.

PART II: REVIEWS

Chapter 6: Academic competitions [Preprint]

This chapter offers a concise overview of academic challenges, tracing their evolution and highlighting their significance in driving research. We initiate with a review of academic challenges from both past and present, outlining the progress made over the years while also weighing their advantages and limitations. Shifting focus to various fields, the chapter presents a cross-disciplinary exploration, showcasing how academic challenges have been implemented and evolved in areas such as machine learning, computer vision, natural language processing, biology, and autonomous driving. Readers will gain a succinct understanding of the landscape of academic challenges and their role in fostering innovation and collaboration across diverse scientific domains.

Chapter 7: Industry/hiring competitions and benchmarks

In this chapter, we shift to the realm of industry, examining the role and impact of AI competitions and benchmarks in a business context. We begin with a retrospective look at industry AI competitions, tracing their evolution over time, and considering both their advantages and limitations. The narrative then shifts to a detailed analysis of challenges specific to various AI technologies, including machine learning, computer vision, and natural language processing. In the latter section, the spotlight is on different business sectors, such as fin-tech and retail, highlighting how industry challenges have to address unique needs and objectives in these areas. The chapter offers readers a comprehensive view of how AI competitions intersect with business goals, driving innovation and potentially guiding hiring processes in the corporate world.

Chapter 8: Competitions and challenges in education

This chapter examines the role of competitions and challenges in educational settings. We begin by discussing the dynamics involved when students participate in competitions, followed by the complexities faced when they take on the mantle of organizing one. A subsequent section explores the evaluation of teaching effectiveness through challenges and the means to measure their pedagogical impact. Four case studies provide deeper insights: the first considers competitions tailored for K-12 students; the second shows the growing trend of hackathons; the third looks at high-stake competitions and their implications; and the last case study offers a perspective on the nuances of challenge organization within an academic setting. Through this chapter, educators, students, and organizers will gain a clearer understanding of how competitions can be effectively integrated and leveraged in educational contexts.

Chapter 9: Benchmarks

This chapter explores the essential topic of benchmarks within the realm of AI and machine learning. We commence with a historical perspective, tracing the origins and evolution of benchmarking practices over the years. The narrative then shifts to the specific applications of benchmarks in machine learning, discussing various modalities, fields, and techniques. A subsequent section addresses the pertinent issues encountered in benchmarking machine learning algorithms, emphasizing challenges in ensuring reproducibility in scientific experiments. The chapter concludes by considering the critical infrastructure required to facilitate robust and effective benchmarking. Through this exploration, readers will gain a comprehensive understanding of the importance, applications, and complexities of benchmarking in the AI domain.

PART III: PRACTICAL ISSUES AND OPEN PROBLEMS

Chapter 10: Competition platforms [Preprint]

This chapter provides an overview of the various platforms available for hosting AI competitions. It begins with a discussion of the most popular platforms, setting the stage for a deeper analysis. The chapter establishes a set of criteria for a structured comparison, making it easy for readers to understand the relative capabilities of each general-purpose AI competition platform at a glance. It also examines domain-specific platforms designed for specific niches, as well as alternative approaches and related services that can enhance the competition experience. Additionally, the chapter evaluates the option of self-hosted competitions and offers guidance for organizers in selecting the most appropriate platform based on their unique needs and objectives. With this comprehensive analysis, readers will be equipped to navigate the landscape of competition platforms and make well-informed decisions for running own challenges.

Chapter 11: Hand-on tutorial on how to create your own challenge or benchmark [Preprint]

This chapter provides a focused tutorial for individuals looking to design and establish their own AI challenge or benchmark. Initially, we cover the general aspects fundamental to any challenge creation, laying the foundation for a comprehensive understanding. Subsequently, we offer a step-by-step guide on using CodaLab Competitions, ensuring users can effectively leverage its features for challenge design. The chapter then transitions into a detailed tutorial for Codabench, elucidating its functionalities and best practices. Through this hands-on approach, readers will acquire the practical knowledge needed to initiate and execute their own challenges or benchmarks using prominent platforms.

Chapter 12: Special designs and competition protocols [Preprint]

This chapter discusses specialized designs and protocols tailored for distinct AI paradigms and circumstances. We first explore the nuances of competitions centered around reinforcement learning, highlighting the unique challenges and considerations inherent to this domain. Adversarial learning is then addressed, focusing on its dynamic nature and the protocols to ensure fair and meaningful competition. The chapter further introduces automated machine learning (AutoML) competitions, emphasizing their distinct structure and evaluation criteria. Another section is dedicated to competitions involving confidential data, providing guidance on safeguarding privacy while maintaining competitive integrity. The chapter concludes with insights into unsupervised learning competitions, detailing their design specifics and challenges. Through this exploration, organizers and participants will gain specialized knowledge pertinent to diverse AI competition scenarios.

Chapter 13: Practical issues: Proposals, grant money, sponsors, prizes, dissemination, publicity [Preprint]

This chapter provides a comprehensive overview of the pragmatic aspects involved in organizing AI competitions. We begin by discussing strategies to incentivize participation, touching upon effective communication techniques, aligning with trending topics in the field, structuring awards, potential recruitment opportunities, and more. We then shift to the essence of community engagement, and into organizational best practices and effective means of disseminating challenge outputs. Lastly, the chapter addresses the logistics, exposing on costs, required manpower, and resource allocation for effectively managing and executing a challenge. By examining these practical problems, readers will gain actionable insights to navigate the multifaceted landscape of AI competition organization, from inception to completion.

Editor Team (alphabetical order)

Isabelle Guyon, Université Paris-Saclay / INRIA / ChaLearn

Adrien Pavao, Université Paris-Saclay 

Evelyne Viegas, Microsoft

Contact: book@chalearn.org