Start Date: February 14 End Date: May 24 Competition URL: Hugging Face

Motivation

Developing a robust system for identifying species of snakes from photographs is an important goal in biodiversity but also for human health. With over half a million victims of death & disability from venomous snakebite annually, understanding the global distribution of the > 4,000 species of snakes and differentiating species from images (particularly images of low quality) will significantly improve epidemiology data and treatment outcomes. We have learned from previous editions that “machines” can accurately recognize (F1 = 86,5% and Top1 Accuracy = 94%) even in scenarios with long-tailed distributions and around 1,600 species. Thus, testing over real Medically Important Scenarios and specific countries (primarily tropical and subtropical) and integrating the medical importance of species is the next step that should provide a more reliable machine prediction.

The difficulty of snake species identification – from both a human and a machine perspective – lies in the high intra-class and low inter-class variance in appearance, which may depend on geographic location, colour morph, sex, or age. At the same time, many species are visually similar to other species (e.g. mimicry).

Our knowledge of which snake species occur in which countries is incomplete, and it is common that most or all images of a given snake species might originate from a small handful of countries or even a single country. Furthermore, many snake species resemble species found on other continents, with which they are entirely allopatric. Knowing the geographic origin of an unidentified snake can narrow down the possible correct identifications considerably. In no location on Earth do more than 126 of the approximately 4,000 snake species co-occur. Thus, regularization to all countries is a critical component of any snake identification method.

Task Description

Given the set of authentic snake species observations – multiple photographs of the same individual – and corresponding geographical locations, the goal of the task is to create a classification model that returns a ranked list of species. The classification model will have to fit limits for memory footprint (ONNX model with max size of 1GB) and prediction time limit (will be announced later) measured on the submission server. The model should have to consider and minimize the danger to human life and the waste of antivenom if a bite from the snake in the image were treated as coming from the top-ranked prediction.

Evaluation Process

This competition provides an evaluation ground for the development of methods suitable for not just snake species recognition. We want you to evaluate new bright ideas rather than finishing first on the leaderboard. Thus, we this year we will award an authorship / co-authorship on a Journal publication and payment for an OpenAccess fee.

The whole evaluation process will be divided into two parts (i) CSV-based evaluation on Hugging Face, and (ii) ONNX/Pytorch model evaluation over our private data.

The test set (private data) will consider different medically important scenarios and overlooked regions.

Metrics

First, we will calculate standard Acc and macro averaged F1. Besides, we will focus on venomous species confusion error, i.e., a number of samples with venomous species confused for harmless and divided by the number of venomous species in the test set.

To motivate research in recognition scenarios with uneven costs for different errors, such as mistaking a venomous snake for a harmless one, this year's challenge goes beyond the 0-1 loss common in classification. We make some assumptions to reduce the complexity of the evaluation. We consider that there exists a universal antivenom that is applicable to all venomous snake bites.