Sclera Segmentation Benchmarking Competition 2020 - SSBC 2020

---------------------------------------------------------------------------------------------------------------------------------------------------------

Sclera biometrics have gained significant popularity among emerging ocular traits in the last few years. In order to establish the sclera as a viable biometric trait and to evaluate its potential for automated biometric systems, several works have been presented in the literature, employing sclera both individually and in combination with the iris. Despite these initiatives, sclera biometrics need to be studied more extensively to ascertain their usefulness. Among the numerous challenges still open in this area, efficient and robust sclera segmentation is among the most important ones, affecting all downstream tasks from normalization to recognition. While a considerable amount of research has been directed towards sclera segmentation in recent years, the performance of existing techniques especially in cross-sensor scenarios and across different acquisition conditions is still not well explored. To investigate these issues, to document recent developments and to attract (and raise) the interest of researchers in this emerging biometric trait, we are organzing SSBC 2020 - a competition (and group benchmarking effort) held in conjunction with IJCB 2020 focusing on the problem of sclera segmentation.

The competition builds on sclera segmentation competitions we organized in the past, namely SSBC 2015, SSRBC 2016, SSBC 2016, SSERBC 2017, SSBC 2018 and SSBC 2019 held in conjunction with BTAS 2015, ICB 2016, BTAS 2016, IJCB 2017, ICB 2018 and ICB 2019 respectively. SSBC 2020 is the next in the series of sclera segmentation challenges and aims to benchmark sclera segmentation models in both cross sensor settings as well as across different acquisition conditions.

The competition is open to everyone. We invite research groups working on ocular biometrics, segmentation problems or other related vision tasks to take part in the competition. All participating researchers (achieving competitive performance) will be invited to co-author a summary paper of SSBC 2020 that will send for consideration at IJCB 2020.

Results:

All togethor 26 research groups registered for SSBC 2020, out of which 13 took part in the final round and submitted a total of 16 segmentation models for scoring.

The top-3 winning solutions of SSBC 2020 are:

• 1st place: UNet-P (Sumanta Das & Ishita De Ghosh)

• 2nd place: FCN8 (R. Lucio, L. A. Zanlorensi Jr., David Menotti)

• 3rd place: RGB-SS-Eye-MS ( Fadi Boutros, Naser Damer, J. H. Grebe and A. Kuijper

Follwing are the team details and resuts. Deatils report can be found at here .

Schedule:

Different Phases of the competition Dates

Site opens 31st May 2020

Registration starts 31st May 2020

Training dataset available 31st May 2020

Test dataset made available 20th, July 2020

Registration closes 10th Aug 2020

Algorithm submission deadline 10th Aug 2020

Results and report announcement 15th Aug 2020

Organizers :

Dr. Abhijit Das,Indian Statistical Institute, Kolkata, India (abhijit.das@inria.fr)

Prof. Umapada Pal, Indian Statistical Institute, Kolkata, India (umapada@isical.ac.in )

Prof. Peter Peer, University of Ljubljana, Ljubljana, Slovenija (peter.peer @fri.uni-lj.si)

Assoc. Prof. Vitomir Štruc , University of Ljubljana, Ljubljana, Slovenija (vitomir.struc @fe.uni-lj.si)

Matej Vitek , PhD candidate, University of Ljubljana, Ljubljana, Slovenija (matej.vitek @fri.uni-lj.si)

Benchmark datasets: The competition aims to benchmark the sclera segmentation task on a dataset containing ocular images captured in the visible spectrum using several mobile devices.

For training two different datasets will be made available to the participants, the first one was acquired with a DSLR camera while the second one was captured by a mobile phone. The first database i.e. MASD consists of 2624 RGB images taken from 82 identities. Images from this dataset were collected from both the eyes of each individual, so there are 164 different eyes in the dataset in total. For each individual, four gaze directions were captured (looking straight, left, right and up) and for each direction 4 images were taken. The subjects in the dataset comprise both male and female subjects, some subjects are wearing contact lenses and images were taken at different times of the day. The database contains images with blinking eyes, closed eyes and blurred eyes. High-resolution images are provided in the database (300 dpi resolution and 7500 x 5000 dimensions). All images are stored in JPEG format. A NIKON D 800 camera and 28300 lenses were used for image capturing. Ground truth segmentation masks of the sclera region were generated for all images in this dataset.

The second database i.e. MSD consists of 500 RGB images from both eyes of 25 individuals (in other words 50 different eyes). For each eye, 10 sample images were captured. The database contains blurred images and images with blinking eyes. The subjects in the dataset comprise both male and female subjects (12 males and 13 females), of different ages and different skin colours, 2 of them were wearing contact lenses and the images were taken at different times of the day. Variation in image quality (blur, lighting condition etc.) and different acquisition conditions was included intentionally in the database to investigate the performance of the framework in non-ideal scenarios. High-resolution images (3264 × 2448) of 96 dpi are included in the database. All the images are stored in JPEG format. The images were captured using a mobile camera with an 8-megapixel rear camera. For development purposes the both the dataset together with the ground truth masks will be made available to the participants at the start of the competition.

For testing purposes the MOBIUS dataset captured at the University of Ljubljana will be used. The segmentation part of the dataset contains around 3000 RGB ocular images (of 35 subjects) with subjects again looking at four different gaze-directions (straight, left, right and up). The images were captured by three different mobile phones and in three different acquisition conditions (good lighting inside, bad lighting inside, good lighting outside). The images correspond to male and female subjects of Caucasian origin. Images from the dataset vary considerably in terms of quality (blur, noise, etc.). All images were manually annotated with segmentation masks for the sclera, iris and pupil. Imperfections in the annotated masks were then corrected using a semi-automatic post-processing procedure. For the competition only the masks for the sclera region will need to be considered. The dataset will be sequestered and made available to the participants solely for the final performance evaluation. Sample images of one subject with corresponding ground truth masks will be made public at the start of the competition.

Description of the evaluation criteria (performance metrics): The algorithms submitted by the participants will be evaluated by the organizers. Participants will need to submit the binary segmentation masks and grayscale probabilistic intensity masks corresponding to the test images to the organizers. These will then be used to score, rank and analyze the participating segmentation models. The evaluation measures used for the competition will be precision, recall, F-measure, and IoU (the F-measure of the submitted binary masks will be considered the primary measure for ranking the submitted models). The manually annotated ground truth (i.e, the sclera region in the ocular images) will be used as the basis for scoring.

How to participate?

Registration for the competition can be done by email. If you would like to register and receive the training dataset, please send an email to abhijit.das@inria.fr with the subject line as "SSBC 2020 registration" with the following information:

Name, Affiliation, Email, Phone number, CV , Mailing Address and signed version of the following form .

How to obtain the test data and submit results?

The instructions for obtaining the testing dataset and the submission of results are available here. Please follow the instructions and submit the required forms before July 20 to obtain the testing data. The (strict) deadline for the submission of final results is August 10.