Home

Iris recognition has been widely accepted as one of the most accurate, stable, and reliable biometric identification technologies. However, traditional iris recognition processes usually impose more constraints to the user cooperation and imaging conditions, which seriously limit the application range of iris recognition. Therefore, there has been much recent work on the non-cooperative or less-constrained iris recognition (either at-a-distance, on-the-move, with minor users' cooperation, within dynamic imaging environments and using mobile devices). Under these circumstances, captured iris regions contain different types of noisy factors, such as occlusions due to eyelids or eyelashes, specular reflections, off-angle, or blur. To make full use of these noisy iris images, efficient and robust iris segmentation have been regarded as the first most important challenge still open to the biometric community, affecting all downstream tasks from normalization to recognition.

In 2007, the Noisy Iris Challenge Evaluation - Part I (NICE.I) was held to benchmark the iris segmentation methods on the Noisy Visible Wavelength Iris Image Database (UBIRIS.v2). And in 2013, the Mobile Iris CHallenge Evaluation – Part I (MICHE I) was held to evaluate the iris segmentation methods developed for iris images captured under uncontrolled settings using mobile devices and built upon a new mobile iris dataset (MICHE-I). The two iris segmentation benchmarking competitions mainly focused on the evaluation of segmentation methods for VIS iris images from caucasian people according to the imaging illumination and ethnic distribution of datasets. Besides, most of submitted methods from NICE.I and MICHE I were developed based on traditional image processing and machine learning methods, rather than deep learning technologies emerged in recent years. Furthermore, in terms of evaluation metrics, only segmentation accuracy of noise-free masks was independently evaluated in the competition, but the localization accuracy of inner and outer boundaries of the iris still relied on indirect evaluation based on the iris recognition performance, which was non-intuitive, time-consuming, and complex (relying on the downstream iris encoding/matching and larger iris datasets).

To investigate these issues, reflect latest developments and attract more interests of researchers in the iris segmentation method, we plan to organize NIR-ISL 2021- a benchmarking challenge held in conjunction with IJCB 2021 focusing on the problem of iris segmentation and localization for NIR iris images from Asian and African people captured in non-cooperative environments. Here we specially split the general iris segmentation task in the conventional iris recognition pipeline into the segmentation of noise-free iris mask and the localization of inner and outer boundaries of the iris, which are narrowly referred to as iris segmentation and iris localization. The main reason for this is to consider that many recent deep learning based iris segmentation methods are only designed to segment the noise-free mask but ignore the localization of iris boundaries. Such an incomplete solution makes it hard to deploy in the conventional iris recognition pipeline. Therefore, the challenge encourages the submission of a complete solution taking the iris segmentation and iris localization into consideration. Of course, the submission of just involving the segmentation of iris mask is also allowed, but the ranking may not be very competitive.

The challenge is open to everyone. We invite research groups working on iris/ocular biometrics, segmentation and localization problems or other related vision tasks to take part in the challenge. A summary paper of NIR-ISL 2021 authored jointly by all participants (achieving competitive performance) will be send for consideration at IJCB 2021. Besides, we will provide some prizes and a certificate sponsored by Tianjin Academy for Intelligent Recognition Technologies(天津中科智能识别产业技术研究院) to the top ranking teams of the challenge as follows:

1st place： $500
2nd place: $300
3rd place: $200

All together 30 research groups registered for NIR-ISL 2021, out of which 14 took part in the final round and submitted a total of 27 valid models for scoring.

According to our ranking rules, each submitted entry was assigned one ranking score per evaluation metric and set of testing data. The final ranking was then obtained by adding all 4(evaluation metrics)x5(datasets)(=20) ranking scores (rank sum). The entry with the smallest sum was placed top in the final ranking.

The top-3 winning solutions of NIR-ISL 2021 are:

• 1st place: Lao Yang Sprint Team (Yiwen Zhang, Tianbao Liu, and Wei Yang, from School of Biomedical Engineering, Southern Medical University)

• 2nd place: SUEP-Pixsur (Dongliang Wu, Yingfeng Liu, Ruiye Zhou, and Huihai Wu, from Shanghai University of Electric Power)

• 3rd place: EyeCool (Hao Zhang, Junbao Wang, Jiayi Wang, and Wantong Xiong, from College of Science, Northeastern University)

Congratulations to them!

Following are the team details and results. More details can be found in the IJCB 2021 summary paper.

All datasets in the challenge can be found in the Github: https://github.com/xiamenwcy/NIR-ISL-2021 .

Complete competition results of NIR-ISL 2021

Summary of Registration information

Supplementary Material

Site opens: February 15, 2021
Registration starts: February 15, 2021
Training dataset available: February 15, 2021
Testing dataset made available: March 1, 2021
Registration closes: April 25, 2021
Prediction results and technical reports submission deadline: April 25, 2021
Results announcement: May 10, 2021

Participates need to register for the competition by filling up a Google Form or sending email. If you would like to register by email, please send to wangcaiyong@bucea.edu.cn with the subject line as “NIR-ISL 2021 registration”, with the following information: (1) team name; (2) names, affiliations of all members; and (3) name, affiliation, email, phone number, CV and mailing address of contact person. A reference number will be assigned to each team by our confirmation email.

The challenge aims to benchmark the iris segmentation and localization task on NIR iris datasets from Asian and African people captured in non-cooperative environments. In this challenge, three distinct iris datasets are used for evaluating different iris segmentation and localization approaches.

The first iris dataset i.e., CASIA-Iris-Asia collects the iris images from Asian people in complex and non-cooperative environments. More specifically, iris images from this dataset are selected from the existing CASIA-Iris-Distance and CASIA-Iris-Complex datasets, where CASIA-Iris-Distance is an Asian iris dataset captured with a CASIA long-range iris camera under NIR illumination, and CASIA-Iris-Complex (also renamed CASIA-Iris-Degradation) is another new collected Asian iris dataset captured with a modified Canon EOS 1300D camera in the complex environments. The CASIA-Iris-Complex contains various degraded iris images, especially occlusion due to eyelids/eyelashes/glasses and off-angle. Therefore, 500 iris images with serious occlusion noise are collected to form a CASIA-Iris-Complex-Occlusion subset, meanwhile 500 iris images with off angle noise are also collected to form a CASIA-Iris-Complex-Off-angle subset. Since the original images from CASIA-Iris-Complex contain the partial face, the eye region is manually extracted with resolution of 640×480 pixels for each selected image. The CASIA-Iris-Distance subset is from Ref [1] and consists of 400 iris images with resolution of 640×480 pixels from the first 40 subjects. For training, 100 iris images are randomly selected from CASIA-Iris-Complex-Occlusion and CASIA-Iris-Complex-Off-angle subsets, respectively. Besides, following Ref [1], the first 300 images from the first 30 subjects are used for training. Hence, 500 iris images from three subsets are mixed to train the iris segmentation and localization model while the rest is used for testing. Ground truth segmentation masks and inner and outer boundaries of the iris were manually labeled for all images in the CASIA-Iris-Asia except for those extremely hard images. In summary, for CASIA-Iris-Distance, the training set contains 296 iris images with ground truths and the testing set contains 99 iris images with ground truths. For CASIA-Iris-Complex-Occlusion and CASIA-Iris-Complex-Off-angle, the training set contains 100 iris images with ground truths and the testing set contains 400 iris images with ground truths, respectively. Some example images and corresponding ground truths are shown in the Fig.1.

The second iris dataset i.e., CASIA-Iris-M1 is a large NIR mobile iris dataset comprising 11,000 images from 630 Asian subjects, splitting into three subsets: CASIA-Iris-M1-S1 (1,400 images), CASIAIris-M1-S2 (6,000 images), and CASIA-Iris-M1-S3 (3,600 images). In this challenge, iris images and corresponding ground truths from Ref [2] are used for training and evaluating the model, where the image resolution is 400×400 pixels. In detail, the training set contains 1,500 images randomly selected from three subsets while the testing set contains another 1,500 disjoint images selected in a similar manner. Some example images and corresponding ground truths are shown in the Fig.2

The third iris dataset i.e., CASIA-Iris-Africa is captured in Nigeria and the first large-scale African iris dataset to the best of our knowledge. Iris images from this dataset have resolution of 1088×640 pixels and are collected under NIR illumination and in less-constrained environments using a hand held IKUSBE30 Public Security Special Iris Collection Device. Various noise factors such as occlusions, reflections, off-angle or illumination variations are introduced in this dataset, making the accurate and robust iris segmentation and localization hard. In this challenge, 370 iris images of different types of noise are selected for training and another disjoint 370 iris images are selected for testing. Ground truth segmentation masks and inner and outer boundaries of the iris were manually labeled for all images. Some example images and corresponding ground truths are shown in the Fig.3

[1] Liu N, Li H, Zhang M, et al. Accurate iris segmentation in non-cooperative environments using fully convolutional networks[C]// Proceedings of the International Conference on Biometrics (ICB). IEEE, 2016: 1-8.

[2] Wang C, Wang Y, Xu B, et al. A Lightweight Multi-Label Segmentation Network for Mobile Iris Biometrics[C]// Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020: 1006-1010.

Once registration is successful, the training dataset and corresponding ground truths will be made available to develop new iris segmentation and localization models. In the experiments, participates need to train the model on 3 different versions of training data:

1. Train with CASIA-Iris-Asia.

2. Train with CASIA-Iris-M1.

3. Train with CASIA-Iris-Africa.

As a result of this setup, participates should have 3 distinct models ready after the training stage. Participates are free to contribute more than a single submission, but each submitted solution should be trained in accordance with the above 3 points.

After training the models and obtaining the testing datasets, please run the trained models on the corresponding testing datasets:

1. For the model trained with CASIA-Iris-Asia, please run it on the testing set of CASIA-Iris-Distance, CASIA-Iris-Complex-Occlusion,
and CASIA-Iris-Complex-Off-angle, respectively.

2. For the model trained with CASIA-Iris-M1, please run it on the testing set of CASIA-Iris-M1.

3. For the model trained with CASIA-Iris-Africa, please run it on the testing set of CASIA-Iris-Africa.

As a result of this setup, participates should have 5 distinct versions of testing results for each submitted solution. The testing results need to contain binary iris masks and inner and outer boundaries of the iris corresponding to the test images. The submitted results should meet the following requirements:

1. All submitted masks and boundaries should be in image form (PNG format) and the same size as the original iris image. Please do not submit results in JPG format, as image artifacts will lead to worse results. The inner and outer boundaries of the iris should be single pixel wide curves.

2. Submit binary mask and boundaries, which should be strictly black-and-white (white for foreground, black for everything else). See below for sample submissions.

3. The submission should follow this tree structure exactly (see below for a sample structure):

a. The submission should be in a single ZIP file named after the assigned reference number (e.g., NIR-ISL2021020101.zip).

If participates are submitting several approaches (different model architectures, etc.), submit them in separate ZIP files,

which can be further named as reference number_X.zip, where X=1,2,.. the order of submitted approaches.

b. The ZIP should contain 3 directories with the names of the training data used in the experiments (“CASIA-Iris-Asia”,

“CASIA-Iris-M1”, “CASIA-Iris-Africa”) as well as a technical paper (pdf, 1~2 pages) describing the proposed approach.

The technical paper file should also be named after the assigned reference number (e.g., NIR-ISL2021020101.pdf or

NIR-ISL2021020101_X.pdf). If participates decide to release their executable programs or models, please directly put them

and related operating instruction manual in the ZIP folder or describe how to access and use them in the technical paper

if they are put somewhere else.

c. Inside each of these directories there should be several subdirectories with the names of the testing datasets (choosing from

“CASIA-Iris-Distance”, “CASIA-Iris-Complex-Occlusion”, “CASIA-Iris-Complex-Off-angle”, “CASIA-Iris-M1”, and

“CASIA-Iris-Africa”).

d. Inside each of these subdirectories there should be 3 more subfolders called “SegmentationClass”,“Inner_Boundary” and

“Outer_Boundary”, which should contain the binary (black-and-white) segmentation masks, inner iris boundary and outer

iris boundary, respectively.

e. Inside the “SegmentationClass”, “Inner_Boundary” and “Outer_Boundary” folders, all submitted masks or boundaries should

have the exact same name as the original image (aside from changing the extension to .png).

4. Send the ZIP file or media sharing link to wangcaiyong@bucea.edu.cn with the assigned reference number or variants (e.g.,

reference number_X) on the subject line.

The algorithms submitted by the participants will be evaluated by the organizers.

For iris segmentation, participants will need to submit the binary iris mask corresponding to the test images to the organizers. The evaluation measures will be the type-I (E¹) and type-II (E²) error rates (please refer to http://nice1.di.ubi.pt/evaluation.htm for more details), as used in the Noisy Iris Challenge Evaluation - Part I (NICE.I). The ground truth of the manually segmented iris region in an iris image will be used as the basis for scoring and ranking.

For iris localization, participants will need to submit the binary inner and outer boundaries of the iris corresponding to the test images to the organizers. The evaluation measures will be the dice index when filling the iris boundaries into iris boundary masks, and Hausdorff distance between the predicted iris boundaries and the corresponding ground truth boundaries (please refer to https://warwick.ac.uk/fac/sci/dcs/research/tia/glascontest/evaluation/ for more details). The ground truth of the manually localized inner and outer boundaries of the iris in an iris image will be used as the basis for scoring and ranking.

For the model complexity, the number of model parameters, the size of model memory (measured in MB), the amount of floating point operations (measured in FLOPs w.r.t the input of 640×480 pixels), and the average running speed (w.r.t the input of 640×480 pixels and GPU/CPU computing devices) are evaluated as additional optional measures of the submitted algorithms. It should be noted that for some traditional methods, only part of measures need to be reported such as average running speed, while for those based on deep learning, it is recommended to report all measures by using the corresponding open-sourced model analyzer w.r.t the different deep learning frameworks (See PyTorch,TensorFlow). Participants will need to submit the measure values of model complexity, programming language and deep learning framework if any.

Finally, each participating team will be assigned one ranking score per evaluation metric and set of testing data. The sum of these ranking scores will be used for the final ranking. The model complexity is not used for scoring, but just reflects the practicability of the submitted models.

Dr. Caiyong Wang, Beijing University of Civil Engineering and Architecture (BUCEA), Beijing, China (wangcaiyong@bucea.edu.cn)
Dr. Yunlong Wang, Institute of Automation, Chinese Academy of Sciences (CASIA), Beijing, China (yunlong.wang@cripac.ia.ac.cn)
Dr. Kunbo Zhang, Institute of Automation, Chinese Academy of Sciences (CASIA), Beijing, China (kunbo.zhang@cripac.ia.ac.cn )
Jawad Muhammad, PhD candidate, Institute of Automation, Chinese Academy of Sciences (CASIA), Beijing, China
Tianhao Lu, Master, Hunan University of Technology (HUT), Zhuzhou, China
Prof. Qichuan Tian, Beijing University of Civil Engineering and Architecture (BUCEA), Beijing, China
Prof. Zhaofeng He, Beijing University of Posts and Telecommunications (BUPT), Beijing, China (zhaofenghe@bupt.edu.cn)
Prof. Zhenan Sun, Institute of Automation, Chinese Academy of Sciences (CASIA), Beijing, China (znsun@nlpr.ia.ac.cn)