Participation in Evaluations of Technology


Leader of the participation in international evaluations of technology using developments of own research, with submissions usually ranked in the first two positions (mostly among ten or more participants). A research problem is usually defined, including an evaluation protocol. Training data is made available to the participants to tune their algorithms. Participants then submit their algorithms, which are tested by the organizers with sequestered data (i.e. not seen previously by the participants).


In several cases, the work is based on research done by supervised MSc or PhD students. The algorithms submitted are in most cases described in publications. Sometimes there is also a joint publication by organizers and participants describing the competition and results.



CROSS-EYED 2017: 2nd Cross-Spectrum Iris/Periocular Recognition

  • Organizer: University of Reading (UK), in conjunction with the International Joint Conference on Biometrics (IJCB-2017)

  • Objective: performance assessment of periocular recognition algorithms, with emphasis on matching of data from different spectra (near-infrared vs. visible)

  • Role: system development, integration and evaluation, applying own postdoctoral research work

  • Results: system ranked 1st/12 in the final submission

  • Web: https://sites.google.com/site/crossspectrumcompetition/

  • Joint publication by organizers and participants describing the competition:

A. F. Sequeira, L. Chen, J. Ferryman, P. Wild, F. Alonso-Fernandez, J. Bigun, T. Freitas Pereira, S. Marcel, V. Kanhangad, K. B. Raja, R. Raghavendra, C. Busch, “Cross-Eyed - Cross-Spectral Iris/Periocular Recognition Database and Competition”, Proc. IEEE/IAPR International Joint Conference on Biometrics, IJCB, Denver, CO, Oct 1-4, 2014



CROSS-EYED 2016: 1st Cross-Spectrum Iris/Periocular Recognition

  • Organizer: University of Reading (UK), in conjunction with the 8th Conference on Biometrics: Theory, Applications, and Systems (BTAS-2016)

  • Objective: performance assessment of periocular recognition algorithms, with emphasis on matching of data from different spectra (near-infrared vs. visible)

  • Role: system development, integration and evaluation, applying own postdoctoral research work

  • Results: system ranked 1st/5 in intermediate results, 1st/7 in the final submission

  • Web: https://sites.google.com/site/crossspectrumcompetition/cross-eyed-2016

  • Joint publication by organizers and participants describing the competition:

A. F. Sequeira, L. Chen, J. Ferryman, F. Alonso-Fernandez, J. Bigun, K. B. Raja, R. Raghavendra, C. Busch, P. Wild, “Cross-Eyed - Cross-Spectral Iris/Periocular Recognition Database and Competition”, Proc. 15th International Conference of the Biometrics Special Interest Group, BIOSIG, Darmstadt, Germany, 21-23 Sept 2016



ICIR2013: First ICB Competition on Iris Recognition

  • Organizer: Institute of Automation, Chinese Academy of Sciences (CASIA), in conjunction with the 6th International Conference on Biometrics (ICB-2013)

  • Objective: performance assessment of iris recognition algorithms, with data including motion blur, non-linear deformation, eyeglasses and specular reflections

  • Role: system development, integration and evaluation, applying own postdoctoral research work

  • Results: system ranked 2nd/13 (1st position among academic participants)

  • Web: http://iris.idealtest.org/2013/ICIR2013.jsp

  • Algorithms based on developments described in:

F. Alonso-Fernandez, J. Bigun, “Periocular Recognition Using Retinotopic Sampling and Gabor Decomposition”, Intl Workshop “What's in a Face?” WIAF, in conjunction with the European Conference on Computer Vision, ECCV, Springer LNCS-7584, pp. 309-318, Firenze, Italy, October 7-13, 2012

F. Alonso-Fernandez, J. Bigun, “Iris Boundaries Segmentation Using the Generalized Structure Tensor. A Study on the Effects on Image Degradation”, Intl Conf on Biometrics: Theory, Apps and Systems, BTAS, Washington DC, September 23-26, 2012

  • Joint publication by organizers and participants describing the competition:

M. Zhang, J. Liu, Z. Sun, W. Su, F. Alonso-Fernandez, V. Némesin, N. Othman, K. Noda, P. Li, E. Hoyle, A. Joshi, “The First ICB Competition on Iris Recognition”, Proc. IEEE/IAPR International Joint Conference on Biometrics, IJCB, Clearwater (Tampa), FL, 29 Sept – 2 Oct, 2014



4NSigComp2010: Forensic Signature Verification Competition (off-line)

  • Organizer: The Netherlands Forensic Institute (NFI), in conjunction with the 12th International Conference on Frontiers of Handwriting Recognition (ICFHR)

  • Objective: performance assessment of signature verification systems on a forensic-like data set of off-line images, including skilled forgeries and disguised signatures

  • Role: coordinator of the submission (incl. system development, integration and evaluation), applying own PhD research work, developed in conjunction with two supervised MSc students

  • Results: system ranked 2nd/10

  • Web: https://www.isical.ac.in/~icfhr2010/CallforParticipation4NSigComp2010.html

  • Algorithms based on developments described in (name of students underlined):

F. Alonso-Fernandez, J. Fierrez, A. Gilperez, J. Ortega-Garcia, “Impact of time variability in off-line writer identification and verification”, Proc. IEEE ISPA, Spec. Sess. on Signal and Image Processing for Biometrics, Salzburg, Austria, 2009-09-16

A. Gilperez, F. Alonso-Fernandez, S. Pecharroman, J. Fierrez, J. Ortega-Garcia, “Off-line signature verification using contour features”, Intl Conf on Frontiers in Handwriting Recognition, ICFHR, Montréal, Québec, August 19-21, 2008



SigComp 2009: Signature Competition, On- and Offline Skilled Forgeries

  • Organizer: The Netherlands Forensic Institute (NFI), in conjunction with the 10th International Conference on Document Analysis and Recognition (ICDAR)

  • Objective: performance assessment of signature verification systems on a forensic-like data set

  • Role: coordinator of the submission (incl. system development, integration and evaluation), applying own PhD research work, developed in conjunction with two supervised MSc students and one PhD Student (Marcos Martinez-Diaz)

  • Results: system ranked 2nd/15 (online signatures), 5th/8 (offline signatures)

  • Web: http://sigcomp09.arsforensica.org/

  • Algorithms based on developments described in (name of students underlined):

F. Alonso-Fernandez, J. Fierrez, M. Martinez-Diaz, J. Ortega-Garcia, “Fusion of static image and dynamic information for signature verification”, IEEE Proc. Intl. Conf. on Image Processing, ICIP, El Cairo, Egypt, 2009-11-07

F. Alonso-Fernandez, J. Fierrez, A. Gilperez, J. Ortega-Garcia, “Impact of time variability in off-line writer identification and verification”, Proc. IEEE ISPA, Spec. Sess. on Signal and Image Processing for Biometrics, Salzburg, Austria, 2009-09-16

A. Gilperez, F. Alonso-Fernandez, S. Pecharroman, J. Fierrez, J. Ortega-Garcia, “Off-line signature verification using contour features”, Intl Conf on Frontiers in Handwriting Recognition, ICFHR, Montréal, Québec, August 19-21, 2008

  • Competition described in the publication:

V. L. Blankers, C. E. van den Heuvel, K. Y. Franke, L. G. Vuurpijl. ”The ICDAR 2009 signature verification competition”. Proc. 10th Intl. Conf. on Document Analysis and Recognition, ICDAR, pp 1403–1407, 2009



LivDet 2009: Fingerprint Liveness Detection Competition

  • Organizer: University of Cagliari (Italy) and Clarkson University (USA), in conjunction with the 5th International Conference on Image Analysis and Processing (ICIAP)

  • Objective: to compare different methodologies for software-based fingerprint liveness detection

  • Role: system development, integration and evaluation applying own PhD research work, in conjunction with a PhD Student (Javier Galbally)

  • Results: system ranked 2nd/4

  • Web: http://prag.diee.unica.it/LivDet09/

  • Algorithm described in the publications (student name underlined):

J. Galbally, F. Alonso-Fernandez, J. Fierrez, J. Ortega-Garcia, “A High Performance Fingerprint Liveness Detection Method Based on Quality Related Features”, Elsevier Future Generation Computer Systems Journal, 28 (1): 311-321, January 2012, ISSN 0167-739X

J. Galbally, F. Alonso-Fernandez, J. Fierrez, J. Ortega-Garcia, “Fingerprint Liveness Detection Based on Quality Measures”, IEEE Proc. Intl. Conf. on Biometrics, Identity and Security, BIDS, Tampa, FL, Sept 22-23, 2009



BSEC 2009: The BioSecure Signature Evaluation Campaign

  • Organizer: TELECOM & Management SudParis, in conjunction with the International Conference on Biometrics 2009 (ICB'09)

  • Objective: to compare signature recognition systems in mobility acquisition conditions

  • Role: system development, integration and evaluation using own technology of the research group, in conjunction with a PhD Student (Marcos Martinez-Diaz)

  • Results: system ranked 2nd/1st of 14 (2 scenarios)

  • Web: http://biometrics.it-sudparis.eu/BSEC2009/

  • Algorithm described in the publications (student name underlined):

F. Alonso-Fernandez, J. Fierrez, M. Martinez-Diaz, J. Ortega-Garcia, “Fusion of static image and dynamic information for signature verification”, IEEE Proc. Intl. Conf. on Image Processing, ICIP, El Cairo, Egypt, 2009-11-07



BMEC 2007: The Biosecure Multimodal Evaluation Campaign

  • Organizer: BioSecure Network of Excellence

  • Objective: to compare fusion algorithms under several conditions (cross-device matching, mobility conditions)

  • Role: coordinator of the submission in the cross-device scenario (submission included as research work in own PhD Thesis), support of a co-worker (Daniel Ramos) in the mobility scenario (incl. system development, integration and evaluation)

  • Results: system ranked 2nd/1st of 14/9 (cross-device, mobility scenarios respectively)

  • Web: http://biometrics.it-sudparis.eu/BMEC2007/

  • Algorithms described in the publications:

F. Alonso-Fernandez, J. Fierrez, D. Ramos, J. Gonzalez-Rodriguez, “Quality-Based Conditional Processing in Multi-Biometrics: Application to Sensor Interoperability”, IEEE Trans. On Systems, Man & Cybernetics, part A, 40 (6): 1168-1179, November 2010, ISSN 1083-4427

F. Alonso-Fernandez, J. Fierrez, D. Ramos and J. Ortega-Garcia, "Dealing with sensor interoperability in multi-biometrics: the UPM experience at the BioSecure Multimodal Evaluation 2007", in Proc. Defense and Security Symposium, Biometric Technologies for Human Identification, BTHI, Proc. SPIE, Orlando, USA, March 2008

  • Joint publication by organizers and participants describing the competition:

N. Poh, T. Bourlai, J. Kittler, L. Allano, F. Alonso-Fernandez, O. Ambekar, J. Baker, B. Dorizzi, O. Fatukasi, J. Fierrez, H. Ganster, J. Ortega-Garcia, D. Maurer, A. A. Salah, T. Scheidat and C. Vielhauer, "Benchmarking Quality-dependent and Cost-sensitive Score-level Multimodal Biometric Fusion Algorithms", IEEE Trans. on Information Forensics and Security, 4 (4): 849-866, Dec. 2009, ISSN: 1556-6013