SVC-OnGoing

On-line signature verification Competition

Based on ICDAR SVC 2021 Competition

The SVC-onGoing competition is based on the ICDAR 2021 Competition on On-Line Signature Verification (SVC 2021). Due to the importance of the competition for the research community, we have decided to establish SVC 2021 as an on-going competition (SVC-onGoing), where researchers can easily benchmark their systems against the state of the art in an open web platform (CodaLab) using large-scale public databases such as DeepSignDB and SVC2021_EvalDB, and standard experimental protocols.

OVERVIEW

The goal of SVC-onGoing is to evaluate the limits of on-line signature verification systems using large-scale public databases and popular scenarios (office/mobile), and the stylus/finger as writing input. On-line signature verification technology is evolving fast in the last years due to several factors such as: i) the evolution in the acquisition technology going from the original Wacom devices specifically designed to acquire handwriting and signature in office-like scenarios through a pen stylus to the current touch screens of mobile scenarios in which signatures can be captured anywhere using our own personal smartphone through the finger, and ii) the extended usage of deep learning technology in many different areas, overcoming traditional handcrafted approaches and even human performance.

Therefore, in this competition the goal is to carry out a benchmark evaluation of the latest on-line signature verification technology using large-scale public databases and both traditional office-like scenarios (pen stylus), but also the challenging mobile scenarios with signatures performed using the finger over a touch screen. SVC-onGoing provides a complete panorama of the state of the art in the on-line signature verification field under realistic scenarios.

task

• Task 1: Analysis of office scenarios using the stylus as input.

• Task 2: Analysis of mobile scenarios using the finger as input.

• Task 3: Analysis of both office and mobile scenarios simultaneously.

In addition, SVC-onGoing simulates realistic operational conditions considering random and skilled forgeries simultaneously in each task.

REGISTRATION

The platform used in the SVC-onGoing competition is CodaLab. Participants need to register to take part in the competition. Please, follow the instructions:

1) Fill up this form including your information.

2) Sing up in CodaLab using the same email introduced in step 1).

3) Join in CodaLab to the SVC-onGoing competition. Just click in the “Participate” tab for the registration

EVALUATION CRITERIA

The evaluation metric considered will be the popular Equal Error Rate (%) similar to most on-line signature verification studies in the literature. We expect to receive scores close to 1 for impostor comparisons and close to 0 for genuine comparisons.

EXPERIMENTAL PROTOCOL

The SVC-onGoing competition is based on two different databases: DeepSignDB and SVC2021_EvalDB. Both databases contain a subset of evaluation that researchers can use to test their systems on CodaLab (DeepSignDB and SVC2021_EvalDB tabs).

Development (DeepSignDB database):

The goal of this stage is to provide the participants with the data needed to train the on-line signature verification systems. Only the DeepSignDB database is considered in this stage of the competition (participants can obtain it after sending us the corresponding license agreement, see details here). In order to simulate similar conditions considered in the final evaluation stage of SVC-onGoing (based on SVC2021_EvalDB), we divide the DeepSignDB database into training and evaluation datasets.

The training dataset is based on 1,084 subjects whereas the evaluation dataset comprises the remaining 442 subjects of the database. For the training of the systems (1,084 subjects), no instructions are given to the participants. They can use the data as they like. Nevertheless, for the evaluation of the systems (442 subjects), we provide the participants with the signature comparisons to run (you will receive an email with them after your registration in SVC-onGoing).

Participants can run their on-line signature verification systems using the signature comparisons files provided to obtain the scores and test the EER performance on the public web platform (CodaLab) created for the competition (DeepSignDB tab). This way participants can obtain a quantitative measure of the performance of the developed systems for the final evaluation stage of SVC-onGoing, using a different database (SVC2021_EvalDB).

Results are updated in CodaLab in real time and they are visible to everyone in a ranking dashboard.

Final Evaluation (SVC2021_EvalDB database):

The final evaluation of SVC-onGoing is carried out using only the novel SVC2021_EvalDB database. The database together with the corresponding signature comparisons files (one file per task) are sent to the participants after signing the corresponding license agreement (see details here). It is important to highlight that all signatures are included in a single folder, and both the nomenclature of the signatures and the signature comparisons files are randomized to avoid cheating. Ground-truth labels are not provided to the participants. In addition, and in order to consider a very challenging impostor scenario, the skilled forgery comparisons included in the corresponding files are optimised using machine learning methods, selecting only the best high-quality forgeries.

Results are updated in CodaLab in real time and they are visible to everyone in a ranking dashboard.

The organizers might need to verify the truthfulness of the scores submitted if necessary.

on-going results

As commented before, this is an on-going competition. Participants can test their developed systems through CodaLab using the signature comparisons files provided with the validation datasets of DeepSignDB and SVC2021_EvalDB. We provide below the on-going results of the competition.