We have multiple metrics to measure systems's behaviour arranged in two main categories: performance and efficiency. 

Performance metrics 

Performance metrics are intended to measure how well systems achieve the proposed task in terms of prediction quality. We differentiate three types of performances: classification performance (binary classification), regression performance (regression errors and correlation), and latency-based measures (to promote systems that detect earlier).

• Binary classification: accuracy, micro, and macro precision, recall, f-score. 

• Regression metrics: root mean squared error (RMSE), Pearson correlation coefficient and P@k.

• Latency-based: Early risk detection metric ERDE or its variants.

The latency-based evaluation is only applied to binary classification subtasks (task1a, task2a and task3a) and multiclass classification tasks (task2c).  It shall be taken into account when a subject is evaluated as positive in the first instance. If in a later round the user is evaluated as negative, this evaluation will not be taken into account as it was previously sent as positive.

Each team will be provided with a token identifier when registering for the task to be used throughout this evaluation campaign. With this token, teams will be able to extract trial, training, and test datasets as well as send their predictions.

Teams can download trial and training data with the token, however, the test data will be provided by the server. So the team has to connect to our server using its token and the server will iteratively provide user writings by rounds. It means that each request will be followed by a server response with a new writing per user.  The server will only provide the next round of writings per user for the current round, so the team should keep a record of all the users and their writings in every round. After each request, the team has to give back to the server its prediction about each user.

NOTE: The server will be opened according to the dates indicated to simulate the evaluation phase with the trial data even if it can be downloaded. 

GET Test Data (and trial data)

For the first GET request, the server outputs the first message of each user. To send a GET request:

• Trial server: URL/{task}/getmessages_trial/{token}

• Test server:  URL/{task}/getmessages/{token}

The output format is the following:

Attributes:

• id_message: internal identifier of the writing

• round: the number of the round (from 1 to unknown)

• nick: alias's subject

• message: message's subject

• date:  the format of the date of the writing is 'YYYY-MM-DD HH:MM:SS'

The first round contains all users in the collection (because all users have at least one message). However, after a few rounds, some users will disappear from the server's response. For example, a user with 10 messages will only appear in the first 10 rounds. Furthermore, the server does not inform the teams that a given user writing is the last one in the user's thread. The last round will be detected by the teams when they receive an empty list from the server. 

After each request, the team has to run its own prediction pipeline and give back to the server its prediction about each individual. The server will provide always the next round of writings, regardless of having received the responses (all users and all runs) for the current round. 

POST predictions of Test Data (and trial data)

Each team has a limited number of three runs for each subtask they participate in. To submit the predictions, each team needs to send a POST request. A team can skip a run if they choose not to use all three runs by not sending a POST request and instead just making a GET request to proceed to the next round. To send a POST request:

• Trial server: URL/{subtask}/submit_trial/{token}/{run}

• Test server:  URL/{subtask}/submit/{token}/{run}

For each subtask, the prediction file to be sent has a different format. 

• For the subtasks associated with binary classification (task1a, task2a, and task3a), predictions will be 0 for “control” (negative) or 1 for “suffer” (positive). The structure would be as follows:

• For the subtasks associated with simple regression (task1b, task2b and task3b), predictions provide a probability for a user to suffer. A value of 0 means 100% negative and a value of 1 would be 100% positive. The structure would be as follows:

• For the subtask associated with multiclass classification (task2c), the predictions must be equal to one of the marked labels (“suffer+against”,  “suffer+in favour”, “suffer+other”, “control”). The structure would be as follows:

• For the subtasks associated with multi-output regression (task2d), the predictions provide the probability that a user belongs to one of the marked labels  (“suffer+against”,  “suffer+in favour”, “suffer+other”, “control”). In total they should add up to 1. The structure would be as follows:

To facilitate participation, we have prepared an example of a client application that communicates with the server. The notebook is here.

IMPORTANT NOTE 1: 

We have provided a repository where you can find the evaluation script and the rest of the scripts related to the competition.

IMPORTANT NOTE 2: 

For all tasks, the procedure is the same, however, as the same dataset is used for all subtasks of each task, each time a GET request is made, the round will advance for the task, whatever subtasks it is part of, as they all start from the same thread. 

Example: If a team participates in subtasks 1a and 1b, they are allowed to make only one GET request for each round, before submitting their predictions for all the subtasks they are participating in. This team, with 3 runs, will have to make a total of 1 GET request and 3 POST requests for each subtask.

• GET request: 

URL/task1/getmessages/{token}

• POST request:

URL/task1a/submit/{token}/{0}

URL/task1a/submit/{token}/{1}

URL/task1a/submit/{token}/{2}

URL/task1b/submit/{token}/{0}

URL/task1b/submit/{token}/{1}

URL/task1b/submit/{token}/{2}

If a team does not submit their predictions using the POST request before making a new GET request, their round will automatically advance and any previous predictions will be invalid. The team must submit predictions for each round using the POST request to make sure their runs are considered valid.