We have multiple metrics to measure systems's behaviour arranged in two main categories: performance and efficiency. 

Performance metrics 

Performance metrics are intended to measure how well systems achieve the proposed task in terms of prediction quality. We differentiate twotypes of performances: classification performance (Binary and multiclass classification) and latency-based measures (to promote systems that detect earlier).

• Binary and multiclass classification: accuracy, micro, and macro precision, recall, f-score.

• Latency-based: Early risk detection metric ERDE or its variants. It shall be taken into account when a subject is evaluated as positive in the first instance. If in a later round the user is evaluated as negative, this evaluation will not be taken into account as it was previously sent as positive.

Each team will be provided with a token identifier when registering for the task to be used throughout this evaluation campaign. With this token, teams will be able to extract trial, training, and test datasets as well as send their predictions.

Teams can download trial and training data with the token, however, the test data will be provided by the server. So the team has to connect to our server using its token and the server will iteratively provide user writings by rounds. It means that each request will be followed by a server response with a new writing per user.  The server will only provide the next round of writings per user for the current round, so the team should keep a record of all the users and their writings in every round. After each request, the team has to give back to the server its prediction about each user.

NOTE: The server will be opened according to the dates indicated to simulate the evaluation phase with the trial data even if it can be downloaded. 

GET Test Data (and trial data)

For the first GET request, the server outputs the first message of each user. To send a GET request:

• Trial server: URL/{task}/getmessages_trial/{token}

• Test server: URL/{task}/getmessages/{token}

• Replace {task} with one of this strings: "task1", "task2", "task3".

The output format is the following:

Attributes:

• id_message: internal identifier of the writing

• round: the number of the round (from 1 to unknown)

• nick: alias's subject

• message: message's subject

• date:  the format of the date of the writing is 'YYYY-MM-DD HH:MM:SS'

The first round contains all users in the collection (because all users have at least one message). However, after a few rounds, some users will disappear from the server's response. For example, a user with 10 messages will only appear in the first 10 rounds. Furthermore, the server does not inform the teams that a given user writing is the last one in the user's thread. The last round will be detected by the teams when they receive an empty list from the server. 

After each request, the team has to run its own prediction pipeline and give back to the server its prediction about each individual. 

POST predictions of Test Data (and trial data)

Each team has a limited number of three runs for each subtask they participate in. To submit the predictions, each team needs to send a POST request. A team can skip a run if they choose not to use all three runs by not sending a POST request and instead just making a GET request to proceed to the next round. To send a POST request:

• Trial server: URL/{task}/submit_trial/{token}/{run}

• Test server:  URL/{task}/submit/{token}/{run}

For each subtask, the prediction file to be sent has a different format. 

• For the subtask associated with multiclass classification (task1), the predictions must be equal to one of the marked labels (“depression”,  “anxiety”, “none”). The structure would be as follows:

• For the subtasks associated with two-level multiclass (task2), the predictions must be equal to one of the marked labels (“depression”,  “anxiety”, “none”). Moreover, the contexts predictions will be "addiction", "emergency", "family", "work", "social" ,"other" or "none".  If a subject is predicted to be healthy ("none"), without anxiety or depression, it is not necessary to send the context and the context sent is ignored. In case more than one context appears, it will be concatenated with the # symbol. The structure would be as follows: 

• For the subtasks associated with binary classification (task3), predictions will be 0 for “control” (negative) or 1 for “suffer” (positive). The structure would be as follows:

To facilitate participation, we have prepared an example of a client application that communicates with the server. The notebook is here.

IMPORTANT NOTE 1: 

We have provided a repository where you can find the evaluation script for 2023 and the rest of the scripts related to the competition.

IMPORTANT NOTE 2 (NEW): 

This year the submision procedure is new. Each time a team makes a GET request to the server, it will be provided with the data related to the round it is in. For the data to be updated to the next round, teams have to make all POSTs request sending their predictions (or empty predictions if they do not want to make use of the 3 runs).

However, if a team is in task 1 and 2, as the same dataset is used, the team must send six POSTs requests to update the round. Example:

• GET request: URL/task1/getmessages/{token} # team get round 1 data

• POST request:

URL/task1/submit/{token}/{0}

URL/task1/submit/{token}/{1}

URL/task1/submit/{token}/{2}

• GET request: URL/task2/getmessages/{token} # team get the same round 1 data as before because it is in task 1 and 2 and it did not submit six predictions requests.

• POST request:

URL/task2/submit/{token}/{0}

URL/task2/submit/{token}/{1}

URL/task2/submit/{token}/{2}

• GET request: URL/task1/getmessages/{token} # team get round 2 data

The team must submit predictions for each round using the POST request to make sure their runs are considered valid.