DATA AND TOOLS
DATA AND TOOLS
Below you can find information to obtain Datasets (Train and Test) and a description of their structure. Moreover, you can also access information on the submission format of your final predictions of the emojis to be associated to a tweet over the Test set.
The Train Dataset includes 250,000 tweets in Italian to train emoji prediction systems. See the Train and Test Datasets and the Label set section of the Task Description to obtain more information on this dataset.
You can download the train dataset as a password-protected ZIP archive from the following link:
https://drive.google.com/file/d/1VSN9H3vZVTIr0qOXYPmiBp6FHsvovRlb/view?usp=sharing
The password of the ZIP archive will be provided to the participants to the ITAmoji Task by sending an email with subject "ITAmoji train dataset password" to itamojievalita@gmail.com
The Train Dataset is a single text file (UTF-8) with one tweet per line in the following JSON Object format:
{"tid":"TWEET_ID","uid":"USER_ID","created_at":"CREATION_DATE","text_no_emoji":"TWEET_TEXT_WITHOUT_EMOJI","label":"EMOJI_LABEL"}The TWEET_TEXT_WITHOUT_EMOJI should be exploited to predict the EMOJI_LABEL.
IMPORTANT: participants have to train their systems exclusively on the 250,000 tweets included in the Train Dataset. It is not allowed to gather additional tweets to extend the train dataset with more training examples.
The following table shows the frequency (number of tweets) for each emoji in the Train Dataset:
The Test Dataset includes 25,000 tweets in Italian that will be used to evaluate the emoji prediction systems developed by ITAmoji participants. See the Train and Test Datasets and the Label set section of the Task Description to obtain more information on this dataset.
The Test Dataset can be downloaded as a password-protected ZIP archive from the following link:
https://drive.google.com/file/d/1Oz78NjhBi1C92lqP1zLfTW7Dkgtmm5oA/view?usp=sharing
The password of the Test Dataset ZIP archive is the same one used for the Train Dataset ZIP archive (ITAmoji participants should already know such password and thus be able to access the Test Dataset). In any case, the password can be obtained by sending an email with subject "ITAmoji test dataset password" to itamojievalita@gmail.com
The Test Dataset is a single text file (UTF-8) with one tweet per line in the following JSON Object format:
{"tid":"ITAMOJI_TWEET_IDENTIFIER","uid":"USER_ID","created_at":"CREATION_DATE","text_no_emoji":"TWEET_TEXT_WITHOUT_EMOJI"}Participants can submit up to three system runs. Each submission should be sent by means of an email to itamojievalita@gmail.com by specifying as subject: "Team: TEAM_NAME Run: RUN_IDENTIFIER" and attaching the emoji prediction results file as an UTF-8 encoded text files in the format described below, in the section "Format of the emoji prediction result file".
The deadline for submitting your runs is September 9th 2018 AoE.
The emoji prediction results file should be an UTF-8 encoded text files where each line describes the emoji(s) predicted for a specific tweet in the Test Dataset.
Each line of the emoji prediction results file should have the following format:
{"tid":"ITAMOJI_TWEET_IDENTIFIER", "label_1":"EMOJI_LABEL_1", "label_2":"EMOJI_LABEL_2", "label_3":"EMOJI_LABEL_3", ... "label_25":"EMOJI_LABEL_25"}where:
"tid" (MANDATORY) is the ITAMOJI_TWEET_IDENTIFIER of the Test Dataset"label_1" (MANDATORY) is the label of the most likely emoji to be associated to the test set tweet (e.g. red_heart, face_with_tears_of_joy, etc.)"label_2" ... "label_25" (OPTIONAL) are the label of the other 24 emojis ordered from the most likely to the less likely to be associated to the test set tweet (e.g. red_heart, face_with_tears_of_joy, etc.)Please, verify that the emoji prediction results file includes one and only one row with emoji predictions for each one of the 25,000 tweets of the Test Dataset. If two lines provide emoji predictions for the same tweet (i.e. have the same "tid"), only the first one will be considered.
Only system runs that will provide the whole ordered set of predicted emoji labels ("label_1" ... "label_25") will be evaluated with respect to Accuracy@5/10/25 and Coverage Error, besides Macro F-score. Please, go to the System evaluation section of Data and tools to get more details about the evaluation of system runs.
On the 26th of September 2018, once evaluated all the runs of ITAmoji 2018, we released the test dataset (25,000 tweets in Italian used to evaluate the emoji prediction systems) with ground truth labels. This dataset can be downloaded as a password-protected ZIP archive from the following link:
https://drive.google.com/open?id=1kbPGdyI3fg6oSQIjx5-91t7ZXfbyxtFu
The password of the Test Dataset ZIP archive is the same one used for the Train Dataset ZIP archive (ITAmoji participants should already know such password and thus be able to access the Test Dataset). In any case, the password can be obtained by sending an email with subject "ITAmoji test dataset password" to itamojievalita@gmail.com
The Test Dataset is a single text file (UTF-8) with one tweet per line in JSON Object format: the ground truth label of each tweet is provided by means of the property "ground_truth_label" of the related JSON Object.
Participants are allowed to use additional data to train unsupervised systems (like word embeddings).
From the following URL https://github.com/fvancesco/acmmm2016 it is possible to download pretrained Twitter embeddings for Italian tweets (with dimension 100 and 300).