Embedded Files
While the LiRo benchmark has several available NLP tasks, in this hackathon we focus on 4 specific tasks. Each task targets a different NLP aspect, but has the same structure, resources, and rules and criteria for judging.
During the hackathon, you can focus on one or even all four of them.
LiRo Challenge Task 1: Named Entity Recognition
LiRo Challenge Task 1: Named Entity Recognition
In this task, given a sentence, you are supposed to label its words as belonging or not to a predefined set of classes. For example, in the sentence "George merge cu trenul Cluj - Timisoara de ora 6:20. ", you would detect that George is a person, that Cluj and Timisoara are geo-political entities and that 6:20 is a time.
We're working with the RONECv2 dataset, and you are expected to maximize the F1 score on the test split of the dataset.
To get a better intuition about NER, you are invited to play around with a base model here.
Check out the technical details for the NER task in this colab notebook!
LiRo Challenge Task 2: Tweet Emotion Detection
LiRo Challenge Task 2: Tweet Emotion Detection
The Tweet Emotion Detection task is simple: given a tweet as a text string, predict whether the tweet shows one or more of the seven predefined emotions of Fear, Anger, Surprise, Joy, etc.
You will be given the REDv2 dataset, and asked to make the best emotion predictions on the test set.
An in-depth description of the task (with code) will be provided to registered participants before the start of the hackathon.
Check out the technical details for the Emotion Detection task in this colab notebook!
LiRo Challenge Task 3: Semantic Text Similarity
LiRo Challenge Task 3: Semantic Text Similarity
The STS task gives you two sentences and asks to rank them on a 0-5 scale of how similar they seem.
You'll be working on the RO-STS dataset, and asked to obtain the maximum Pearson correlation on the test set.
Check out the technical details for the STS task in this colab notebook!
LiRo Challenge Task 4: Sentence Segmentation
LiRo Challenge Task 4: Sentence Segmentation
Think splitting a text in sentences is easy? Think again. While state-of-the-art models like NLP-Cube, Spacy, Stanza, Trankit, etc. do provide 95% segmentation accuracy, they miss some pretty obvious cases given they are all trained on the same Universal Dependencies - Romanian RRT dataset.
We're thinking that it's time that we reach 99% accuracy, and your task is to get there. You'll be given an unsegmented text and your task is to split the text into individual sentences.
You'll have access to the same Romanian RRT dataset like all the other tools, but you'll be evaluated on a hidden test set. You will be provided an additional small dataset to use as you see fit, from the same domain as the hidden test set.
Check out the technical details for the sentence segmentation task in this colab notebook!
Judging criteria and final notes
Judging criteria and final notes
For these tasks, you will implement the training and prediction of a model of your choice. Inputs and outputs are standardized, and precise metrics are given.
You will be judged based on the performance of the model, ingenuity of design, speed of operation (includes model size, not only run time), and, last-but-not-least, readability of your code.
All the code you submit for this hackathon will be made public on the LiRo GitHub repo, and the winners in each task will have their scores inserted in the LiRo benchmark as the new State-of-The-Art in Romania.
Page updated
Google Sites
Report abuse