People’s opinions provide valuable insights for individuals and organizations. Typically, works on Portuguese primarily focus on Document Level Sentiment Analysis or Aspect-Based Sentiment Analysis (as done in the ABSAPT tasks on IberLEF 2022 and IberLEF 2024). However, these tasks only partially solve the Sentiment Analysis task, while the most complete subtask, known as Aspect Sentiment Quad Prediction (ASQP), has no works or datasets available in Portuguese.

We propose to create an Aspect Sentiment Quad Prediction dataset for TripAdvisor reviews written in Portuguese, following what was done in the ABSAPT tasks. Four sub-tasks will be available: Aspect Term Extraction, Aspect Category Detection, Opinion Term Extraction (as Target-Oriented Opinion Words Extraction), and finally Aspect Sentiment Quad Prediction.

The first task comprehends only the identification of the aspects present within a review. The second task comprises the identification of the category of each given aspect in a sentence, while the third task requires the extraction of all opinion terms for each aspect. The last task consists in the extraction of all four elements, given a sentence, resulting in multiple quadruples of (Category, Aspect, Sentiment, Polarity) for each available review.

The availability of resources written in Portuguese is scarce, specially annotated datasets, which limits the amount of research done for this language. 

This task will contribute to the progress of Portuguese NLP, as there is a demand for developing new methods and tools in the area.

Previous Aspect-Based Sentiment Analysis competitions, such as SemEval [1, 2, 3] consist of multiple subtasks that may be considered a complete ASQP task, when unified. For Portuguese, the biggest dataset is from the ABSAPT-2024 competition [4], that contains only the Aspect Term Extraction and Sentiment Orientation Extraction subtasks.

Our previous competitions, ABSAPT-2022 and ABSAPT-2024, engaged a total of nine teams from Brazil, Portugal and Vietnam. For the last edition, we had a training dataset with 1320 reviews and 4,828 annotations about more than 80 aspects of the accommodation sector, composed of a union of the datasets from the works [5] (reviews about accommodations in Porto Alegre, Brazil) and [6] (drawn from Rutilio's dataset). 

In this task, we expect to present new sentiment analysis tasks for the Portuguese language, with more than 1,000 unique annotated reviews. To the best of our knowledge, the only ASQP datasets available are from English or Chinese reviews, so this will be the first dataset available in a low-resource language.

We have 8,067 reviews about accommodations in Paris, Las Vegas, and New York City. They were extracted from TripAdvisor and kindly given to us by Professor Rutilio Rodolfo Lopes Barbosa [7].

The already annotated examples from ABSAPT-2024 will be used as a starting point, to which we will add the annotations of the aspects categories, the sentiment terms, and the polarities for the pairs of aspect-sentiment.

We aim to have more than 1,000 unique reviews, annotated by at least two annotators each, with a higher number of examples (as each review may contain multiple aspects, and each aspect may be related to multiple sentiment terms).

Participating teams will receive manually annotated training and test datasets. Submissions for the test dataset will be evaluated on several metrics, such as Accuracy, Precision, Recall, and F1.

The ATE and OTE subtasks will be evaluated on Precision, Recall, and F1, while the ACD will be evaluated on all four metrics, with F1 being considered the main metric for all of them.

The main sub task (ASQP) will be evaluated with Precision, Recall and F1, also using F1 as the main metric. For this task, the predicted quadruples will be considered correct only if all four components are predicted correctly, meaning that if a single component is predicted wrong, that prediction will be considered as an error.

The dataset split will be stratified based on both the categories and polarities, trying to keep the number of each category and each polarity as balanced as possible, while keeping each unique review in a single set (without any of its multiple annotations being used in other sets).

We aim to keep 60% of the annotations for training, and the other 40% split as test for the two groups of subtasks (20% for  used for both ATE and ASQP, and 20% for both OTE and ACD).

Each example will be a quadruple, containing the aspect term, sentiment term, category, and polarity. An example of an annotation is:

{

"category": "general", 

"aspect": "hotel", 

"sentiment": "sujo", 

"polarity": "NEG"

}

where “polarity” and “category” are classes, and the “aspect” and “sentiment” are terms found on the review. Each unique review may contain one or more annotated quadruples, and each term may also be present on multiple annotations.

The expected target is anyone interested in Sentiment Analysis, specially the newer and more complete variations of it. We hope for substantial engagement of academics, researchers, students, industrial teams, and practitioners of private companies.

1. M. Pontiki and D. Galanis and J. Pavlopoulos and H. Papageorgiou and I. Androutsopoulos and S. Manandhar. SemEval-2014 Task 4: Aspect Based Sentiment Analysis. In Proceedings of the 8th International Workshop on Semantic Evaluations (SemEval-2014), pages 27–35, Dublin, Ireland, 2014. Association for Computational Linguistics.

2. M. Pontiki and D. Galanis and H. Papageorgiou and S. Manandhar and I. Androutsopoulos. SemEval-2015 Task 12: Aspect Based Sentiment Analysis. In Proceedings of the 9th International Workshop on Semantic Evaluations (SemEval-2015), pages 486-495, Denver, Colorado, USA, 2015. Association for Computational Linguistics.

3. M. Pontiki and D. Galanis and H. Papageorgiou and I. Androutsopoulos and S. Manandhar and M. AL-Smadi and M. Al-Ayyoub and Y. Zhao and B.  Qin and O. De Clercq and V. Hoste and M. Apidianaki and X. Tannier and N. Loukachevitch  and E. Kotelnikov and N. Bel and S. M. Jiménez-Zafra and G. Eryigit. SemEval-2016 Task 5: Aspect Based Sentiment Analysis. In Proceedings of the 10th International Workshop on Semantic Evaluations (SemEval-2016), pages 19–30, San Diego, California, USA, 2016. Association for Computational Linguistics.

4. Thurow Bender, A., Gomes, G., Lopes, E., Araujo, R., de Freitas, L., and Corrêa, U.. Overview of ABSAPT at IberLEF 2024: Overview of the Task on Aspect-Based Sentiment Analysis in Portuguese. Procesamiento Del Lenguaje Natural, v. 73, p. 315-322, sep. 2024. 

5. L. A. de Freitas. Feature-level sentiment analysis applied to Brazilian Portuguese reviews. PhD thesis, Pontifícia Universidade Católica do Rio Grande do Sul (2015).

6. U. B. Corrêa. Análise de sentimento baseada em aspectos usando aprendizado profundo: uma proposta aplicada à língua portuguesa. PhD thesis, Universidade Federal de Pelotas (2021).

7. López Barbosa, R.R., Sánchez-Alonso, S. and Sicilia-Urban, M.A. (2015), "Evaluating hotels rating prediction based on sentiment analysis services", Aslib Journal of Information Management, Vol. 67 No. 4, pp. 392-407. https://doi.org/10.1108/AJIM-01-2015-0004