Introduction

In the information explosion era, choosing articles to read becomes a challenge. Thus, many recommendation systems are developed for this purpose. In financial opinion mining [1], investors are looking for analytical reports (opinions) that would lead to profitable outcomes. Therefore, we propose pilot tasks for evaluating the rationales [2] by using forecasting skills [3] as a proxy. In this ERAI shared task, we focus on amateur investors' opinions, and pay attention to two settings: (1) Pairwise Comparison and (2) Unsupervised Ranking. Our goal is to sort out the opinions that would lead to higher maximal potential profit (MPP) and lower maximal loss (ML).

Task Description

(1) Pairwise Comparison

In the pairwise setting, there are two given opinions with MPP and ML labels. Models are asked to determine (i) whether the given opinion 1 will lead to higher MPP than the given opinion 2 and (ii) whether the given opinion 1 will lead to higher ML (more loss)* than the given opinion 2. Thus, both would be binary classification tasks. We will use accuracy to evaluate the performances.

* Label "1": "ML1" < "ML2"; Label "0": "ML1" > "ML2"

(2) Unsupervised Ranking

In the unsupervised ranking setting, a pool of investors' opinions will be given, and the participants need to rank them with unsupervised methods. The goal is to find out the top 10% of posts that will lead to higher MPP. We will use the average MPP of the selected posts as the evaluation metrics. Our previous work [2] provides an example and the details of evaluation metrics (MPP and ML).

Results:

Registration

Our datasets are collected from PTT and Mobile01. The posts are written in Chinese, and we also provide the translated version in English. The datasets are available now.

Important Dates

• Registration Open: July 08, 2022

• Dataset Released: July 08, 2022

• System's Outputs Submission Deadline: Sep 12, 2022

• Release of Results: Sep 15, 2022

• Shared Task Paper Submissions Due: Oct 9, 2022.

  • Submission System: https://softconf.com/emnlp2022/FinNLP/

• Camera-Ready Version of Shared Task Paper Due: Oct 22, 2022

Accepted papers proceedings will be published at ACL Anthology.

1. The reviewing process will be single-blind.

2. Shared task participants will be asked to review other teams' papers during the review period.

3. Submissions must be in electronic form using the FinNLP-2022 paper submission software linked above.

4. At least one author of each accepted paper is required to attend the workshop to present the work. Authors will be required to agree to this requirement at the time of submission.

5. Time zone: Anywhere On Earth (AOE)

Reference:

1. Chen, Chung-Chi, Hen-Hsen Huang, and Hsin-Hsi Chen. From Opinion Mining to Financial Argument Mining. Springer Nature, 2021.

2. Chen, Chung-Chi, Hen-Hsen Huang, and Hsin-Hsi Chen. "Evaluating the Rationales of Amateur Investors." Proceedings of the Web Conference 2021. 2021.

3. Zong, Shi, Alan Ritter, and Eduard Hovy. "Measuring Forecasting Skill from Text." Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020.