Exaggerated numeral detection (ExNum) shared task aims to identify unreasonable numerals in market comments and news headlines. Although there are many discussions on misinformation and false information from the word aspect, only a few studies pay attention to the false information from the numeral. For example, the market may have different reactions to the quick news ``Apple's earnings increase 100%'' and ``Apple's earnings increase 10%''. However, the additional ``0'' in the former quick news may just be mistyped. As shown in Xie et al. [1], changing a word will lead to different outcomes in financial forecasting systems. We think that it is also the same in the case of changing a numeral. Given numerals are important in financial narratives, we want to invite more research groups to discuss it.
The idea of exaggerated numeral detection is proposed by Chen et al. [2] with the automatic-generated instances, and we extend this idea with manually-created instances in the proposed shared task. In ExNum, there are two subtasks based on the publicly available datasets, Numeracy-600K [2] and NQuAD [3]. Numeracy-600K includes 600K market comments in English, and NQuAD contains 70K Chinese news headlines and articles. We use these datasets as a training set for participants, and will manually create exaggerated instances to test their systems. The goals of the systems are (1) to identify whether the given market comments contain exaggerated numerals and (2) to identify whether the numerals in a given news headline are correct based on the news article. The former needs common sense toward numerals, and the latter is a kind of fact-check setting. We believe that we will get a better understanding of the properties of exaggerated numeral information with participants' discussions.
[1] Xie, Yong, et al. "A Word is Worth A Thousand Dollars: Adversarial Attack on Tweets Fools Stock Prediction." NAACL-2022.
[2] Chen, Chung-Chi, et al. "Numeracy-600K: Learning numeracy for detecting exaggerated information in market comments." ACL-2019.
[3] Chen, Chung-Chi, et al. "NQuAD: 70,000+ Questions for Machine Comprehension of the Numerals in Text." CIKM-2021.
The main challenge of these tasks is that we only have correct instances in the training set, but we need to identify the incorrect or exaggerated numerals when testing.
Subtask 1: Identify whether the given market comments contain exaggerated numerals (English)
The training set of this subtask is Numeracy-600K, and there are 600K market comments and 600K blog titles. Since market comments are published by professional information vendors, we think that all of them are correct. In the test set, we will manually create some market comments with exaggerated numerals, and the participants' systems are expected to identify which market comments contain exaggerated numerals.
Here are some instances:
Input: CHINA H1 GDP +3 PCT Y/Y
Output: May be Correct (0)
Input: CHINA H1 GDP +30 PCT Y/Y
Output: Hardly Happened/ May be exaggerated (1)
Subtask 2: Identify whether the numerals in a given news headline are correct based on the news article (Chinese)
The training set of this subtask is NQuAD, and there are about 70K news articles and titles. All articles and titles contain at least one numeral. In the training set, we also only have the correct instance. In the test set, we will manually create some incorrect instances with exaggerated (false) numerals, and the participants' systems are expected to identify whether the given title contains exaggerated numerals when given a news article.
Here are some instances:
Input Title: 《油價》經濟擔憂影響 NYMEX原油下跌1.5%
Input Article: 紐約商業交易所（NYMEX）1月原油期貨12月15日收盤下跌1.17美元或1.5%至每桶76.11美元，因對經濟擔憂的影響，歐洲ICE期貨交易所（ICE Futures Europe）近月布蘭特原油下跌1.49美元或1.8%至每桶81.21美元。
Output: Correct (0)
Input Title: 《日股》日經挫225點；國防股飆、海運股續揚
Input Article: 美國聯準會(Fed)如預期升息兩碼(50個基點)、不過持續維持「鷹派」姿態，引發美股昨日(14日)紅翻黑，也拖累日經225指數15日走跌，終場下跌0.37%或跌104.51點，收28,051.70點，結束連2個交易日走揚態勢。東證股價指數(TOPIX)終場跌0.18%(跌3.52點)，收1,973.90點，3個交易日來首度走跌。
Output: False/Exaggerated (1)
Obtain Training Set by Register: https://forms.gle/Q1s6BxJuqyjeshHB6
Test Data Release: Jan. 15, 2023
System Result Submission Deadline & Ground Truth Release: Feb. 8, 2023
Evaluation Result Release: Feb. 12, 2023
System Paper Submission Deadline: Feb. 16, 2023
Submission System: https://easychair.org/conferences/?conf=thewebconf2023iwpd
Notification: March 06, 2023
Camera-ready version ready: March 20, 2023
The proceedings of the workshop will be published jointly with the conference proceedings.