FinSBD-2 Shared Task
Introduction
Sentences
Sentences are basic units of the written language. Detecting the beginning and end of sentences, or sentence boundary detection (SBD), is the foundational first step in many Natural Language Processing (NLP) applications such as POS tagging; syntactic, semantic, and discourse parsing; information extraction; or machine translation.
Despite its important role in NLP, Sentence Boundary Detection has so far not received enough attention. Previous research in the area has been confined to only formal texts (news, European Parliament proceedings, etc.) where existing rule-based and machine learning approaches are extremely accurate so-long the data is perfectly clean. No sentence boundary detection research to date has addressed the problem in noisy texts extracted automatically from machine-readable files (generally PDF file format) such as financial documents.
One type of financial document is the prospectus. Financial prospectuses are official PDF documents in which investment funds precisely describe their characteristics and investment modalities. The most important step of extracting any information from these files is to parse them to get noisy unstructured text, clean the text, format the information (by adding several tags) and finally, transform it into semi-structured text, where sentence and list boundaries are well marked.
These prospectuses also contain many visual demarcations indicating a hierarchy of sections including bullets and numbering. There are many sentence fragments and titles, and not just complete sentences. The prospectuses more often than not contain punctuation errors. And in order to structure the dense information in a more easily read format, lists are often used.
Lists
This year, we have included the task of extracting lists due to their unique structure and common occurrence in financial documents.
A list can be similar to a sentence that enumerates several items of the same category. For example, the “Simple List” from Figure 1 can be easily read as one normal sentence. However, looking at Figure 2, the list cannot be read as one sentence; although it is one unit, because there are multiple sentences included and there is a visible hierarchy of information. It is therefore important to make the distinction between sentences and lists and, for these lists, to create a hierarchy that organizes the items. Mastering this distinction and item hierarchy can pave the way for more accurate information extraction.
Task Description
Last year we organized the first edition of FinSBD focusing on extracting well-segmented sentences from Financial prospectuses in PDF format by detecting their beginning and ending boundaries in two languages: English and French. In addition to an improved version of the previously proposed task, this year we are extending this task to include the detection of lists and list items, as well as their hierarchy.
FinSBD'2 is split into two sub-tasks:
Extracting sentence boundaries, including list and list item boundaries.
Organizing the lists items hierarchically.
For each given PDF, a JSON will be provided containing:
text extracted by us (key "text")
sentence boundaries (key "sentence")
list boundaries (key "list")
list item boundaries (key "item")
list item boundaries of level 1 (key "item1")
list item boundaries of level 2 (key "item2")
list item boundaries of level 3 (key "item3")
list item boundaries of level 4 (key "item4")
Item boundaries overlap with item boundaries of different levels. Each item level represents its depth within the list.
Boundaries are represented by indexes of starting and ending characters that the system has to predict.
We also included the PDF coordinates of each boundaries as metadata (which can be used for visualization on PDF if needed).
Example:
{
"text": "Ce document fournit des informations essentielles aux investisseurs ...",
"sentence": [ {"start": 17, "end": 53, "coordinates": ... }, ... ],
"list": [ {"start": 1080, "end": 1267, "coordinates": ... }, ... ],
"item": [ ... ],
"item1": [ ... ],
"item2": [ ... ],
"item3": [ ... ],
"item4": [ ... ]
}
Sub-task 1 consists in predicting boundaries of sentences, lists and list items.
Sub-task 2 consists in predicting boundaries of item1, item2, item3 and item4. We can also see sub-task 2 as refining item boundaries into 4 classes of boundaries (item = item1 + item2 + item3 + item4).
Last year, participants were only given indexes of tokens. This year, we are providing indexes of characters as well as coordinates of boundaries to allow different kind of character or word tokenization and/or possible usage of spatial and visual cues. Therefore, we hope to encourage novel approaches based on multimodality, especially since lists are often spatially structured to convey information visually.
Improved annotation guidelines will also be provided to explain how the new and richer dataset was created. Participants can choose to work on both languages, or submit systems for one language only. They can participate in one or both sub-tasks.
This task is open to everyone. The only exception are the co-chairs of the organizing team, who cannot submit a system, and who will serve as an authority to resolve any disputes concerning ethical issues or completeness of system descriptions.
Evaluation
For each sub-task, the evaluation metrics will be computed based on boundaries which are pairs of character indexes ("start" and "end").
The F-score will be the official metric and an evaluation script will be provided to all the teams.
Prize
A USD$1000 prize will be rewarded to the best-performing teams.
Registration
Register here at https://forms.gle/NixDGuVjrdFMjYhR9
Once registered, we will share the training data with you.
Starter kit with evaluation script can be found here https://github.com/finsbd/finsbd2
Important Dates
13 March 2020 - First announcement of the shared task and beginning of registration30 March 2020 - Release of training data and scoring script1 May 2020 - Test set made available18 May 2020 - Registration deadline18 May 2020 - Systems' outputs collected31 May 2020 - Shared task title and abstract due8 June 2020 - Shared task paper submissions due15 June 2020 - Camera-ready version of shared task paper due5-10 January 2021 - FinNLP 2020 Workshop (Kyoto, Japan or virtual)
Contact
For any questions on the shared task please contact us on fin.sbd.task@gmail.com
Shared Task Organizing committee - Fortia Financial Solutions
Abderrahim AIT-AZZI
Willy AU
Bianca CHONG
Dialekti VALSAMOU-STANISLAWSKI