Germeval Task 2017
Shared Task on Aspect-based Sentiment in Social Media Customer Feedback
Introduction
In the connected, modern world, customer feedback is a valuable source for insights on the quality of products or services. This feedback allows other customers to benefit from the experiences of others and enables businesses to react on requests, complaints or recommendations. However, the more people use a product or service, the more feedback is generated, which results in the major challenge of analyzing huge amounts of feedback in an efficient, but still meaningful way.
Thus, we propose a shared task on automatically analyzing customer reviews about “Deutsche Bahn” - the german public train operator with about two billion passengers each year.
Example:
“RT @XXX: Da hört jemand in der Bahn so laut ‘700 Main Street’ durch seine Kopfhörer, dass ich mithören kann. :( :( :(“As shown in the example, insights from reviews can be derived on different granularities. The review contains a general evaluation of the travel (The customer disliked the travel). Furthermore, the review evaluates a dedicated aspect of the train travel (“laut” → customer did not like the noise level).
Consequently, we frame the task as aspect-based sentiment analysis with four sub tasks:
Tasks
Subtask A) Relevance Classification
Determine whether a social media post contains feedback about the "Deutsche Bahn" or if the post is off-topic/contains no evaluation.
Example:
Ehrlich die männer in Der Bahn haben keine manieren?<tab>relevant<tab>negative<tab>Atmosphäre#Haupt:negativeIn the given post, the task is to identify that the post is relevant
Subtask B) Document-level Polarity
Identify, whether the customer evaluates the "Deutsche Bahn" or travel as positive, negative or neutral.
Example:
Re: Ingo Lenßen Guten morgen Ingo...bei mir kein regen aber bahn fehr wieder nicht....liebe grusse ....<tab>relevant<tab>negative <tab>Sonstige_Unregelmässigkeiten#Haupt:negativ In the given post, the task is to identify the posts' polarity : negative
Subtask C) Aspect-level Polarity
Identify all aspects which are positively and negatively evaluated within the review. In order to increase comparability, the aspects are previously divided into categories (see data). Consequently, the aim of the subtasks is to identify all contained categories and their associated polarity.
Example:
"Alle so ""Yeah, Streik beendet"" Bahn so ""Okay, dafür werden dann natürlich die Tickets teurer"" Alle so ""Können wir wieder Streik haben?""" <tab>relevant<tab>neutral <tab>Ticketkauf#Haupt:negativ Allgemein#Haupt:positiveIn the given post, the task is to identify the aspects (and their polarity): Ticketkauf#Haupt:negativ Allgemein#Haupt:positive
Subtask D) Opinion Target Extraction
Identify the linguistic expression in the posts which are used to express the aspect-based sentiment (subtask C). The opinion target expression is defined by its starting and ending offsets.
Example:
<Document> <text>@m_wabersich IC 2151? Der fährt nicht. Ich habe Ihnen die Alternative bereits genannt. /je</text> <Opinions> <Opinion category="Sonstige_Unregelmässigkeiten#Haupt" from="26" to="37" polarity="negative" target="fährt nicht"/> </Opinions></Document>In the given post the task ist to identify the target expression fährt nicht.
The whole data is available in bot TSV and XML. Subtask D) however, can only be done using the XML format, as the spans of the opinion target expression are not available in TSV. For more detail see the data section.
Instructions for Technical Reports
Technical reports must be submitted in PDF format via the Easychair system (link t.b.a.). Submissions should use the ACL LaTeX style files or MS Word style files tailored for the GSCL 2017, with which GermEval 2017 will be co-located. The corresponding style files are available here: http://gscl2017.dfki.de/callPapers.php
The minimum length for the report is 2 pages (bibliography excluded). The maximum length is 4 pages (bibliography excluded), plus an unlimited number of references. Papers should describe the methods used clearly, and in sufficient detail in order to ensure reproducibility.
The system papers will be reviewed by other task participants in the same task. We expect that task participants are happy to review papers within the same task. Paper reviewing will be single-blind, though you can use the team name, name and/or affiliation.