As part of the workshop, we will also be hosting a shared task on Review Opinion Diversification. The shared task aims to identify opinions from online product reviews. By identification of opinions, we don’t just mean string matching with a predefined list. Instead, we reward two systems equally whether they recognize ‘this product is cost-effective’ as an opinion or, instead, ‘this product is inexpensive’ or ‘this product is worth the money.’ We have an annotated dataset of 80+ products, with more than 10,000 reviews in totality, each review being labelled with its constituent opinions in the form of one opinion matrix per product.
A supervised task to predict the helpfulness rating of product reviews based on review text. For a review which 3 users rated as helpful and 2 users found not-helpful, will be 3/5.
Subtask B judges a system on its ability to tell whether a given review R1 contains a given opinion O1 or not. While R1 can be easily identified by its Reviewer ID, opinions are not labeled with words. Instead, they are identified by the other reviews that they appear in. Therefore, we ask the participants to provide an opinion matrix as output, which we will evaluate using several verified metrics.
This subtask aims at producing, for each product, top-k reviews from a set of reviews such that the selected top-kreviews act as a summary of all the opinions expressed in the reviews set.