As part of the EYRE '18 workshop co-located with CIKM 2018, two shared tasks related to entity retrieval are organized, in order to assess strengths and weaknesses of existing systems, compare performance of techniques, and enhance communication among researchers and developers.

Task 1: Entity Summarization

An entity in a knowledge base is involved in a set of triple-structured facts that describe the entity. When there are many facts, entity summarization is the task of selecting a size-constrained subset of triples that best represent the entity.

For this shared task, we use the Entity Summarization BenchMark (ESBM). The benchmark has provided a Java tool for evaluating system-generated entity summaries based on a crowdsourced gold standard. Please note that the benchmark is based on two datasets: DBpedia and LinkedMDB. Participants can report their results (i.e., F-measure under k=5 and k=10) for either or both datasets, but are encouraged to report both.

We would be happy to add your results to the website of ESBM, assuming you report your results for both datasets.

Accepted Systems

• Dongjun Wei, Shiyuan Gao, Yaxin Liu, Zhibing Liu and Longtao Huang. MPSUM: Entity Summarization with Predicate-based Matching. [Source]
• Hermann Kroll, Denis Nagel and Wolf-Tilo Balke. BAFREC: Balancing Frequency and Rarity for Entity Characterization in Linked Open Data. [Source]
• Zhang Xiang, Yang Qingqing and Liu Binchu. L2RSUM: Using Ranking Algorithm to Summarize Entities. [Source]
• Eun-Kyung Kim and Key-Sun Choi. Entity Summarization Based on Formal Concept Analysis. [Source]

Task 2: Entity Search

Entity search is the task of returning a ranked list of entities in a knowledge base in response to a free text user query.

For this shared task, we use the DBpedia-Entity v2 benchmark. The benchmark has provided a crowdsourced gold standard for evaluating system-generated ranked entities. Please note that the benchmark categorizes queries into four groups: named entity queries, IR-style keyword queries, natural language questions, and queries that seek a particular list of entities. Participants can report their results (i.e., NDCG@10 and NDCG@100) for any subset of categories, but are encouraged to report all. NDCG should be calculated using trec_eval.

We would be happy to add your results to the website of DBpedia-Entity v2, assuming you follow the provided train/test split and report your results for all the four categories of queries.

Accepted Systems

• Bo Ma, Yating Yang, Tonghai Jiang, Xi Zhou and Lei Wang. Graph-based Reranking Approach for DBpedia Entity Search. [Source]
• Shahrzad Naseri, Sheikh Muhammad Sarwar and James Allan. Semantic Driven Fielded Entity Retrieval. [Source]
• Xinshi Lin, Kwun Ping Lai, Zihao Wang and Wai Lam. Entity Retrieval via Query Graph Inference. [Source]

Submission Guidelines

Each participating system should submit a short paper (up to 4 pages) describing the system and reporting the results. Participants are encouraged to publish the outputs and/or source codes of their systems, and include a link to those resources in their papers.

Manuscripts should be submitted electronically, as PDF formatted using the ACM camera-ready templates; please use sample-sigconf.tex as the template. Papers should be submitted through the EYRE '18 online submission system.

Papers will be reviewed very lightly by the organizers to ensure the presentation quality. At least one author of each accepted paper is expected to attend the workshop and present.

Important Dates

• Submission: 6 August 2018 (Anywhere on Earth)
• Notification: 13 August 2018 17 August 2018
• Camera-ready: 27 August 2018 3 September 2018

Organizers

• Gong Cheng, Nanjing University, China
• Kalpa Gunaratna, Samsung Research America, USA
• Jun Wang, University College London, UK