Full-day Tutorial at ESWC 2013, Montpellier, France

27 May 2013.
Montpellier, Building 31 of the Triolet Campus

Download the lecture SLIDES and the hands on session SLIDES


26 May 2013. Slides available.
24 May 2013. Schedule Updated.
26 Apr 2013. Schedule Updated.


In many application scenarios useful semantic content can hardly be created (fully) automatically, but motivating people to become an active part of this endeavor is still an art more than a science. In this tutorial we will introduce the most popular approaches to ‘human computation’, as a means to realize semantic content management architectures in which human and computational intelligence are seamlessly interwoven.

The term ‘human computation’, coined by Luis von Ahn in 2004, refers to the process by which specific steps of a technical task – traditionally tackled in Computer Science via an automatic algorithm – are outsourced to humans. The tutorial will focus on two prominent representatives of human computation, ‘games with a purpose’ and ‘microtask crowdsourcing’; we will explain the core notions underlying them, and present support technology and tools, as well as a series of technical and socio-economical challenges and open issues related to their application in given scenarios. The two human-computation instances share many commonalities in terms of types of tasks they are designed to address, and the methods they apply to deal with questions related to quality assurance, resource management, and workflow design. The tutorial will describe a selection of such state-of-the-art methods in crowdsourcing research, and their implementation in platforms such as Amazon’s Mechanical Turk and OntoGame. Furthermore the tutorial will analyze the motivation and incentive mechanisms that are used to crowdsourcing-driven projects, and introduce guidelines and best practices. The discussion will be framed by numerous examples and case studies in entity linking, entity resolution, semantic annotation and conceptual modeling.


 Elena Simperl.

 Gianluca Demartini.

 Maribel Acosta.







Human computation fundamentals (60 min)

Introduction of the core concepts and definitions behind human computation and crowdsourcing, including task design, interface and experience design, quality assurance, resource management, incentives and motivators. Examples of hybrid human-machine systems for semantic technologies, databases, and information retrieval.


Elena Simperl

Coffee break


Games with a purpose (30 min)

Types of games, main backend functionality (players matching, consensus finding, dealing with spam). Gamification features and their application to real-world scenarios.

Examples of games and lessons learned from their development and deployment.


Elena Simperl


Micro-task Management and Automation (60 min)


Main features and functionalities of crowdsourcing platforms, as well as extensions for complex tasks, automatic evaluation, work assignment, spam management. Incentives and motivation factors.

Design patterns: Majority agreement, Iterative crowdsourcing, Find-Fix-Verify

Hybrid Human-Machine systems.

11:30 – 12:30

Gianluca Demartini

Lunch break

12:30 – 14:00

Paid Crowdsourcing Quality Control (60 min)

·       Influence of task UI Design on Quality

·       Payment strategies

·       Qualification tests, Honey Pots, Master workers

·       Model Crowd Workers

Future Directions for Quality Control

14:00 – 15:00

Gianluca Demartini

Applications in semantic data management (30 min)

Examples of recent applications of crowdsourcing to semantic data management problems such as ontology mapping and entity linking.

15:00 – 15:30

Elena Simperl

Coffee break 


Amazon Mechanical Turk hands-on (60 min)

·       Requester toolkit, HIT design, qualifications, approval of tasks.

·       Use of the MTurk web interface for requesters to easily crowdsource basic micro-tasks.

·       Use of the MTurk Java SDK with examples of programmatic task creation, status review, and result recollection.

·       Applications to semantic applications and tools for ontology editing, Linked Data interlinking and query processing.


Maribel Acosta

Conclusion (30 min)

Summary of the tutorial and discussion with participants on open issues and applications of human computation to semantic technologies.

17:00 – 17:30