Using crowdsourcing for Image Analysis
Project "Crowdsourcing for error detection in cortical surface delineations"
Background
Crowdsourcing is a new approach to large-scale data annotation that is based on outsourcing cognitive tasks to anonymous workers from an online community. While it has already been successfully used in several medical imaging applications, such as the classification of neuroimaging literature, see http://brainspell.org/, reference data and training data generation as well as the assessments of surgical skills , it has - to our knowledge - not yet been applied in the context of neuroimage data annotation.
Purpose
With the recent trend towards big data analysis, neuroimaging datasets have grown substantially in the past years. While larger datasets potentially offer important insights for medical research, one major bottleneck is the requirement for resources of medical experts needed to validate automatic processing results. To address this issue, the goal of this paper was to assess whether anonymous non-experts from an online community can perform quality control of MR-based cortical surface delineations derived by an automatic algorithm.
Methods
So-called knowledge workers from an online crowdsourcing platform were asked to annotate errors in automatic cortical surface delineations on 100 central, coronal slices of MR images. They were instructed to annotate errors in the following way:
Results
On average, annotations for 100 images were obtained in less than an hour. When using expert annotations as reference, the crowd on average achieves a sensitivity of 82% and a precision of 42%. Merging multiple annotations per image significantly improves the sensitivity of the crowd (up to 95%), but leads to a decrease in precision (as low as 22%).
Conclusion
Our experiments show that the detection of errors in automatic cortical surface delineations generated by anonymous untrained workers is feasible. Future work will focus on increasing the sensitivity of our method further, such that the error detection tasks can be handled exclusively by the crowd and expert resources can be focused on error correction.
Publications: