Data generation
Who generated the data?
Non-pathologists (NPs) generated the bulk of annotations in the single-rater datasets; approximately half of their data was corrected and approved by a pathologist. Multi-rater datasets contain annotations from NPs as well as pathologists.
Annotation classes
In our paper, we group annotations into classes and super-classes, based on clinical reasoning. We found that class grouping impacts interrater variability statistics and accuracy of model training.
The dataset you download from here contains raw classes, providing you with the flexibility to explore different class grouping schemes to fit your research question.
How was the data generated?
NPs were shown suggestions for nucleus segmentation boundaries and classifications. Low-power tissue region annotations from our previous study were combined with high-power suggestions of nucleus boundaries generated using image processing heuristics.
Point annotations were used to confirm correct suggestions where the heuristic provides correct segmentation. Nuclei that were inaccurately segmented were annotated using bounding boxes. The user interface used for annotation and the annotation protocol are described in detail here and in the supplement of our paper.
Extent of pathologist involvement
Pathologists were involved in the data generation & approval:
Correcting & approving region-level (low power) annotations used for classification suggestions
Answering questions & providing feedback
Final approval of annotations on a FOV-by-FOV basis
Pathologists also annotated a multi-rater evaluation set, which can be used to assess class-specific reliability of NPs.
Any manual segmentation data?
Our crowdsourcing approach is designed to avoid asking participants to manually trace nuclear boundaries. Nonetheless, to validate the segmentation accuracy of approved algorithmic suggestions, we asked one practicing pathologist to manually trace boundaries in a limited number of FOVs. All nuclear boundaries in FOVs in the Unbiased Control prefixed by "SP.3_#_U-control_#_" are manually traced.
