Important Dates (2017):
    Apr.:
    Jun.:
    Jul.:
    Aug.:
    Aug.:
    Sep.:
    Nov.:
Release of training datasets.
Release of validation datasets.
Submission of short papers, reporting proposed method & preliminary results.
Release of testing datasets (& performance evaluation).
Contacting top performing methods for preparing slides for oral presentation.
Challenge at MICCAI (Quebec City) - Announcing results.
Extended LNCS paper submission deadline.
SCOPE

The Multimodal Brain Tumor Segmentation (BraTS) challenge has always been focusing on the evaluation of state-of-the-art methods for the segmentation of brain tumors in magnetic resonance imaging (MRI) scans.

BraTS 2017 utilizes multi-institutional pre-operative MRI scans and focuses on the segmentation of intrinsically heterogeneous (in appearance, shape, and histology) brain tumors, namely gliomas. Furthemore, this year, in order to pinpoint the clinical relevance of this segmentation task, BraTS’17 also focuses on the prediction of patient overall survival, via integrative analyses of radiomic features and machine learning algorithms.

The datasets used in this year's challenge have been updated, since BRATS'16, with more routine clinically-acquired 3T multimodal MRI scans and all the ground truth labels have been manually-revised by expert board-certified neuroradiologists.

BraTS 2017 runs in conjunction with the MICCAI 2017 conference, on the 14th of September, as part of the Full-Day MICCAI 2017 BrainLes Workshop, while it builds upon its 5 previous successful instances: 
                              1. BraTS 2012 (Nice, France) - [proceedings]
                              2. BraTS 2013 (Nagoya, Japan) - [proceedings]
                              3. BraTS 2014 (Boston, USA) - [proceedings]
                              4. BraTS 2015 (Munich, Germany) - [proceedings]
                              5. BraTS 2016 (Athens, Greece) - [proceedings]
CLINICAL RELEVANCE
Gliomas are the most common primary brain malignancies, with different degrees of aggressiveness, variable prognosis and various heterogeneous histological sub-regions, i.e. peritumoral edema, necrotic core, enhancing and non-enhancing tumor core. This intrinsic heterogeneity of gliomas is also portrayed in their imaging phenotype (appearance and shape), as their sub-regions are described by varying intensity profiles disseminated across multimodal MRI scans, reflecting varying tumor biological properties. Due to this highly heterogeneous appearance and shape, segmentation of brain tumors in multimodal MRI scans is one of the most challenging tasks in medical image analysis.

There is a growing body of literature on computational algorithms addressing this important task. Unfortunately, open data sets for designing and testing these algorithms are not currently available, and private data sets differ so widely that it is hard to compare the different segmentation strategies that have been reported so far. Critical factors leading to these differences include, but not limited to, i) the imaging modalities employed, ii) the type of the tumor (GBM or LGG, primary or secondary tumors, solid or infiltratively growing), and iii) the state of disease (images may not only be acquired prior to treatment, but also post-operatively and therefore show radiotherapy effects and surgically-imposed cavities).

Towards this end, BraTS is making available a large dataset with accompanying delineations of the relevant tumor sub-regions.
 

Fig.1: Glioma sub-regions.

Shown are image patches with the tumor sub-regions that are annotated in the different modalities (top left) and the final labels for the whole dataset (right). The image patches show from left to right: the whole tumor (yellow) visible in T2-FLAIR (Fig.A), the tumor core (red) visible in T2 (Fig.B), the enhancing tumor structures (light blue)visible in T1Gd, surrounding the cystic/necrotic components of the core (green) (Fig. C). The segmentations are combined to generate the final labels of the tumor sub-regions (Fig.D): edema (yellow), non-enhancing solid core (red), necrotic/cystic core (green), enhancing core (blue).

(Figure from the BraTS IEEE TMI paper.)


TASKS

1. Segmentation of gliomas in pre-operative scans.
Description: The participants are called to address this task by using the provided clinically-acquired training data to develop their method and produce segmentation labels of the different glioma sub-regions. The sub-regions considered for evaluation are: 1) the "enhancing tumor" (ET), 2) the "tumor core" (TC), and 3) the "whole tumor" (WT) [see Fig.1]. The ET is described by areas that show hyper-intensity in T1Gd when compared to T1, but also when compared to “healthy” white matter in T1Gd. The TC describes the bulk of the tumor, which is what is typically resected. The TC entails the ET, as well as the necrotic (fluid-filled) and the non-enhancing (solid) parts of the tumor. The appearance of the necrotic and the non-enhancing tumor core is typically hypo-intense in T1-Gd when compared to T1. The WT describes the complete extent of the disease, as it entails the TC and the peritumoral edema (typically depicted by hyper-intense signal in FLAIR).

2. Prediction of patient overall survival (OS) from pre-operative scans.
Description: Once the participants produce their segmentation labels in the pre-operative scans, they will be called to use these labels in combination with the provided multimodal MRI data to extract imaging/radiomic features that they consider appropriate, and analyze them through machine learning algorithms, in an attempt to predict patient OS. The participants do not need to be limited to volumetric parameters, but can also consider intensity, morphologic, histogram-based, and textural features, as well as spatial information, and glioma diffusion properties extracted from glioma growth models.

The generalizability of the proposed methods will firstly be evaluated in an unseen validation dataset that will be provided to the participants during June 2017. Finally, an unseen test dataset will be provided (without accompanying ground truth labels) for a time-window of 48 hours in August 2017, after which the participants will have to send their results to the organizers for performance evaluation.


DATA
Ample multi-institutional routine clinically-acquired multimodal MRI scans of glioblastoma (GBM) and lower grade glioma (LGG), with pathologically confirmed diagnosis and available OS, will be provided as the training, validation and testing data for this year’s BraTS challenge. These multimodal scans describe a) native (T1) and b) post-contrast T1-weighted (T1Gd), c) T2-weighted (T2), and d) T2 Fluid Attenuated Inversion Recovery (FLAIR) volumes, and were acquired with different clinical protocols and various scanners from multiple (n=13) institutions, mentioned at the bottom of this page (Data Contributors). All the imaging datasets have been segmented manually, by one to four raters, following the same annotation protocol, and their annotations were approved by experienced neuro-radiologists. Annotations comprise the whole tumor, the tumor core (including cystic areas), and the Gd-enhancing tumor core, as described in the BraTS reference paper, published in IEEE Transactions for Medical Imaging (also see figure above). The provided data will be distributed after their pre-processing, i.e. co-registered to the same anatomical template, interpolated to the same resolution (1 mm^3) and skull-stripped.

The data provided during BraTS'17 differs significantly from the data provided during the previous BraTS challenges. The only data that have been previously used and will be utilized again (during BraTS'17) are the images and annotations of BraTS'12-'13, which have been manually annotated by clinical experts in the past. The data used during BraTS'14-'16 (from TCIA) have been discarded, as they described a mixture of pre- and post-operative scans and their ground truth labels have been annotated by the fusion of segmentation results from algorithms that ranked highly during BraTS'12 and '13. This year, expert neuroradiologists have radiologically assessed the complete original TCIA glioma collections (TCGA-GBM, n=262 and TCGA-LGG, n=199) and categorized each scan as pre- or post-operative. Subsequently, all the pre-operative TCIA scans (135 GBM and 108 LGG) were manually annotated by experts for the various glioma sub-regions and included in this year's BraTS datasets.

Participants are not allowed to use additional private data (from their own institutions) for data augmentation, since our intentions are to provide a fair comparison among the participating methods. The only case that this will be considered as a valid contribution is if they also report results using only the BraTS'17 data and discuss any potential difference in the results.

The overall survival (OS) data, defined in months, will be included in a comma-separated value (CSV) file with correspondences to the pseudo-identifiers of the imaging data.

All participants will be presented with the same test data, which will be made available through an on-line platform, such as SMIR, during 1-21 August and for a limited controlled time-window (48h), before the participants are required to upload their final results. The top-ranked participating teams will be invited before the end of August to prepare slides for a short oral presentation of their method during the BraTS challenge.


EVALUATION

In this year's challenge, two reference standards are used for the two tasks of the challenge: 1) manual segmentation labels of tumor sub-regions, and 2) clinical data of overall survival.


For the segmentation task, and for consistency with the
configuration of the previous BraTS challenges, we will use the "Dice score", and the "Hausdorff distance". Expanding upon this evaluation scheme, in BraTS'17 we will also use the metrics of "Sensitivity" and "Specificity", allowing to determine potential over- or under-segmentations of the tumor sub-regions by participating methods.

Since the BraTS'12-'13 are subsets of the BraTS'17 test data, we will also calculate performance on the '12-'13 data to allow a comparison against the performances reported in the BraTS reference paper.
 

Fig.2: Methods evaluation from previous BraTS benchmarks. Hausdorff scores for two tumor sub-regions. Black squares indicate the mean scores, which were used here to rank the methods. The Hausdorff distances are reported on a logarithmic scale. (Figure from the BraTS reference paper.)

For the task of predicting patient survival, we intend to follow two evaluation schemes, based on classification and regression principles. For the evaluation based on the classification principle, we will divide the provided data in three groups based on survival, i.e. long-survivors (>18 months), short-survivors (<6 months), and mid-survivors (between 6 and 18 months), and then use the Kaplan–Meier estimator (incl. Hazard ratio and p-value) to evaluate the predictions of the participating teams. For the evaluation based on the regression principle, we intend to use the Intraclass Correlation Coefficient to evaluate the predictions.

For both tasks, we will announce a 3-weeks evaluation period (1–21 August), during which the participants will be able to request different dates for the test data to be released to them. Note that each team should analyze the test data using their local computing infrastructure and submit their segmentation results 48-hours later.


PARTICIPATION SUMMARY & DATES

Training Data availability.
The co-registered, skull-stripped, and annotated training data set will be available via an online platform, such as SMIR (www.smir.ch), by the end of April.

Validation Data availability.
The independent set of validation scans will be made available to the participants during June.

Short Paper submission.
Participants will have to evaluate their methods on the training and validation datasets, and submit a short paper describing preliminary results, as well as their segmentation method, by the end of July (2-6 LNCS pages; will be reviewed lightly by the organizers). This unified scheme should allow for appropriate preliminary comparisons. Participants who wish to submit a significant longer version to the MICCAI 2017 BrainLes Workshop - BraTS'17 will be part of this workshop at MICCAI in Quebec - can submit this longer manuscript instead. Short papers will be part of the workshop proceedings distributed by the MICCAI organizers.

Test Data availability & Performance Evaluation.
The test scans will be made available to each participating team for a limited controlled time-window (48h) during 1-21 of August. The participants will analyze the images using their local computing infrastructure and will have to submit their segmentation results 48h later to the online platform (e.g. SMIR).

Oral Presentations.
The top-ranked participants will be contacted before the end of August to prepare slides for their presentations.

Announcement of Final Results.
Results of the challenge will be reported during the BraTS'17 challenge in Quebec city, which will run as part of a joint event with the MICCAI 2017 BrainLes Workshop and the MICCAI 2017 White Matter Hyperintensities challenge.

 Post-conference LNCS paper.
The top-ranked methods will be invited to submit papers to the LNCS proceedings of the BrainLes Workshop.
Joint post-conference journal paper.
In the weeks following the challenge, the participating teams will be invited to contribute to a joint manuscript, describing all participating methods and summarizing the results of the MICCAI BraTS 2017 challenge, that will be submitted to a high-impact journal in the field.

Feel free to send any communication related to the BraTS challenge in brats@miccai2017.org


Organizing committee

Spyridon Bakas, Ph.D., [main contact]
Center for Biomedical Image Computing and Analytics (CBICA), University of Pennsylvania, USA

Technical University of Munich (TUM), Germany

Christos Davatzikos, Ph.D.,
Center for Biomedical Image Computing and Analytics (CBICA), University of Pennsylvania, USA

University of Bern, Switzerland

Cancer Imaging Program (CIP), National Cancer Institute (NCI), National Institutes of Health (NIH), USA


Data Contributors

Cancer Imaging Program (CIP), National Cancer Institute (NCI), National Institutes of Health (NIH), USA

Center for Biomedical Image Computing and Analytics (CBICA), University of Pennsylvania, PA, USA

University of Alabama at Birmingham, AL, USA

Roland Wiest, M.D.
University of Bern, Switzerland
University of Debrecen, Hungary
 
Marc-Andre Weber, M.D.
Heidelberg University, Germany
Subpages (2): BraTS 2015 BraTS 2016