Participation in the BioNLP shared task is free and open to all, academia and industry, individuals and groups. This page provides instructions and general guidelines for participation; please see the main page for descriptions of the tasks and other information.
Sample data can be downloaded without registration, but registration is required for training and development data access, use of the development test server, and final submission.
Registration does not require commitment to participate, and while registrants are requested to provide names and other identifying information to the organizers, this information will not be published. The BioNLP shared task allows anonymous participation: final results will be initially published without identifying information, and participants wishing to remain anonymous may withdraw from the task at this point.
Participants wishing to train and test machine-learning based systems during this phase may find the data of the previous shared task, the BioNLP'09 shared task on event extraction, useful. Please note that while the general setup and data format of this former task are the same and the event types correspond to those of the GENIA main task, some details such as file naming conventions are different.
During this phase, an online submission system accepting system outputs for development test data and returning evaluation results will be available. This system is identical to the submission system for final results. We strongly encourage participants to test submitting their system outputs to the online development test set evaluation system to avoid surprises in final submission.
The submission system for final results will be opened at the same time as the test data is made available. To encourage early testing of submission to assure that the format of the final submission is correct, the system accepts any number of submissions. However, to prevent fine-tuning against the test set, the final submission system does not provide immediate feedback on results. Additionally, only the last submission from each participant is considered in evaluation; possible prior submissions are simply discarded.
Final results will be announced to all participants two days after the close of the submission system.
Please note that while the shared task test data will be made available for further testing through the web interface after the shared task, the full gold annotations for this data will not be released at this time to allow the test data to continue to serve as the basis for stable evaluation after the task in a way that minimizes risk of overfit or otherwise unrealistic results. Participants wishing to perform manual error analysis are encouraged to do so on the development test data, which should have statistically identical properties to the test data. A date for the release of gold annotations will be set later.
After completion of the evaluation, all participants are encouraged to write a manuscript describing their system, analysis, and results for submission to the BioNLP 2011 shared task workshop, about which the detail will be announced later. The manuscript submission deadline is April 2011. Detailed instructions for authors will be made available before the end of the evaluation phase.
The BioNLP 2011 shared task workshop will be held in summer 2011.
supporting task on coreference.
Participants are also encouraged to make use of available event extraction systems, and may submit results from their own previously introduced systems. However, it is not sufficient for participation to submit the output of a system developed by another group (even if retrained), and participants making use of event extraction systems introduced by others in their own system should carefully evaluate the contribution of their proposed extensions or modifications to the performance of the base system.
We ask the participants to observe the following minimal restrictions.