Data Upload

We will be using Globus for data submission. 

Each participating team should have an individual who used Globus to access the training data. We will be establishing restricted Globus collections - one for each team - to be used to submit the data.

By March 25,  each team's contact person should receive an email invitation to the team's Globus collection. You will then use the Globus tools to upload your data to that collection. We have prepared a brief guide to uploading files from Globus. 

If you have any difficulties with any aspect of the upload process, please email Harry Hochheiser.

Format

Participants are allowed to submit up to 3 system outputs on the test set for each subtask. Each submission must include in the file name the team name, the subtask, the number of the submission, and the cancer type (brca for breast cancer, ovca for ovarian cancer, mela for melanoma) in this format:  team_subtask_submissionNumber_cancerType, e.g. BCH_subtask1_submission1_brca, BCH_subtask1_submission1_ovca, BCH_subtask1_submission1_mela.

Please, write the system output for each submission per cancer type in a separate json file. Each file should contain a list of patients. Each patient contains a list of [chemo, relation, timex3]. For example:

patient_01:

['taxol', 'begins-on', '2013-06-17']

['taxol', 'ends-on', '2013-09']

...

patient_02:

...

System Description

Alongside the test set system output, please upload a short description of your system to address the relevant points below.

Main points:
1. Team name; total number of team members; country, name of organization

2. Which subtasks did you submit results

3. Brief description of the used approaches/methods

3.a. Which option best describes the core of your system? Non-Machine Learning, Machine Learning (non Deep Learning), Deep Learning (not transformer language models such as BERT), Deep Learning (with transformer language model such as BERT), Deep Learning (large language models), Hybrid method (combination of various approaches), Other

3.b. External data used other than provided by the task organizers

3.c. What language representations did you use?

3.d. If you used pre-trained language models, please, specify which ones.

3.e. Which features did you use in your system? e.g. unigrams, n-grams, word length, etc.

3.f. Which NLP components did your system exploit (at the core, preprocessing, or postprocessing)? 

3.g. Did you use the baseline system the organizers provided? If yes, describe how.

3.h. If you applied supervised machine learning systems, which ones

3.i. If you applied unsupervised machine learning, name your method.

3.j. If you used pattern matching or rule based approaches, describe. Same for  dictionary lookup/string matching techniques

3.k. Is your system a domain adaptation of a previous one?

3.l.Is your system general or specific for biomedical data?

3.m. Which programming languages did you use to implement your system/s

3.n. Which software did you use? e.g. tensorflow, huggingface, etc.

Additional points:

4. How long did you work on the shared task?

5. Please, grade the degree of difficulty of the shared task - very easy, easy, average degree of difficulty, difficult, very difficult

6. Would you be interested in participating in this task (or a similar one) again?

7. How did you hear about the task?