Embedded Files
Getting Started
Getting Started
- First, you should sign up for the dataset, you should first visit our dataset page for the instruction.
- Then, you should register to be in the competition here. Once you are registered, the system will give you an automatically generated key. You will need this key to submit predictions.
- Please note that signing up for the dataset and the competition are two separate processes so that participants may chose to only look at the data and opt out of the competition (though participants are strongly encourage to participate).
Evaluation
Evaluation
The predictions will be evaluated using the following metrics:
Predictions will be ranked using the aggregated score:
Submitted predictions are evaluated daily at 11.59 PM EST. If you have submitted multiple sets of predictions, only your latest set of predictions of that day will be evaluated.
Prediction Submission
Prediction Submission
Please use this form to submit your predictions for evaluation and leaderboard. To submit your predictions for evaluation, please login here (under the top-right menu) using your email and the key you received during the registration.
Please follow the submission guideline below:
- The predictions must be a comma-separated list of decimals between 0 and 1, inclusive.
- 1 represents the "true" class (i.e. efficiently completed Block B) and 0 represents the "false" class
- For AdjustedAUC, the submitted decimals are used
- For AdjustedKappa, the submitted decimals are rounded to 0 and 1 using .5 as the threshold (.5 is rounded up)
- The predictions must be ordered in the same order as the list of student ids in hidden_label.csv
- Any decimal places after the 4th digit may be discarded.
Final Evaluation
Final Evaluation
In order to be included in the final evaluation:
- Participants must submit their codes using this form before December 27, 2019 11.59 PM EST.
- Only the last submission of each participant before the conclusion of the competition (December 15, 2019 at 11:59 p.m. EST) will be considered for the final evaluation.
- The final submissions will be evaluated using the final test set, which is different from the leaderboard set, as described here.
- Clarification: the prediction submission already include predictions for both the leaderboard set and the final test set. Only the leaderboard set are evaluated so far, and never the test set. The final evaluation will be done only on the final test set.
The five winners will be featured at an AIED/EDM2020 conference workshop. Additionally, the winners' findings will be highlighted in an invited special issue of the Journal of Educational Data Mining, anticipated for publication in 2021.
Frequently Asked Questions
Frequently Asked Questions
Q: Why are my predictions not evaluated?
A: We only evaluated submitted predictions once a day at 11.59 PM EST. If you have multiple submissions, only your latest submission of each day will be evaluated.
Q: Can I share the dataset with my friends/colleagues/peers?
A: To share the dataset, please share the dataset form (https://forms.gle/VWLcHDuJ8sEtkyB48) instead of sharing the link to the dataset directly.
Q: Can I participate as a team?
A: Yes.
Q: Why are there two separate signups, one for the dataset and one for the competition itself?
A: During our last competition, there were several participants who would like to look at the dataset but were hesitant to participate in the competition. Thus, we decided to separate the dataset request from the competition so that participants may look at the data without participating in the competition (though you're strongly encouraged to).
Q: I lost my key. How could I recover it?
A: To recover your key, contact us using the email you want to recover the key at assistments.data.mining.team [at] gmail [dot] com
Contacts
Contacts
If you have any questions or concerns, please do not hesitate to contact us:
assistments.data.mining.team [at] gmail [dot] com
Page updated
Google Sites
Report abuse