RecoGym Challenge100 -Preliminary Results

Thank you to everyone who participated in the RecoGym challenge. We are still working on establishing the final set of winners, but we are releasing an initial analysis which shows how close the competition is. The following plot contains all entries that we were able to obtain results for:

The above plot shows an initial 8 seed evaluation for all teams for the RecoGym challenge (colours represent the different seeds, the confidence intervals are 95%). The evaluation involved testing each agent training on 1000 users and testing on 10 000 users. We shortlisted the top 10 teams that exceeded click through rates of 1.5% for a more thorough evaluation (see the list below).

For the final evaluation we will use more seeds, this should tighten the confidence intervals and better separate the solutions. After this evaluation we will declare any entry within noise of the winning entry as a joint winner. This more extensive evaluation, which is already in progress, will be completed very shortly. Thank you for your patience!

Teams that are shortlisted are:

• Team CyberAgent_AdEcon
• AlpT96
• FabiCim
• HanShotFirst
• yasui
• ykaneko1992
• AdremDataLab
• ADReM_Tim
• Team MontiPuleri
• bx

Note: The Newbraw team's solution had very long training times - well in excess of the 5 hours allowed, so it was disqualified (we managed to run only 3 seeds due to hitting run time limits).