Evaluation

The Predictive Challenge 2014 has finished. Here are the results!

Flavio Figueiredo is with Universidade Federal de Minas Gerais, Brazil. Gitte Vanwinckelen is with the Department of Computer Science, KU Leuven, Belgium.

Data availability

The deadline for the competition have passed, but you still can get access to the dataset. You need to print, sign, and send to ecmlpkdd@chartbeat.com a scanned copy of this agreement in order to be able to get the training and testing data.

Evaluation

Evaluation of task 1 was made in terms of RMSE (Root Mean Squared Error) in the prediction of log(1+SUM_VISITS_48H); participants will be ranked by decreasing RMSE. Evaluation of tasks 2 and 3 wasmade also in terms of RMSE, and participants ranked respectively by decreasing RMSE on log(1+SUM_TWITTER_48H) and log(1+SUM_FACEBOOK_48H).

The code that computes the logged RMSE is available: rmse_code.txt

After the competition ends, participants will be able to see the distribution of errors per website in order to gain further insights on their systems.

Paper

Participants will be asked to submit a description of their systems on a paper, for details see workshop »»

Submitting your prediction (deadline passed)

You need to upload to http://hacks.chartbeat.com/ecml/ three files in comma-separated format, containing records of the form <PAGE_ID, SUM_VISITS_48H>, <PAGE_ID, SUM_TWITTER_48H> and <PAGE_ID, SUM_FACEBOOK_48H> for each of the 30,000 URLs in the secret evaluation data, example:

H000P000,101

H000P001,50

...

SUM_VISITS_48H is the prediction of the total visits received in the first 48 hours by the webpage (in this case, page H000P000 is predicted by you to acquire 101 visits in its firs 48 hours), SUM_TWITTER_48H is the same for Twitter, and SUM_FACEBOOK_48H the same for Facebook.

You can submit intermediary predictions. Intermediary predictions will be used to create a ranking that will allow you to see how their system compares with the one of the competitors. The last prediction submitted by each person/team will be the one taken into account for deciding on the winner.