Evaluation-as-a-Service: a serious game
Traditional challenges in data science distribute data sets and tasks and then receive the results of the participant algorithms. In the paradigm named "Evaluation-as-a-Service" (EaaS) no data are distributed but usually the algorithms are moved towards the data. Often, the data are made available in a cloud and can not be downloaded whereas the code can be installed in a virtual machine or Docker container. This setup has advantages because it can potentially scale to very large data sets, it can work on confidential data as the participants may only see the training data and quickly changing data can be evaluated. Optimization on test data can also be prevented. The availability of code and data together makes the results fully reproducible if the setup can be kept for example in a cloud. The availability of code, particularly if it is split into components can not only help research. It can also have applications in education as it reduces the entry barrier of working on very large data sets for students. The cloud limits the amount of infrastructure knowledge required and the availability of code makes it possible to only exchange small components or modify combinations of components. Influence of each component can be evaluated easily, which means that understanding the underlying solutions should become easier. Hopefully this will allow more collaboration and many links between research and education.
~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~~
Henning Müller studied medical informatics at the University of Heidelberg, Germany, then worked at Daimler-Benz research in Portland, OR, USA. From 1998-2002 he worked on his PhD degree at the University of Geneva, Switzerland with a research stay at Monash University, Melbourne, Australia in 2001. Since 2002 Henning has been working in medical informatics at the University Hospitals of Geneva where he habilitated in 2008 and was named titular professor in 2014. Since 2007 he has been a professor in business informatics at the HES-SO Valais in Sierre and since 2011 he has been responsible for the eHealth unit in Sierre. Henning was coordinator of the Khresmoi project, scientific coordinator of the VISCERAL project, initiator of theImageCLEF benchmark. He has authored over 400 scientific papers, is in the editorial board of several journals and reviews for many journals and funding agencies around the world. For 2015-2016 Henning is a visiting professor at the Martinos Center in Boston, MA, USA part of Harvard Medical School and the Massachusetts General Hospital (MGH) working on collaborative projects in medical imaging and system evaluation among others in the context of the Quantitative Imaging Network of the National Cancer Institutes