Work in progress to collect information for Isabelle Guyon's NeurIPS 2022 D&B track keynote [DOWNLOAD SLIDES].
This work was done in collaboration with:
Computer Vision: Sergio Escalera sergio.escalera.guerrero@gmail.com, Hui Zhang zhanghui@4paradigm.com, Chen-Lin Zhang zhangchenlin@4paradigm.com, Wei-Wei Tu tuweiwei@4paradigm.com, Julio Jacques Jr. juliojj@gmail.com
Character Recognition: Birhanu Belay birhanu-hailu.belay@universite-paris-saclay.fr, Haozhe Sun haozhe.sun@universite-paris-saclay.fr , Hui Zhang zhanghui@4paradigm.com
Language: Melvin Johnson melvinp@google.com, Dipanjan Das dipanjand@google.com, Suneet Dhingra suneet@google.com, Romain Egelé romainegele@gmail.com
Reinforcement Learning : Aleksandra Faust sandrafaust@google.com, Olivier Pietquin pietquin@google.com, Shiyu Huang huangshiyu@4paradigm.com, Wei-Wei Tu tuweiwei@4paradigm.com
Medical data: Magali Richard magali.richard@univ-grenoble-alpes.fr, Gustavo Stolovitzky gustavo.stolo@gmail.com, Romain Egelé romainegele@gmail.com
Tabular data: Isabelle Guyon guyon@google.com, Romain Egelé romainegele@gmail.com, Frank Hutter fh@cs.uni-freiburg.de
Time Series: Zhen Xu zhxu.cs@gmail.com
NAS: Colin White colin@abacus.ai, Frank Hutter fh@cs.uni-freiburg.de
Meta-datasets: Eleni Triantafillou etriantafillou@google.com, Hugo LaRochelle hugolarochelle@google.com, Vincent Dumoulin vdumoulin@google.com
NeurIPS D&B track papers: Joaquin Vanschoren joaquin.vanschoren@gmail.com
Benchmark platforms: Joaquin Vanschoren joaquin.vanschoren@gmail.com
Multi-Modal: Chen-Lin Zhang zhangchenlin@4paradigm.com, Wei-Wei Tu tuweiwei@4paradigm.com
AI4Science: Hao Zhou zhouhao@4paradigm.com, Wei-Wei Tu tuweiwei@4paradigm.com
Graph: Huan Zhao zhaohuan@4paradigm.com, Wei-Wei Tu tuweiwei@4paradigm.com
If you want to contribute, please contact:
Isabelle Guyon guyon@google.com
Do not try to be exhaustive, select "editor's picks".
Indicate for all dataset or benchmark:
Link to resource (make sure this link includes download instructions, documentation, license).
Reference and link to paper describing the dataset/benchmark and baseline results.
Date created.
Comments:
why did you chose it?
what is its purpose?
what is its historical or practical importance?
what are its key difficulties?
what are its potential flaws?
Quantitative numbers:
For datasets:
Volume = Size in GB (indicate whether compressed or not and compression ratio, if known)
Number of examples.
"Intrinsic dimension": Dimension of state-of-the-art embedding (that is a learned representation available for transfer learning) or number of features (if tabular data with no known embedding).
Number of classes or labels
For RL benchmarks
State space "Dimension": Number of variables of the environment or dimension of state-of-the-art embedding (that is a learned representation)
Action space dimension: Number of available actions.
Isabelle Guyon Keynote @NeurIPS'22
NeurIPS has been in existence for more than 3 decades, each one marked by a dominant trend. The pioneering years saw the burgeoning of back-prop nets, the coming-of-age years blossomed with convex optimization, regularization, Bayesian methods, boosting, kernel methods, to name a few, and the junior years have been dominated by deep nets and big data. And now, recent analyses conclude that using ever bigger data and deeper networks is not a sustainable way of progressing. Meanwhile, other indicators show that Machine Learning is increasingly reliant upon good data and benchmarks, not only to train more powerful and/or more compact models, but also to soundly evaluate new ideas and to stress test models on their reliability, fairness, and protection against various attacks, including privacy attacks.
Simultaneously, in 2021, the NeurIPS Dataset and Benchmark track was launched and the Data-Centric AI initiative was born. This kickstarted the "data-centric era". It is gaining momentum in response to the new needs of data scientists who, admittedly, spend more time on understanding problems, designing experimental settings, and engineering datasets, than on designing and training ML models.
We will retrace the enormous collective efforts made by our community since the 1980's to share datasets and benchmarks, putting forward important milestones that led us to today's effervescence. We will pick a few hot topics that have raised controversy and have engendered novel thought-provoking contributions. Finally, we will highlight some of the most pressing issues that must be addressed by the community.