Contributor: Eleni Triantafillou, Vincent Dumoulin
The term 'meta-dataset' refers to a dataset of datasets, that are assembled in a cohesive benchmark, to serve as an environment for training and / or evaluating different capabilities of models, including few-shot learning, meta-learning, multi-task learning, transfer learning and domain generalization / adaptation.
List of meta-datasets obtained by drawing upon and extending the comprehensive related work section from Meta-Album.
Visual Decathlon
BSCD-FSL
Comments: The Broader Study of Cross-Domain Few-Shot Learning Benchmark aims at measuring few-shot learning and meta-learning capabilities in the context of significant mismatch between the source and target task distributions. Meta-training is performed on mini-ImageNet, and target learning tasks are sampled from one of four datasets: CropDiseases, EuroSAT, ISIC, and ChestX.
Meta-Dataset
Comments: A dataset of 10 diverse datasets for studying few-shot learning and meta-learning. There are two common scenarios: 1) train only on (the training classes of) ImageNet and 2) train on the (training classes of) all training datasets (there are 8 in total). In both cases, evaluation happens on tasks from the test classes of each dataset. Requeima et al proposed to add 3 additional test-only datasets (CIFAR-10, CIFAR-100 and MNIST) for a richer evaluation on entirely held-out sources.
Meta-Album
Comments: Meta-Album is a meta-dataset for studying few-shot learning and meta-learning. It is a rolling benchmark, expanded over a series of competitions, where the test set of the previous year's competition consitutes legal training data for next year's competition. The current version contains 40 datasets. Compared to other few-shot learning benchmarks, Meta-Album has a larger number of small datasets from very diverse domains, and intentionally excludes datasets that are heavily-used in the community like ImageNet, to avoid giving an unfair advantage to approaches developed for those datasets.
Visual Task Adaptation Benchmark (VTAB)
VTAB+MD
WILDS / WILDS 2.0
Comments:
WILDS contains 10 datasets that reflect a diverse set of realistic distribution shifts, for studying the problems of domain generalization and subpopulation shift (as well as settings that are hybrids of these two). Notably, in contrast to the other meta-datasets in this survey, the inputs of WILDS tasks come both from visual and textual modalities in different cases. For instance, for their Toxicity Classification dataset, the input is a text comment, whereas for their Tumor Identificaiton dataset the input is a (patch of an) image of a lymph node section from a patient.
WILDS 2.0 extends 8 of the 10 datasets in the WILDS benchmark to include curated unlabeled data that would be realistically obtainable in deployment, allowing to study unsupervised domain adaptation.
DomainNet
DomainBed