ImageNet Workshop 2021

A NeurIPS 2021 workshop on

ImageNet: past, present, and future

Summary

Since its release in 2009, ImageNet has played an instrumental role in the development of deep learning architectures for computer vision, enabling neural networks to greatly outperform hand-crafted visual representations. ImageNet also quickly became the go-to benchmark for model architectures and training techniques which eventually reach far beyond image classification. Today’s models are getting close to “solving” the benchmark. Models trained on ImageNet have been used as strong initialization for numerous downstream tasks. The ImageNet dataset has even been used for tasks going way beyond its initial purpose of the training of classification models. It has been leveraged and reinvented for tasks such as few-shot learning, self-supervised learning and semi-supervised learning. Interesting re-creation of the ImageNet benchmark enables the evaluation of novel challenges like robustness, bias, or concept generalization. More accurate labels have been provided. About 10 years later, ImageNet symbolizes a decade of staggering advances in computer vision, deep learning, and artificial intelligence.

We believe now is a good time to discuss what’s next: Did we solve ImageNet? If not, what is there left to do? What are the main lessons learnt thanks to this benchmark? What should the next generation of ImageNet-like benchmarks encompass? Is language supervision a promising alternative? How can we reflect on the diverse requirements for good datasets and models, such as fairness, privacy, security, generalization, scale, and efficiency?

Our workshop is featured in the December 2021 issue of the Computer Vision News.

← Click to check out!

Schedule & contents

The workshop has taken place on Monday 13 December - Tuesday 14 December 2021, depending on your time zone.

The recordings are available below.

ImageNet Workshop 2021

How to attend the workshop (closed)

You can attend the live event through the NeurIPS virtual site for our workshop. Access to the page requires registration at NeurIPS. You will find links to the Zoom and Gathertown sessions on the virtual site.

You can watch the Zoom events either through the livestream on the virtual page or on Zoom.
You can watch the Livestream events can only through the livestream on the virtual page.
You can join the poster session here on Gathertown.

How to actively participate in the workshop (closed)

We highly encourage the active participation of the attendees. Attendees can participate in various ways.

Invited talks, spotlight talks and panel sessions

Register questions and topics for discussion before the workshop: Google forms (link)
Post questions and topics for discussion during the live events: Rocketchat (requires registration)

Invited speakers will answer the selected questions during the panel sessions.

Poster sessions

Join the Gathertown during the poster sessions and interact with the authors, speakers, other attendees, and organisers.

After the workshop

We plan to publish the recorded events from the workshop.

The NeurIPS Code of Conduct applies to our workshop.

Important dates (closed)

~~Submission system open: 26 August 2021~~

~~Deadline for tracks #1 and #2: 18 September 2021~~ ~~25 September 2021, 23:59 AOE (final; no extension)~~

~~Deadline for track #3: 5 December 2021 (Please submit to~~ ~~imagenetworkshoppc@gmail.com~~)

~~Author notification: 15 October 2021 22 October 2021~~

~~Workshop date: 13 December 2021~~

~~Feel free to join~~ ~~the mailing-list~~ ~~to be notified of updates, or simply check back later.~~

Call for papers (closed)

We call for research work on and around ImageNet to be presented at the workshop, see “Scope and topics” below for details. There are three submission tracks. All accepted papers, including the ones for the main track, are non-archival; that is, there are no formally published proceedings. Nonetheless, we will make all accepted papers available on OpenReview.

1. Main track

We welcome original research papers in the main track. The submissions must be at most 5 pages in length, excluding the references and appendix, and use our modified NeurIPS 2021 style file (see below). Reviewers are not obliged to read the appendix. Please submit a single PDF file that includes the main paper, references, and any appendix. The review process is double-blind, so please ensure that the submission is appropriately anonymised.

2. Extended abstract track

We call for extended abstract submissions that present open problems, novel-yet-not-thoroughly-tested ideas, and any perspective on the future of ImageNet and related research. The submissions must be at most 3 pages in length, excluding the references, and use our modified NeurIPS 2021 style file (see below). The review process is double-blind, so please ensure that the submission is appropriately anonymised.

3. Published work track

To encourage lively discussion around ImageNet, we allow the presentation of previously published papers, if they are highly relevant to the topic. Please note that it is not possible to submit papers accepted at the NeurIPS’21 main conference, as per the NeurIPS guidelines this year. There is no formal style and page limit on this track; authors may submit a published paper as-is. The review process is single-blind. Authors of the accepted papers will be able to present their work during the workshop.

Prizes for best papers

We will award the best papers for the main track (track 1) with a cash prize of 1000 USD. Stay tuned for further updates on the prizes!

Reviewer registration for authors

Given the expected size of the workshop, we will likely be short of reviewers. Authors will be given an option to register a subset of authors as reviewers in the OpenReview submission form. We strongly encourage at least one author for each paper to be registered as a reviewer. The paper reviewing will take place between late September and early October. We aim to assign at most two papers per reviewer.

Links and resources

Submission link: Closed. Track 3 is still open; please submit to imagenetworkshoppc@gmail.com.

Template files:

Scope and topics

All three tracks welcome any submission on the topics below. The list is non-exhaustive; if unsure, please contact the organizers at imagenet21@googlegroups.com.

ImageNet as a benchmark

How is ImageNet serving the community as a benchmark?
Did we solve the ImageNet classification task?
How can we spot and remedy the remaining errors?
What are the remaining challenges not captured by ImageNet?
How can we measure the research progress on robustness, bias, and other out-of-domain generalization issues using ImageNet?
How to fix errors in ImageNet labels and compare results across different labels?

ImageNet as a pre-training dataset

Is ImageNet still the best dataset for pre-training?
What could a better pre-training dataset look like? What are good properties?
Zero- and few-shot learning on or using ImageNet
Unsupervised and self-supervised learning
Semi-supervised learning
Transfer learning

Explorative and innovative ideas even without state-of-the-art (SOTA) results

We encourage submissions that are groundbreaking but are difficult to get published at conferences because they do not get SOTA performances.

Beyond ImageNet

Recipes for building a large-scale dataset
Socially responsible benchmarks and datasets: fairness, privacy, security, robustness, scalability, efficiency, and environmental friendliness
Efficiency and quality aspects of annotations and crowdsourcing
Going beyond classification labels for supervising visual models - e.g. language descriptions for images
Post-ImageNet benchmarks and datasets

Feel free to join the mailing-list to be notified of updates, or simply check back later.