Workshop Tracks
The ARTS workshop has two track for submitting papers.
Challenge Track: Papers from participants in Adversarial Nibbler Challenge describing their experiences and red-teaming approaches
Workshop Topics Track: Empirical and Position papers related to workshop topics
In empirical papers, authors are invited to share novel findings, preliminary results, and post-hoc analyses. In position papers, authors can offer new perspectives, ideas or theoretical comments that argue for challenges, benefits, best practices, and strategies for the study of red teaming and adversarial testing. The main goal of both types of papers is to offer arguments and cases for discussions among the presenters to probe the concepts and interplay of the presented work and positions.
All submissions should be in English. Papers should be between 2 pages (position papers) and 4 (empirical papers) formatted in the main proceedings style and submitted via Easychair. All submissions will be peer reviewed by the workshop Program Committee. Submissions will be published in the workshop proceedings.
Track 1: Adversarial Challenge Track
Organizers of the workshop are hosting an adversarial data challenge called Adversarial Nibbler, which is a data-centric AI hackathon for discovering a diverse set of safety vulnerabilities (i.e. adversarial examples) in current state-of-the-art Text-to-Image (T2I) models that can ultimately help improve their safety. A typical bottleneck in safety evaluation is achieving a wide coverage of different types of challenging examples in the evaluation set, i.e., identifying “unknown unknowns” or long-tail problems. All datasets collected during the challenge will be made publicly available under a CC-BY-SA license with which we aim to facilitate model training, optimization, and safety evaluation and provide greater awareness of these issues and assist developers in improving the future safety and reliability of generative AI models.
While this challenge interrogates Text-to-Image models, the primary target for participants is on the text component of the system, i.e., finding text prompts that look safe and pass by safety filters, but nonetheless cause models to generate unsafe images.
The challenge is supported by Kaggle and MLCommons. It is hosted on the Dynabench platform and is part of the DataPerf challenge suite.
Check Parrish et al., 2023 for detailed description of the challenge.
Join the challenge at: https://www.dataperf.org/adversarial-nibbler (Note, you don't need ML or CS background to participate. The challenge has been designed to be accessible to a wide range of researchers and developers with and without a traditional AI/ML background.)
Track 2: Workshop Topics Track
We welcome short paper submissions (4 pages, excluding references and supplementary materials). These submissions may cover topics including, but not limited to:
Best practices in adversarial testing for generative models;
The effectiveness of adversarial approaches across language and vision-language models;
FATE-related concerns on current red-teaming + adversarial testing processes;
Remaining blind spots and unknown-unknowns in adversarial testing;
Generalizability in red teaming and adversarial testing approaches;
Assessing the effectiveness of adversarial approaches.