ICML 2024 Workshop
Next Generation of Sequence Modeling Architectures
Fri 26 Jul, 9 a.m. CEST - Straus 3, Messe Wien Exhibition Congress Center, Vienna, Austria
Link to the event on ICML website
Description
Description
This workshop will bring together various researchers to chart the course for the next generation of sequence modeling architectures. The focus will be on better understanding the limitations of existing models like transformers, recurrent neural networks, and state space models (e.g., S4, Mamba, LRU) and describing existing open problems. We will touch on topics such as memory, long-range context and in-context learning, optimization stability of these architectures, and their ability to represent different class problems. We will also cover interpretability and pragmatic aspects of making these models efficient and perform well: how they should be scaled up and the trade-offs and limitations imposed by current hardware. We will place additional emphasis on building both theoretical and also empirical understanding of the sequence models at scale; for example, this could be a better understanding of the scaling properties of these models concerning data, number of parameters, and amount of time the model spends at the inference.Â
This workshop will bring together various researchers to chart the course for the next generation of sequence modeling architectures. The focus will be on better understanding the limitations of existing models like transformers, recurrent neural networks, and state space models (e.g., S4, Mamba, LRU) and describing existing open problems. We will touch on topics such as memory, long-range context and in-context learning, optimization stability of these architectures, and their ability to represent different class problems. We will also cover interpretability and pragmatic aspects of making these models efficient and perform well: how they should be scaled up and the trade-offs and limitations imposed by current hardware. We will place additional emphasis on building both theoretical and also empirical understanding of the sequence models at scale; for example, this could be a better understanding of the scaling properties of these models concerning data, number of parameters, and amount of time the model spends at the inference.Â
Submission
Submission
Submissions must present original research that has not been previously published. Submitted manuscripts should be composed of a main body, which can be up to four pages long, followed by an unlimited number of pages for references and appendices, all in a single file. All submissions must be anonymous and should not include any information that violates the double-blind review process, including citing authors' prior work or sharing links in a way that can reveal the identities of authors to potential reviewers. Submissions that do not conform to these instructions may be desk-rejected at the Program Committee's discretion to ensure a fair review process for all potential authors. After submission and during the review period, authors can post their papers as technical reports on arXiv or other public forums. For details concerning the format of the papers, please see the LaTeX style files on Overleaf. Submissions should be made through OpenReview.
Quick Links: Openreview , LaTeX template
Submissions must present original research that has not been previously published. Submitted manuscripts should be composed of a main body, which can be up to four pages long, followed by an unlimited number of pages for references and appendices, all in a single file. All submissions must be anonymous and should not include any information that violates the double-blind review process, including citing authors' prior work or sharing links in a way that can reveal the identities of authors to potential reviewers. Submissions that do not conform to these instructions may be desk-rejected at the Program Committee's discretion to ensure a fair review process for all potential authors. After submission and during the review period, authors can post their papers as technical reports on arXiv or other public forums. For details concerning the format of the papers, please see the LaTeX style files on Overleaf. Submissions should be made through OpenReview.
Quick Links: Openreview , LaTeX template
Speakers
Speakers
Organizers
Organizers
EPFL
Google Deepmind
ELLIS Intitute & MPI-IS, Tuebingen AI Center
ETH Zurich
IDEAS NCBR