# Omni-Modal LLMs

OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLM

Technical Report

arXiv, Webpage, Github, Model (10K+ Downloads!)

Hanrong Ye*†, C.-H. Huck Yang†, Arushi Goel†, Wei Huang†, Ligeng Zhu†, Yuanhang Su†, Sean Lin†, An-Chieh Cheng†, Zhen Wan†, Jinchuan Tian†, Yuming Lou†, Dong Yang†, Zhijian Liu, Yukang Chen, Ambrish Dantrey, Ehsan Jahangiri, Sreyan Ghosh, Daguang Xu, Ehsan Hosseini Asl, Danial Mohseni Taheri, Vidya Murali, Sifei Liu, Jason Lu, Oluwatobi Olabiyi, Frank Wang, Rafael Valle, Bryan Catanzaro, Andrew Tao, Song Han, Jan Kautz, Hongxu Yin†*^, Pavlo Molchanov^

Building omni-modal LLMs is MORE than just mixing tokens. At NVIDIA, we explored deeper possibilities in building truly omni-modal systems — leading to OmniVinci-9B, which introduces three key innovations: OmniAlignNet is a unified vision–audio alignment module powered by contrastive learning; Temporal Embedding Grouping & Constrained Rotary Time Embedding enables absolute and relative temporal representation across multimodal tokens. To support this, we curated a 24M-sample omni-modal dataset and developed a new large-scale data engine for efficient labeling. OmniVinci-9B outperforms Qwen2.5-Omni across omni-modal, vision, and audio benchmarks.

# Egocentric Multi-Modality Foundation Model

🦅MM-Ego: Towards Building Egocentric Multimodal LLMs

Accepted by International Conference on Learning Representations (ICLR) 2025

arXiv, TL:DR

Hanrong Ye*, Haotian Zhang*, Erik Daxberger, Lin Chen, Zongyu Lin, Yanghao Li, Bowen Zhang, Haoxuan You, Dan Xu, Zhe Gan, Jiasen Lu, Yinfei Yang

We introduce a foundation model for egocentric video understanding, contributing from three key perspectives: (a) a data engine that can automatically transform human narrations into 7 million egocentric QA samples, (b) a "Memory Pointer Prompting" mechanism designed for long egocentric video comprehension, and (c) the curation of a challenging egocentric video understanding benchmark. With the data, benchmark, and model, we successfully build MM-Ego, an egocentric multimodal LLM that shows powerful performance on egocentric video understanding.

# Any-to Any Multi-Modality Foundation Model

X-VILA: Cross-Modality Alignment for Large Language Model

Technical Report 2024

arXiv

Hanrong Ye, De-An Huang, Yao Lu, Zhiding Yu, Wei Ping, Andrew Tao, Jan Kautz, Song Han, Dan Xu, Pavlo Molchanov, Hongxu Yin

X-VILA is a foundation model for cross-modality understanding, reasoning, and generation in the domains of video, image, language, and audio. X-VILA demonstrates the ability to perceive (see, hear, and read) multi-modality inputs, and generate (draw, speak, and write) multi-modality responses. We curate an any-to-any SFT dataset with 1.6M samples, and design a visual-textual joint alignment mechanism that significantly improves the input-output consistency.

# Multi-Modality Generation for Segmentation

SegGen: Supercharging Segmentation Models with Text2Mask and Mask2Img Synthesis

Accepted by European Conference on Computer Vision (ECCV) 2024

Website, Preprint

Hanrong Ye, Jason Kuen, Qing Liu, Zhe Lin, Brian Price, Dan Xu

Can generation models help boost segmentation performance on prevalent segmentation benchmarks? We propose a new training data generation method for image segmentation, which is the first data generation method that pushes the performance limits of state-of-the-art semantic/panoptic/instance segmentation models to a significant extent. We first leverage text to generate diverse segmentation masks, and then use the masks to synthesize the corresponding realistic images. This new generation framework avoids the "chicken or egg dilemma" and thus is able to produce very high-quality synthetic data, which transfers to significantly improved segmentation performance. Notably, in terms of the ADE20K semantic segmentation mIoU, Mask2Former R50 is largely boosted from 47.2 to 49.9 (+2.7); Mask2Former Swin-L is also significantly increased from 56.1 to 57.4 (+1.3). We also observe strong gains on COCO panoptic segmentation (+0.7 PQ) and instance segmentation (+0.8 AP). Moreover, training with our synthetic data makes the segmentation models more robust towards unseen domains.

# Simultaneous 2D/3D Multi-Task Scene Perception

DiffusionMTL: Learning Multi-Task Denoising Diffusion Model from Partially Annotated Data

Accepted by Computer Vision and Pattern Recognition Conference (CVPR) 2024

Website, Paper, Github

Hanrong Ye and Dan Xu

We design a joint multi-task denoising diffusion framework for significantly improving prediction quality under a partially annotated multi-task learning setting, where the initial multi-task predictions are noisy due to the severe lack of ground-truth supervision. 

# Simultaneous 2D/3D Multi-Task Scene Perception

TaskExpert: Dynamically Assembling Multi-Task Representations with Memorial Mixture-of-Experts

Accepted by International Conference on Computer Vision (ICCV) 2023

Paper, Cite

Hanrong Ye and Dan Xu

TaskExpert is a novel multi-task mixture-of-experts model that enables learning multiple representative task-generic feature spaces and decoding task-specific features in a dynamic manner. Specifically, TaskExpert introduces a set of expert networks to decompose the backbone feature into several representative task-generic features. Then, the task-specific features are decoded by using dynamic task-specific gating networks operating on the decomposed task-generic features. To establish long-range modeling of the task-specific representations from different layers of TaskExpert, we design a multi-task feature memory that updates at each layer and acts as an additional feature expert for dynamic task-specific feature decoding. More details are coming.

# Simultaneous 2D/3D Multi-Task Scene Perception

TaskPrompter: Spatial-Channel Multi-Task Prompting for Dense Scene Understanding

Accepted by International Conference on Learning Representations (ICLR) 2023

Paper, Github, Cite

Hanrong Ye and Dan Xu

TaskPrompter jointly models task-generic and task-specific representations, as well as cross-task representation interactions simultaneously in one single module. Its compact design establishes a new SOTA performance while reducing computation costs. Novelty well received by all reviewers (ratings: 8,8,6,6,6). We further propose a new joint 2D-3D multi-task benchmark based on Cityscapes. TaskPrompter can generate predictions for 3D detection, segmentation, and depth with one model, one training, and one inference. Our 3D detection performance largely surpasses the previous best single-task results on Cityscapes. 

# Simultaneous 2D/3D Multi-Task Scene Perception

Inverted Pyramid Multi-task Transformer for Dense Scene Understanding 

Accepted by European Conference on Computer Vision (ECCV) 2022

Github, Demo, arXiv, Cite

Hanrong Ye and Dan Xu

We propose a novel end-to-end Inverted Pyramid multi-task (InvPT) Transformer to perform simultaneous modeling of spatial positions and multiple tasks in a unified framework. Our method achieves superior multi-task performance on NYUD-v2 and PASCAL-Context datasets respectively and significantly outperforms previous SOTA.

Journal Extension: InvPT++: Inverted Pyramid Multi-Task Transformer for Visual Scene Understanding

Accepted by IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 2024

Github, Demo, arXiv, Cite

Hanrong Ye and Dan Xu

In the journal extension, we further propose two types of Cross-Scale Self-Attention modules, i.e., Fusion Attention and Selective Attention, based on InvPT. The special attention modules are proposed to facilitate cross-task interaction across different feature scales. The new designs help InvPT++ achieve notably better performance and efficiency.