Poster Sat, Apr 25, 2026 • 6:30 AM – 9:00 AM PDT Pavilion 4 P4-#3610

OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLM

Hanrong Ye ⋅ Chao-Han Huck Yang ⋅ Arushi Goel ⋅ Wei Huang ⋅ Zhen Wan ⋅ Jinchuan Tian ⋅ An-Chieh Cheng ⋅ Ligeng Zhu ⋅ Yuanhang Su ⋅ Yuming Lou ⋅ Yong-Xiang Lin ⋅ Dong Yang ⋅ Sreyan Ghosh ⋅ Zhijian Liu ⋅ Yukang Chen ⋅ Ehsan Jahangiri ⋅ Ambrish Dantrey ⋅ Daguang Xu ⋅ Ehsan Hosseini-Asl ⋅ Seyed Danial Mohseni Taheri ⋅ Vidya Nariyambut Murali ⋅ Sifei Liu ⋅ Yao Lu ⋅ Oluwatobi Olabiyi ⋅ Yu-Chiang Frank Wang ⋅ Rafael Valle ⋅ Bryan Catanzaro ⋅ Andrew Tao ⋅ Song Han ⋅ Jan Kautz ⋅ Hongxu Yin ⋅ Pavlo Molchanov

Project Page

Abstract

Advancing machine intelligence requires developing the ability to perceive across multiple modalities, much as humans sense the world. We introduce OmniVinci, an initiative to build a strong, open-source, omni-modal LLM. We carefully study the design choices across model architecture and data curation. For model architecture, we present three key innovations: (i) OmniAlignNet for strengthening alignment between vision and audio embeddings in a shared omni-modal latent space; (ii) Temporal Embedding Grouping for capturing relative temporal alignment between vision and audio signals; and (iii) Constrained Rotary Time Embedding for encoding absolute temporal information in omni-modal embeddings. We introduce a curation and synthesis pipeline that generates 24M single-modal and omni-modal conversations. We find that modalities reinforce one another in both perception and reasoning. Our model, OmniVinci, improves over Qwen2.5-Omni with +19.05 on DailyOmni (cross-modal understanding), +1.7 on MMAR (audio), and +3.9 on Video-MME (vision), while using just 0.2T training tokens — a 6× reduction compared to Qwen2.5-Omni’s 1.2T. We finally demonstrate omni-modal advantages in downstream applications spanning robotics, medical AI, and smart factory.