MLA: A Multisensory Language–Action Model for Multimodal Integration and Forecasting in Robotic Manipulation

💡Abstract

Vision-language-action models (VLAs) have shown generalization capabilities in robotic manipulation tasks by inheriting from vision-language models (VLMs) and learning action generation. Most VLA models focus on interpreting vision and language to generate actions, whereas robots must perceive and interact within the spatial-physical world. This gap highlights the need for a comprehensive understanding of robotic-specific multisensory information, which is crucial for achieving complex and contact-rich control. To this end, we introduce a multisensory language–action (MLA) model that collaboratively perceives heterogeneous sensory modalities and predicts future multisensory objectives to facilitate physical world modeling. Specifically, to enhance perceptual representations, we propose an encoder-free multimodal alignment scheme that innovatively repurposes the large language model itself as a perception module, directly interpreting multimodal cues by aligning 2D images, 3D point clouds, and tactile tokens through positional correspondence. To further enhance MLA’s understanding of physical dynamics, we design a future multisensory generation post-training strategy that enables MLA to reason about semantic, geometric, and interaction information, providing more robust conditions for action generation. For evaluation, the MLA model outperforms the previous state-of-the-art 2D and 3D VLA methods by 12% and 24% in complex, contact-rich real-world tasks, respectively, while also demonstrating improved generalization to unseen configurations. Project website: sites.google.com/view/openmla.

Vision-language-action models (VLAs) have shown generalization capabilities in robotic manipulation tasks by inheriting from vision-language models (VLMs) and learning action generation. Most VLA models focus on interpreting vision and language to generate actions, whereas robots must perceive and interact within the spatial-physical world. This gap highlights the need for a comprehensive understanding of robotic-specific multisensory information, which is crucial for achieving complex and contact-rich control. To this end, we introduce a multisensory language–action (MLA) model that collaboratively perceives heterogeneous sensory modalities and predicts future multisensory objectives to facilitate physical world modeling. Specifically, to enhance perceptual representations, we propose an encoder-free multimodal alignment scheme that innovatively repurposes the large language model itself as a perception module, directly interpreting multimodal cues by aligning 2D images, 3D point clouds, and tactile tokens through positional correspondence. To further enhance MLA’s understanding of physical dynamics, we design a future multisensory generation post-training strategy that enables MLA to reason about semantic, geometric, and interaction information, providing more robust conditions for action generation. For evaluation, the MLA model outperforms the previous state-of-the-art 2D and 3D VLA methods by 12% and 24% in complex, contact-rich real-world tasks, respectively, while also demonstrating improved generalization to unseen configurations. Project website: https://sites.google.com/view/open-mla

⚙️Model Architecture

We propose MLA, a multisensory language–action model that collaboratively processes diverse sensory inputs and predicts their corresponding future states to enhance physical-world modeling for robotic control. To avoid introducing additional modality-specific encoders that lack pretraining alignment with LLM’s embeddings, MLA adopts an encoder-free multimodal alignment mechanism, repurposing the initial transformer blocks of the LLM as a perception module to directly interpret visual, geometric, and tactile cues. In particular, we project 3D points and the spatial positions of the tactile gripper onto 2D image planes using camera parameters, thereby constructing cross-modal positional mappings. These positional correspondences serve as positive pairs for token-level contrastive learning, aligning multimodal features within the LLM’s embedding space. This position-guided consistency constraint enhances the multimodal representations of our MLA model and supports more comprehensive physical-world perception. To further enhance the LLM’s understanding of physical robotic scenes, we propose a future multisensory generation post-training strategy.  Specifically, the lightweight transformer-based decoders and tailored generation scheme are designed to process the LLM’s final-layer features and generate the future states of multiple modalities, including 2D images, 3D point clouds, and tactile signals. Through this predictive process, MLA is able to reason about physical dynamics from multiple dimensions, encompassing semantic information, geometric structures, and object-centric interactions.

📊Real-world Experiments

a) Impact of Input Modalities and Alignment Strategies in the Encoder-Free Multimodal Alignment Scheme. As shown in the Figure left a), we first examine the role of different input modalities and alignment strategies under the following configurations: (Ex1) 2D image input only, (Ex2) 2D image + 3D point cloud with simple token-level concatenation, (Ex3) 2D image + 3D point cloud + tactile signals with simple token-level concatenation, (Ex4) all modalities with image-level contrastive alignment, and (Ex5) our proposed all modalities with token-level contrastive alignment. 

b) Impact of Contrastive Loss Position. As shown in Figure left b), we investigate the effect of applying contrastive loss at different layers of the LLaMA-2 backbone. Specifically, we select the 4th, 8th, 12th, and 32nd layers for cross-modal alignment during the SFT and post-training stages. 

c) Impact of Different Generation Modalities in Future State Generation. As shown in Figure left c), building upon the MLA model following SFT, we further evaluated three ablation variants during post-training: (1) without image generation, (2) without point cloud generation, and (3) without tactile signal generation. 

🌟Demonstrations

We present the inference and execution processes of the MLA model across multiple tasks. The MLA demonstrates the ability to accurately infer real-world actions and maintain stable interactions with diverse objects. It is worth noting that, since the Franka arm operates under force control, we adopt a slower execution speed for some tasks that are particularly sensitive to tactile feedback in order to ensure stability.

🤖Setup Details

For single-arm tasks, we employ a Franka Research 3 robotic arm equipped with a ROBOTIQ adaptive gripper as the end-effector. Visual observations are provided by two Intel RealSense D455 cameras, one positioned at a right-front third-person viewpoint and the other mounted on the wrist. In addition, two Tashan TS-E-A tactile sensors are attached to the fingertips of the gripper to capture tactile feedback.

For dual-arm tasks, we utilize two parallel Franka Emika arms with the same end-effector configuration. The observation setup includes an additional front-facing RealSense D455 camera along with two wrist-mounted cameras, ensuring comprehensive multi-view perception.

📊Real-world Experiments

📃Pre-training Data

🏆Simulation Experiments