Embedded Files
Code is currently available at anonymous GitHub ↑, will be at GitHub after review.
We present OmniVIC: a universal variable impedance controller (VIC) enhanced by a vision language model (VLM), which improves safety and adaptation in any contact-rich robotic manipulation task to enhance safe physical interaction. The system integrates a VLM with a VIC to enable safe and adaptive manipulation in contact-rich tasks. The VLM processes multimodal inputs, including visual observations, natural language instructions, and real-time force/torque feedback, to generate context-aware impedance parameters (stiffness and damping) for the VIC.
Overall Video:
Overall Video:
omniVIC_all.mp4Experiments in simulation:
Experiments in simulation:
Highlight moment:
(during a push task)
OmniVIC_sim.mp4Experiments in the real world:
Experiments in the real world:
Real-world experiments:
Real-world experiments:
Top: Task 1: The robot gently closes the top drawer.
Top: Task 1: The robot gently closes the top drawer.
Middle: Task 2: The robot moves along the negative Y-axis while maintaining the same height in Z-axis, where there is a ramp in the middle of the path to introduce contact.
Middle: Task 2: The robot moves along the negative Y-axis while maintaining the same height in Z-axis, where there is a ramp in the middle of the path to introduce contact.
Bottom: Task 3: the same as Task 2, but with two different ramp shapes to test multiple phases of contact.
Bottom: Task 3: the same as Task 2, but with two different ramp shapes to test multiple phases of contact.
OmniVIC_real_exp1.mp4OmniVIC_real_exp2.mp4OmniVIC_real_exp3.mp4Bibtex:
"
}
"
Page updated
Google Sites
Report abuse