RoboCup Best Paper Award, IROS 2024
[ArXiv]
Jake Varley∗, Sumeet Singh∗, Deepali Jain∗, Krzysztof Choromanski∗,
Andy Zeng, Somnath Basu Roy Chowdhury, Avinava Dubey, Vikas Sindhwani
We present an embodied AI system which receives open-ended natural language instructions from a human, and controls two arms to collaboratively accomplish potentially long-horizon tasks over a large workspace. Our system is modular: it deploys state of the art Large Language Models for task planning,Vision-Language models for semantic perception, and Point Cloud transformers for grasping. With semantic and physical safety in mind, these modules are interfaced with a real-time trajectory optimizer and a compliant tracking controller to enable human-robot proximity. We demonstrate performance for the following tasks: bi-arm sorting, bottle opening, and trash disposal tasks. These are done zero-shot where the models used have not been trained with any real world data from this bi-arm robot, scenes or workspace. Composing both learning- and non-learning-based components in a modular fashion with interpretable inputs and outputs allows the user to easily debug points of failures and fragilities. One may also in-place swap modules to improve the robustness of the overall platform, for instance with imitation-learned policies.
Fig. 1: The comparison of our method with other approaches: Vision-Language-Action Models and Code Gen AI techniques.
Multi-Modal Embodied AI Platform
A more detailed scheme of our system, consisting of various modules, in particular the: (a) LLM Task Planner, (b) Perception Module (open-vocabulary detector producing point cloud inputs for subsequent modules) and (c) Control Module: PCT Grasping Encoder, Trajectory Optimizer and Compliant Controller, is given below.
Fig. 2: Embodied AI Bi-Arm Platform: closer look.
Multiple Levels of Safety
Our system exhibits safety on multiple levels: semantic (provided in particular by LLMs capable of processing complex textual instructions) as well as physical, given by SQP-driven trajectory optimization (e.g. to avoid collisions) and compliant controllers (for the regulation of the applied forces). All those safety layers are orchestrated together to provide interpretable and physically safe execution of the task.
Fig 3: Visualization of th SQP-based trajectory optimizer in action.
Fig. 4: Testing the compliance of the fingers of our Embodied AI Bi-Arm Platform.
Fig. 5: General compliance tests of our Embodied AI Bi-Arm Platform.
Highly-Dexterous Bi-Arm Manipulation
Our system is capable of solving highly dexterous tasks, beyond the reach of Vision-Language-Action models or Code Gen AI methods. It can also solve tasks requiring planning, with different phases of the solution corresponding to different manipulation primitives. We show it below on the example of the bottle opening and pick-up-placing task, where the robot is asked to place certain objects in the garbage bin.
robot.unscrew_cap(bottle_text='detergent bottle',bottle_cap_text='detergent bottle cap')
Fig. 6: Our Embodied-AI Bi-Arm Platform in action for dexterous tasks requiring high-degree of precision and arm coordination as well as tasks requiring longer-horizon planning.
Point Cloud Transformers & Blackbox Optimization
The Embodied AI Bi-Arm Platform successfully leverages several novel techniques, both on the modeling and optimization side. Those include in particular Point Cloud Transformers to encode point cloud inputs for optimal grasp pose and Blackbox/ES optimization techniques that are applied to train them (in contrast to regular backpropagation strategies; we found ES optimization much more stable).
Fig. 7: Top: Grasp-pose module, driven by the Point cloud Transformer (PCT) encoder, in action. Bottom left: Point Cloud input to the PCT. Bottom right: Pictorial description of the Blackbox/ES algorithm used to train the PCT module. The optimization algorithm computes an unbiased estimation of the gradient of the Gaussian smoothing of the reward function, by querying in parallel workers to evaluate different perturbed versions of the current instantiation of the Point Cloud Transformer model.