Simulation-Assisted Learning for Efficient Bin-Packing of Deformable Packages in a Bimanual Robotic Cell

Omey M. Manyar*, Hantao Ye*, Meghana Sagare, Siddharth Mayya, Fan Wang and Satyandra K. Gupta

*Equal Contribution

Code

Paper

Abstract: Bin-packing is an important problem in the robotic warehouse domain. Traditionally, this problem has been studied only for rigid packages (e.g., boxes or rigid objects). In this work, we tackle the problem of bin-packing with deformable packages that have become a popular choice for fulfillment needs. We present a system that incorporates a dual robot arm bimanual setup, uniquely combining suction and sweeping motions to stably and reliably pack deformable packages in a bin. Additionally, we propose a comprehensive action prediction framework to optimize for bin-packing efficiency by predicting optimal actions for both robots involved. Our methodology leverages a two-pronged learning strategy, where initially, we train a model in a self-supervised manner to predict a scoring metric indicative of bin-packing efficiency and then leverage an online optimization scheme to compute optimal actions in real-time. The model is pre-trained in simulation in MuJoCo and fine-tuned on small-scale real-world data. Our packing score prediction model predicts a bin-packing score [0,1] with an MSE of 0.003. Real-world experiments validate our method's adaptability to novel scenarios and its effectiveness in packing operations.

Bin Packing with Proposed Bi-Manual Robotic Cell

Model Overview

Learning Architecture for Action Prediction

Packing Score 1 represents the quality of package placement
Packing Score 2 represents the quality of bin-sweep and final package arrangement
Bin State extractor is computed by our perception pipeline using RGB-D images

Perception Pipeline

Cell Overview (With Cameras)

Two RealSense D415 cameras are used
One is used for Bin State Estimation and another one for Pick Pose Estimation

Pickup-pose Estimation

Segmentation Performance

YOLOv8 is trained for package segmentation in RGB images. In addition to pose estimation, the oriented bounding box in RGBD is used to measure package dimensions.

Bin State Estimation

The density-based spatial clustering algorithm (DBSCAN) and the plane patch detection algorithm are used to extract the position and orientation of the package stack.

Bin state:

x = 0.080 m

θ = 7.895 deg

Bin state:

x = 0.127 m

θ = 14.687 deg

Bin state:

x = 0.255 m

θ = 19.128 deg

Bin state:

x = 0.337 m

θ = 20.409 deg

Simulation

The simulation is running under MuJoCo 3.1.2
Flexcomp is used to construct the deformable packages
- Each package constructed by flexcomp could be represented as a 4*3*2 point grid connected by equally constrained edges from their neighbors
- Young's modulus and Poisson ratio could be set for each package by the MuJoCo plugin: mujoco.elasticity.solid
  - Young's modulus is computed by stiffness (K) and package dimensions (l * w * h) : (K * h) / (l * w)
  - Poisson's ratio is set at 0.3 based on overall package characteristics

Real Data Collection

Stiffness Computations

To determine the stiffness (K) of each package, the pick-place robot moves linearly down in the z-direction under impedance control while recording external forces on its suction gripper. The stiffness is calculated using the recorded force on the robot's flange (Fz) and the z-deviation (Δz) from its commanded goal pose.