Yexin Hu*, Alexandra Gillespie*, Akhil Padmanabha, Kavya Puthuveetil, Wesley Lewis, Karan Khokar, Zackory Erickson
*indicates equal contribution
Accepted at 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
Project Video
Robocap Iros.mp4Abstract
Liquids and granular media are pervasive throughout human environments, yet remain particularly challenging for robots to sense and manipulate precisely. In this work, we present a systematic approach at integrating capacitive sensing within robotic end effectors to enable robust sensing and precise manipulation of liquids and granular media. We introduce a parallel-jaw gripper with embedded capacitive sensing arrays that enable a robot to directly sense the materials and dynamics of liquids inside of diverse containers, including some visually opaque. When coupled with model-based control, we demonstrate that the proposed system enables a robotic manipulator to achieve state-of-the-art precision pouring accuracy for a range of substances with varying dynamics properties.
Basic Methodology
To perform precision pouring using the capacitive sensor, a robot must be able to understand the relationship between the capacitive value and the weight of the substance that has been poured out to determine when to stop pouring and rotate back to the vertical orientation. But the challenge is that even if the robot has already stopped pouring and rotated back, the substance will still be poured out due to inertia. To take this into consideration, our pouring pipeline is separated into two parts. The first is to determine when to stop pouring. We use the Overpoured Weight Estimation (OWE) function, which is a second-order polynomial to fit the relationship between the target weight and stopping weight, which could directly map the target weight to the stopping weight. Once we have our target weight, then we can use this model to estimate when the robot should stop pouring. The second part is the pouring control process, here we trained a neural network model that could take capacitive signals in a 0.1s interval as input, and output the weight changes during that interval. During the pouring process, we record the real-time capacitive readings and input them into the Poured Weight Estimation (PWP) model to estimate the weight changes for each 0.1s interval during the pouring process. We accumulate all those weight changes to estimate the weight of the substance that has already been poured out. Once the weight reaches the stopping weight, the robot stops pouring and rotates back, thus achieving precision pouring.
Experimental Setup
Example of Data Collection Trials for Classification
classification.mp4Example of Data Collection Trials for Pouring
pouring_1.mp4Real-Time Evaluation of Learned Model for Pouring
pouring_2.mp4A. Substance and Container Classification Data Collection Details
As described in Section IV-A, the RoboCAP Gripper closes around the container over the course of two seconds, yielding 200 individual readings for each of the ten electrodes due to the sampling frequency of 100 Hz. The microcontroller stops collecting values at the end of the two seconds, at which point the RoboCAP Gripper is fully closed around the container. Since the containers in our dataset have different diameters, we set the open and close positions manually. The RoboCAP Gripper was then commanded to open and this cycle was repeated for ten open-close trajectories for each substance-container combination. This comprised one set of data collection. All the containers were grasped at the same height, 1 cm from their base.
B. Sample Scatter Plots for Overpoured Weight VS Stop Weight
C. OWE Sample Fitting Curves
Fig. .1 shows two sample fitting curves of the OWE model for water and rice. In these plots, we can observe that the overpouring weight for both substances generally increases as the robot continues to rotate its end effector to pour out more substance.
D. Substance-Container Classification: Generalization
In practice, a robot may encounter substances or containers that were not included in its initial training datasets. The generalization of capacitive end-effector sensing to novel, out-of-distribution containers or substances presents a promising yet challenging direction for future exploration, including the ability to classify new types of liquids and granular media outside the initial training distribution. One approach to do so is to predict material properties such as viscosity, density, surface tension, and thermal conductivity, rather than relying solely on discrete class labels. These continuous physical properties can provide a more nuanced understanding and categorization of materials. Further, employing machine learning techniques, such as transfer learning and few-shot learning, could enhance the gripper’s ability to generalize from known substances to new ones by leveraging similarities in material properties.
E. Substance-Container Classification: Electrode Ablations
To further motivate the selection of ten electrodes on the RoboCAP Gripper, we investigated the performance of our RoboCAP classifier with fewer electrodes testing on the last-day dataset. These results can aid researchers with more informed design decisions while integrating capacitive sensing in robotic end-effectors. For the six electrode case, we selected the 1st, 3rd and 5th electrodes from each panel. For the two electrode case, we selected the 1st and the 5th electrodes of the right panel (pouring side) since the electrodes from this panel have a relatively larger variation as the substance pours out. Lastly, for the one electrode case, we selected the third electrode on the right panel.
As seen in Table .1, increasing the number of electrodes leads to improved classification performance. However, classification performance when using smaller arrays of sensor remains competitive in performance and there exist several opportunities for future research and commercialization in terms of miniaturizing these arrays, such as using flexPCBs, to fit onto existing end effector fingertips.
F. Substance-Container Classification: Increasing Substance-Container Combinations
To further explore the ability of these methods to scale to more substance-container combinations that a robot may encounter in the real world, we present results, displayed in Fig. .2, on how classification accuracy varies when substance-container combinations are added. While container accuracy consistently stays above 95%, substance accuracy declines as more classes are added. This performance decline is expected as the classification problem becomes increasingly challenging with more classes. To tackle this problem, as well as predicting continuous physical properties of containers and substances beyond discrete material classification. Additionally, future work may investigate multimodal sensing strategies, such as combining force-torque or haptic sensing in combination with capacitive sensing, to improve recognition performance of liquids and granular media.
G. Pouring: Electrode Ablations
We conducted an ablation study to explore the impact of the capacitive sensor’s design on pouring performance. The first set of experiments we ran for this study was with six electrodes. For this, we selected the 1st, 3rd and 5th electrodes from each panel and disabled the other electrodes during both the training and test trials. The second set of experiments were with two electrodes. Here we selected the 1st and the 5th electrodes of the right panel (pouring side) since the electrodes from this panel have a relatively larger variation as the substance pours out. All other electrodes are disabled for these experiments.
For each case described above, we collected 10 complete pouring trials in the same way as described in Sec. V-B, for each of the five substances for training, totaling 50 trials. Out of these, 80% of data collected was used for actual training and 20% for validation. We also collected 24 additional pouring trials per substance, in the same way as described in Sec. V-D for fitting the OWE function. For test, we conducted 5 trials for each substance at all four different target weights (50g, 75g, 100g, and 125g), totaling 100 trials. Table .2 shows the mean error and standard deviation for the two cases and our proposed design. Compared to the cases with the reduced number of electrodes, our proposed design exhibited better or on-par performance for all liquid and granular media substances. We further find that reducing the number of electrodes, or in some cases having information from only two electrodes, can also accurately predict the poured-out weight in a precision pouring context.
Overall, as the number of electrodes decreases, performance gradually declines. One reason for this can be that capacitive sensors are sensitive to environmental changes and electromagnetic interference (EMI). We find that using a larger number of electrodes helps improve robustness to noise in learning-based models. Without alternative signals from other electrodes, a model can erroneously learn noise patterns from a few number of electrodes, resulting in instabilities and lower performance during test time in new scenarios.
H. Pouring: OWE Function Ablation
To motivate the inclusion of both the PWP model and OWE function in our controller, we ablate our RoboCAP Controller method to exclude the OWE function and demonstrate performance when we do not account for the over-poured weight. In this ablation, the results of which are summarized in Table .3, we use the PWP model to rotate the RoboCAP Gripper back to stop pouring when the final target weight wtarget, instead of the stop weight wstop, has been reached.
Without the OWE function, the ablated version of our RoboCAP Controller method overpoured each substance in amounts ranging from 18.6-26.4 g, compared to errors of 2.0-5.4 g for our full controller with both the PWP model and OWE function. These results show that effective controllers for robotic precision pouring must reason about the inertia of the substance being poured.
I. Pouring: Loss Term Ablations
To motivate our use of the offset start and end times,OS and OE respectively, when computing the change in ground truth weight ∆w, as well as the corresponding auxiliary loss terms 3, 4, and 5, we present results for the ablation of these terms when training the PWP model. Unlike our complete PWP model, which is trained using the full loss objective (1) and predicts ∆ˆw, OS, and OE, our ablated PWP model is trained only using loss term (2) and only predicts ∆ˆw. Without the use of OS and OE, the ablated PWP model cannot account for the time delay between when a substance leaves the lip of the source container and when it reaches the target container and is measured by the weighing scale. As a result, the ablated PWP model maps capacitive sensor readings to the change in weight in the target container, while the complete PWP model maps to the change of weight in the source container. We also collected new data (120 pouring trials) to fit an OWE function, as described in V-D, and conducted 100 test trials, as in Appendix F, for the ablated PWP to be used for control. Results comparing the complete PWP and ablated PWP, when used as part of our control pipeline to precisely pour the five substances, are summarized in Table .4. We found that pouring performance degrades when the PWP does not reason about the offset start and end times, OS and OE, reinforcing the impact of these terms in the proposed PWP loss function.
J. Dielectric Constants for Containers and Substances
K. Data Processing Details: Os and Oe
Although Os and Oe are time indices, we use linear interpolation to make these two variables trainable. Below is the weight interpolation function we used:
L. Data Processing Details: W
Since the weighing scale has some noise and the impulse caused by falling substance will cause a certain amount of deviation between the reading of the weighing scaler and the actual reading, (1) we use a low pass filter to smooth the training weight data to reduce the error caused by the noise; (2) if the weight at a certain moment is larger than the weight 1 second later(we set an offset to 1 to ensure that the noise of the weighing scale does not cause this phenomenon), we manually adjust the weight at that moment to the weight 1 second later to reduce the impact of the impulse. Code for the lowpass filter and weight adjusting is shown below: