Satoshi Funabashi, Atsumu Hiramoto, Naoya Chiba,
Alexander Schmitz, Shardul Kulkarni and Tetsuya Ogata
Waseda University
2025 IEEE International Conference on Robotics & Automation (ICRA), 19–23 May, Atlanta, USA
Abstract
To achieve a desired grasping posture (including object position and orientation), multi-finger motions need to be conducted according to the the current touch state. Specifically, when subtle changes happen during correcting the object state, not only proprioception but also tactile information from the entire hand can be beneficial. However, switching motions with high-DOFs of multiple fingers and abundant tactile information is still challenging. In this study, we propose a loss function with constraints of touch states and an attention mechanism for focusing on important modalities depending on the touch states. The policy model is AE-LSTM which consists of Autoencoder (AE) which compresses abundant tactile information and Long Short-Term Memory (LSTM) which switches the motion depending on the touch states. Motion for cap-opening was chosen as a target task which consists of subtasks of sliding an object and opening its cap. As a result, the proposed method achieved the best success rates with a variety of objects for real time cap-opening manipulation. Furthermore, we could confirm that the proposed model acquired the features of each subtask and attention on specific modalities.
Paper
IEEE Xplore:
arXiv: https://arxiv.org/abs/2503.07757
Accepted at ICRA2025
Bibtex
@ARTICLE{9681179,
author={Funabashi, Satoshi and Isobe, Tomoki and Hongyi,
Fei and Hiramoto, Atsumu and Schmitz, Alexander and
Sugano, Shigeki and Ogata, Tetsuya},
journal={IEEE Robotics and Automation Letters}, title={Multi-
Fingered In-Hand Manipulation With Various Object Properties
Using Graph Convolutional Networks and Distributed Tactile
Sensors},
year={2022},
volume={7},
number={2},
pages={2102-2109},
doi={10.1109/LRA.2022.3142417}}
Overview Video
Method
The proposed model combines an AE and an LSTM. The AE compresses high-dimensional tactile data into low-dimensional features of a size similar to that of the joint angle values and torque values. The LSTM generates motion by predicting future information through deep predictive learning. Such a two-stage model combining feature extractor and motion generation models is widely used in robot manipulation and can exhibit high generalization performance with reasonable training data. We provided two AEs: one for obtaining tactile features of the entire finger and palm and another for obtaining local tactile features of only the thumb. This structure aims to improve the success rate of tasks by separately providing tactile features of only the thumb, considering the role of the thumb in the task of cap-opening. During the training of the LSTM, the model should learn to focus on input data at the timing of switching motions and generate the switching motions. The loss function used for training the model is the mean squared error, which aims to minimize the difference between the teaching data and the model’s output. In addition to this, the proposed method introduces imposing constraints using the squared error of the hidden features of LSTM at the start and end of specific sub-tasks.
Within a sequence of motions to perform a task, the modalities of greater significance for each motion constantly change. Therefore, an attention mechanism is introduced to weigh each modality based on the past and present states of the motions. In this study, an attention mechanism is applied to the input values to LSTM with the purpose of focusing on modalities, as shown in Fig. 2. The attention mechanism A(t) for each modality uses the calculated value from the following equation:
Results & Analysis
To evaluate the proposed constraints on the loss function and an attention mechanism, we prepared three types of baseline model. The first model, AE-LSTM, uses only constraints on the loss function; the second model, AE-LSTM, utilizes only the attention mechanism; and the third model, AE-LSTM without employing the proposed method, for comparing the success rate of cap-opening motions. Moreover, we quantitatively evaluate the partial success rate and complete success rate of cap-opening motions. Here are the definitions:
• Complete success: If the cap is opened and recognized, and the motion is immediately stopped, it is considered a complete success.
• Partial Success: If the cap-opening motion is successful but not recognized, resulting in the continued execution of the motion, it is considered a partial success.
The proposed AE-LSTM model (model I) using constraints on the loss function and an attention mechanism achieved the highest partial success rate and complete success rate. This indicates that the proposed method has acquired generalization performance for initial positions and objects. By imposing constraints on the loss function (model II), it was possible to switch to a stop motion immediately after the cap was opened, leading to an improvement in the complete success rate. On the other hand, for models (III and IV) without constraints, even if the cap was opened, they often continued the motion without switching to a stop. Additionally for model I and III, by focusing attention on appropriate modalities for each motion the accuracy of motion was enhanced, resulting in improved partial success rates for both sliding and cap- opening motions. Conversely, for models (II and IV) without the attention mechanism, the accuracy of sliding motions was low, leading to frequent failures in cap-opening due to the inability to move the object smoothly left and right. From those results it can be inferred that constraints on the loss function contributed to improving the complete success rate, while the attention mechanism contributed to increasing the partial success rate. Therefore, the proposed method, which combines both constraint and attention techniques, could achieve the most successful in-hand manipulation.
To assess whether the attention mechanism appropriately switches attention to modalities during motion generation, we visualized the weighting values for each modality by the attention mechanism in the proposed model. The horizontal axis represents the time at intervals of 10Hz, and the vertical axis represents the output values of the attention mechanism. During the cap-opening motion, attention is directed to joint angles and whole tactile features. Throughout the sequence of motions, the object’s posture may sometimes be parallel to the hand and other times diagonal, causing the position for opening the cap to be difficult. Therefore, it is presumed that attention to these modalities during the cap-opening motion is for accurately capturing the position to open the cap. Additionally, immediately after the cap-opening motion, attention is focused on the tactile features of the thumb. This seems to be for confirming the state of the cap in order to switch motions. Furthermore, during the sliding motion, attention is directed to joint angles. If the accuracy of sliding motion is low, it is difficult to move the object and the motion tends to fail. This demonstrates that focusing attention on joint angles enables precise motion. These results imply that the attention mechanism appropriately directs attention to important modalities for each sub-task.
To analyze the effect of constraints on the proposed loss function, we investigated the latent variables of LSTM in the proposed model through principal component analysis (PCA) and analyzed whether the motions for sub-tasks were embedded in the latent space. The horizontal and vertical axis represents the first and the second principal component, respectively. Each plot connects the transition of latent variables at each timestep from the start to the end of the task with a straight line. Imposing constraints on the loss function forced the latent variables of LSTM to trace a loop trajectory with the timing of motion switching. This suggests that the proposed constraints works and has contributed to improving the ability to switch motions. With the cap-opening motion positioned to the left (dark and light green dots) and the sliding motion to the right (light blue dots) on the horizontal axis, it can be inferred that the cap-opening manipulation was achieved stably by the model which could embed these two main motions in different latent spaces clearly.
Acknowledgements
This work was supported in part by Japan Science and Technology Agency, ACT-I Information and Future Acceleration Phase under Grant JPMJPR18UP and in part by Moonshot R&D under Grant JPMJMS2031.
