LIST OF CONTRIBUTORS

In robotic-assisted minimally invasive surgery, surgeons need real-time tool tracking solutions to efficiently guide them during the endoscopic procedures since they cannot depend on visual or tactile examination due to the reduced field of view. Current endoscopic vision methods such as marker or marker-less tracking or other methods such as electromagnetic tracking are usually robust. But, the task becomes complicated when visual occlusion or depth perception impede the surgeon’s judgment. In previous work, we introduced the pneumatically attachable flexible rail, a soft robot that guides surgical tools and intraoperative imaging devices across soft tissues. Here, we present the integration of a curvature sensing optical fibre that relies on Bragg grating sensing into this soft device to determine the location of the utilized tool in regards to the PAF rail.

Optimizing the sensitivity of a tactile sensor to a particular range of stimuli magnitudes usually compromises the sensor’s widespread usage. We present a tactile sensor capable of dynamically tuning its stiffness for enhanced sensitivity across a range of applied forces. Taking inspiration from the Eustachian tube in the mammalian ear, the sensor exploits an adjustable pneumatic back pressure to control the effective stiffness of its 20mm diameter elastomer interface. An internally translocated fluid is coupled to the membrane and optically tracked to measure physical interactions at the interface.

With tunable stiffness and a soft interface, the sensor is intended for comfortable interaction with skin during human-robot interaction. For example, the sensor is undergoing a feasibility study on its use in palpating breast tissue in search of stiff nodes. The sensor is capable of both detecting tactile cues on the surface of hard and soft objects at a super-resolution in the sub-millimeter scale, and can distinguish with 95% accuracy between stimuli with similar stiffnesses (0.181 N/mm difference) over a large range (0.1 to 1.1 N/mm) from only a 0.6 mm deep palpation. This corresponds to a maximum change in contact pressure of 0.5 kPa, which is within the range of what is considered to be comfortable for a physical examination . In its most sensitive regime, the sensor is capable of detecting forces in the region of 0.02 N.

These properties are demonstrated to be relevant for tissue palpation by using the sensor to locate and estimate the size of a synthetic hard node embedded 20 mm deep in a soft silicone sample. The results suggest that the sensor is a good candidate for tactile tasks involving unpredictable or unknown stimuli.

Cystoscopies are an invasive procedure to inspect the bladder by passing a thin camera through the urethra. Despite being a common procedure, they carry complications such as urinary retention due to swelling and urinary tract infections. Improving training of surgeons and using new technologies for skill acquisition, such as medical phantoms, can avoid putting patients at unnecessary risk. In this work, we propose a sensorised soft bladder phantom, which aims to realistically represent the structure of a human bladder when performing cystoscopies. Mechanoreceptive sensors are embedded into a biomimetic and physiologically representative structure. The phantom includes the ability to expand from empty to filled state, and detect and categorise force readings based on their magnitude.

The phantom consists of three main components: an inner bladder (made from latex), an outer bladder (made from silicone) and the sensors. The inner bladder consists of folds and ridges, which represent the mucosa layer of the bladder when empty, whereas the outer bladder represents the bladder when full. The inner bladder expands when filled with water, but is limited by the size and shape of the outer bladder. The outer bladder is placed within a PLA support structure.

The sensors for the phantom are based on previously established methods of translocating fluid to measure displacement with adjustable mechanical stiffness. Each sensor includes ‘pockets’ within its silicone membrane, which deforms when pressure is applied, which translocates fluid within a tube against compressed air back pressure. The displacement of the fluid is tracked by a camera and mapped to the magnitude of force applied.

This work replicates different states of a bladder (empty and full) in one phantom, allowing the user to imitate multiple steps of a cystoscopy. On top of this, the data provided to the user through the phantom’s sensing capability could prove useful when undertaking future cystoscopies.

Robots have shown great promise at performing sophisticated tasks in unpredictable environments where humans are prone to potential hazards. As biological organism, such as animal, utilizes sensing systems to routinely adapt to changing environments for survival, the necessity of robots that can detect danger and self-heal from damage has been recognized. Unlike conventional robots built with rigid joints and linkages that employ algorithm-based adaptation to increase sustainability, soft robots are intrinsically damaged resistant to impacts, owing to the elastomeric materials with intrinsically good toughness and resilience. Advancements of innovative materials, such as self-healing elastomers, have further accelerated the development of damage-tolerant robots that can withstand severe damages, such as cuts or punctures. To apply damage intelligence in soft robotic systems, it is critical to equip those robots with stretchable strain sensors that can provide reliable dynamic sensing performance for feedback control as well as possess tough and self-healable material properties for damage resilience. However, the pursuit of applying self-healing sensors in robotics still remains challenging, and the challenge partially originates from the complexity of the self-healing component itself. Herein, we introduce a simple yet powerful autonomous self-healing optical sensing mechanism with networks of Self-Healing Lightguides for Dynamic Sensing (SHeaLDS). SHeaLDS is a tough, transparent, and autonomously self-healing polyurethane urea elastomer that exploits the intrinsic damage resilience of light propagation in an optical waveguide. The use of stretchable photonic sensing fibers, what we have previously called Light Lace, opens up new and simpler materials and structural requirements. Pristine SHeaLDS has the sensitivity (~1.76 dB cm^-1) over large strain range (g~30%), and, with combined optimization of material and structural design for hyperelastic deformation of the sensor and autonomous self-healing capacity, SHeaLDS exhibits reliable dynamic sensing at larger strains (g~140%) with no drift or hysteresis, puncture resistance, and ability to self-heal from cuts at room temperature with no external intervention. As a demonstration of utility, we introduce damage resilient and intelligent robots that heal cuts as well as detect this damage and adapt their controller weights accordingly. A soft quadruped robot protected by SHeaLDS detects and self-heals from extreme damage (e.g., six cuts on one leg) in one minute and adapts its behaviors (e.g., movement direction) based on the damage condition autonomously through feedback control.

Prosthetics are a valuable tool to improve quality of life of those with Upper Limb Differences (ULDs). However, over 50% of UL prosthetics are rejected, commonly because of discomfort. Discomfort is typically caused by Residual limb (RL) movement within the socket (causing interfacial pressures) and RL volume changes. Current solutions include socket design improvements and sensorised sockets to quantify patient discomfort, but these approaches are not mechanically robust to daily wear. One improvement could be an electronic-skin (e-skin)-based prosthetic liner. E-skins have the potential for human skin-like sensing and flexibility, and this solution would give users and prosthetists a more contextual, in situ understanding of user discomfort. The Neatskin, a soft tactile sensor with salt water-filled channels between electrodes, could be suitable for such prosthetic liners. Discrete Electrical Impedance Tomography (DEIT) and a neural network are used to sense, locate and quantify deformations. This approach provides a low signal to noise ratio and high spatial resolution whilst using a low number of electrodes. However, this has not been tested on larger versions of the design without increasing the number of electrodes (to allow for scalability), and has only been tested with single-point deformations on a flat surface. How adaptable these trained neural networks are to non-flat surfaces, and to objects different to the training surface, is unknown. These tests are crucial to establishing the capabilities of using e-skins for prosthetic health solutions.

This work investigates the effectiveness of larger pressure sensors using DEIT and neural networks for prosthetic applications. We present an e-skin made from silicone, with a salt water-filled channel network in a branch-like design, large enough to wrap around a forearm. Our e-skin uses the same number of electrodes as the fingerprint-sized NeatSkin. A single layer neural network was trained to locate and quantify deformations via linear regression. A 5.5mm diameter probe with an Interlink Electronics FSR400 Force Sensing Resistor attached was used to press into the skin. The probe was attached to a UR5 arm to implement deformations with varying force, angle, and speed. To examine the versatility of trained neural networks for e-skins, our e-skin was trained and tested on different flat and curved surfaces, and also trained on one surface before being tested on others.

Results and findings will be presented at the workshop, proving the potential of e-skin solutions like the NeatSkin for improving prosthetic designs and residual limb health.

During Robotic-assisted partial nephrectomy (RAPN), a drop-in Ultrasound (US) probe is used to identify the resection margins. However, despite facilation by the robot, scanning the kidney proves challenging due to slippage and requires a highly skilled surgeon. In previous work [1], we presented a Pneumatically Attachable Flexible (PAF) rail to enable stable, trackguided US scanning of the kidney during RAPN. In this research, we investigate autonomous control during the US scanning using the PAF rails using fibre-optic shapesensing data as the input for path-planning. In this study, we assess the performance of real-time curvature sensing of the PAF rails with rigid and soft phantoms. We then use shape sensing data to plan the trajectory of a robotguided US probe performing an autonomous scan of a kidney phantom.

The past decade has seen an explosion of novel continuum soft robotic platforms. Being entirely composed of soft materials, makes these robots robust and safe, but at the same time, it renders their modeling [4], control [5], and shape sensing [6] substantially more complex.

The latter is especially complex because it is both a technological and algorithmic challenge. On the one hand, sensors must not obstruct the natural behavior or reduce the compliance of soft robots. On the other hand, non-collocated and nonlinear sensors require algorithms for the measurements to be interpreted and connected to a description of the robot shape.

Several sensing modalities have been considered to implement shape sensing, such as resistive, capacitive, optical, and visual. Magnetic sensors are a promising solution. Magnetic sensors are compact, highly sensitive, and can be easily integrated into existing soft robot designs allowing an integration without interfering with the robot’s softness. The authors of use co-axial pairs of magnets and sensors embedded in the robot to estimate planar and 3D deformations in their work. Such simple arrangements greatly simplify the analysis, allowing for connecting readings to shape through direct interpolation. Nevertheless, relying on isolated pairs also strongly limits the density and the amount of information gathered through this method.

This paper proposes to use permanent magnets and multiple magnetoresistive sensors placed at arbitrary locations within the soft segment. We then present a data-driven approach for making sense of the complex relationship between measurements - so achieving proprioception with magnetic sensors.

First, we train shallow neural networks to predict the measurements of the magnetoresistive sensors from a parameterization describing their relative pose with respect to the magnets. We then optimize the configuration estimate - and thus the sensor positions - to minimize the error between the predicted and actual sensor measurements. The result is the robot configuration estimate that better explains the sensor readings. This way, we introduce a priori information on the modes of deformation of a continuum soft robot, effectively removing the kinematics from the black box.

We provide experiments showing that this architecture requires small training sets to achieve a mean precision of 82.5%when applied to 3D shape proprioception of a soft segment. Leveraging a diverse training set, even an error as low as 7% is possible

Soft robots are flexible and lightweight; therefore, they can adapt to disturbed environments and work cooperatively with humans. To acquire tactile information about a target during the work of the soft robot, a tactile sensor is necessary. In recent years, soft tactile sensors consisting of liquid metal and silicone rubber have been studied in soft robotics owing to their flexibility. In previous studies, the intersections of liquid metal channels arranged in a grid pattern were used as sensing elements. However, this structure limits the placement of the sensing elements to a flat surface. In order to place the sensing elements in non-flat areas such as fingertips of a soft robot, each element must be able to be freely embedded. This study proposes a flexible tactile sensor where channels filled with liquid metal are embedded perpendicular to the silicon rubber surface. Each sensing element is composed of a single channel filled with liquid metal; therefore, each element can be freely arranged independently. The channel is arched, and the diameter is 90 μm, the width of the entire channel is 0.4 mm, therefore it can be placed in a high density. The arch-shaped channels are formed using a mold made of polyethylene filament, which is inexpensive and easy to fabricate. We fabricated a sensor with 4×4 channels embedded at 2.0 mm intervals, imitating the human two-point discrimination threshold. In this sensor, the applied force deforms the silicon rubber, which results in the deformation of the internal liquid metal channels. The length of the channel perpendicular to the sensor surface decreases, and the cross-sectional area increases, according to the applied force. Since the resistance of the channel filled with liquid metal is proportional to the length and inversely proportional to the cross-sectional area, the deformation of the channel decreases the resistance. By measuring the change in resistance, the applied force can be detected. To evaluate the ability to identify shapes in contact with the sensor, we measured the change in resistance when a circular and a cross-shaped indenter was pressed into the sensor. The experimental results show that the resistance of the sensing element in contact with the indenter changes about twice as much as that of the non-contact sensing element and this tactile sensor discriminates the indented shapes.

Large scale robotic skin with tactile sensing ability is emerging with promising potential for use in task- and social-based touch applications in human-robot systems. Nonetheless, to build seamless integration of tactile sensation into a full robot body requires overcoming challenges in modeling, fabrication, processing, and efficiency. Recent developments in vision-based tactile sensing and related learning methods are promising, but they have been mostly designed for small scale use such as by fingers and hands in manipulation tasks. In addition, a recent data-driven method using the Sim2Real approach has been widely exploited for use in data-hungry applications such as touch-based interactions. However, vision-based soft tactile sensing Sim2Real frameworks lack proper physics modeling of soft contact mechanics, which is crucial for efficiently mirroring simulation to reality. In this paper, we introduce a multi-physics simulation pipeline, we call SimTacLS, for realistic rendering of tactile images in a simulation environment, taking into account geometrical and mechanical properties of physical contact, such as local/global soft skin deformation upon interaction with external stimuli. In the simulation environment, the system utilizes two data domains (virtual images and skin deformation) as a labelled dataset to train a tactile deep neural network (TacNet) to extract high-level robotic tactile information. Furthermore, to equip TacNet for real applications, we proposed a generative adversarial network (GAN)-based transformer network (called R2S-TN) to minimize differences between real and virtual tactile images. SimTacLS opens new possibilities in online learning of transferable tactile-driven robotics tasks from virtual worlds to actual scenarios without compromising accuracy

Recent applications of strain sensors such as, electronic skins and human-friendly wearable sensors require the design to achieve wide-sensing range, precise reproducibility, durability, and biocompatibility. The current work proposes a robust liquid-type strain sensor based on glycerol and graphene quantum dots (Gly-GQDs) as sensing units. The fabrication process followed a simple, cost-effective, and scalable approach of molding and casting using Ecoflex with sinusoidal channel consisting of 2 mm diameter. The as-assembled device showed remarkable sensitivity with gauge factor of 10.2 at stretching of 150%, excellent linear response (COD 0.993), minimum memory effect of 1.8%, ability to track input force up to the speed of 5 Hz, the lowest possible limit of detection (1%), and stability (up to 8000 cycles) and complete structural stability towards water contact making it strong candidate for the underwater applications. Furthermore, its biocompatibility was proven using rigorous biological characterizations. Finally, the device was tested in a variety of wearable applications. Its excellent performance enables it to be a strong candidate for wearable application.

In Japan, labor scarcity is becoming more serious due to the declining birth rate and aging population, and automation of factories is strongly required, especially in food industries a demand for automation of food packing and sorting. Various grippers for food handling are proposed. During grasping and manipulation, grasped objects should be identified; in addition, states of grasping and manipulation, such as success or failure, should be recognized. Employing sensors, it is possible to measure bending, strain, contact and proximity. It is noted that, if many sensors are mounted on a gripper, the gripper increases in size, may become less soft and an exorbitant amount of wiring and measuring devices are required.

In this study, a novel method is proposed capable of measuring strain and proximity by a single capacitance-based sensor. The method can also determine the grasping state and the type of object grasped by the soft gripper based on the measured strain and proximity. The single soft capacitance-based sensor is installed on the surface of a soft gripper. The sensor is connected to an LCR meter. High-frequency AC and low-frequency AC are applied alternating to simultaneously measure strain and proximity by a single sensor. The strain of the sensor is estimated from the capacitance measured at low frequency, and this strain was used to estimate the expansion level of the gripper chamber. The proximity of an object with high permittivity is estimated from the capacitance measured at high-frequency, and this proximity was used to determine whether the grasped object has high permittivity or not. First, we identified the appropriate frequencies to measure strain and proximity. Next, the method was proposed to estimate whether the gripper succeeded to grasp a particular object from the measured values of strain. In addition, the method was proposed to determine whether the grasped object is the target object based on proximity measurements. In this research, the target object is assumed as a cucumber, and it is determined whether the grasped object is the cucumber. Finally, the proposed method was experimentally validated. As the results show our method succeeded in appropriately determining all three states: (1) the state where the soft gripper grasped the cucumber, (2) the state where an object other than the cucumber grasped, and (3) the state where a gripper failed to grasp an object.

This study focuses on effects of structure differences on soft tactile sensor output. The sensor's sensitivity will change when a part of the structure such as a silicone membrane is hurt or damaged, and the sensor output will alter even if the identical input is applied. In that case, active sensitivity adjustment must be achieved to let the sensor correctly work. To achieve the active sensitivity adjustment, we must investigate how the structural differences alter the sensor output. In this presentation, we present the two kinds of structure of our tactile sensors and experimental data.

In our study, we fabricated two tactile sensors. First, we explain their common features. Both have three chambers filled with air in its body. The three chambers are covered with a thin and soft silicone rubber membrane, in which strain gauges are embedded. When an indenter pushes or rubs the membrane, the strain of the strain gauges can be measured. When the air pressure increases, the soft membrane inflates, changing its sensitivity.

For example, We fabricated two types of tactile sensors. Each chamber of Type A is divided by a diaphragm, and not connected, thus each chamber's air pressure can be controlled independently. In the other model (Type B), the diaphragms between the chambers have a hall, connecting all three chambers, thus we cannot control the inner air pressure of each chamber independently. Therefore, the structure difference is the diaphragm's hall in this presentation.

In the experimental setup, an indenter was fixed on a linear stage and could be moved vertically. The tactile sensors were also set on another linear stage, which was moved horizontally. The height of the indenter was set so that it rubs the surface of the silicone rubber membrane. We measured the strain by the strain gauges under different air pressure conditions. As an experimental result, we found that the time series data of strain in the two sensors are different from each other, which is caused by the structure difference. Also, we found that the sensor sensitivity changes when the air pressure changes. Future work includes building a method to compensate the structure difference and numerical analysis of our proposed sensor.

In this presentation, we report the design and experimental results of a flexible tactile sensor that utilizes the phase change of gallium. Taking advantage of the low melting point of gallium, which is about 29.8 ° C, we have developed a flexible tactile sensor that can change its sensitivity. In our tactile sensor, Gallium was sandwiched between the silicon membrane and the silicon base. In the liquid state of gallium, deformation due to external force applied to the membrane is not directly transmitted to the base. Conversely, in the solid state, the deformation is transmitted directly to the base. Thus, we expect that the base's strain in the liquid state will be different from that in the solid state even if the identical forced displacement is applied.

Two strain gauges and a thermistor were embedded inside the base to measure the base's strain and temperature. Particularly, the two strain gauges were placed perpendicularly to enable us to distinguish the direction of the external force induced by the indenter. The gallium can be heated and cooled by Peltier device attached to the tactile sensor.

As an experimental setup, we fixed a tactile sensor and an indenter on linear stages. The indenter on the linear stage was moved to press the tactile sensor, and its moving direction can be relatively changed by rotating the tactile sensor.

We compared the strain date in eight different directions from which the indenter was pushed in both the gallium's liquid and solid state. From the experimental data, we found that the sensitivity can be changed by changing gallium's phase. However, we concluded that it is difficult to completely distinguish the direction of the stimulus only from the strain output, because there are similar strain patterns in some directions.

Then, we investigated the sensitivity change due to the phase change. We recorded data during the heating and cooling periods. We found that the sensitivity change abruptly occurred around the melting point, which means that it is difficult to continuously change the sensitivity. Future works include investigating other liquids such as Magneto Rheological liquid.

Current robotic systems lack the resilience that we can observe in nature. If part of the robot breaks, often the whole system fails. While some approaches have been proposed, the majority of them are focusing on updating the control policy. Unfortunately, this process is often quite complex and sometimes not even able to counteract the damage at all. On the side, biological systems are remarkably resilient. They use morphological features to deal with damage (often referred to as morphological computation). A particularly impressive example is whiskers. In our previous works, we introduced an artificial whisker sensor that exhibited resilience against physical damage by allowing its morphology to adapt. This work aims to see whether or not this idea could enable other intelligent sensing abilities such as texture discrimination tasks. More specifically, changing the morphology of the whisker helps to identify mismatching between prior knowledge in the magnitude spectrum (probably in frequency bandwidth) of the sensor signal (strain gauge) and, therefore, to classify textures with the perceived ones. This allows the sensor to recover the tactile perception on texture discrimination after the whisker is physically damaged without the need of computationally expensive re-classification. This work is expected to shed a light on a new generation of robots that automatically work in the open world where self-maintenance against uncertainties is needed.

Artificial tactile sensing allows a system to be aware of its surroundings and react to its changes. Optical-based soft artificial skins are a promising approach to achieve advanced sensing capabilities. In particular, when a waveguide is a soft optically transparent material, the light emitted by a photoemitter (PE) is transmitted through the soft medium, and deformations in the substrate cause variations of detected light intensity read by a photoreceiver (PR). This principle has been adopted for several single-point soft sensors, also in soft robots. Indeed, from a transduction viewpoint, electromagnetic noise does not influence light, thus allowing its use in the harshest environments. We present the results

and ongoing work of our group in this area, as we believe that the roadmap for optical skin perceptive robots is just at its beginnings and many challenges are still open. For example, one aspect concerns sensorizing large areas in soft robots, where we see two possible approaches, i.e., creating large patches, or standalone modules that can be smoothly combined.