My research interests are in the fields of Haptic Perception, Tactile Sensing, Machine Learning, Manipulation, and Human-Robot Interaction. I am particularly interested in the domain of assistive robotics. I have also actively worked on Teleoperation Systems, Control Systems, and Psychophysics.

1. Food Manipulation for Assistive Feeding

Eating is an activity of daily living (ADL) and losing the ability to self-feed can be devastating. Through this project, we are developing algorithms and technologies towards a robotic system that can autonomously feed people with upper-extremity mobility impairments.

Eating free-form food is one of the most intricate manipulation tasks we perform in our daily lives, demanding robust nonprehensile manipulation of a deformable hard-to-model target. Automating food manipulation is daunting as the universe of foods, cutlery, and human strategies is massive. To understand how humans manipulate food items during feeding and to explore ways to adapt their strategies to robots, we collected human trajectories by asking them to pick up food and feed it to a mannequin. From the analysis of the collected haptic and motion signals, we demonstrate that humans adapt their control policies to accommodate to the compliance and shape of the food item being acquired. We develop a taxonomy of manipulation strategies for feeding to highlight such policies and propose a set of haptic classifiers to generate policies based on a class of food items. Our analysis of success and failure cases of human and robot policies further highlights the importance of adapting the policy to the food item.

Successful robotic assistive feeding depends on reliable bite acquisition and easy bite transfer. The latter constitutes a unique type of robot-human handover where the human needs to use the mouth. This places a high burden on the robot to make the transfer easy. We believe that the ease of transfer not only depends on the transfer action but also is tightly coupled with the way a food item was acquired in the first place. To determine the factors influencing good bite transfer, we designed both skewering and transfer primitives and developed a robotic feeding system that uses these manipulation primitives to feed people autonomously. Our experiments and analysis show that an intelligent food item dependent skewering strategy improves the bite acquisition success rate and that the choice of skewering location and fork orientation improves the ease of bite transfer significantly.

More details here.

Some relevant videos below:

  • This video shows an autonomous robotic feeding system that demonstrates the integration and use of perception using multiple complementary modalities of computer vision and tactile sensing, perception, learning, motion planning, and robot control in order to feed people with upper-extremity mobility impairments. Our robotic system uses fine manipulation planning and leverages the complementary capabilities of multiple sensing modalities such as vision and haptics to automatically acquire bites for a variety of food. items. Our system can use different features from the plate to visually identify the food type which enables a user to gain the ability to self-select the next bite. Also, our system can decide which discrete manipulation primitive to instantiate such as a food item dependent skewering approach angle as well as select its continuous parametrization such as skewering position, rotation and how much force to apply. The robot uses visual servoing to not only skewer a food item but also feed a user by tracking their mouth.
  • This video shows highlights of some of the interesting manipulation strategies that people use for manipulating food for feeding a target. It also shows how the arm trajectories and haptic signals vary for these various manipulation strategies.
  • This video shows that bite transfer and bite acquisition are not independent and that how a robot chooses to skewer an item during bite acquisition may affect how it transfers the item to a person for feeding later. We developed an SP-Net that finds out the best position and rotation to skewer an item such that it can successfully acquire the bite and eventually transfer it to a human user easily. Our results show that a skewering strategy based on a food item’s shape, size and physical properties outperforms the baseline approach of skewering at the center in terms of the bite acquisition success rate, especially for long, slippery, and heterogeneous food items. Our human participant studies show that transfer depends on acquisition. Angled skewering combined with angled transfer performed significantly better than vertical skewering combined with horizontal transfer for easy bite transfer. Also, people tend to avoid hitting the tines of the fork while biting a long and slender carrot and thus, where a robot skewers an item can affect the ease of bite transfer as well.

2. Rapid Haptic Perception using Force and Thermal Sensing

Tactile sensing can enable a robot to infer properties of its surroundings. Recent research has focused on robots that haptically perceive the world through exploratory behaviors that occur over tens of seconds. During manipulation, many opportunities arise for robots to gather information about the environment from brief (<= 2 seconds) contact due to simple motions (e.g., linear). The goal of our work was to enable robots to infer haptic properties under these conditions using force and thermal sensing.

We used a data-driven approach with various machine learning methods. Key challenges were obtaining adequate haptic data for training and developing methods that performed well on haptic data that differed from the training data due to common real-world phenomena. For example, haptic sensory signals vary significantly due to the robot, including its velocity, stiffness, and sensor temperature.

To collect suitable data, we used a variety of platforms, including simplified robots, handheld human-operated devices, and a mobile robot. We also generated synthetic data with physics-based models. Through careful empirical evaluation, we identified machine learning methods that better handled common signal variations. We also used physics-based models to characterize common perceptual ambiguities and predict the performance of data-driven methods. Overall, our research demonstrates the feasibility of robots inferring haptic properties from brief contact with objects in human environments. By using force and thermal sensing, our methods rapidly recognized materials, detected when objects moved, detected contact with people, and inferred other properties of the robot’s surroundings.

Link to Ph. D. Dissertation coming soon!

Some relevant videos below:

  • The video shows real-time sparse haptic map generation using whole-arm tactile skin while the robot is reaching into a cluttered environment and grabbing a bunch of keys. Initially, the robot has no idea about the environment and uses model-predictive control to reach its goal while maintaining the contact forces below a certain threshold. However, during the initial attempt to reach, it comes in contact with trunks and rigid objects in the environment which it identifies using HMMs and maps using brown dots (right). After the initial attempt fails, the robot updates its knowledge of the environment by identifying the location of the rigid objects in the environment like trunks using haptic mapping. Using this updated knowledge in its next turn, the robot successfully avoids them using a motion planner to successfully reach the goal and grab the keys.
  • The video is a collection of slides of my talk at RSS 2015, on heat-transfer based recognition of materials for short-duration contact and varying initial conditions.
  • The video shows the performance of a stretchable fabric-based force and thermal sensing skin for haptic perception during common non-prehensile manipulation tasks.
  • The video shows a mobile robot with a linear actuator touching various objects relevant to ADLs (activities of daily living) and IADLs (Intrumental activities of daily living) in the bathroom, bedroom, and kitchen of a household. There is a multimodal sensor attached at the end of the linear actuator which collects various multimodal (force, thermal) signals during contact with the object. The robot also records the kinematic motion signals using an encoder at the other end of the linear actuator. The video shows some examples of objects when the robot collected the data at varying velocities and times of the day over a period of three days. The experiments were performed on the objects as they were found (in-situ). These signals were used for multimodal tactile perception of objects in the home.
  • The video shows overhead view of our robot Cody with a stretchable, flexible tactile sensor array on its forearm and end-effector. The robot reaches through instrumented (with force-torque sensors for ground truth of contact force) clutter to a pre-specified goal location. Using the tactile sensing, including sensing that covers the articulated joints, the robot is successful. Whereas, without tactile sensing the robot fails to reach the goal.

  • The video shows the rapid categorization performance using HMMs while the robot reaches into clutter made of trunks and leaves. The robot uses data from the forearm tactile skin for online categorization. A taxel (tactile-pixel) is marked as a green dot if it is categorized as a leaf and as a brown dot if it is categorized as a trunk.

  • The video shows the performance of online Haptic Classification using information from incidental contact during goal-directed motion. The classification uses features such as Maximum Force, Contact Area, and Contact Motion for Classification purposes. These features are obtained from the Artificial Skin attached on the forearm of the Robot 'Cody'. The right side of the video shows an image representation of the unrolled skin Taxel array wherein darker pixels imply higher forces.

  • The video shows the performance of a PR2-robot equipped with a fabric-based tactile skin during goal-directed motion in unknown cluttered environments using online Haptic Classification and a bidirectional RRT planner. The RRT planner initially has no knowledge of the environment. During the motion, if the robot comes in contact with an obstacle and if the classification algorithm determines it to be a fixed obstacle, the RRT goes back to the initial position and re-plans with the updated knowledge of the environment. The robot can thus create a haptic map of the environment while reaching the goal. The RRT planner does not care if the obstacle is movable. I have included two cases in the above video to show the performance of PR2 with the same goal but different starting positions.

3. Combining Tactile Sensing and Vision for Haptic Mapping

We consider the problem of enabling a robot to efficiently obtain a dense haptic map of its visible surroundings using the complementary properties of vision and tactile sensing. Our approach assumes that visible surfaces that look similar to one another are likely to have similar haptic properties.

Some relevant videos below:

  • The robot 'DARCI' uses a tactile sleeve and a Kinect to create a dense haptic map while reaching into a cluttered environment. As the robot comes in incidental contact with the objects in the environment, it acquires local haptic information using the sleeve and propagates the local information to update its estimate of the haptic properties of the visible surface using the Kinect.
  • The video summarizes the basic idea, i.e. combining vision and touch using a dense conditional random field (CRF). It also shows a short clip of the robot DARCI generating a dense haptic map while reaching into a cluttered evironment. Our method combines the data from a tactile-sensing sleeve and a Kinect using the dense CRF.

4. Antagonistic Muscle based Robot Control for Physical Interactions

Robots are ever more present in human environments and effective physical human-robot interactions are essential to many applications. But to a person, these interactions rarely feel biological or equivalent to a human-human interactions. It is our goal to make robots feel more human-like, in the hopes of allowing more natural human-robot interactions. In this work, we examine a novel biologically-inspired control method, emulating antagonistic muscle pairs based on a nonlinear Hill model. The controller captures the muscle properties and dynamics and is driven solely by muscle activation levels. A human-robot experiment compares this approach to PD and PID controllers with equivalent impedances as well as to direct human-human interactions. The results show the promise of driving motors like muscles and allowing users to experience robots much like humans.

5. Haptic interaction during partner dance-based exercise for physical assistance

In this work, our long-term goal is to enable a robot to engage in partner dance for use in rehabilitation therapy, assessment, diagnosis, and scientific investigations of two-person whole-body motor coordination. Partner dance has been shown to improve balance and gait in people with Parkinson's disease and in older adults, which motivates our work. During partner dance, dance couples rely heavily on haptic interaction to convey motor intent such as speed and direction. We conducted human-subject studies to evaluate the feasibility of a wheeled mobile robot as a partnered stepping platform and to analyze its acceptance among older adults for physical assistance.

Some relevant videos below:

  • Here, we investigate the potential for a wheeled mobile robot with a human-like upper-body to perform partnered stepping with people based on the forces applied to its end effectors. Blindfolded expert dancers performed a forward/backward walking step to a recorded drum beat while holding the robot's end effectors. We varied the admittance gain of the robot's mobile base controller and the stiffness of the robot's arms. High admittance gain and high arm stiffness conditions resulted in significantly improved performance with respect to subjective and objective measures. Biomechanical measures such as the human hand to human sternum distance, center-of-mass of leader to center-of-mass of follower (CoM-CoM) distance, and interaction forces correlated with the expert dancers' subjective ratings of their interactions with the robot. In response to a final questionnaire, 1/10 expert dancers strongly agreed, 5/10 agreed, and 1/10 disagreed with the statement "The robot was a good follower." 2/10 strongly agreed, 3/10 agreed, and 2/10 disagreed with the statement "The robot was fun to dance with." The remaining participants were neutral with respect to these two questions.
  • To potentially facilitate healthy aging by engaging older adults in partner dance-based exercise, older adults would need to be accepting of partner dancing with a robot. Using methods from the technology acceptance literature, we conducted a study with healthy older adults to investigate their acceptance of robots for partner dance-based exercise. Participants were generally accepting of the robot for partner dance-based exercise, tending to perceive it as useful, easy to use, and enjoyable. Through a qualitative data analysis of structured interview data, we also identified facilitators and barriers to acceptance of robots for partner dance-based exercise. Throughout the study, our robot used admittance control to successfully dance with older adults, demonstrating the feasibility of this method. Overall, our results suggest that robots could successfully engage older adults in partner dance-based exercise.

6. Bio-inspired Robot Arm and Hand Coordinated Grasping/Manipulation Control

We developed a novel control law to exhibit human-motion characteristics in redundant robot arm systems for reaching tasks. This newly developed method nullifies the need for the computation of pseudo-inverse of Jacobian while the formulation and optimization of any artificial performance index is not necessary. The proposed control law models the time-varying properties of the muscle stiffness and damping as well as the low-pass filter characteristics of human muscles. The newly developed control law uses a time-varying damping shaping matrix and a bijective joint muscle mapping function to describe the human motion characteristics for reaching motion like quasi-straight line trajectory of the end-effector and symmetric bell shaped velocity profile. The aspect of self-motion and repeatability, which are inherent in human-motion, are also analyzed and successfully modeled using the proposed method. Experiment results show the efficacy of the newly developed algorithm in describing the human-motion characteristics. We performed extensive simulations by extending the above control law for hand-arm coordination tasks in reach-to-grasp tasks for grasping objects of different shapes and sizes.

Some relevant videos below:

  • The video shows the performance of the a Hand-Arm system while coordinating to reach and grasp objects of different shapes. The Arm is a 7-DOF redundant system while the hand is a 4-fingered 12-DOF hand. The reach-to-grasp task is carried out using my newly developed control law as described here. The simulation is carried out using RoboticsLab Software.
  • The video shows the performance of the same 7-DOF Robot Arm in executing human-like reaching motion with compliance. It exhibits quasi-straight line trajectory of the end-effector and a symmetric bell-shaped velocity profile.

  • The video shows the performance of a 7-DOF redundant robot-arm system in maintaining the end-effector position while external disturbances (force correspond to red lines in video: force applied using mouse) perturb the robot's motion. I implemented a Task-Space Disturbance Observer to analyze its effect on the Null-Space Motion of the 7-DOF Robot Arm.

  • The video shows the system characterization of a 7-DOF Robot Arm in KIST, South-Korea. The performance of the arm is compared between cases without compensation and with friction and gravity compensation using my method.

7. Analysis of Stability and Performance of Telesurgical Systems

Telerobotic surgical systems, although researched quite extensively, still have stability and performance issues. This is mainly because little consideration has been given to the effect of hardware or environment characteristics as well as practical surgical constraints in previous stability and performance analyses. This work focuses on the stability aspect in greater depth by considering the effect of surgical soft-tissue environments and the damping of the master device and hence, employs passivity approach which relates the haptic device damping with the environment characteristics. There have been a few models proposed to describe the visco-elastic behavior of soft tissues, including the popular Maxwell and Voigt models. Choosing a best soft-tissue model is still unresolved. This research provides passivity analysis of teleoperation of soft tissues, based on the Kelvin model that can overcome the disadvantages of the Maxwell or Voigt models. The passivity analysis results in new criterion for design of the haptic interface as well as the control. The effect of discretization method on the passivity criteria is also discussed. Simulation results show that this newly developed criterion increases the domain of environments that can be passively interacted by the haptic master. This newly developed criterion is, therefore, a less conservative criteria than that developed based on the popular Voigt environment model, which signifies that less device damping is required to maintain the passivity of the teleoperation system which would in turn increase its performance.

Another important issue in the performance of telesurgical systems is the practical constraint of force feedback because it is difficult to use force sensors in surgical environments. This is because of the inherent noise, and limitations in the sensor-deployable place and space. A scheme to replace the direct use of force sensors in the slave side may alleviate this problem. This work, hence, develops a method for environment force estimation using the concept of disturbance observer. Simulation results show efficacy of the proposed method. The slave device successfully tracks the position of the master device, and the estimation error quickly becomes negligible. The experimental results validate the theory and simulation outcome.

Link to M.S. Thesis