Research
Figure 1. Overview of my research topics
Topic A:
Control of Teleoperation Systems for Beating Heart Surgery
Beating-heart surgery enables intraoperative assessment of the heart and could eliminate negative effects such as the risk of long-term cognitive loss and stroke that arrested-heart operations often suffer from, therefore is a promising alternative to arrested-heart surgery.
However, operating on a beating heart is extremely challenging due to the heart’s fast and quasi-periodic motions (the heart movement velocity and acceleration are approximately 210 mm/s and 3800 mm/s2, respectively). Manual tool position compensation according to the heart motion will not only lead to the human operator’s fatigue and exhaustion but also increase the risks of tool-tissue collision and tissue injury.
To facilitate beating-heart surgery and minimize the risks of tool-heart collision and tissue injury, I developed a series of telerobotic control systems to simultaneously compensate for the beating heart’s motion and provide the human operator with non-oscillatory haptic feedback, to allow him/her to operate on a “stationary” heart. Figure 2 shows the objectives of the proposed systems.
Research Objectives
Figure 2. Objectives of the teleoperation systems for beating-heart surgery
I. Impedance-control-based motion compensation and haptic feedback
A bilateral impedance-controlled telerobotic system is proposed for beating-heart surgery. The method only uses the measured interaction forces without any need for vision-based heart motion estimation, active observer, or motion prediction to compensate for the heart motion and provide steady force feedback. Specifically, two reference impedance models and two nonlinear bilateral adaptive impedance controllers were proposed for the master and slave robots respectively (shown in Figure 3).
Figure 3. Bilateral impedance-controlled telerobotic system for beating-heart surgery. (a) The telerobotic system. (b) Reference impedance models for master and slave robots. (c) Nonlinear bilateral adaptive impedance controllers for the master and slave robots.
I-(a): 1-DOF System Evaluation
Figure 4 shows the experimental setup for the 1-DOF task which was performing beating-heart anchor deployment for mitral valve annuloplasty under the guidance of US images. The experimental results show that the proposed system reduced the excess force application rate by 70% and increased the success rate of anchor deployment by 100% compared to manual attempts.
Figure 4. Experimental setup for 1-DOF anchor deployment
I-(b): 3-DOF System Evaluation
Figure 5 shows the 3-DOF physiological motion compensation setup. The experimental evaluations demonstrated that the proposed method could be used in robots with significant dynamics and achieve accurate performance for surgical applications that need low and constant contact forces during beating-heart interventions. It should be noted that due to the flexibility of the designed reference impedance model for the slave robot, the bilateral impedance control method is more suitable for surgeries that require less tool-tissue interaction forces such as mitral valve annuloplasty, blunt resection, ablation, etc.
Figure 5. Experimental setup for 3-DOF task
Related publications:
L. Cheng, M. Sharifi, M. Tavakoli, “Towards Robot-Assisted Anchor Deployment in Beating-Heart Mitral Valve Surgery,” The International Journal of Medical Robotics and Computer Assisted Surgery, vol. 14, no. 3, pp. e1900, 2018. Also, selected for poster presentation at the International Symposium on Medical Robotics, Atlanta, GA, 2018.
L. Cheng and M. Tavakoli, “Switched-Impedance Control of Surgical Robots in Teleoperated Beating-Heart Surgery,” Journal of Medical Robotics Research, vol. 3, no. 03n04, pp. 1841003, 2018.
L. Cheng, J. Fong, M. Tavakoli, “Semi-Autonomous Surgical Robot Control for Beating-Heart Surgery,” IEEE 15th International Conference on Automation Science and Engineering, Vancouver, Canada, 2019.
II. Image-based position and impedance hybrid control approaches
Figure 6. System concept of the proposal
When there is no contact between the surgical tool and the heart tissue, the impedance-control-based motion compensation method will not be applicable as it needs the measured tool-tissue interaction forces. Therefore, I proposed the image-based position and impedance hybrid control approach to achieve motion compensation and non-oscillatory haptic feedback by designing an ultrasound (US) image-based position controller for the slave robot and an impedance controller for the master robot (Figure 6). The US imaging was used to provide visual feedback and measure the beating heart's position. The impedance model for the master robot is used to provide the human operator with non-oscillatory force feedback during tool-tissue interaction.
The presence of US imaging introduced a non-negligible time delay to the system due to image acquisition and processing. To address this issue, an extended Kalman filter (EKF)- and a neural network (NN)-based heart motion predictors were designed.
II-(a): Extended Kalman Filter-Based Motion Predictor
The control strategy took advantage of the quasi-periodicity of the point of interest (POI) position, modeled the delayed POI position as a time-varying Fourier series, and predicted the POI position by EKF to compensate for the time delay. Video 1. shows the system's experimental performance.
Related Publications:
L. Cheng and M. Tavakoli, “Ultrasound Image Guidance and Robot Impedance Control for Beating-Heart Surgery,” Control Engineering Practice (A Journal of IFAC), vol. 81, pp. 9-17, 2018.
Video 1. System experimental performance
II-(b): Neural Network-Based Motion Predictor
To investigate whether the motion prediction accuracy can be improved or not, a NN-based heart motion prediction method was proposed. It has been demonstrated that a NN model can approximate any continuous function and it has been successfully used for forecasting of many time series in many applications. Also, NN has the advantage that it can approximate nonlinear functions without any prior information on the data series, which makes it suitable for the application of quasi-periodic beating-heart motion prediction. Figure 7 shows the schematic of the proposed steps for heart motion prediction. To prove the concept, an experimental setup was developed (Figure 8), and the experimental results were compared to the results of the EKF-based motion predictor system and the no-motion predictor system (Figure 9). It can be concluded that the NN predictor works better than no predictor or EKF predictor.
Figure 7. Schematic of the NN-based heart motion prediction
Figure 8. Experimental setup for the NN-based heart motion prediction system
Figure 9. Experimental results of the systems with no motion predictor, EKF predictor, and NN predictor. MASE is the abbreviation of Mean Absolute Synchronization Error. AFM is the abbreviation of the Average Forces applied by the human operator on the Master robot. AFS is the abbreviation of the Average Forces applied by the Slave robot on the simulated heart.
Related Publications:
L. Cheng and M. Tavakoli, “Neural-Network-Based Heart Motion Prediction for Ultrasound- Guided Beating-Heart Surgery,” IEEE 15th International Conference on Automation Science and Engineering, Vancouver, Canada, 2019.
L. Cheng and M. Tavakoli, “Neural Network-based Physiological Organ Prediction and Robot Impedance Control for Teleoperation Beating-Heart Surgery,” Biomedical Signal Processing and Control, doi.org/10.1016/j.bspc. 2021.102423, 2021.
III. Surgical training and cooperation for multi-surgical robots
To achieve haptics-enabled surgical training and cooperation in beating-heart surgery, I developed an impedance-controlled multi-master/single-slave telerobotic system by designing a multi-user shared control architecture and a multilateral impedance-controlled strategy for this architecture (Figure 10). The desired objectives of the proposed system include:
providing position guidance to the trainees during the training procedure
providing force feedback to all human operators (trainer and trainees) regardless of their levels of authority over the slave robot
motion compensation for the heart’s motion
reflecting only the non-oscillatory force portion of the slave-heart tissue interaction force to all human operators
To this end, virtual fixtures and a dominance factor were introduced, and a reference impedance model with adjusted parameters was designed for each master or slave robot (Figure 11). The proposed impedance-based control methodology was evaluated experimentally and shown in videos 2 and 3.
Figure 10. The multilateral teleoperation system for beating-heart surgical training and cooperation
Figure 11. Parameter adjustments for the reference impedance models
Related Publications:
L. Cheng and M. Tavakoli, “A Multilateral Impedance-Controlled System for Haptics-Enabled Surgical Training and Cooperation in Beating-Heart Surgery,” International Journal of Intelligent Robotics and Applications, vol. 3, no. 3, pp. 314-325, 2019.
L. Cheng and M. Tavakoli, “COVID-19 Pandemic Spurs Medical Telerobotic Systems: A Survey of Applications Requiring Physiological Organ Motion Compensation,” Frontiers in Robotics and AI, doi:10.3389/frobt.2020.594673, 2020. (Invited paper)
Video 2. Training tasks performed by a multilateral teleoperation system
Video 3. Cooperation tasks performed by a multilateral teleoperation system
Topic B:
AR-Guided Robotic Assistant for Surgeries
Visualization remains one of the major challenges in minimally invasive procedures because the surgeon's vision is narrowed by the small openings from the patient.
Augmented Reality (AR) has emerged as a transformative technology in the field of robotic surgery, by overlaying virtual information onto the surgeon's view of the patient's anatomy, providing real-time visual guidance during surgery. This includes displaying anatomical structures, critical landmarks, and diagnostic imaging directly within the surgeon's field of view, allowing for better spatial understanding and precision in complex procedures.
However, applying AR in robotic surgeries introduces several challenges including accurate registration and alignment of virtual overlays with the patient's anatomy, maintaining real-time synchronization between AR visualizations and the surgical field, potential disruptions in the surgeon's workflow due to the need to shift focus between the AR display and the patient. My research is going to breakthrough those barriers.
I. Robot-Assisted Fibula Osteotomies in Mandible Reconstruction Surgery
When a tumor approaches or invades a patient’s mandible, a mandibulectomy surgery to remove all or part of the mandible and the tumor tissue around it may be performed. Specifically, for segmental mandibulectomy, where an entire segment of the mandible is removed, a bony autograft is necessary so as to reconstruct the length and the form of the mandible to produce the best functional and aesthetic results. Fibula free-flap reconstruction is used widely by taking a fibula bone segment, soft tissue from the calf, and the fibular vascular pedicle. The fibula bone segment should be shaped to match, as closely as possible, the piece of the mandible that was removed. The main difficulty of this segmentation is to be able to make the fibula bone which is straight into a curved bone piece so as to match the mandibular resection.
To provide guidance to the surgeon during fibula segmentation according to the reconstruction surgical plan and to improve the fibula bone cutting accuracy, an admittance-controlled robotic assistant incorporating 3D AR visualization and haptic virtual fixtures (VF) was proposed (Figure 12). The admittance controller was used to reduce the surgeon’s hand tremor. The VF and AR were used to provide haptic and visual guidance to the surgeon respectively.
Related Publications:
L. Cheng, J. Carriere, J. Piwowarczyk, D. Aalto, N. Zemiti, M. de Boutray, and M. Tavakoli, “Admittance-Controlled Robotic Assistant for Fibula Osteotomies in Mandible Reconstruction Surgery,” Advanced Intelligent Systems, 2020, 2000158 (1-8).
Video 4. AR guided robot-assisted fibula osteotomies
II. AR-Guided Dental Training
In December 2019, an outbreak of novel coronavirus pneumonia occurred and subsequently attracted worldwide attention when it bloomed into the COVID-19 pandemic. To limit the spread and transmission of the novel coronavirus, governments, regulatory bodies, and health authorities across the globe strongly enforced shut down of educational institutions including medical and dental schools. The adverse effects of COVID-19 on dental education have been tremendous, including difficulties in the delivery of practical courses such as restorative dentistry.
As a solution to help dental schools adapt to the pandemic, I worked with Tactile Robotics Ltd, Canada, and developed a compact and portable teaching-learning platform that can facilitate remote and distancing-aware teaching and learning in dentistry. The platform complements traditional methods by seamlessly integrating and synchronizing 3D AR video, audio, feel, and posture, and can bring orientation between the instructor and the students in real-time by using the designed DT-Rightway articulator. Most importantly, as the platform is packaged in a small portable suitcase, it can be used anywhere by connecting to the cloud-based data storage network to retrieve procedures and performance metrics.
Related Publications:
L. Cheng, A. Maddahi, M. Kalvandi, Y. Maddahi, and M. Tavakoli, “Application of DenTeach in Remote Dentistry Teaching and Learning during the COVID-19 Pandemic: A Case Study,” Frontiers in Robotics and AI, doi: 10.3389/frobt.2020.611424, 2021.
Summary of my research
My research endeavors to push the boundaries of Control and Automation for Medical Robots, with a focal point on Teleoperation, Haptics, Non-linear Control, and Medical Imaging. By amalgamating these domains, the ultimate aim is to empower medical professionals with advanced tools that amplify their skills, improve patient care, and drive the evolution of medical robots into a new era of efficacy.
Although within the realm of healthcare in the past, my research underscores the universality of control and automation principles. The methodologies and insights generated from my previous studies can be seamlessly translated to a multitude of applications, such as intelligent manufacturing, industry 4.0, electric transportation, and unmanned aerial vehicles, which I am enthusiastic to explore and interested in.
