Title: Reinforcement Learning with Approximated Rewards for Signal Temporal Logic Specifications
Keywords: motion planning, formal specifications, reinforcement learning
Students: Job van der Werf
Supervisors: Dr. Zengjie Zhang and Dr. Sofie Haesaert
Defense Date: 27.05.2024
Affiliation: Eindhoven University of Technology, Eindhoven, Netherlands
Abstract: The goal of this paper is to create a structured method to generate reward functions for reinforcement learning from STL specifications. Three papers that are related to this goal have been analyzed and recreated in the same simulated environment for comparison. Three ideas for using STL robustness in creating a reward function have been tested and compared to a handcrafted reward function. None of the methods can outperform the handcrafted reward function given the same number of training steps.
Title: Generating Temporal Logic Specifications for Autonomous Navigation using ChatGPT
Keywords: motion planning, large language model, prompt engineering
Students: Chenyuan Qian
Supervisors: Dr. Zengjie Zhang and Dr. Zhiyong Sun
Defense Date: 08.11.2023
Affiliation: Eindhoven University of Technology, Eindhoven, Netherlands
Abstract: This report primarily introduces a method for using ChatGPT to generate Signal Temporal Logic specifications for autonomous navigation. The report introduces a method to calculate differences between various specifications by converting STL specifications into tree structures. Subsequently, the paper proposes a ChatGPT-based approach for generating STL specifications. By inputting natural language descriptions of navigation requirements, ChatGPT can give executable STL specifications and extract information about the targets and obstacles. Finally, several generated scenarios are provided.
Title: An Analysis of Predator Influence on Flocking Behaviour
Keywords: behavior modeling, multi-agent systems, flocking
Students: Noortje Hagelaars
Supervisors: Dr. Zengjie Zhang and Dr. Zhiyong Sun
Defense Date: 26.04.2023
Affiliation: Eindhoven University of Technology, Eindhoven, Netherlands
Abstract: This report considers the complex nature of flocking dynamics, in particular regarding prey-predator interactions. To get a deeper understanding on the progression of these interactions, a simple predator is added into the existing Flock2orientation-based flocking model mirroring starling murmurations, to analyse flock splitting induced by a predator. In the simulation, the predator continuously attacks the flock centre, which leads to repeatedly splitting and merging of the flock. The behavioural display of flock splitting is analysed by clustering the subflocks through k-means and graph clustering approaches. Although the results show that the predator indeed influences the flocking dynamics by creating more and smaller subflocks, further insights can be achieved by creating a more realistic predator model and optimising the clustering approach.
Title: Generating Weather-Specific Features Toward Perception Error Models for Automated Driving
Keywords: feature generation, autonomous driving, perception error modeling
Students: Shao-Hsuan Hung
Supervisors: Dr. Zengjie Zhang and Dr. Zhiyong Sun
Defense Date: 05.12.2023
Affiliation: Eindhoven University of Technology, Eindhoven, Netherlands
Abstract: This project implements a procedure that generates perception error models (PEMs) under various weather conditions for the future study of autonomous vehicles (AVs) behaviors in various conditions. The machine learning-based PEM is capable of enhancing the efficiency of virtual testing for AVs without the need for computationally expensive sensor and perception models. However, the previous work has two limitations: First, it does not consider the impact of various weather conditions on the PEM. Second, the proposed procedure for preparing the perception dataset is time-consuming for collecting the dataset. In this work, the implemented procedure is formed by two pipeline: (1) a perception dataset processing pipeline that allows us to gather the perception dataset w.r.t. weather conditions and sensor modalities with just one command. (2) We design a hidden Markov model (HMM)-based PEM and corresponding training pipeline to generate weather-specific PEMs. This pipeline is intended to integrate with nuScenes, a public AV dataset, and Baidu-Apollo, an open-source driving software. We then demonstrate the procedure by generating weather-specific PEMs using data from 60 scenes in the nuScenes dataset. Our results highlight that weather-specific PEMs generated by the procedure are necessity for virtual testing to simulate more realistic environmental conditions.
Title: Autonomous Driving Using Reinforcement Learning
Keywords: autonomous vehicles, motion planning, reinforcement learning, reward shaping
Students: Lin-Chi Wu
Supervisors: Dr. Zengjie Zhang and Dr. Zhiyong Sun
Defense Date: 25.08.2023
Affiliation: Eindhoven University of Technology, Eindhoven, Netherlands
Abstract: Reinforcement learning (RL) is an effective approach to motion planning in autonomous driving, where an optimal driving policy can be automatically learned using the interaction data with the environment. Nevertheless, the reward function for an RL agent, which is significant to its performance, is challenging to determine. Conventional work mainly focuses on rewarding safe driving states but does not incorporate awareness of the risky driving behaviors of vehicles. In this paper, we investigate how to use risk-aware reward shaping to leverage the training and test performance of RL agents in autonomous driving. Based on the essential requirements that prescribe the safety specifications for general autonomous driving in practice, we propose additional reshaped reward terms that encourage exploration and penalize risky driving behaviors. A simulation study in OpenAI Gym indicates the advantage of risk-aware reward shaping for various RL agents. Also, we point out that proximal policy optimization (PPO) is likely to be the best RL method that works with risk-aware reward shaping.
Title: Implementation of Real-time Control by ROS and ROS2
Keywords: robot operating system (ROS), benchmarking
Students: Robert Lefringhausen and Qiaoyue Yang
Supervisors: M. Sc. Zengjie Zhang and Dr. Dirk Wollherr
Defense Date: 14.07.2020
Affiliation: Technical University of Munich, Munich, Germany
Abstract: In this project, a real-time controller for an inverted pendulum and a model of an inverted pendulum are implemented in ROS 1 with the ROS control package and in ROS 2. Afterwards, the real-time control performance of both ROS versions is compared. Several experiments are run with both systems. The performance measurements of both controllers running on a real-time operating system and on a non-real-time operating system are compared. The influence of CPU stress on the real-time performance is also investigated. The comparison of the performance data shows that ROS 2 has a lower latency and is less influenced by system stress compared to ROS 1. For ROS 2 the difference between the real-time and the nonreal-time operating system is significant whereas there is only a small difference for ROS 1.
Title: Object Distance Measurement and Virtual View Synthesis Based on Stereo Vision Technique
Keywords: sensor fusion, stereo vision, coordination transformation
Students: Haoming Song
Supervisors: M. Sc. Zengjie Zhang and Dr. Dirk Wollherr
Defense Date: 20.09.2019
Affiliation: Technical University of Munich, Munich, Germany
Abstract: Stereo vision is used to produce two stereoscopic pictures of a given object, it consists of two cameras of parallel optical axes separated horizontally from each other by a small distance, and these two cameras are combined together in a single frame. The distance between the cameras and the object can be measured depending upon the distance between the positions of the objects in both pictures, the focal lengths of both cameras as well as the distance. By comparing information about a scene from two vantage points, depth information can be extracted by examining the relative positions of objects in the two panels. This is similar to the biological process of Stereopsis. Throughout this work, two webcams are used to find the object distance. After the calculation of the depth image, a synthesis-virtual view can be generated. Six tests were executed and the measured object distances were 35 cm, 40 cm, 50 cm, 60cm, 72.5 cm, and more than 100 cm. The measured distances were compared with those measured at the same time by a depth camera and there were good agreement between them with percentage error ranging from 0:3315% to 3:576%.
Title: Robot Collision Detection by Recurrent Neural Network
Keywords: collision detection, deep learning, signal processing
Student: Zhong Chen
Supervisors: M. Sc. Zengjie Zhang and Dr. Dirk Wollherr
Defense Date: 19.02.2019
Affiliation: Technical University of Munich, Munich, Germany
Abstract: As the presence of robotic technologies in daily life grows rapidly, research on Human-Robot Interaction becomes of utmost importance. Physical Human-Robot Interaction is still considered as one of the major challenges in robotics, it is easily affected by mechanical vibration and noise in real-world scenarios. In this report, a Recurrent Neural Network(RNN) classification method is proposed to discriminate physical interaction and collision. The dataset has 3 classes and each contains thousands of samples from 7 robot joints. The experimental average accuracy rate of the recurrent neural network was 95.00%. Finally, the classification results of the proposed method are compared with other machine learning methods.
Title: Dual optimization for human-robot collaborative control
Keywords: optimal control
Student: Xinchen Du
Supervisors: M.Sc. Zengjie Zhang and Dr. Dirk Wollherr
Defense Date: 20.11.2018
Affiliation: Technical University of Munich, Munich, Germany
Abstract: Human-robot collaboration is considered an important trend in research and in industry. Dual optimal control for the system was previously conducted in Cartesian space, where optimization is simplified as a linear problem by building an impedance system model. To get rid of requirements of impedance control (such as constraints of sampling width), optimization in joint space is investigated. With the dynamic model in joint space, the control problem is transformed into solving a nonlinear HJB equation, which can be numerically solved by techniques like policy iteration or dynamic programming. In the collaboration, the leadership between human and robots is assigned by setting their weight matrices in the cost function.
Title: Stable Impulse Controller for Robot Manipulators under Stiff Collisions
Keywords: impulse control, collision-reaction control, switched systems
Student: Jiaxin Huang
Supervisors: M.Sc. Zengjie Zhang and Dr. Dirk Wollherr
Defense Date: 20.11.2018
Affiliation: Technical University of Munich, Munich, Germany
Abstract: Stiff collisions can cause oscillation during the motion of robot manipulators. It can not only affect the stability of the system but also lead to mechanical damage to the manipulators. On account of the discontinuousness at the impact moments, the control laws in modern control theory are not suitable for this case. A new control strategy needs to be designed, which can avoid oscillation. In this report, a hybrid controller is provided, which is designed for the continuous dynamics and discontinuous dynamics separately. The performance of the hybrid controller is evaluated by weak stability as well as strong stability. The system stability is also confirmed by simulation.
Title: Robot Collision Detection and Classification Based on Time-Frequency Analysis Transformations
Keywords: collision detection, machine learning, signal processing
Student: Wenqian Xu
Supervisors: M.Sc. Zengjie Zhang and Dr. Dirk Wollherr
Defense Date: 22.10.2018
Affiliation: Technical University of Munich, Munich, Germany
Abstract: A common method to analyze the robot collision is using a Neural Network (NN) and Support Vector Machine (SVM). In this research, aiming to get a higher accuracy in predicting the robots' reactions to different environmental situations, the reaction signals will be transformed into time-frequency maps and classified with the help of a Convolutional Neural Network (CNN). The research proves that the prediction rate is improved compared with the traditional method.
Title: Robot Collision Detection and Classification Based on Support Vector Machine
Keywords: collision detection, machine learning, signal processing
Student: Hang Xu
Supervisors: M. Sc. Zengjie Zhang and Dr. Dirk Wollherr
Defense Date: 05.04.2018
Affiliation: Technical University of Munich, Munich, Germany
Abstract: Physical human-robot interaction is still considered one of the major challenges in robotics. It is easily affected by mechanical vibration and noise in the test process, which makes the test waveform more complicated. In order to discriminate interaction and collision, a set of time domain and frequency domain features and wavelet features will be generated and then selected by 2 tests and a recursive feature elimination method. Meanwhile, a classification system based on Support Vector Machine(SVM) will be implemented.
Title: Robot Collision Detection by Machine Learning Method
Keywords: collision detection, machine learning, signal processing
Students: Pengfei Xia, Yuan Yuan, and Xiaohui Hu
Supervisors: M. Sc. Zengjie Zhang and Dr. Dirk Wollherr
Defense Date: 11.01.2018
Affiliation: Technical University of Munich, Munich, Germany
Abstract: Nowadays more and more robots are used to service humans and it is of great importance to take physical Human Robot Interaction (pHRI) into consideration. In previous studies, interaction and collision were hard to distinguish. In this Experiment, interaction and collision can be discriminated by three different machine learning methods and Support Vector Machine(SVM) has the best training effect.