Generating the Future
Emotion Detection by Physiological Sensors
This project aims to build a multi-sensor system to measure the emotional states of humans. The technology can be used to measure the user engagement when audiences are watching a film or listening a lecture online. We plan to combine the booming Artificial Intelligence Technics like deep learning and machine learning with the traditional sensor field to understand deeply in the emotion of human when they interact with computers and other digital devices.
Now the system has integrated a Galvanic Skin Response(GSR) sensor, a Ear-clip Heart Rate Meter and a Self-labeling button through an Arduino unit. The galvanic skin response (GSR, which falls under the umbrella term of electrodermal activity, or EDA) refers to changes in sweat gland activity that are reflective of the intensity of our emotional state, otherwise known as emotional arousal. The button is designed to let the subject to label their feelings(strong or nature) by themselves when they engage in a event.
This project with be part of my PhD thesis : Measuring the User Engagement and Attention during Private Visual Behavior Using Physiological Sensors and Eye Tracker. A private visual behavior happens if you watch something alone without interacting and being with someone else. Private visual behaviors are not just limited to watching films alone on TV or mobile devices. It also includes activities like driving and online chatting which need your visual system to focus. Compared with public visual behaviors like watching a movie in cinema, private visual behavior is more common and makes more influence in our daily life.
More work will be presented in the near future : )
Figure 1: Physiological Signal Collection System(Software)
Figure 2: Physiological Signal Collection System(Hardwar)
UAV Obstacle Avoidance by Stereo Camera
This project aims to establish a real-time vision algorithm to detect obstacles on the path of Unmanned Aerial Vehicle (UAV) and avoid collision. Autonomous collision avoidance system of UAV demands both accuracy and speed to detect obstacles. In this project, a stereo vision system is firstly developed to calculate depth map and a self-adapted segment algorithm based on disparity distribution is designed to segment obstacle from background. To avoid collision, a simplified obstacle model and collision boundary approach is adopted. This approach simplifies a 3D obstacle into a 2D plane and updates its geometric and depth information in real-time to execute collision avoidance.
In experiments, several kinds of obstacles such as artificial obstacles, trees, and human are used to test the robustness of this algorithm. Because of the high computation cost of image processing, especially for embedded system, a Graphics Processing Unit (GPU) was combined along with a Central Processing Unit (CPU) to guarantee real-time performance. Experiment results show a significant gain in both computational and detecting performance. On a NVIDIA TK1 embedded board, this algorithm can detect obstacles and send avoiding instruction at a 20 fps rate.
This project was fund by the 2016 NUAA Graduates Innovation Experiment Fund. As the Co-PI (Principle Investigator) of this project, I designed segment algorithm as well as the target recognition method for the vision part. The flight control system and collision avoidance method was designed and implemented by Zhang Xiang, who is now a flight control engineer in DJI. To view the experimental result of this project, please click the page of ' video'.
Figure 1: Structure of experimental UAV
Figure 2: Real-time obstacle segmentation and avoidance
Autonomous Take-off and Landing for Multi-rotor Aircraft using vision method
In this project, we designed a vision method to automatically launch and land a multi-rotor aircraft on a preset target using Camshift algorithm and Kalman filter. Compared with traditional GPS and INS navigation system, vision navigation system has a bunch of advantages such as low cost and less-dependent on hardware, which is fairly suitable for providing position data in complex electromagnetic environment.
In this project, frame is firstly captured through airborne camera and transmitted to ground station to be analyzed. Secondly, Camshift algorithm is implemented to calculate the position of a ground object. However, Camshift algorithm is easy to lapse when target is temporarily lost by camera. Thus, histogram intersection is used to determine whether the target that be tracked is lost by UAV. As soon as the target is lost, Kalman filter is used to predict the central position of target. At last, the flight control computer sends the flight control signal based on the data from ground station to guarantees that the multi-rotor aircraft can stably launch and land. The result of experiment indicates that this method can work well if the ground target is a solid rectangle.
This project was a part of my undergraduate thesis and funded by 2014 NUAA Innovation Training program. As the PI of this project, I developed the recognition program on the ground station. The flight control system and image transmission module were established by Wang shiyong and Xue bayang. To view the experimental result of this project, please click the page of 'video'.
Figure 3: Automatic take-off
Figure 4: Vision analyse software of ground station
Salient Object Detection using Depth-map
Traditional graph-based saliency detection methods based on CIElab distance often have problems of inconsistency and misdetection if some parts of the background or objects have relatively high contrast with its surrounding areas. To overcome this problem and provide more accurate results in varying conditions, in this research we proposes a multi-feature-based saliency detection algorithm using depth values to refine inconsistent parts in saliency map. Firstly, coarse saliency maps are computed through multi-feature-fused manifold ranking. Compared with traditional manifold ranking, this method is more robust to varying scenes. Then, a self-adaptive segment method based on depth distribution is used to filter the less significant areas and enhance the salient objects. Finally, the saliency-depth consistency check is implemented to suppress the highlighted areas in the background and enhance the suppressed parts of objects.
Figure 5: Examples of results computed by different methods
Figure 6: Illustration of the main stages of proposed algorithm
The results according to the comparison with eight state-of-the-art methods show that the proposed algorithm significantly improves the quality of saliency maps in the challenging natural scenarios. This research was conducted by myself independently under the supervision of my inspector Dr. Zhong Yang.
Visual Guidance and Pattern Recognition applied on Automated Guided Vehicle
This project is going to develop an automated guided vehicle to recognize pre-trained objects or patterns on the way of its route. It is now in the second term and got second round fund from NUAA 2017 Graduates Innovation Experiment Fund in June, 2017 (only 50% of the initial projects). At present stage, our AGV has been installed and can avoid obstacles through stereo camera. Currently, I am cooperating with one of my colleague, Song Jiarong, to optimize ImageNet, a large network, to realize real-time recognition of different traffic signs. To view the video of obstacle avoidance of our AGV, please click the page of 'video'.
Figure 7: Structure of Automated Guided Vehicle (AGV)
Figure 8: Structure of optimized convolutional neural network
Figure 8: Results of traffic sign recognition